With the GA of Portworx Data Services I needed a way to connect some test applications with Apache Kafka. Kafka is one of the most asked for Data Services in PDS. Deploying Kafka is very easy with PDS but I wanted to show how it easy it was for a data team to connect their application to Kafka in PDS. I was able to find a kafka-python library, so I started working on a couple of things.
- A python script to create some kind of load on Kafka.
- Containerize it, so I can make it easy and repeatable.
- Create the kubernetes deployments so it is quick and easy.
This following github repo is the result of that project.
https://github.com/2vcps/py-kafka
See the repo for the steps on setting up the secret and deployments in K8s to use with your PDS Kafka, honestly it should work with any Kafka deployment where you have the connection service, username and password.
Check out the youtube demo I did above to see it all in action.