Remoting over Apache Kafka 2.0: Built-in Kubernetes support
I am Long Nguyen from FPT University, Vietnam. My project for Google Summer of Code 2019 is Remoting over Apache Kafka with Kubernetes features. After a successful Phase 1, finally the 2.0 version of the plugin has been released. The 2.0 version provides seamless integration with Kubernetes environment.
2.0 version features
-
Start a simple Apache Kafka server in Kubernetes.
-
Dynamically provision Remoting Kafka Agent in Kubernetes.
-
Helm chart to bootstrap the whole system in Kubernetes.
Start a simple Apache Kafka server in Kubernetes
Use of the plugin requires that users have a configured Apache Zookeeper and Apache Kafka server, which could be intimidating for people who just want to try out the plugin. Now, users can start a simple, single-node Apache Kafka server in Kubernetes environment with just one button click.
On the Global Configuration page, users can input Kubernetes server information and credentials. When users click Start Kafka on Kubernetes button, Jenkins will create a Kubernetes client from the information and then apply Apache Zookeeper and Apache Kafka YAML specification files from resources. After downloading images and creating containers, it will automatically update Apache Zookeeper and Apache Kafka URLs into respective fields.
Dynamically provision Remoting Kafka Agent in Kubernetes
With previous version, users have to manually add/remove nodes so it is hard to scale builds quickly. Kubernetes plugin allows us to dynamically provision agents in Kubernetes but it is designed for JNLP agent. With this new version, Remoting Kafka agent can also be provisioned automatically in Kubernetes environment.
Users can find the new feature in Cloud section in /configure. Here users could input Kubernetes connection parameters and desired Remoting Kafka agent properties including labels. When new build with matching labels gets started and there are no free nodes, Cloud will automatically provision Remoting Kafka agent pod in Kubernetes to run the build.
Helm Chart
Helm chart for Remoting over Apache Kafka plugin is based on stable/jenkins chart and incubator/kafka chart. You can follow the instruction here to install a demo ready-to-use Helm release. Your kubectl get all should look like this:
NAME READY STATUS RESTARTS AGE
pod/demo-jenkins-64dbd87987-bmndf 1/1 Running 0 2m21s
pod/demo-kafka-0 1/1 Running 0 2m21s
pod/demo-zookeeper-0 1/1 Running 0 2m21s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/demo-jenkins NodePort 10.108.238.56 <none> 8080:30386/TCP 2m21s
service/demo-jenkins-agent ClusterIP 10.98.85.184 <none> 50000/TCP 2m21s
service/demo-kafka ClusterIP 10.109.231.58 <none> 9092/TCP 2m21s
service/demo-kafka-headless ClusterIP None <none> 9092/TCP 2m21s
service/demo-zookeeper ClusterIP 10.103.2.231 <none> 2181/TCP 2m21s
service/demo-zookeeper-headless ClusterIP None <none> 2181/TCP,3888/TCP,2888/TCP 2m21s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/demo-jenkins 1/1 1 1 2m21s
NAME DESIRED CURRENT READY AGE
replicaset.apps/demo-jenkins-64dbd87987 1 1 1 2m21s
NAME READY AGE
statefulset.apps/demo-kafka 1/1 2m21s
statefulset.apps/demo-zookeeper 1/1 2m21sHow to Contribute
You are welcome to try out the plugin and integrate it into your current setup. If you find out any bug or if you would like to request new feature, you can create ticket at JIRA. If you would like to contribute code directly, you can create pull requests in the GitHub page below.
About the author