Remoting over Apache Kafka plugin with Kafka launcher in Kubernetes

Long Nguyen July 11, 2019

I am Long Nguyen from FPT University, Vietnam. My project for Google Summer of Code 2019 is Remoting over Apache Kafka with Kubernetes features. This is the first time I have contributed for Jenkins and I am very excited to announce the features that have been done in Phase 1.

Project Introduction

Current version of Remoting over Apache Kafka plugin requires users to manually configure the entire system which includes Zookeeper, Kafka and remoting agents. It also doesn’t support dynamic agent provisioning so scalability is harder to achieve. My project aims to solve two problems:

Out-of-the-box solution to provision Apache Kafka cluster.
Dynamic agent provisioning in a Kubernetes cluster.

Current State

Kubernetes connector with credentials supported.
Apache Kafka provisioning in Kubernetes feature is fully implemented.
Helm chart is partially implemented.

Apache Kafka provisioning in Kubernetes

This feature is part of 2.0 version so it is not yet released officially. You can try out the feature by using the Experimental Update Center to update to 2.0.0-alpha version or building directly from master branch:

git clone https://github.com/jenkinsci/remoting-kafka-plugin.git
cd remoting-kafka-plugin/plugin
mvn hpi:run

On the Global Configuration page, users can input Kubernetes server information and credentials. Then they can start Apache Kafka with only one button click.

Kafka provisioning in Kubernetes UI

When users click Start Kafka on Kubernetes button, Jenkins will create a Kubernetes client from the information and then apply Zookeeper and Kafka YAML specification files from resources.

Kafka provisioning in Kubernetes architecture

Helm Chart

Helm chart for Remoting over Apache Kafka plugin is based on stable/jenkins chart and incubator/kafka chart. As of now, the chart is still a Work in Progress because it is still waiting for Cloud API implementation in Phase 2. However, you can check out the demo chart with a single standalone Remoting Kafka Agent:

git clone -b demo-helm-phase-1 https://github.com/longngn/remoting-kafka-plugin.git
cd remoting-kafka-plugin
K8S_NODE=<your Kubernetes node IP> ./helm/jenkins-remoting-kafka/do.sh start

The command do.sh start will do the following steps:

Install the chart (with Jenkins and Kafka).
Launch a Kafka computer on Jenkins controller by applying the following JCasC.

jenkins:
  nodes:
    - permanent:
        name: "test"
        remoteFS: "/home/jenkins"
        launcher:
          kafka: {}

Launch a single Remoting Kafka Agent pod.

You can check the chart state by running kubectl, for example:

kubectl get all -n demo-helm
NAME                                    READY   STATUS    RESTARTS   AGE
pod/demo-jenkins-998bcdfd4-tjmjs        2/2     Running   0          6m30s
pod/demo-jenkins-remoting-kafka-agent   1/1     Running   0          4m10s
pod/demo-kafka-0                        1/1     Running   0          6m30s
pod/demo-zookeeper-0                    1/1     Running   0          6m30s

NAME                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/demo-0-external           NodePort    10.106.254.187   <none>        19092:31090/TCP              6m30s
service/demo-jenkins              NodePort    10.101.84.33     <none>        8080:31465/TCP               6m31s
service/demo-jenkins-agent        ClusterIP   10.97.169.65     <none>        50000/TCP                    6m31s
service/demo-kafka                ClusterIP   10.106.248.10    <none>        9092/TCP                     6m30s
service/demo-kafka-headless       ClusterIP   None             <none>        9092/TCP                     6m30s
service/demo-zookeeper            ClusterIP   10.109.222.63    <none>        2181/TCP                     6m30s
service/demo-zookeeper-headless   ClusterIP   None             <none>        2181/TCP,3888/TCP,2888/TCP   6m31s

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/demo-jenkins   1/1     1            1           6m30s

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/demo-jenkins-998bcdfd4   1         1         1       6m30s

NAME                              READY   AGE
statefulset.apps/demo-kafka       1/1     6m30s
statefulset.apps/demo-zookeeper   1/1     6m30s

Next Phase Plan

Implement Cloud API to provision Remoting Kafka Agent. (JENKINS-57668)
Integrate Cloud API implementation with Helm chart. (JENKINS-58288)
Unit tests and integration tests.
Release version 2.0 and address feedbacks. (JENKINS-58289)

About the author

Long Nguyen

Long is a Software Engineering student at FPT University, Vietnam. He started to contribute for Jenkins from Google Summer of Code 2019 for the project Remoting over Apache Kafka with Kubernetes features