Easily reuse Tekton and Jenkins X from Jenkins

James Strachan April 21, 2021

What is Tekton?

Tekton is a powerful and flexible open-source framework for creating CI/CD systems, allowing developers to build, test, and deploy across cloud providers and on-premises systems.

Why use Tekton?

Tekton pipelines have a number of benefits:

they are cloud native and designed from the ground up for kubernetes
each Tekton Pipeline is fully declarative and completely self described; it does not depend on any separate out of band Jenkins controllers, plugins or plugin/controller configurations
each Pipeline Task runs as a stand alone kubernetes Pod which is completely independent of any other pods and pipelines and are fully scheduled by Kubernetes to maximise resilience and optimize resource usage. A bad pipeline cannot take down another one & the kubernetes scheduler manages them all
each step can be any command in any container image with whatever secrets, volume mounts, environment variables and resource limits you need
there is no need to bundle a JVM or Jenkins Remoting container into the pod so you can keep resources and cost down

Why use Jenkins and Tekton together?

Jenkins is the most popular open source automation server around. Lots of developers use it every day to get things done. Jenkins can now be used to automate Tekton pipelines too which helps teams digitally transform to more cloud native solutions for their CI and CD. In such a case, you can use Tekton pipeline engine while getting all benefits from Jenkins as an orchestrator, user interface and the reporting engine.

Introducing the Tekton Plugin for Jenkins

The Tekton Client plugin for Jenkins lets you easily use Jenkins to automate creating and running Tekton pipelines. It bridges the Kubernetes learning gap and allows invoking Tekton Pipelines and resources through Jenkins. This allows users to not have much of the Kubernetes specific knowledge beforehand and work.

Its a single Jenkins plugin to install - so it’s easy to use.

For background check out the blog post Bridging the Gap with Tekton-client-plugin for Jenkins by the founder of the plugin Vibhav Bobade.

Requirements

The Tekton Client plugin for jenkins assumes you have access to a kubernetes cluster.

The kubernetes cluster should have Tekton pipelines installed.

If you have not yet installed Tekton you could use this tekton helm chart

The Jenkins controller should also have kubernetes RBAC access to be able to create Tekton resources and watch them and their associated pods and pod logs.

If you are running your Jenkins controller inside Kubernetes then an easy way to setup the RBAC is to install the Jenkins Resource Helm Chart in the same namespace as your Jenkins controller.

Another option is to use an installation of Jenkins X and let it setup a Jenkins controller via GitOps

Specifying the Tekton pipelines

You can configure the Tekton pipeline via:

a file path in a git clone block
a URL to a tekton YAML file
a block of YAML

We recommend defining Tekton pipelines as YAML files and checking them into a git repository so that you can use GitOps and follow the Pipeline As Code pattern.

This means that you can version your pipelines in git. It also means you can benefit from the various IDE plugins available for Tekton such as VS Code and IDEA so that you get auto completion, formatting and validation while editing the YAML.

So you can use the usual Git provider support in Jenkins to clone the git repository that contains then Tekton YAML file then reference the file by name.

Reusing Pipelines from the Tekton Catalog

The Tekton Catalog defines a ton of Tekton Tasks you can reuse in your pipelines

We have found when it comes to a microsevices style architecture you end up with lots of repositories and pipelines. Then using a Pipeline As Code pattern with GitOps we want to Version Everything but also make it easy for any repository to use any version of any task or pipeline.

e.g. you may have many repositories using the current version of a pipeline but want to try out a new change to the pipeline in just 1 repository to verify it works well; then if it does, incrementally roll that change out to more repositories.

This can make it hard trying to reuse as much as you can across the different git repositories while also minimising the number of versions and forks of git repositories you have and simplifying the maintenance of all of the pipelines.

We have found on the Jenkins X project that a nice way to do this via GitOps such that we reference versioned Tekton Tasks and Pipelines in git so that they are easy to reuse or override.

So we reuse Tasks and Pipelines via the uses: image notation which lets us keep all of our Tekton Tasks and Pipelines in vanilla Tekton YAML; so that the IDE completion and validation works - but we can easily reuse Tasks or steps from libraries while also Versioning Everything

Note that if wish to reuse steps/tasks via the uses: image notation then you must click the Tekton Catalog flag in your Job definition which will then resolve the uses: clause with the actual step/task.

What is Jenkins X?

The Jenkins X project automates your CI/CD on kubernetes to help you accelerate:

Automated CI/CD pipelines lets you focus on your actually application code while Jenkins X automatically creates battle tested Tekton CI/CD pipelines for your project which are managed via GitOps so that its super easy to keep your pipelines up to date across your repositories or to upgrade or override pipelines or steps for specific repositories.
Automatic promotion of versioned artifacts via GitOps through your Environments such as Staging, Pre-production and Production whether they are running in the same kubernetes cluster or you are using multiple clusters for your environments
Preview Environments lets you propose code changes via Pull Requests and have a Preview Environment automatically created, running your code in kubernetes to get fast feedback from your team before agreeing to merge changes to the main branch
ChatOps comment on Pull Requests to give feedback, approve/hold changes, trigger optional pipelines for additional testing and other ChatOps commands

All of the above is implemented in reusable Tekton pipelines.

Reusing Jenkins X Pipelines

So how can we reuse automated CI/CD pipelines from Jenkins X project from Jenkins?

Make sure you have the Tekton Client plugin for Jenkins installed in your Jenkins server.

Using a working template

If you want to start with a working example then

Create A Git Repository From This Template
add a new Frestyle project to your Jenkins server
enable the Git source code management for your new github.com repository
click Add build Step (near the bottom of the page) and then select Tekton : Create Resource (Raw)
make sure that FILE is selected for the input and enter the name .lighthouse/jenkins-x/release.yaml for the file name
if you are using a Jenkins X cluster enter jx for the namespace
ensure that Enable Tekton Catalog is checked
now save the pipeline - it should look something like this:

Jenkins Console

Now if you trigger the pipeline you should see it create a Tekton Pipeline and you should see the output of the tekton pipeline in the Jenkins console. The pipeline is actually running as a completely separate Pod in kubernetes; the Jenkins controller just tails the log into the console.

In a Jenkins X cluster this pipeline should just work (reusing all the cloud resources and IAM roles setup by the Terraform) but in an arbitrary kubernetes cluster you may get issues around not being able to push images or promote due to lack of GitOps environments being defined which we can help you work through via the Jenkins X slack room

Using an existing repository

You can configure a Pull Request or Release pipeline in your project by copying the YAML file for the language pack you wish to use.

e.g. if you are using maven then copy pullrequest.yaml or release.yaml into your projects source code then reference it from your Jenkins Job:

Then follow the above instructions for setting up a Freestyle project for your git repository and referencing the file name for your pipeline.

Overriding steps

Being able to reuse steps from libraries of pipelines is awesome; but sometimes you need to change things. The assumptions, commands, arguments, environment variables or approaches used for every step in a library may not quite match what you need on a specific application. You may need to run steps before/after steps in the library or you may need to override a specific step to do something different.

You can easily customize any inherited step in any shared pipeline or add custom steps before/after any step.

The fact that all the Tekton YAML is fully declarative makes it super easy to modify things via your IDE with validation and smart completion and not have to use a scripting language and understand complex shared pipeline libraries.

The easiest way to try overriding a step is to install the jx binary to your $PATH then use the jx pipeline override command which will create a new locally overridden step you can then just edit in your IDE.

Then at any time you can view the effective pipeline when you make local changes

Comparing the Kubernetes and Tekton plugins

Those of you using Jenkins on a Kubernetes cluster are probably using the kubernetes plugin right now.

Here is an example of how to use a Jenkinsfile with a pod YAML file so that you can run commands in different containers in the pod.

What this means is that:

a kubernetes pod is created based on the pod YAML file which is scheduled by kubernetes
the Jenkinsfile runs on the Jenkins controller talking over Jenkins remoting to the pod to tell it to run commands in different containers. The pod includes the jnlp container which does the remoting between the Jenkins controller and the pod

This has a few issues:

each container in the pod must have a shell so that jnlp can invoke commands. This may mean you have to create your own images
it can be a little slow to start since there is chattiness with the Jenkins controller and the pod - whereas with Tekton pods just start and run locally without any coodination with the Jenkins controller
you have to maintain 2 files: the Jenkinsfile and the pod.yaml and it’s hard to share/override both of those files across multiple repositories as you need to make changes (e.g. overriding environment variables/images/commands/resource limits on demand on steps).

Though one downside of the tekton approach is that by default there is no automatic synchronisation of state; after a Task in tekton completes there’s no automatic upload of state to the Jenkins controllers disk. You can always add a step in your Task to upload workspace state to the Jenkins controller if that’s what you want.

Though remember that tekton plugin doesn’t take anything away; so you can mix and match the kubernetes and tekton plugins to suit your needs.

Conclusion

We are really excited about the combination of Jenkins, Tekton and Jenkins X letting developers pick the best tool for the job while becoming more cloud native and increasing the automation help reduce the amount of manual work creating and maintaining pipelines while also helping to improve the quality and practices of our CI/CD.

Please try it out and let us know how you get on!

About the author

James Strachan

James is a long time open source contributor, created the Groovy programming language and Apache Camel integration framework. For the past few years he’s been working on CI/CD with Kubernetes.