SCM API turns 2.0 and what that means for you

Way back in September 2013 we announced the Literate plugin, as an experimental new way of modeling branch development in Jenkins.

When you are performing an experiment, the recommendation is to do just enough work to let you perform the test. However, the culture in Jenkins is to always try and produce reusable components that others can use in ways you have not anticipated.

So when releasing the initial version of the Literate plugin we also separated the Literate specific bits from the SCM specific concepts and multi-branch concepts. These were lower level concepts were organized into the following plugins:

In addition, we released updates to three of the more common SCM plugins which included implementations of the SCM API:

While there was some interest in the Literate plugin, it did not gain much traction - there are only 39 Jenkins instances running the Literate plugin as of December 2016.

In terms of the reusable components, we had only made a minimal implementation with some limitations:

Roll forward to November 2015 and Jenkins Pipeline got a release of the Pipeline Multibranch. It seems that pairing Pipeline with Branch API style multi-branch is much more successful than Literate - there are close to 60,000 instances running the pipeline multi-branch plugin as of December 2016.

There also were two new SCM plugins implementing the SCM API:

Unlike the previous implementations of the SCM API, however, these plugins do not maintain a local cache of the repository state. Rather they make queries via the GitHub / BitBucket REST APIs on demand.

The above design decision exposed one of the initial MVP compromises of the SCM API plugin: very basic event support. Under the SCM API 1.x model, the only event that an SCMSource can signal is something changed, go look at everything again. When you are accessing an API that only allows 5,000 API calls per hour, performing a full scan of the entire repository just to pick up a change in one branch does not make optimum usage of that 5,000 calls/hour rate limit.

So we decided that perhaps the SCM API and Branch API plugins have left their Minimum Viability Experiment state and the corresponding limitations should be addressed.

So what has changed in the SCM API 2.0.x and Branch API 2.0.x release lines? These plugin releases include:

In addition, we have upgraded the following plugins to include the new fine-grained event support:

Ok, that was the good news. Here is the bad news.

We found out that the GitHub Branch Source and BitBucket Branch Source plugins had made invalid assumptions about how to implement the SCM API. To be clear, this was not the plugin developers fault: at the time there was no documentation on how to implement the SCM API.

But fixing the issues that we found means that you have to be careful about which specific combinations of plugin versions you have installed.

Upgrading should make multi-branch projects much better. When you are ready to upgrade you must ensure that you upgrade all the required plugins. If you miss some, just upgrade them and restart to fix the issue.

If you are using the Blue Ocean plugin

Other plugins that may require updating:

After an upgrade you will see the data migration warning (see the screenshot in JENKINS-41608 for an example) this is normal and expected. The unreadable data will be removed by the next scan / index or can be removed manually using the Discard Unreadable Data button. The warning will disappear on the next restart after the unreadable data has been removed.

SCM API 2.0.x adds fine-grained event support. This should significantly improve the responsiveness of multi-branch projects. This should significantly reduce your GitHub API rate limit usage.

If you are using the GitHub Branch Source or GitHub Organization Folders plugins then upgrading will significantly reduce the API calls made by Jenkins to GitHub.

If you are using any of the upgraded SCM plugins (e.g. Git, Mercurial, GitHub Branch Source, BitBucket Branch Source) then upgrading will significantly improve the responsiveness to push event notifications.

You should read the new documentation on how plugin developers are supposed to implement the SCM API

The persistent reader may be wondering what happens now to the Literate plugin.

For me, the logical heir of the Literate Plugin is the Pipeline Model Definition plugin. This new plugin has the advantage of an easy to read pipeline syntax with the extra functionality that I suspect was preventing people from adopting Literate.

The good news is that the Pipeline Model Definition already has 5000 installations as of December 2016 and I expect up-take to keep on growing.