Machine Learning Plugin project - Coding Phase 3 blog post
Good to see you all again !
This is my final blog post about coding phase 3 in Jenkins Machine Learning Plugin for GSoC 2020. Being at the end of GSoC 2020, we had to finish all the pending issues and testing before a stable release in the main repository. Throughout this program, there were lots of learning and hard work will make this plugin valuable to the Data Science and Jenkins community.
Summary
With combining all of the work in phase 1, 2 and 3, initial version of Machine learning plugin( 1.0 ) was successfully released in Jenkins plugin repository. An interesting feature which allows users to connect to their existing programming language kernels more than connecting to only IPython kernel was introduced in this phase. It can be selected in multiple steps with different kernel. Images and graphs produced by Jupyter notebooks will be saved in user preferred folder in the workspace that can be used for reporting/analytic purposes later. Hoping this blog summarizes the Machine Learning’s features and future contributions. Thank you for your interest and support !!!
Main features of Machine Learning plugin
-
Execute Jupyter notebooks directly
-
Run different language scripts using multiple build steps
-
Convert Jupyter Notebooks to Python
-
Configure Jupyter kernels( IPython, IRKernel, IJulia etc) properties
-
Support to execute Notebooks/scripts on Agent
-
Extract graph/map/images from the code
-
Each build step can be associated with a machine learning task
-
Support for Windows and Linux
Future improvements
-
Improving performance of the plugin
-
Try to implement JENKINS-63377
-
Support parameterized definitions in Notebooks JENKINS-63478
-
Increasing testing code coverage
Multiple language kernel support
If there are existing kernels in the system, user will be able to configure in the global configurations in order to apply in the builder/step configuration.
Some popular interactive kernels
-
IPythonforpython -
IRKernelforR -
IJuliaforJulia -
IJavascriptforjavascript
More kernels and installation guides are found here. https://github.com/jupyter/jupyter/wiki/Jupyter-kernels
Dump images and graphs
Text output will be displayed in the console log. At the same time images/graphs/heat maps and HTMLs will be saved in the workspace. An action is shown in the left panel to display images in realtime. Due to the Content Security Policy of jenkins, some HTMLs which contain harmful javascript may not render in jenkins UI.
Fixed bugs
There were more bugs identified and fixed with many interactive testings. Setting the working directory of kernels was a big issue while getting datasets/files by script. Zeppelin process launcher was bypassed to fix this issue.
Patch version released
A major bug which was created while setting the process working directory had patched in the v1.0.1. The latest release is more stable now.
Acknowledgement
Machine Learning plugin had been developed under GSoC 2020 program. A huge thanks to Bruno P. Kinoshita, Marky Jackson, Shivay Lamba, Ioannis Moutsatsos and Org admins for this wonderful experience. I would be grateful for contributing this plugin continuously and more in Jenkins.
About the author