This post describes executing a CI test job on merge requests that need review and then autoland jobs that are approved on Launchpad hosted git based projects. The primary tool used is jenkins-launchpad-plugin.
The workflows are currently in use by a number of projects the Canonical Server team manages, including:
The examples used below are the actual jobs used by pycloudlib.
Requirements Link to heading
Before creating the automation jobs there are a number of pieces that need to be in place to enable this work.
Launchpad Bot Account Link to heading
A Launchpad bot account is required to vote on merge requests and to do the merging itself. This bot requires:
- A registered SSH Keys with Launchpad
- Commit rights to the projects needing review to vote on merges and autoland
Jenkins Link to heading
Jenkins instance configured with:
- The parameterized-trigger plugin, as the name implies, gives the ability to trigger other jobs with parameters.
- Below I use jenkins-job-builder to deploy jobs more easily. This requires a bot account on the jenkins instance to deploy jobs.
- Optionally, the pipeline job plugin if you wish to run a test job with a pipeline based job. These are handy for more complex test jobs where a user wishes to build, unit test, and integration test as a part of CI rather than doing a simple test (e.g.
Jenkins Slaves Link to heading
The slaves used for this work will need:
- Direct access to Launchpad
- git configured to allow
Jenkins Launchpad Plugin Link to heading
The final step is the main library to get all of this to work is installing and configuring jenkins-launchpad-plugin.
Install Link to heading
Install is not as simple as it could be for now, but the below would get it on a system.
Versions of Ubuntu from 18.04 (Bionic) onward no longer have the python-jenkins package and require to install it from pip (e.g.
pip install python-jenkins). Note that even in pip it is called python-jenkins and not only jenkins, which is an entirely different package.
Configuration Link to heading
Configuration of jenkins-launchpad-plugin is kept in
$HOME/.jlp/jlp.config. The configuration lays out the configuration for connected to the Jenkins server and Launchpad credentials.
Of note is the last section for allowed users to launch CI. Keep in mind that launching CI against a user’s code means that he or she is capable of launching arbitrary code on the CI system.
CI Jobs Link to heading
The next section describes the individual jobs required to review merges requests in the ‘Needs Review’ state, run CI, and vote on them.
My examples will use Jenkins Job Builder YAML for each job, which are production examples used by the pycloudlib project.
CI Trigger Link to heading
The trigger job is something that will run every 15 mins to look for merge requests with ‘Need review’ status. Those merge requests will then get CI run against them by launching the corresponding CI test job.
This job will only start testing on jobs that have not already been reviewed or are not already getting reviewed. As stated earlier, the user must be in the allowed users list as well.
Note that this job will not launch CI against a merge request unless the user is in the ‘allowed_users" list. That list as shown above in the configuration of jenkins-launchpad-plugin can include individual users or Launchpad groups. It is highly recommended to have a single CI user’s group so the list
CI Test Link to heading
Next, comes the testing. For all intents and purposes the actual test can be whatever a projects wants: a build test, unit test, lint tests, integration test, etc. The important part here is noticing the expected parameters and publisher portions of the job.
The CI trigger job will launch this job with the information about the merge request including the revision to be tested, the repo name, branch name, and URL of the merge proposal.
set -e if any command fails during the testing then the CI will be considered a failure. The publisher triggers then launch the voting job with the appropriate test result value.
Launchpad Vote Link to heading
Finally, it is time to vote on the merge proposal. The CI job passes in information about the merge request and places a vote on the actual merge proposal.
Autoland Jobs Link to heading
The final section describes how to autoland approved merge requests by looking for approved merge requests, running a final test against them, and landing the code. Once again, these examples use Jenkins Job Builder’s YAML based configuration.
Autoland Trigger Link to heading
Similar to the CI job, the trigger job will look for merge requests, only this time in the ‘Approved’ state and launch the autoland test job.
Autoland Test Link to heading
The autoland test job should then attempt the merge, but not push, while also running any applicable tests. This prevents any final issues sneaking in when a merge request is not quite fully updated to trunk.
Similar to the above, this test job can include any tests a project finds applicable.
Autoland Link to heading
Finally, comes the autoland, which does a number of validating items before actually committing.
First, the autoland requires a test result from the previous job. If the test job had failed, then the autoland will add a comment to the merge proposal and mark it ‘Needs fixing’ with the URL of the test job.
The autoland will also ensure that the latest revision matches the reviewed revision to prevent any additional commits from getting added.
Next, the autoland will review the commit message to verify that it is correctly formatted. It expects at a minimum a short summary line up to 74 characters. If a longer commit message is required then immediately after the short summary a blank line and then the longer summary with lines up to 74 characters.
At the end of a commit message a list of bugs can be included that are fixed as a single list or each on its own line.
If the author of the code is someone other than the person submitting the merge request, then the separate author can be specified in the commit message. When doing the autoland the author specified here will be made the author of the final commit instead of the person who submitted the merge request.
The autoland will then attempt the merge and if any part of the merge fails the process will reject the merge, again marking the merge request as ‘Needs fixing’ and add a message about what failed.
Finally, if any bugs were specified then the project’s tasks for the bugs will be marked “Fix Committed” and a message added to them with the commit hash, project, and branch name where the fix is located.