At this years Embedded Linux Conference we learnt about kernelci.org – a project that performs automated testing of upstream kernels. We were so impressed that we immediately got involved and are now contributing our resources to perform automated boot tests of the latest kernels within our board farm.
The Linux kernel supports a large variety of platforms yet there is limited coverage in its upstream testing. Thus when developers add new features to the kernel it’s very difficult to ensure that those new features don’t break support for existing platforms – especially as it’s impractical/impossible to expect those developers to have access to the full range of devices the kernel supports. The kernelci.org project aims to improve upstream testing and to allow bugs to be detected as early as possible. In this post we’ll find out a little more about how it all works.
A good overview of the project can be found in Kevin Hilman’s “Upstream Kernel Testing” presentation, though here is a summary of how it works:
- A Jenkins installation monitors a large number of upstream kernel GIT trees which include the mainline and stable trees. When changes are pushed to these trees, a build job is triggered to build kernels for a variety of boards. Once the builds are complete the artifacts are published online.
- As soon as new kernels are built, the kernelci infrastructure will instruct a variety of (there are 7 at present) board farms (also known as labs) to deploy and boot the kernel on real hardware. Upon successfully booting the kernels, additional tests are run on the board.
- Upon completing the boot tests, the infrastructure will publish the results online and to relevant mailing lists. Thus often providing very quick feedback to the community.
In addition to simply running tests and reporting if builds, boots or tests fail – if a failure does occurs then it will also attempt to determine which commit caused the issue via the automated use of git bisect – very clever indeed. As an example of the value kernelci.org can bring, take a look at a summary of recent boot logs for the linux-next kernel here. You’ll see that upon detecting that a new version of the kernel was available it automatically built and booted kernels on 92 unique boards resulting in a total of 389 boots. It observed that at least 15 boots failed and when this occurred it attempted to figure out which commit caused the regression. You can see that this triggered a bug report which was later fixed.
Kernelci.org says “It is our mission to detect, bisect, report and fix regressions on upstream Kernel trees before they even reach «mainline».”, it’s clear by looking at their bug tracking page that they are achieving this goal.
At Embedded Bits we already use a board farm for automated testing, thus it seemed natural to make this resource available to kernelci.org such that we can assist in improving the quality of upstream kernels. We currently average 120 boots a day and strive to add additional boards over time.