During the development of an embedded Linux based product it’s not uncommon to reach a point where the features of the product are seemingly demanding too much from the CPU. When this happens and features start to stutter the first point of call is often ‘top‘ – this often results in the dreaded realization that the system has or is near to 0% idle time. At this point it’s very easy to get mislead by the myriad of statistics output from performance measuring utilities such as ‘top’ which often lead to optimization efforts that have little effect. This post explores the ‘I/O wait’ statistic and demonstrates how a system with 0% idle time doesn’t mean you’re demanding too much CPU.
A few weeks ago the engineering team of Embedded Bits attended the Embedded Linux Conference Europe (ELCE) held this year in Düsseldorf, Germany. As described by The Linux Foundation: “The conference is the premier vendor-neutral technical conference for companies and developers using Linux in embedded products. This conference, now in it’s 8th year, has the largest collection of sessions dedicated exclusively to embedded Linux and embedded Linux developers. ELCE is embedded Linux experts talking about solutions to your embedded Linux problems. ELCE consists of 3 days of presentations, tutorials and Bird-of-a-Feather sessions.”
Attending the event provides our engineers with a valuable opportunity to keep up to date with the ever changing Linux landscape and to mix with the community shaping its future.
This year we also presented a talk titled ’12 Lessons Learnt in Boot Time Reduction’, slides for this talk are available here with videos expected to follow in due course.
One of the challenges of boot time reduction is understanding why the boot time of a device may vary with each reboot, this is important because we strive not only for minimal boot times but for consistently minimal boot times. This post uses a Xilinx Zynq platform to demonstrate how we can measure, understand and find the causes of boot time variability. We’ll also provide an insight into how we use automation at Embedded Bits to improve the process.
To explore boot time variability we’ll be using Xilinx’s Zynq-7000 based ZC702 evaluation kit. The Zynq range of SoC’s cleverly combine a dual-core Cortex A9 MPCore with programmable logic (Artix-7 FPGA). The ZC702 is provided with a ‘Base Targeted Reference Design (TRD)‘ (a Linux distribution on an SD card) – we’ll use this to perform our investigation against.
By its very nature, the only way to measure variability is to measure the boot time over and over again during successive runs (time consuming!). At Embedded Bits where possible we install development boards into our board farm – along with providing benefits such as board sharing and collaborative working, it crucially provides automation – we’ll take advantage of this as we explore boot time variability. We’ll start by instructing the farm to repeatedly reboot the board whilst capturing boot logs.
As a means to demonstrate our boot time reduction skills, last November we put together a demo which shows an Embedded Linux device cold-booting in less than a second.
The demo consists of an ARM Cortex-A9 based device connected to a camera, 7-segment display and HDMI display. The device uses the camera along with OpenCV based software to count the number of yellow balls present on the table beneath and display the count on the 7-segment display. The device also outputs the camera image and ball detection illustration on the LCD display.
The device is able to do all this within one second of software reset, here is the video:
Whilst recently making changes to an embedded Linux distribution I came across a modest but very powerful feature of the kernel’s tried and test Kconfig system. What I discovered was a script that allows for scripted modification of kernel configuration files, i.e. .config and *_defconfig files.
The script, which first appeared in the 2.6.29 kernel can be found in the kernel’s scripts/ directory and its usage is very straight forward. Here are some examples:
# Enable timing information on printk's ./scripts/config --enable CONFIG_PRINTK_TIME # Change the path of the initramfs ./scripts/config --set-str CONFIG_INITRAMFS_SOURCE ../fs # Increase the size of the kernel log buffer ./scripts/config --set-val CONFIG_LOG_BUF_SHIFT 14 # Enable LZO compression for SquashFS in ~/test_defconfig ./scripts/config --file ~/test_defconfig --enable CONFIG_SQUASHFS
Whilst investigating ways to improve the cold boot time of embedded Linux I came across a little known control parameter of the USB stack known as ‘delay_use‘. It’s a parameter that describes the amount of time given to Mass Storage Devices to allow them to ‘settle down’ before being used. This article examines ‘delay_use’ and identifies how it may be used to reduce boot time and improve responsiveness.
I recently came across the following disconcerting message in my kernel’s boot output:
Truncating RAM at 40000000-5fffffff to -57ffffff (vmalloc region overlap). ... Kernel command line: console=ttySC0,115200 mem=512M ... Memory: 384MB = 384MB total
Which is the kernel’s way of saying “I understand there may be some RAM here – but I’m not going to use it all”. So what is the cause of this warning? And what do we need to do to reclaim that lost RAM?
Today I encountered a bug that was quite difficult to find regarding strings. In order for strings to work they must be null-terminated, and this implies that an array of characters can contain a string with a length equal to the array size minus one, because there must be space for the null character. I found out that, when initializing array of chars with strings, the compiler does not complain if just the null character doesn’t fit.
Before I started playing with the BeagleBoard XM I’ve had never booted a board directly from an MMC card and I didn’t have a clue what an ‘MLO’ file was. After some research on the internet it seemed apparent that it was used in place of the traditional first stage boot loader: XLoader. In fact it in most cases it is XLoader – a quick invocation of my toolchain’s string implementation seemed to correlate with this:
$ arm-none-linux-gnueabi-strings /media/boot/MLO | grep X-Loader ()*+,-./0123456789:;< =>?Texas Instruments X-Loader 1.5.0 (Mar 27 2011 - 17:37:56) X-Loader hangs
I was curious as to why it was named MLO, how the board boots into this Image and how I can create my own MLO with some bare metal code. This post answers these questions and results in a very simple executable MLO file. It will probably satisfy those readers who like to understand and write all the code that runs on a board. It’s very easy to use a boot loader like UBoot to obtain and execute an image from an MMC card – but it adds boot time, duplication and complexity. Besides it’s fun to get close to the metal…
Last year at the ELCE 2010 conference in Cambridge I performed a talk about reducing boot times of embedded Linux devices. The video of this talk has now been posted on-line:
ELCE 2010 – The Right Approach to Minimal Boot Times ELCE Video (Best viewed in Chrome)
The accompanying slides for this presentation can be found here