Archive

The support period of the Lucid Lynx on the Asus UL20A is about to end in May, so I was looking for a new OS. I gave a try to Debian Testing, or Linux Mint Debian Edition, to be precise. But not Pangolin. Sorry.

It was my smoothest Linux install ever. All hardware worked out of the box. Volume and brightness buttons? Check. Wifi? Check. Two-finger scrolling? Check. I also applied the usual MATE fixes for mutt and the power-button. Debian comes with a 486-kernel by default, so it handles only one processor core. One can upgrade manually to a multi-core 686-kernel with apt-get.

aptitude search linux-image

And what do I find?

linux-image-rt-686-pae

A PREEMPT_RT patched kernel, kindly compiled by Debian. Time to plunge into real-time again!

I put together a Linux syscall tutorial project a year ago (see github) which contained a small program performing a periodic real-time task and printing statistics like average/min/max dt and its standard deviation. It’s called rt and uses the setitimer() system call to generate periodic SIGALRM signals. Time to measure latencies of a 1ms periodic task on a RT kernel!

To my utter disappointment, performance was fine, until the tapeta daemon changed my wallpaper, that is. At which point a 300% latency appeared. That’s right, period time 4ms instead of 1ms. Same as earlier, without an RT kernel. What’s wrong?

I was not using real-time scheduling, that’s what. It turned out Linux processes run at static priority zero by default, where the actual scheduling priorities are dynamic, depending on interactivity, nice level, sleep/runtime, etc. This is called the SCHED_OTHER policy. In other words, one is at the mercy of the Completely Fair Scheduler.

On the other hand, a process’ static priority can be increased, and the scheduling policy can be changed to SCHED_FIFO with the sched_setscheduler() system call. In this case the only one whose mercy we are at, is another SCHED_FIFO or SCHED_RR (Round Robin) scheduled process with equal or higher static priority, lurking in the background somewhere. Important to note, though, a process needs to run as root to be able to change its own or another process’ scheduling. There is also a command line utility called chrt, to change the scheduling policy of processes.

By the way. I’m not using threads. Threads are evil. People who use them mostly condemn themselves to a thousand hells of corrupted data, deadlocks and, at the end of the day, slower performance, thanks to cache misses and dirty cache lines. And the uClibc library in many embedded systems does not support POSIX threads. Thank goodness. I’m particularly angry with threads, as every damn RT-Linux tutorial you look at, spends 75% with setting up the bloody things. And you even have to worry about which thread your signals will be delivered to. What about learning fork() and pipe() instead? KEEP IT SIMPLE STUPID!

But enough of my rants and dodgy theories, let’s see the practice. First let’s see, how the skeleton of a periodic task looks like.

And now let’s see what happens when a process becomes real-time. I’m measuring latencies of a 1ms task on Raspberry Pi, actually without a PREEMPT_RT kernel. I start rt as a normal process, then change its scheduling to SCHED_FIFO with chrt. See the code of rt on github.

A line of statistics is printed after every second with the following columns: n (number of events), Mean dt, SD (standard deviation), and Min/Max dt. Whilst the program was running, after the 12th line of data above, I changed rt’s scheduling with the command below.

The effect was truly dramatic. With normal scheduling I was missing about 2% of the events, with an average latency of 20% (0.2ms), in the worst case up to 600% (6ms). After changing to real-time scheduling, not a single event went lost, average latency dropped to 2% (0.02ms!), and the worst case delay dropped to 31% (0.31ms).

All this is causing 7-8 % processor load on a Raspberry Pi running two ssh sessions and omxplayer playing an internet radio. Single core, no PREEMPT_RT kernel. You probably need the latter in a production system which must not miss a beat for weeks.

So Linux is definitely able to run an ECU. That’s where my new ARIA25 board will excel. After someone designs an ECU for me…