What's going on in the workshop?

Tag Archives: embedded

A little embedded Linux magic comes in unexpectedly handy. Years ago, I started using Acronis True Image to back up my office PC. Other backup packages are available, as they say, but it’s worked well enough for me. This week I had reason to use it in earnest.

The PC is set up with two main partitions on the disk. There’s one for the operating system (Windows 10) and applications, and another with data on it, analogous to the /home partition on a Linux system. Inevitably, it turned out that the boundary between these partitions was in the wrong place. The operating system partition was full, and things were starting to fail. Time to make sure the backup is up-to-date and then to restore to differently-sized partitions.

The operation fell at the first hurdle. Acronis True Image will create a ‘recovery’ USB stick from which the PC boots. This contains a small Linux distribution (Acronis Linux) and a GUI for managing the recovery operation. My backups are on a network drive, which happens to be an Apple Time Capsule. Would the Acronis recovery system connect to the Time Capsule? Would it ever. I couldn’t even persuade it to ask for the username and password, though it could see the Time Capsule on the network.

In Acronis Linux, pressing ctrl-alt-F2 in time-honoured fashion brings up a command line. Yay! Power! A bit of digging reveals a command called asamba which can connect to SMB shares. Using that, I could connect to the Time Capsule absolutely fine and mount it to the local filesystem. So why couldn’t the GUI do it? Annoying.

My next trick was to copy the backup from the Time Capsule to a USB hard drive using another Linux PC. It took a while for the several hundred gigabytes, but it got there. Acronis recovery will read USB drives, so the next step should be easy, right? No. My USB drive happened to be formatted ext4. Though Acronis Linux is perfectly capable of mounting ext4 filesystems, the recovery GUI only acknowledges the presence of FAT and NTFS discs. Aargh!

The solution was to mount the USB drive on my Linux PC, then share it using Samba. Acronis recovery found and let me log on to the Samba share, and could thus see the drive and backups.

Changing the partition sizes was the next problem. The GUI only allows you to shrink partitions or to grow them in to free space. My disk didn’t have any free space. It was all partitioned and in use. I thought that shrinking the data partition would create free space that the operating system partition could grow in to, but I was caught out by the presence of a 500MB ‘recovery’ partition which Windows had sneaked in between the two. I don’t know what it’s for but I didn’t feel like taking the risk of destroying it. There are no tools in the Acronis GUI for deleting partitions and starting again, so I was stuck.

Command line to the rescue! Ctrl-alt-F2 again, and there’s a perfectly good copy of fdisk available. It was a moment’s work to delete the existing recovery and data partitions, then ctrl-alt-F1 to the GUI. Restart the restore process and there’s free space available for the operating system partition to grow in to, and plenty of room to create new recovery and data partitions.

The restore took a few hours, as expected, but was successful. The only wrinkle was that Acronis recovery had marked the recovery partition as active rather than the boot partition, so the PC complained that there was no operating system. Another trip in to fdisk to mark the boot partition active sorted that out, and now I’m typing this very text on the PC restored to working order.

Footnote: there is another Acronis restore tool intended for restoring backups to dissimilar hardware. This may have handled the partition resizing more elegantly, but I didn’t try it.

There are many ways of connecting sensors and devices to a Raspberry Pi. One of the most popular is the I2C bus. It’s great for devices which don’t need to transfer too much data, like simple sensors and motor controllers, and it’s handy because lots of devices (up to 127, or even more) can be connected to the same pair of wires, which makes life really simple for the experimenter. I’ve mentioned using the I2C bus in another blog post, because sometimes a bit of software fiddling is needed to get it to work.

It fits on top of the Pi, and has lots of useful features including a student-proof power supply, real time clock, accelerometer, space for a Zigbee module, analogue inputs, diagnostic LEDs and four motor driving outputs, all wired to convenient connectors.

The analogue inputs and motor outputs are implemented by a PIC microcontroller connected to the I2C bus. The software for the PIC was written by an undergraduate several years ago. It works well, but seems to have some odd habits. I found that it would apparently work, but sometimes an attempt to read data from the PIC would just fail, or return wrong data, and sometimes data would get written to the wrong register. At first I suspected a wiring problem, but examining the SDA and SCL signals with a scope showed nothing wrong. I tested another device on the same bus – a Philips PCF8575 I/O expander – and it worked perfectly every time. That narrowed the problem down to the PIC. Since there was nothing I could do about the PIC’s software, I had to find a workaround.

I spent some time experimenting with where the communications seemed to go wrong. Reading from an I2C device usually involves two separate operations on the bus. The first one tells the I2C device which register address we want to read, and the second does the actual read. The diagram below shows the sequence. The ‘control byte’ in each case sends the address of the I2C device (0x30 in this case) plus a bit indicating read or write.

I found a pattern in the failures. From time to time, the write operation which sets the register address would fail, reporting ‘I/O error’. After that, reading the data would return the wrong value. I modified my code so that if the write operation failed, it would retry a couple of times before giving up. It turned out that retrying was always successful, if not on the first attempt then on the second. However, the data read would still return the wrong value. The value returned was always the address of the register I wanted! It seemed as if something was getting stuck somewhere in the I2C system. Whether it was in the Linux drivers, or the PIC software, I don’t know, and I didn’t spend long enough to find out. My assumption is that the PIC software is sometimes just too busy to respond to the I2C operations correctly.

I tried the retry strategy again, and it turned out that the second attempt to read the data byte always got the right value. The algorithm to read reliably looks like this, in pseudo-code:

if (write_register_address() fails)
retry up to 3 times;
read_data();
if (we had to retry writing register address)
read_data();

In practice I was using the Linux I2C dev interface to implement this. Yes, it’s a bit of a nasty hacky workaround, but it did get the communications working reliably.

There was another device I wanted to talk to: an Arduino Mini running a very simple sketch to return some sensor data. This also used the I2C bus. There are handy tutorials about how to get an Arduino to behave as an I2C slave device, like this one. The I2C interface is implemented nicely by the Wire library. Implementing a slave involves responding to two events: onReceive and onRequest.

The onReceive event is called when data, like the register address, is written to the slave, and the onRequest event is called when the master wants to read data. My initial code looked like this:

This worked most of the time, but after a few thousand transactions, it would appear to ‘lock up’ and ignore any attempt to change registers – it would always return the same register, and in fact no more onReceive events were ever generated. Of course, it turned out to be my fault. When reading data in the onReceive event code, it turns out to be important to make sure that data is actually available, like this:

Over the last few years, there has been a lot of controversy in the Linux world about systemd. As I understand it, systemd is intended to be a better-engineered, more powerful version of the motley collection of little programs and scripts which keeps the essential services on a Linux system running.

The controversy arises because the original 1970s Unix way of doing things was to rely on a motley collection of little programs and scripts for everything, each of which was simple but well understood, and to knit them together to form a complete operating system. Systemd takes a different approach, using larger and more sophisticated components which are more dedicated to particular tasks, such as managing services or network connections. This is supposed to make it more efficient and easier to manage in the twenty-first century.

I’ve been doing some work recently on an embedded Linux system which runs on the latest version of Debian Linux, version 8 (‘Jessie’). Debian Jessie fully supports systemd to the extent that it seems to be the default way of doing things. I thought I’d experiment with it a bit.

When working on an embedded Linux system, I very frequently want to have a piece of my software run reliably at startup, get restarted if it fails, and be able to output logging information to an easily-managed place. In this case, my software provides a D-Bus interface to a piece of industrial electronics.

In the past I’ve relied on copying and pasting scripts from other pieces of software, and managing log files has always been a bit of a mess. It’s hard to do these things right, so re-inventing the wheel is too risky, which means that the best strategy is to copy somebody else’s scripts. I have never counted the hours of my time which have been wasted by dealing with awkward corner cases and peculiar bugs due to recycled scripts behaving in ways I hadn’t anticipated.

What does it look like with systemd? There are some helpful tutorials out there, including this one from Alexander Patrakov, so it didn’t take me too long to put together a service file which looks like this:

I’ve changed the names to protect the innocent, but the contents of the file are pretty self-explanatory. The [Unit] section just includes a description which is readable to a human being. The [Service] section describes the service itself. In this case it’s of type dbus, which means that systemd will check that the service name (com.martin-jones.gateway in this case) gets correctly published on to D-Bus. The Restart=always setting means that my software gets restarted if it exits. The [Install] section just indicates that this service should run when the system comes up in multi-user mode (like the old runlevel 5).

Having created this file, I simply copied it into /etc/systemd/system/my_dbus_gateway.service and, lo and behold, my new service worked. It was immediately possible to manage the service using commands like

Now for logging. I’d heard that systemd would log the stdout and stderr outputs of services into its own journal, and forward that to syslog as required. It does, but there’s a subtlety. Output from stderr appears in /var/log/syslog immediately line-by-line, but output from stdout gets aggressively buffered. This means that it gives the appearance of not working at all unless you explicitly flush the stdout buffer in your code using something like

fflush(stdout)

That’s the only wrinkle I came across, though.

In summary, using systemd’s facilities has made my life as an embedded Linux developer much, much easier and hopefully more reliable. That’s a good thing. My top tips for getting your software working under systemd are these:

Create your .service file using the recipe above and the documentation

Working with Lattice FPGAs recently, I had a need to program one but couldn’t find my ‘proper’ (Chinese clone, bought from eBay) programming adapter. When I started the Diamond Programmer software, though, it claimed it could see a USB programming adapter. It turned out that I’d left an FTDI ‘FT2232H Mini Module‘ attached to the PC. I use the module for all sorts of little debugging exercises: most often as a dual serial port for serial port debugging, but it also works for programming Parallax Propeller microcontrollers.

As luck would have it, the Diamond software recognises the unadulterated FT2232H as a legitimate USB programmer, and pressing the ‘Detect Cable’ button finds it. Note that if you plug in a new USB device, the Diamond Programmer software needs restarting before it can see it.

The FT2232H has two ports, A and B, and these appear as ports FTUSB-0 and FTUSB-1 in the Diamond software. All that remained was to figure out the wiring. Fortunately, there are a lot of clues in the schematics of various Lattice evaluation boards, particularly the MachXO2 Pico Board and the iCE40 Ultra Breakout Board.

Here’s the wiring, both for SPI and JTAG, referred to the pins on the Mini Module. I chose to use port B since it was more convenient for my prototype board. Translating the wiring to port A is left as an exercise for the reader.

Today’s announcement of the Shellshock Bash vulnerability had me worried. I run lots of Debian Linux systems, and they’re not all the latest version. Many are still Debian Squeeze (version 6) which no longer gets security updates as standard. That’s my fault, of course, and I should have upgraded, but I haven’t. Yet. Now I’m more motivated to do it. However, upgrading to Debian Wheezy (version 7) isn’t something I wanted to do in a hurry, especially on remote machines.

Debian have thought of people like me, and there is a ‘Long Term Support‘ option for Debian Squeeze, which is great, and includes the necessary security update to Bash. The trouble is, it only supports i386 and amd64 processors, and the machines I’m worried about are ARM (specifically armel) ones.

I was left with one option: build the new Bash from source. Fortunately, Debian Squeeze LTS has the source available, so I was able to do this. Here’s how. This might be useful to other Debian ARM users who are none too fastidious about keeping up to date.

which was a royal pain due to lack of disc space. Beware, these packages want about 180MB of disc space (plus about 80MB for the package downloads) so might need some care on a small system. I started by installing packages individually, doing ‘apt-get clean’ after each one, but texlive-latex-base is an absolute monster and I had to do some filesystem reshuffling to get it to install. I hope you don’t have to.

During the build (repeating the dpkg-buildpackage command above) the patch for ‘CVE-2014-6271‘ was mentioned, which was reassuring. The actual build process took a while – about half an hour on a 1GHz-ish ARM chip (a SheevaPlug).

The build completed successfully, so I was able to install the new package:

I’ve recently had a need to access SD memory cards from an Atmel AVR microcontroller. There are plenty of libraries out there which can do it, which saved me a lot of time, but choosing the right one and getting it to work wasn’t entirely straightforward.

I looked at sdfatlib, which is intended for use with the Arduino. The Arduino also uses an AVR microcontroller but the code is mostly written in C++. My project (using Atmel Studio) was set up to build a lot of complex code in C, and I didn’t fancy trying to convince it to build C++ and link successfully. The compiler balked at the first mention of the word ‘class’, so I decided not to pursue it.

Next in the candidate list there was FatFs, which is written in C but targeted at all sorts of microcontrollers including the AVR. It looked more generic than I needed, and I didn’t want to spend too much time dealing with the low-level SPI stuff to talk to an SD card, so I moved on. I would certainly consider it for future projects though.

The final choice was sd-reader, written and released by Roland Riegel. This was the answer to my wishes: all written in C and well tested, specifically targeted at SD cards on AVR microcontrollers. It compiled as part of my project without a problem. However, there were some issues to deal with.

Firstly, it’s set up assuming that it has sole control of the SPI interface. My project already used SPI for another peripheral, so I had to do some trickery in sd_raw_config.h and my existing code to dynamically reconfigure the SPI port for each peripheral each time it was selected. I replaced the macro:

#define select_card() PORTB &= ~(1 << PORTB0)

with

#define select_card() ({SPCR=0x50; nSD_CS_PORT &= ~nSD_CS_MASK;})

which forced SPCR, the SPI control register, to the right value for the SD card. I had to do something similar in my other peripheral code. This all worked fine but a small change to sd_raw_init() in sd_raw.c was needed because the initialisation of the card takes place with a very slow clock speed, so that accesses SPCR as well.

I wanted to enable access time stamping in my application, so I set

#define FAT_DATETIME_SUPPORT 1

in fat_config.h. The documentation indicates that functions fat_set_file_modification_date() and fat_set_file_documentation_time() are available, but in fact they’re not. They’re declared static and used internally by the sd-reader code. The way to get time stamps to work is to define a function:

is claimed by the documentation to leave dir_entry filled in with the directory’s details if the directory already exists. It does, but returns 0 indicating an error, so there’s no way to tell if it was impossible to get the directory or it just already existed. I worked round this by attempting to open the directory first, and only if that failed, attempting to create it.

Caveats: I may not have been using the most up-to-date version of the software, or I may have misinterpreted the documentation, or these things might have changed since I wrote this text.

I’ve just successfully got Debian Linux running on a Raspberry Pi, having built it entirely from the balloonboard.org distribution. Why might you want to do this? Well, I did it because it gives me a small, clean Debian Linux installation which I can then customise. Here’s how I did it.

Create yourself an SD card with two partitions on it: one smallish FAT partition for the boot files, and a big ext4 one for the root filesystem. Don’t forget to format them.

Into the FAT partition copy the contents of build/kernel/rpi-initrd-boot. Into the ext4 partition untar the file build/rootfs/debianrootstrap.modules.tgz.

Now put the SD card into your Pi and switch it on. This should boot into a ‘recovery kernel’ which has a minimal root filesystem in its initial ramdisk, just enough to sort out the ‘proper’ root filesystem. I used the console serial port to work with it. The HDMI and USB ports might also work but I haven’t tried them. It should come up with a login prompt. Log in as root, password rootme.

Now to configure the root filesystem:

mount /dev/mmcblk0p2 /mnt/root
chroot /mnt/root

Finish the Debian installation (this has to be done now because some aspects of it need to run on the ARM processor, so it’s not easy to have your PC do it). It will ask you questions about time zones and things:

/var/lib/dpkg/info/dash.preinst install
dpkg --configure -a

Set a password for the root account so you can log in

passwd root

and add the serial port to the list of secure ports which are considered safe for root logins:

echo /dev/ttyAMA0 >> /etc/securetty

And shut things down

halt

That last step will probably produce an error message, but it doesn’t matter as long as it’s written everything to the SD card.

Now put the SD card back in your PC, and copy the contents of build/kernel/rpi-boot into the FAT partition. That contains the real kernel which will mount the newly-minted root filesystem. Put it back into the Pi and boot. It will ask you to change the root password at first login.

It worked for me, though I had to ‘ifdown eth0’ and ‘ifup eth0’ again to get Ethernet to work. From that point on I was able to install Debian packages normally.