What I Wish I’d Known When I Was an Embedded Linux Newbie

Here are some tips compiled from our seasoned engineers on what they wish
they'd known about embedded Linux back when they were "newbs".
Newcomers and seasoned veterans alike should get some good nuggets of
information and possibly a fun perspective looking back at our own humble
beginnings. We'll try not to overwhelm you as we make our way through
the list. We're not here to rewrite the books, but we do want to provide
a personal perspective. If you're in the camp of people who've
been using desktop Linux, just be aware that embedded Linux is a different
animal, especially when it comes to space constraints, different CPU
architecture (ARM), resilience to sudden power outages and inability to
install any mainline Linux kernel or distribution you please. Or, maybe
you're in the microprocessor camp moving toward a more generalized and
capable embedded Linux system. Either way, we'll assume you have at
least some knowledge of Linux as we walk through this guide.

Getting Around

One of the first things you've probably already come to notice in an
embedded Linux product is the lack of a nice desktop environment. No GUI
here, my friend! Here, the command line is king. Just as you're
probably very proficient with a mouse, you'll come to develop muscle
memory to make you as proficient if not more using a keyboard. Right now,
you're probably feeling more like this finger-typer here, and
that's okay! We're here to tell you things are going to get
better.

Serial Console (RS-232)

One of the first things you'll use is a serial console. Yes,
"serial console" as in the archaic Microsoft Hyperterminal program
to connect to your board to get a command prompt, or shell in Linux-speak.
The serial console is the most trusted method of communicating with the
board since it's not prone to network failure as is SSH and Telnet,
plus you'll be able to see both startup and shutdown messages. For
connecting via serial console, take a look at minicom (Linux/Mac/Cygwin) or
PuTTY (Windows). A serial console must be configured to talk at the same
rate of speed as the board you're connecting to. It seems the most
common is 115200 Bps with no parity bit and no hardware or software flow
control. This is what minicom setup (minicom -s) looks like on a Mac:

Understanding UARTS like RS-232, RS-485

It's advisable to brush up on your UART communication skills, since
connecting to RS-232 serial console is much the same as connecting to a
standard TTL-level UART or RS-485. The following are some resources that may help your
understanding:

Secure Shell

A heavily accessed tool in the toolbelt is secure shell, and you'll want
to be sure to become efficient with it. It comes down to two main commands
that use the same underlying technology: SSH and SCP.

SSH

SSH allows you to connect to your board's shell prompt using the
network. It uses port 22 by default, and you can set up port forwarding in
your router to allow for remote access to the board. Typical usage is very
simple:

1 ssh @
2 ssh root@192.168.1.50

SCP

SCP can be thought of as "Secure Copy". This allows you to
transfer files to and from your board. If you can ssh to your board, you
can copy files. Typical usage:

scp @:
scp foobar.txt root@192.168.1.50:/root/

Screen and Tmux

Instead of having multiple windows open to the same board, try using screen
or tmux. Each of these programs can provide multiple shells for a single
connection with the added bonus of keeping your shells active and open even
when your connection is closed. Screen usually comes preloaded, is
smaller than tmux and is usually sufficient. Tmux is more powerful and
has its own advantages. Since screen is so readily available,
it's nice to dedicate muscle memory to it.

Editing Files

Since you'll be spending a lot of time in the command line, you'll
want to try to become proficient at using a non-GUI editor. There are two
main contenders here: vim and Emacs. Do a Google search for vim vs. Emacs
and you'll quickly find there are two separate camps, each very
passionate about their preferred editor. The engineers here at Technologic
Systems are mostly in the vim camp, but encourage you to decide for
yourself which is right for you.

Vi and Vim

Most embedded Linux boards will come preloaded with at least vi,
the original vim. Vim is simply "vi improved". A vim
install takes a bit more storage, but in this author's opinion, it's
totally worth every extra byte.

[Note: Derek Wyatt has a fantastic and entertaining set of videos for learning
Vim, from basics to advanced usage.]

Emacs

Emacs has a more traditional word editor feel than vim, which many beginners
find comforting. If it's not installed by default, it easily can be as
it's in most package management systems like apt-get and yum.

There's a guided tour of Emacs at GNU.org if you're interested
in learning more.

Disk and File Corruption

You've seen this message before. Admit it, you've unplugged a disk
without ejecting it first because you think that message is for chumps, and
you've never had any problems with data on your USB thumbdrive. In
the embedded Linux world, this is an absolute must, as disks are prone to
data and filesystem corruption when the carpet is pulled from underneath.
This includes sudden power outages. You can attempt to use fsck to repair
the disk, which works for a while, but you'll inevitably find yourself
in a situation where it cannot repair the disk any longer. We've
written a whitepaper on filesystem corruption and ways to prevent it from
happening.

A utility to get in the habit of using is sync, which commits buffer cache
to disk. You can read more about it in the sync man page.

NFS

Standing for Network File System, NFS is a great tool to use for accessing
files between two computers on a network. The definition is simple, but
the implications are many. For example, you can set up (or export) an NFS
share containing a tarball of a Linux distribution and set up your board to
load it. You can also set up an NFS share to edit your project files in an
IDE on your desktop machine and then compile it on the embedded board
without the need for manual copy/paste. The Ubuntu community has a really
nice article on this: "Setting Up NFS
How To".

Programming

Whoa. What an overwhelming topic. It's an exciting one, and
certainly important, but there are so many different languages, platforms,
memory addresses, processors, styles, etc., it'd be very difficult not to
overwhelm a reader in this article. Let's just say that in the
embedded Linux world, the language of choice is usually C because of
low-level hardware optimization. In fact, the Linux kernel is written in C,
a decision that Linux creator Linus Torvalds has a very non-apologetic stance on.
Because of limited resources, efficient and fast (aka inexpensive) code is
sought after. For example, floating point arithmetic is generally frowned
upon because it involves a lot of layers, including the kernel, to make
floating point happen, which is very expensive and time consuming. If your
CPU has a hardware floating point unit, then it's not really a problem.

One pitfall newbies might fall into is NIH (not invented here)
syndrome, where they feel they have to re-invent the wheel. In actuality,
there are a ton of existing resources and existing libraries (such as key/value
storage like Berkeley DB, send/receive data via HTTP or FTP). A new
development project ought to be preceded by an extensive search or research
phase before jumping in. Some Linux API calls exist that a lot of people
don't realize, like sched_setscheduler(), mlock(), mlockall(), etc.
Also, having an understanding about /dev/mem makes development in userspace
much easier and forgoes the need to develop kernel drivers. Resist the
temptation to put more and more of your embedded application into the
kernel and stick with the bare minimum and you'll almost always be
better off in the long term. Along those lines, don't think of your
device as a monolithic application and instead separate things into
processes. Not everything needs to be optimized, so don't be afraid to
use tmpfs filesystem or pipes to communicate or shell scripts. Really, it
comes down to not underestimating the virtue of simplicity. Use layers,
but use them strategically, being sure not to add them until necessary, else
your system almost always will get slower, have difficulty troubleshooting
failure modes and result in higher demands on hardware. Layering is an
under-utilized advantage of embedded programmers coming from
microcontrollers and over-utilized by developers coming from a PC or
sysadmin background.

ctags and cscope

If you're doing anything related to software development, be sure to
take the time to learn about ctags and cscope. These give you a method
for jumping to and from function and variable declarations across your
project files and can work together. Both vim and Emacs work with ctags
(etags for Emacs). You can learn more about ctags and cscope by reading
our guide, Tag
Jumping with ctags and cscope.

Kernel Compiling

Eventually in your embedded Linux travels, you'll come across some
instructions that say you need a particular kernel module. When you find
out your kernel doesn't have it, it's intimidating to think of what
you'll need to do in order to compile it yourself from scratch.
But, it's not actually as scary as you might think, and it's a very good
way to improve your understanding of Linux. Each single board computer or
computer on module will have slightly different instructions and different
versions of the kernel, and Technologic Systems' engineers have worked
hard to write guides on several of their products. The guides aren't
incredibly long, and they go over things like choosing the module to compile via
make menuconfig and inserting the module via insmod when you've
successfully compiled them. Take a look at the following guides as a
starting point, and then reach out to a support channel to find out more
specific information about your board:

NAND Flash Considerations

NAND flash is really fast, but don't write to it a lot as it's not
as fantastic as it seems. eMMC flash is more reliable, doubly so when
configured in SLC mode.

Checking Space

Disk space is usually constrained in an embedded Linux device. You may
have the ability to increase storage using an SD card or USB thumbdrive,
but likely, you'll be booting from an onboard flash drive that might
have only 4GB or less. Make sure you're aware of file sizes and their
units—for example, MB is different than MiB (MiB = Mebibyte = 1024KiB; MB
= Megabyte = 1,000KB). Commands like du and df
help identify how the storage is being utilized. You also can remove and
purge unnecessary locales or language translations to free up space. If
you're in a Debian-like environment, be sure to use tools like apt-get
clean and autoremove to keep things on the up and up.

Reading/Writing Images

Undoubtedly, you'll need to restore a factory image or load a prepared
image onto your embedded Linux device. Usually the preferred tool for
generating these images is dd. We have a guide that will walk you through
how to do this in various platforms. See How to Write an SD Card Image
(Linux, Windows, Mac OSX).

Distributions and Tarballs

Since Linux is a file-based OS, you can create a tarball of your
distribution and extract it either to a prepared partition or NFS
filesystem and it'll boot. You can create a tarball using a command
similar to:

Recommended Reading

Let's switch to a first-person view here for a moment. When I first started in
embedded Linux, this book was an invaluable resource for me. I'd recommend
this be on the required reading list for newly hired employees or interns.
It explains in very simple terms exactly what's happening in an
embedded Linux platform, much like the products we sell at Technologic
Systems. So, please take a look and read it: Embedded Linux Primer: A
Practical Real-World Approach by Christopher Hallinan.

Conclusion

Hopefully you feel like you've come away with some great tips to get
you going in the embedded Linux world. There's still so much to learn,
so get out there and start exploring!