Isolation and Integrity Techniques on Embedded Linux Devices

Introduction

This article is the first in a series of articles that will be published in our blog, covering isolation and integrity techniques. The series focuses on Linux Internet of Things (IoT) devices, but much of the material we will cover is suited for embedded Linux in general, and regular Linux machines such as desktops and servers.

A basic tenet of operating system security is to keep running processes isolated from each other, and to ensure that processes run with the lowest possible level of privilege. Otherwise, attackers would have a much easier time of executing privilege escalation attacks, gaining fuller control of the system after gaining a foothold through an initial attack. For instance, an attacker could break into a normal user account, and then try to obtain administrative privileges, or break into an exposed network service and then try to compromise other functionality on a device. These two cases exemplify two types of attacks:

Vertical escalation - accessing a device without credentials or with normal, unprivileged user credentials, and then find a way to gain administrator privileges. With administrator privileges, the attacker can access anyone's private data, plant malware, or configure the system in undesirable ways. Attackers can use flaws in software, such as buffer overflow vulnerabilities, to perform this type of privilege escalation.

Horizontal, or lateral movement - accessing the device with one set of unprivileged credentials (possibly obtained in a previous attack), and then accessing configurations or data that belong to other accounts. In a real world analogy, this would be an attacker logging into his or her own account on a banking website, and then going from there into the account of some other user (probably not to to deposit funds into the other user's account). A horizontal attack could happen, for example, due to problems like improper permissions settings on files and directories.

Isolation techniques can prevent both of these types of attacks from happening, and integrity checking can help detect when attacks have occurred, so that remediation steps could be taken.

In this series of articles, we will examine some of the major isolation techniques and integrity checking methods available on Linux, with practical examples on isolating popular web servers and network services.

Since we focus on IoT, our main concern is with isolating processes from each other, rather than isolating human users. We will examine Linux distributions common on IoT, such as Ubuntu and Debian variants, alongside more common desktop and server distributions.

Outline

In this article, we will cover the following subjects:

Discretionary Access Controls (DAC) - on Linux, file system permissions are implemented using the DAC approach.

chroot - a way of creating a limited view of the file system for a process we want to restrict.

Mandatory Access Controls (MAC) - generally less permissive than regular DAC controls, these isolation mechanisms restrict access to a wider choice of system objects. We will cover two popular Linux MAC systems - SELinux and AppArmor.

This will be continued with future articles on the the following topics:

Jail packages - those make it easier and more user-friendly to deploy and combine the other building blocks here, including the kernel isolation mechanisms above and chroot. Minijail and Firejail are two jail packages that we will examine.

With this plan in mind, let's dive in. We begin with an overview of Linux Discretionary Access Controls.

Discretionary Access Control (DAC)

DAC is the simplest and most common method of restricting processes under one account from accessing data and functionality belonging to another account, or to the system. This method is mainly used for isolating different users' files from each other, but on a system where different processes run under different user accounts, it can be used for process isolation.

Open a command-prompt on any Linux machine, and do a ps aux command. You'll see that different processes have different owners:

ps aux: This is the command that shows information about all system processes.

| grep: We're taking the output of "ps aux", and using it as the input for the "grep" filter utility.

-v: We want to look for the inverse of what the search term is.

^root: We're looking for all lines that have the word "root" at the beginning. But, since we're also using the "-v" option for grep, we're actually looking for all lines that do NOT begin with the word "root".

The first thing to note here is that the processes of certain system daemons are owned by non-root users. This helps prevent hackers from using those daemons to exploit the system. The processes that are associated with these daemons have the privileges to perform their own specific jobs, but nothing beyond that. The second thing to notice is that there are also quite a few processes that the logged-in user "donnie" owns. This separation prevents the user from accessing other processes, to do things that would affect either the system or other users.

Now, let's see how this would work out in a practical use-case. Let's say that we've set up an Apache webserver on a Linux machine. We can do a ps aux command to see what the Apache processes look like. On a Red Hat/CentOS machine, the process name is "httpd". The original process is owned by the "root" user. But, all spawned processes - the ones that actually respond to client connection requests - are owned by the "apache" user.

The purpose of this is to prevent someone from using the Apache daemon to access files, directories, and processes that either the "apache" or the "www-data" user doesn't have permission to access.

Now, let's say that our web content files contain some super-secret data that we only want for certain authorized users to see. When we look at the permissions on the /var/www/html directory, which is the default location for storing web content files, we see that only someone with root privileges can write to the directory, but that anyone can view and read the directory contents.

This means that anybody who has access to this machine can view the super-secret data in the web content files. We can fix that with two simple commands. All we need to do is to change ownership of the /var/www/html directory to either "apache" or "www-data", depending upon whether we're on the CentOS or the Ubuntu server. Then, we'll change permissions so that only either the "apache" or the "www-data" user can access the directory.

cd /var/www
sudo chown -R apache: html

or

sudo chown -R www-data: html
sudo chmod 770 html

NOTE: Normally, leaving the /var/www/html directory set with it's default ownership and world-readable permissions isn't a problem. In the above example, we were changing things only because we decided to store secret data in the website content files.

In general, process ownership is something that's determined by either a program's internal code or by a configuration file. Most of the time, these things are already set up by the package maintainer, so the end user doesn't have to worry about it. Of course, in this example, whoever maintains the website would need to have proper sudo privileges in order to update the web content, but that's a topic for another discussion.

Using DAC is an easy and convenient way to protect users from other users. Unprivileged users can set permissions on their own files and directories in order to restrict who can access them. Administrators can create shared directories for specific user groups, and set the permissions so that only members of the group can access them.

For process isolation, DAC has only limited uses. The sad reality is that many processes need to run with root user privileges in order to do what they need to do, and there's nothing that we can do to change that. Whenever you develop your own programs, it's best to set them up to run without root user privileges, if at all possible. Sometimes, a program only needs root user privileges upon initial startup. Whenever that's the case, if possible, write the program in such a way that it will drop into unprivileged mode once the initial startup has completed.

In practice, Discretionary Access Controls are difficult to apply to their full potential on embedded devices, as they require separating all the device's functionality into distinct services and components running under individual restricted accounts, and then defining the exact permissions necessary for each component. Unfortunately, many IoT devices run much of their software under the root account, which bypasses DAC checks completely, and even if not, the access controls are usually configured to be too permissive, allowing attackers who take over one component of the device to investigate or change much of the device's inner workings.

chroot

The next method of process isolation, which is available on all major Unix and Linux distros, is called chroot. Essentially, this means that we're setting up a directory structure that appears to a certain process or user as if it's the root of the filesystem. Our objective is to create a "chroot jail" from which users and processes theoretically can't escape, in order to prevent them from accessing other areas of the filesystem outside of the jail.

We will examine the possible ways of applying chroot to the Apache Webserver and a few popular network services, as an example.

Apache Webserver with chroot

The traditional way to set up a chroot filesystem is to create a directory tree that will serve as the simulated root filesystem, and then copy into it everything we need to run the jailed process. For example, let's say that we want to set up an Apache webserver in chroot mode. Back in the good old days, we would had to have created the chroot directory structure and copied the Apache executable file and all of its associated library files into it. After creating the web content files within this directory structure, we then would have had to configure the virtual hosts to recognize this chroot filesystem. The whole process was quite daunting, and didn't scale well at all.

Then, someone invented the mod_chroot module as an Apache plug-in, which somewhat simplified the process. Ever since Apache version 2.2.10, chroot capabilities have been built right into the Apache executable. However, the official Apache documentation page for the "ChrootDir" directive still tells us:

Note that running the server under chroot is not simple, and requires additional setup, particularly if you are running scripts such as CGI or PHP. Please make sure you are properly familiar with the operation of chroot before attempting to use this feature.

The whole point of setting up Apache in chroot mode is to prevent attackers and users from escaping into the main Linux filesystem. The goal is to prevent anyone from either accessing sensitive data that's elsewhere on the system, or from using malicious program files that they may have uploaded to attack other parts of the filesystem. Unfortunately, even though setting up Apache in chroot mode is a daunting task that's not for the faint-hearted, the rewards just aren't there. Running Apache in chroot mode might keep the script-kiddies at bay, but experienced hackers can still break out of the chroot filesystem and wreak havoc. One way they can do this is simply by running recursive cd commands. Once out of the chroot filesystem, an experienced hacker could access sensitive files on other parts of the filesystem, or run malicious programs that could affect the whole system. Clearly, we need something that's easier to set up, and that's more effective than chroot for enhancing the security of an Apache webserver. This can be done by the different Linux jail packages, and we will have a look at those in later articles.

BIND DNS Server in chroot Mode

Contrary to what we've seen with Apache, setting up a BIND DNS server in chroot mode is just as easy as can be, as long as you do it on a Red Hat-type operating system, and as long as you follow one simple tip.

NOTE: "BIND" stands for "Berkeley Internet Name Domain", and you install it by installing the "bind" package.

Red Hat-type operating systems, such as Red Hat Enterprise Linux or CentOS, come with an available bind-chroot package that automatically sets up the chroot environment for your DNS server. The trick though, is that you don't want to install bind-chroot at the same time that you install the bind package. Rather, install the bind package first, then set up your named.conf file and your zone files in the normal manner. Once you're done with that, just install the bind-chroot package, and all of your zone files and configuration files will automatically get moved over into the chroot filesystem. Nothing could be easier. Still though, chroot is still chroot, which means that it's relatively easy for an experienced hacker to defeat. Fortunately, Red Hat operating systems already come with SELinux, that can protect the DNS system from attacks with much more effectiveness. Ubuntu and SUSE systems come AppArmor, which would also be more effective. We'll look at both of these later in this article.

VSFTP and chroot

VSFTP is probably the most common implementation of the File Transfer Protocol. Although there's no easy way to chroot the VSFTP daemon, you can easily confine users to their own specified upload/download directories. If you only want for certain users to be confined to a chroot jail, you can just create a file with the list of specified user names. If you want for all users to be confined, just go into the vsftpd.conf file and uncomment the following line:

chroot_local_user=YES

Then, if you want for certain people to not be confined, just create a file with the list of their user names.

NOTE: This type of chroot is strictly for confining users within a specified directory, and has nothing to do with process isolation.

SSH, SFTP, and chroot

You can also confine Secure Shell users to their own specified directories by adding a ChrootDirectory stanza to the normal sshd_config file.

If you want for users to be able to log into an SSH command-prompt and have an interactive session, you'll need to copy all executable, library, and configuration files over into that user's home directory. However, if all you want to do is to restrict SFTP users from leaving their home directories, all you'll need to do is to place the proper ChrootDirectory directives into the sshd_config file. If you want for users to have SFTP access without SSH access, look for the Subsystem sftp /usr/libexec/openssh/sftp-server line, and change it to:

Summary

Overall, chroot is useful mainly because it restricts visibility and encourages creating limited filesystem spaces for selected services. On the other hand, it requires careful preparation to apply and deploy correctly, and has limited value against a determined attacker who intends to circumvent it. For this reason, chroot is usually recommended only as part of a wider toolset, such as that included in the Linux jail packages we will review later; those are also much simpler in use.

Mandatory Access Control

At the beginning of this document, we looked at Discretionary Access Control, or DAC, which allows users to set their desired permissions settings on files or directories. In this section, we'll look at Mandatory Access Control, or MAC, and see how we can use it for process isolation.

To understand the difference between DAC and MAC, imagine that you have a super-secret job and that you have to access super-secret documents in order to perform that job. With nothing but Discretionary Access Control, you would be able to take one of your super-secret books and hand it off to some member of the general public who isn't authorized to see it. With Mandatory Access Control, there are rules that would prevent you from doing that. With operating systems, things work pretty much the same way.

Mandatory Access Control systems can do three things for us. They can:

Control what system resources a process is allowed to access. For process isolation, MAC is easier to deal with and much more effective than messing around with trying to set up a chroot.

Control what files and directories different users are allowed to access, based on security classifications.

Implement a Role Based Access Control system, based on a user's job role.

Of these three things, we'll only cover the first one, since that is the topic of this document. (Besides, covering all three would require a whole book.)

There are several MAC systems in use, but we'll only cover the two that are currently in most widespread use, which are SELinux and AppArmor.

SELinux

Security-enhanced Linux, or SELinux, comes already installed on all Red Hat-type Linux distros (these include Red Hat Enterprise Linux, CentOS Linux, Oracle Linux, Scientific Linux, and Fedora Linux). You can theoretically install SELinux on any other Linux distro, such as Debian or Ubuntu. Android devices use SELinux as an important security building block. If you are building embedded devices with the Yocto project, you can add SELinux support by using the meta-selinux recipe. Keep in mind that SELinux requires file system support for extended attributes (xattr), and can have a negative impact on the device's memory consumption and CPU performance.

More than anything, SELinux is a glorified labeling system that places labels on all system resources. SELinux policies then dictate which system resources a particular process is allowed to access. To demonstrate, let's take a look at the Apache daemon on a CentOS virtual machine.

The -Z option of the ps command allows us to see the SELinux labels for the processes. On Red Hat-type systems, the Apache daemon is configured to spawn five different child processes when it starts up. At the start of each line, we see the process label. For our present purposes, we're going to ignore the system_u:system_r part, because that part doesn't have anything to do with process isolation. All we're concerned about for now is just the httpd_t part, which is the "type" of the process. Now, let's go into the /var directory and look at the label for the www directory.

As with the ps command, the -Z option allows the ls command to show SELinux labels. Here, we see that the type for the www directory is set to httpd_sys_content_t. If you've guessed that processes of type httpd_t would be able to access files in a directory of type httpd_sys_content_t, then you've guessed correctly.

The type on home directories is user_home_dir_t. You might guess that the Apache daemon would not be allowed to access a directory with this type, but how could you be sure? Let's search for an SELinux boolean to see if you're right.

There are actually several different booleans that control home directory access for other daemons, but the one for "httpd" is the only one that we want to see for now. Because of this rule, the Apache web server daemon can't access files in users' home directories, even if the users decide to set their directories with world-readable DAC permissions. If necessary, you could change this rule, but you normally won't want to. By controlling filesystem access in this manner, SELinux is giving you the benefits of running Apache in chroot mode. But, SELinux is much easier to set up and is much harder for intruders to override. In addition to controlling directory access, you can also configure SELinux so that the Apache daemon won't be able to run any type of PHP or CGI scripts. In the modern world, there's a good chance that you'll need to use PHP scripting, but it's nice to know that you can disallow it if you don't need it.

On the x86/x86_64 versions of Red Hat-type distros, SELinux comes already installed and in "enforcing" mode. This means that SELinux will prohibit any action that its policies deem illegal from taking place. If you're dealing with IoT devices, you'll need to be aware that the ARM version of CentOS comes with SELinux in "permissive" mode, which means that illegal actions will be allowed to take place, but they will be logged. To place SELinux on an ARM device into "enforcing" mode, you'll need to edit the /etc/sysconfig/selinx file. When you open the file in your text editor, look for the line that says:

SELINUX=permissive

Change it to say:

SELINUX=enforcing

This change will cause SELinux to go into enforcing mode every time you boot the system, but it won't change the current state of SELinux. To change the running state to enforcing, do:

sudo setenforce 1

On our Raspberry Pi that's running CentOS, we noticed that even after editing this file, it still booted up with SELinux in Permissive mode. If you run CentOS on any IoT device, be sure to check the status of SELinux every time you boot up the device.

The other thing that you'll need to know about CentOS for ARM devices is that the files and directories might not all be labeled correctly after you place the device into enforcing mode. Or at least, that's what happened on our Raspberry Pi, and we only found out because certain actions that should have been legal were being prohibited. That's not a big deal, because with the small filesystems of ARM devices, it won't take long to fix. All you need to do is to run these two simple commands:

cd /
sudo restorecon -R *

That should fix all of your problems.

The major advantage of SELinux is that it comes with a pre-configured set of policies that provide protection for most system and network services right out-of-the box, so you normally don't have to jump through hoops to protect any additional services that you might want to install later. One exception to this rule is Dropbear, which can be used as a replacement for OpenSSH. To see this, just do the ps -eZ command on a CentOS device that's running Dropbear. You'll see something that looks like this:

The fact that it's an "unconfined" type means that there are no SELinux restrictions on this service. It is possible to write your own custom policy module, in which you could create a dropbear_t type with its own set of restrictions and labels, but that's not something for the faint-hearted. One of the disadvantages of SELinux is that the language for writing custom policies is quite complex, and the learning curve is quite steep. On the bright side, whole books have been written about SELinux, so good documentation about how to do these things does exist.

Also, keep in mind that the default set of policies that come installed when you install a Red Hat-type operating system only places restrictions on network and system services. User-mode programs, such as Firefox or LibreOffice, run in "unconfined" mode. But, fear not, we'll look at how to deal with those applications in just a few moments.

AppArmor

AppArmor, the second MAC system at which we'll look, is available on SUSE Enterprise, OpenSUSE, and all Ubuntu-type systems. Embedded devices built with the Yocto project can add AppArmor support by using the meta-security recipe.

Rather than assigning labels to all processes and files, AppArmor uses profiles to allow or disallow users or processes from accessing certain resources. Each process that you want to protect requires its own profile. Sometimes, a profile will get installed at the same time that you install a software package, and sometimes it won't. You can also install a separate software package that consists of additional AppArmor profiles that will cover some of the packages that don't come with one automatically. But, a lot of software packages don't have any pre-configured profile at all, which means that you'll have to create one yourself. There is an available utility that helps you do that, but it's still a somewhat tedious process.

To show how this works, let's look at some snippets from the /etc/apparmor.d/usr.sbin.rsyslogd file, which is the profile for the rsyslog daemon. First, let's look at the section that enables certain kernel capabilities:

What's important to note here is that if something isn't listed in an AppArmor profile, it's automatically prohibited. So for the rsyslog daemon, all kernel capabilities except for these are disabled. Next, let's look at a typical access rule.

/etc/rsyslog.conf r,

This means that the rsyslog daemon is allowed to read the "rsyslog.conf" file, but it's not allowed to write to this file. Note that all AppArmor rules have to end with a trailing comma. Now, let's look at a rule that also allows rsyslog to write to files:

/var/log/** rw,

This means that rsyslog can read from or write to any files that are in the /var/log directory, as well as all subdirectories.

There are several different directories for which this profile allows rsyslog to have either read or write access. Since AppArmor profiles act as a "whitelist", rsyslog doesn't have read or write access to any files or directories that are not listed in this profile.

It is possible to install AppArmor on non-Ubuntu operating systems, as long as they use the GRUB bootloader. That's because you need to configure GRUB to pass a kernel parameter at boot time that will enable AppArmor. Linux distros for ARM CPUs, such as Raspbian, don't use the GRUB bootloader, and modifying the kernel configuration file that is there has no effect.

Of course, there's a lot more to AppArmor than what we can present here, but you now should at least have the gist of it.

Conclusion

In this article, we covered some of the basic isolation mechanisms on Linux, and two of the most common Mandatory Access Control systems. Although powerful, we have seen their shortcomings as well: all of them need careful preparation, configuration to deploy, and otherwise they can be rendered useless by a determined attacker. Each one of them can also negatively affect the device's intended functionality if misconfigured.

The mechanisms we reviewed above provide some of the building blocks for a secure embedded Linux system, but should only be treated as building blocks, and complemented with other security features such as active monitoring for attack detection and prevention.

In the next articles, we will examine several related subjects, including the common monitoring tool AIDE, some the security building blocks offered by the Linux kernel, and the more user-friendly ways to apply them using Linux jail packages, and other ways to package software applications for effective isolation. All of these tools should help anyone who designs and deploys an IoT device to correctly apply the available security features to achieve security in depth, isolating potentially vulnerable services from the rest of the system and limiting the damage an attacker can inflict on the device when remotely exploiting its components over the network.

Share this post

Written By

Leo Dorrendorf

Leo Dorrendorf is a security researcher with experience in the academy and the industry, including a diversity of topics from reverse-engineering and breaking to designing and implementing connected systems. Currently part of the VDOO security team, Leo deals with creating engines for automated threat modeling, binary scanning, and requirement generation which incorporates a growing number of standards from the world of embedded security.