As a server administrator, I recently discovered a severe oversight of mine, one that was so big that I didn’t consciously see it for years.

What can Unprivileged Unix Users do on your server?

Any so-called “unprivileged Unix users” who have SSH access to a server (be it simply for the purpose of rsync’ing files) is not really “unprivileged” as the word may suggest. Due to the world-readable permissions of many system directories, set by default in many Linux distributions, such an user can read a large percentage of directories and files existing on the server, many of which can and should be considered as a secret. For example, on my Debian system, the default permissions are:

/etc: world-readable including most configuration files, amongst them passwd which contains plain-text names of other users/boot: world-readable including all files/home: world-readable including all subdirectories/mnt: world-readable/src: world-readable/srv: world-readable/var: world-readable
etc.

Many questions are asked about how to lock a particular user into their home directory. User “zwets” on askubuntu.com explained that this is besides the point and even “silly”:

A user … is trusted: he has had to authenticate and runs without elevated privileges. Therefore file permissions suffice to keep him from changing files he does not own, and from reading things he must not see. World-readability is the default though, for good reason: users actually needmost of the the stuff that’s on the file system. To keep users out of a directory, explicitly make it inaccessible. […]

Users need access to commands and applications. These are in directories like
/usr/bin, so unless you copy all commands they need from there to their home directories, users will need access to
/bin and
/usr/bin. But that’s only the start. Applications need libraries from
/usr/lib and
/lib, which in turn need access to system resources, which are in
/dev, and to configuration files in
/etc. This was just the read-only part. They’ll also want
/tmp and often
/var to write into. So, if you want to constrain a user within his home directory, you are going to have to copy a lot into it. In fact, pretty much an entire base file system — which you already have, located at
/.

I agree with this assessment. Traditionally, on shared machines, users needed to have at least read access to many things. Thus, once you give someone even just “unprivileged” shell access, this user — not only including his technical knowledge but also the security of the setup of his own machine which might be subject to exploits — is still explicitly and ultimately trusted to see and handle confidentially all world-readable information.

The problem: Sometimes, being “unprivileged” is not enough. The expansion towards “untrusted” users.

As a server administrator you sometimes have to give shell access to some user or machine who is not ultimately trusted. This happens in the very common case where you transfer files via
rsync (backups, anyone?) to/from machines that do not belong to you (e.g. a regular backup service for clients), or even those machines which do belong to you but which are not physically guarded 24/7 against untrusted access.
rsync however requires shell access, period. And if that shell access is granted,
rsync can read out all world-readable information, and for that matter, even when you have put into place
rbash (“restricted bash”) or
rssh (“restricted ssh”) as a login shell. So now, in our scenario, we are facing a situation where someone ultimately not trusted can rsync all world-readable information from the server to anywhere he wants to simply by doing:

1

rsync user@host:/.

One may suggest to simply harden the file permissions for those untrusted users, and I agree that this is a good practice in any case. But is it practical? Hardening the file permissions of dozens of configuration files in
/etc alone is not an easy task and is likely to break things. For one obvious example: Do I know, without investing a lot of research (including trial-and-error), which consequences
chmodo-rwx/etc/passwd will have? Which programs for which users will it break? Or worse, will I even be able to reboot the system?

And what if you have a lot of trusted users working on your server, all having created many personal files, all relying on the world-readable nature as a way to share those files, and all assuming that world-readable does not literally mean ‘World readable’? Grasping the full extent of the user collaboration and migrating towards group-readable instead of world-readable file permissions likely will be a lot of work, and again, may break things.

In my opinion, for existing server machines, this kind of work is too expensive to be justified by the benefits.

So, no matter from which angle you look at this problem, having ultimately non-trusted users on the system is a problem that can only be satisfactorily solved by jailing them into some kind of chroot directory, and allowing only those tasks that are absolutely neccessary for them (in our scenario,
rsync only). Notwithstanding that, and to repeat, users who are not jailed must be considered as ultimately trusted.

The solution: Low-level system utilities and a minimal jail

For above reasons regarding untrusted users, ‘hardening’ shell access via
rbash or even
rssh is just a cosmetic measure that still doesn’t prevent world-readable files to be literally readable by the World (you have to assume that untrusted users will share data liberally).
rssh has a built-in feature for chroot’ing, but it was originally written for RedHat and the documentation about it is vague, and it wouldn’t even accept a chroot environment created by debootstrap.

Luckily, there is a low-level solution, directly built into the Linux kernel and core packages. We will utilize the ability of PAM to ‘jailroot’ a SSH session on a per-user basis, and we will manually create a very minimal chroot jail for this purpose. We will jail two untrusted system users called “jailer” and “inmate” and re-use the same jail. Each user which will be able to
rsync files, but either will not be able to escape the jail, nor see the files of the other.

The following diagram shows the directory structure of the jail that we will create:

1

2

3

4

5

6

7

8

9

10

/

|-- bin

|-- home

| |-- jailer

| \-- inmate

|-- lib

| \-- i386-linux-gnu

| \-- cmov

|-- usr

\-- bin

The following commands are based on Debian and have been tested in Debian Wheezy.

First, create user accounts for two ultimately untrusted users, called “jailer” and “inmate” (note that the users are members of the host operating system, not the jail):

1

2

adduser jailer

adduser inmate

Their home directories will be
/home/jailer and
/home/inmate respectively. They need home directories so that you can set up SSH keys (via
~/.ssh/authorized_keys ) for passwordless-login later.

The installed configuration file is
/etc/security/chroot.conf . Into this configuration file, add

1

2

jailer /home/minjail

inmate /home/minjail

These two lines mean that after completing the SSH authentication, the users jailer and inmate will be jailed into the directory
/home/minjail of the host system. Note that both users will share the same jail.

Third, we have to enable the “chroot” PAM module for SSH sessions. For this, edit
/etc/pam.d/sshd and add to the end

1

session required pam_chroot.so

After saving the file, the changes are immediately valid for the next initiated SSH session — thankfully, there is no need to restart any service.

Making a minimal jail for chroot

All that is missing now is a minimal jail, to be made in
/home/minjail . We will do this manually, but it would be easy to make a script that does it for you. In our jail, and for our described scenario, we only need to provide
rsync to the untrusted users. And just for experimentation, we will also add the
ls command. However, the policy is: Only add into the jail what is absolutely neccessary. The more binaries and libraries you make available, the higher the likelihood that bugs may be exploited to break out of the jail. We do the following as the root user:

1

2

3

cd /home

mkdir minjail

cd minjail

Next, create the home directories for both users:

1

2

mkdir -p ./home/jailer

mkdir -p ./home/inmate

Next, for each binary you want to make available, repeat the following steps (we do it here for
rsync , but repeat the steps for
ls ):

1. Find out where a particular program lives:

1

2

which rsync

$ /usr/bin/rsync

2. Copy this program into the same location in the jail:

1

2

mkdir -p ./usr/bin

cp /usr/bin/rsync ./usr/bin

3. Find out which libraries are used by that binary

1

2

3

4

5

6

7

8

ldd /usr/bin/rsync

linux-gate.so.1 => (0xb7798000)

libacl.so.1 => /lib/i386-linux-gnu/libacl.so.1 (0xb7774000)

libpopt.so.0 => /lib/i386-linux-gnu/libpopt.so.0 (0xb7767000)

libc.so.6 => /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xb7603000)

libattr.so.1 => /lib/i386-linux-gnu/libattr.so.1 (0xb75fd000)

/lib/ld-linux.so.2 (0xb7799000)

4. Copy these libraries into the corresponding locations inside of the jail (linux-gate.so.1 is a virtual file in the kernel and doesn’t have to be copied):

1

2

3

4

5

6

mkdir -p ./lib/i386-linux-gnu/i686/cmov

cp /lib/i386-linux-gnu/libacl.so.1 ./lib/i386-linux-gnu

cp /lib/i386-linux-gnu/libpopt.so.0 ./lib/i386-linux-gnu

cp /lib/i386-linux-gnu/i686/cmov/libc.so.6 ./lib/i386-linux-gnu/cmov

cp /lib/i386-linux-gnu/libattr.so.1 ./lib/i386-linux-gnu

cp /lib/ld-linux.so.2 ./lib

After these 4 steps have been repeated for each program, finish the minimal jail with proper permissions and ownership:

1

2

3

4

5

6

chown root:root ./bin ./lib ./usr

chown jailer:jailer ./home/jailer

chown inmate:inmate ./home/inmate

chmod 751 ./home

chmod 750 ./home/jailer

chmod 750 ./home/inmate

The permission 751 of the
./home directory (
drwxr-x--xrootroot) will allow any user to enter the directory, but forbid do see which subdirectories it contains (information about other users is considered private). The permission 750 of the user directories (
drwxr-x--- ) makes sure that only the corresponding user will be able to enter.

We are all done!

Test the jail from another machine

As stated above, our scenario is to allow untrusted users to
rsync files (e.g. as a backup solution). Let’s try it, in both directions!

Testing file upload via rsync

Both users “jailer” and “inmate” can rsync a file into their respective home directory inside the jail. See:

1

2

3

4

5

rsync -avz myfile.tar jailer@host:~

sending incremental file list

sent 75 bytes received 11 bytes 8.19 bytes/sec

total size is 24,070,782 speedup is 279,892.81

To allow password-less transfers, set up a public key in
/home/jailer/.ssh/authorized_keys of the host operating system.

Testing file download via rsync

This is the real test. We will attempt do download as much as possible with rsync (we will try to get the root directory recursively):

Here you see that all world-readable files were transferred (the programs ls and rsync and their libraries), but nothing from the home directory inside of the jail.

However,
rsync succeeds to grab the user’s home directory. This is expected and desired behavior:

1

2

3

4

5

6

7

8

9

rsync -avz jailer@host:~ .

receiving incremental file list

jailer/

jailer/.bash_history

jailer/myfile.tar

sent 53 bytes received 23,773,208 bytes 1,901,860.88 bytes/sec

total size is 24,070,805 speedup is 1.01

Testing shell access

We have seen that we cannot do damage or reveal sensitive information with
rsync . But as stated above,
rsync cannot be had without shell access. So now, we’ll log in to a bash shell and see which damage we can do:

1

ssh jailer@host "/bin/bash -i"

Put
"/bin/bash -i" as an argument to use the host system’s bash in interactive mode, otherwise you would have to set up special device nodes for the terminal inside of the jail, which makes it more vulnerable for exploits.

We are now dumped to a primitive shell:

1

bash-4.2$

At this point, you can explore the jail. Try to do some damage (Careful! Make sure you’re not in your live host system, prefer an experimental virtual machine instead!!!) or try to read other user’s files. However, you will likely not succeed, since everything you have available is Bash’s builtin commands plus
rsync and
ls , all chroot’ed by a system call to the host’s kernel.

If any reader of this article should discover exploits of this method, please leave a comment.

Conclusion

I have argued that the term “unprivileged user” on a Unix-like operating system can be misunderstood, and that the term “untrusted user” should be introduced in certain use cases for clarity. I have presented an outline of an inexpensive method to accomodate untrusted users on a shared machine for various purposes with the help of the low-level Linux kernel system call
chroot() through a PAM module called
pam_chroot.so as well as a minimal, manually created jail. This method still is experimental and has not entirely been vetted by security specialists.

Trackbacks/Pingbacks

[…] SSH connection has a number of security implications. As I have argued in a previous blog post “Unprivileged Unix Users vs. Untrusted Unix Users”, having access to a shell on a server is problematic if the user is untrusted (as is always the […]