Provide a simple way of mounting almost the entire installed operating system read-only, atomically snapshot it, or share it between multiple hosts to save maintenance and space. Instead of spreading RPM package content all over the place in the filesystem, and artificially separate /bin from /usr/bin and /lib from /usr/lib, move all content to /usr and provide only symlinks in the root filesystem.

Provide a simple way of mounting almost the entire installed operating system read-only, atomically snapshot it, or share it between multiple hosts to save maintenance and space. Instead of spreading RPM package content all over the place in the filesystem, and artificially separate /bin from /usr/bin and /lib from /usr/lib, move all content to /usr and provide only symlinks in the root filesystem.

/usr on its own filesystem provides a lot of valuable options in custom setups. For historic reasons, we split-off more and more tools from /usr and put them in /. But, advanced features in today's systems can not really bootup with an empty /usr anymore. More and more fails in subtle ways in such setups.

/usr on its own filesystem provides a lot of valuable options in custom setups. For historic reasons, we split-off more and more tools from /usr and put them in /. But, advanced features in today's systems can not really bootup with an empty /usr anymore. More and more fails in subtle ways in such setups.

−

Instead of moving more tools to /, we today already require /usr to be mounted from inside the initramfs, to be available before the real 'init' starts. The split of the root filesystem an /usr serves no purpose in Linux anymore and only complicates or prevents simple and more flexible setups.

+

Instead of moving more tools to /, we today already require /usr to be mounted from inside the initramfs, to be available before the real 'init' starts. The split of the root filesystem and /usr serves no purpose in Linux anymore and only complicates or prevents simple and more flexible setups.

== Owner ==

== Owner ==

Line 16:

Line 21:

== Current status ==

== Current status ==

* Targeted release: [[Releases/17 | Fedora 17 ]]

* Targeted release: [[Releases/17 | Fedora 17 ]]

−

* Last updated: 2011-11-04

+

* Last updated: 2012-02-07

−

* Percentage of completion: 10%

+

* Percentage of completion: 100%

== Detailed Description ==

== Detailed Description ==

Line 34:

Line 39:

* /var - persistent data; non-shareable;

* /var - persistent data; non-shareable;

* /run - volatile data; non-shareable; mandatory tmpfs filesystem

* /run - volatile data; non-shareable; mandatory tmpfs filesystem

−

−

In the process of moving /bin and /sbin to /usr/bin, /usr/sbin can be moved also to /usr/bin.

<pre>

<pre>

Line 42:

Line 45:

|-- usr

|-- usr

| |-- bin

| |-- bin

−

| |-- sbin -> bin

+

| |-- sbin

| |-- lib

| |-- lib

| `-- lib64

| `-- lib64

Line 48:

Line 51:

|-- var

|-- var

|-- bin -> usr/bin

|-- bin -> usr/bin

−

|-- sbin -> usr/bin

+

|-- sbin -> usr/sbin

|-- lib -> usr/lib

|-- lib -> usr/lib

`-- lib64 -> usr/lib64

`-- lib64 -> usr/lib64

Line 54:

Line 57:

== Benefit to Fedora ==

== Benefit to Fedora ==

−

* Simpler and cleaner file system layout

+

* Simpler and cleaner overall file system layout, with full compatibility.

* Clear separation of operating system and host specific resources.

* Clear separation of operating system and host specific resources.

−

* No confusion about tools install locations and no $PATH fiddling.

+

* Improve compatibility with other Unixes/linux, no confusion about tools install locations, no $PATH fiddling, all possible paths to a binary will always work. All binaries will be available on both /usr and / thus minimizing compatibility problems.

+

* Improve compatibility with build systems such as GNU autotools who never have been aware of the /usr split in the first place

+

* Minimize difference to other Unixes, such as Solaris, which already did the same move

* install a fresh F17 with the usrmove feature included (not yet available)

−

+

−

* install a fresh F17

+

-> see symbolic toplevel links:

-> see symbolic toplevel links:

<pre>

<pre>

/lib -> usr/lib

/lib -> usr/lib

/lib64 -> usr/lib64

/lib64 -> usr/lib64

−

/sbin -> usr/bin

+

/sbin -> usr/sbin

/bin -> usr/bin

/bin -> usr/bin

−

/usr/sbin -> bin

</pre>

</pre>

+

+

Ensure that basic shell operations work.

+

# /sbin/ifconfig |/bin/grep -i ip

+

== User Experience ==

== User Experience ==

<!-- If this feature is noticeable by its target audience, how will their experiences change as a result? Describe what they will see or notice. -->

<!-- If this feature is noticeable by its target audience, how will their experiences change as a result? Describe what they will see or notice. -->

−

* less toplevel directories

+

* fewer toplevel directories

== Dependencies ==

== Dependencies ==

<!-- What other packages (RPMs) depend on this package? Are there changes outside the developers' control on which completion of this feature depends? In other words, completion of another feature owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate? Other upstream projects like the kernel (if this is not a kernel feature)? -->

<!-- What other packages (RPMs) depend on this package? Are there changes outside the developers' control on which completion of this feature depends? In other words, completion of another feature owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate? Other upstream projects like the kernel (if this is not a kernel feature)? -->

−

* initramfs (dracut)

+

* initramfs ([https://fedoraproject.org/wiki/Dracut dracut])

* changes in selinux policies

* changes in selinux policies

* repackaging of packages with content in /bin, /sbin, /lib*

* repackaging of packages with content in /bin, /sbin, /lib*

−

* move consolehelper real binaries from /usr/sbin/* to /usr/lib/<pkgname>/<tool> or /usr/libexec/<tool> and change consolehelper to look in these places.

* We do not support to bootup with an empty /usr today, so moving things to /usr and have compat links in the rootfs should be low risk.

* We do not support to bootup with an empty /usr today, so moving things to /usr and have compat links in the rootfs should be low risk.

−

* If things turn out to get difficult, we can delay the creation of the final /bin /sbin /lib /lib64 compat links to a later release. The symbolic links created in %post and added to the filelist with %ghost provide the compatibility then.

== Documentation ==

== Documentation ==

<!-- Is there upstream documentation on this feature, or notes you have written yourself? Link to that material here so other interested developers can get involved. -->

<!-- Is there upstream documentation on this feature, or notes you have written yourself? Link to that material here so other interested developers can get involved. -->

* This page is the primary source of documentation

* This page is the primary source of documentation

−

* [http://www.freedesktop.org/wiki/Software/systemd/separate-usr-is-broken Why /usr on a separate partition is broken currently ]

<!-- The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concerns. If there are any such changes involved in this feature, indicate them here. You can also link to upstream documentation if it satisfies this need. This information forms the basis of the release notes edited by the documentation team and shipped with the release. -->

<!-- The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concerns. If there are any such changes involved in this feature, indicate them here. You can also link to upstream documentation if it satisfies this need. This information forms the basis of the release notes edited by the documentation team and shipped with the release. -->

* With this release, packages will not install files anymore in the following directories: /bin /sbin /lib /lib64 and /usr/sbin.

* With this release, packages will not install files anymore in the following directories: /bin /sbin /lib /lib64 and /usr/sbin.

−

+

* Fresh installations of this release, will have the following symbolic links in the toplevel directory:

−

* Fresh installations of this release, will have the follwing symbolic links in the toplevel directory:

+

/bin -> usr/bin

/bin -> usr/bin

/sbin -> usr/sbin

/sbin -> usr/sbin

Line 149:

Line 242:

and for 64bit architectures

and for 64bit architectures

/lib64 -> usr/lib64

/lib64 -> usr/lib64

−

additionally there is

−

/usr/sbin -> bin

−

−

* If you update from a prior release, packages will install symbolic links from file locations in /bin /sbin /lib and /lib64, which they were located previously in to their counterpart in /usr. As soon as one of the toplevel directories only contains symbolic links, this directory can be removed and replaced by a symbolic link pointing to the corresponding directory in /usr.

−

* To simplify the previous step, you can boot with the follwing boot options on the kernel command line, which will invoke a transition script in dracut:

+

Steps to upgrade to Fedora 17 using yum directly:

−

** "rdupdateusrmove" : will convert those directories, which only contain symbolic links to a symbolic link

** "rdforceupdateusrmove": will rename those directories to a <dir>.bak and create the symbolic links

+

== Comments and Discussion ==

== Comments and Discussion ==

Line 182:

Line 270:

==== Why don’t you fix the /usr situation by putting all the relevant binaries in /bin /sbin /lib and /lib64? ====

==== Why don’t you fix the /usr situation by putting all the relevant binaries in /bin /sbin /lib and /lib64? ====

−

and

+

Then you would share the toplevel directory hierarchy among all hosts. Hosts would need to mount /etc and /var for host-only versions. Especially /etc/fstab is not accessible, without adding information to the initramfs on how to mount it. For every host-only additional top level directory like /opt and /srv, you would have to have a mountpoint.

+

==== So, why don’t you just mount /usr from the initramfs and leave the files where they are? ====

==== So, why don’t you just mount /usr from the initramfs and leave the files where they are? ====

Ok, so imagine you have a /usr mounted from a network location and you want to update a package. So maybe you mount the master copy of /usr on your master machine and update /usr with your package manager. Then you provide a new copy of the master /usr to the other machines, when they reboot. They all have the new updated /usr now. But what about /sbin /bin /lib and /lib64? They still have the old binaries. No glibc security update for them. So, every machine has to update these directories via rsync or such (rpm will not work with a readonly /usr). This doubles the maintenance to keep both parts of the system in sync.

Ok, so imagine you have a /usr mounted from a network location and you want to update a package. So maybe you mount the master copy of /usr on your master machine and update /usr with your package manager. Then you provide a new copy of the master /usr to the other machines, when they reboot. They all have the new updated /usr now. But what about /sbin /bin /lib and /lib64? They still have the old binaries. No glibc security update for them. So, every machine has to update these directories via rsync or such (rpm will not work with a readonly /usr). This doubles the maintenance to keep both parts of the system in sync.

Line 198:

Line 287:

Then you would share the toplevel directory hierarchy among all hosts. Hosts would need to mount /etc and /var for host-only versions. Especially /etc/fstab is not accessible, without adding information to the initramfs on how to mount it. For every host-only additional top level directory like /opt and /srv, you would have to have a mountpoint.

Then you would share the toplevel directory hierarchy among all hosts. Hosts would need to mount /etc and /var for host-only versions. Especially /etc/fstab is not accessible, without adding information to the initramfs on how to mount it. For every host-only additional top level directory like /opt and /srv, you would have to have a mountpoint.

−

==== Can’t we just move things to /usr and merge */bin with */sbin maybe later? ====

+

[[Category:FeatureAcceptedF17]]

−

We can do that, but then we would have the work twice for some packages and their %post section would generate even more compat symlinks for the transition process.

+

−

+

−

==== Why do you want to merge /sbin with /bin? ====

+

−

Because, we felt, that if you merge directories anyway, we can merge even more and get rid of things, which have no clear usage anymore. /sbin is not containing static binaries anymore. /sbin should not contain daemons, which are started by root manually. What is left, are mostly tools, which do not work for normal users, because they have no write access for the devices on which they operate. But most of them have read access and some of them really print valuable information even for the non-root user.

+

−

+

−

You might also want to read this [http://thread.gmane.org/gmane.linux.redhat.fedora.devel/155511/focus=155792 excellent mail from Lennart Poettering]

+

−

+

−

==== Why should root not start daemons from the terminal? ====

+

−

If you are root, then you can easily generate a service file and run your daemon in a clean managed way.

+

−

+

−

When services are spawned it is essential that this happens from a clean execution context that is fully detached from the context of the user that asks for it. Otherwise context settings of that user are inherited by the daemon that better should not be. Examples for this are: resource limits, environment variables and similar, more problematic are audit trails, SELinux contexts or scheduling settings. Some daemons are written in a style that resets many of these process settings on initialization. However this is never complete, simply because new process settings are added more quickly to the kernel's struct task than any userspace developer could make sure to reset. On top of that, some of the settings in struct task cannot be reset at all. The effective result is that the services are executed with inherited settings that better should not be inherited thus making behaviour of the daemon highly dependent on the user context starting it and skewing audit trails and suchlike. The solution is to fork off all services from a defined, clean execution context, like systemd does it, from PID1, so that all process settings are in a defined state and the same on every invocation.

+

−

+

−

If you just want to test a daemon, you can start it even with the daemon not being in your $PATH all the time. And if you want to regularly run it, then you really should make it a service, which can be started on your demand by the service manager (systemd).

+

−

+

−

[[Category:FeatureReadyForWrangler]]

+

<!-- When your feature page is completed and ready for review -->

<!-- When your feature page is completed and ready for review -->

<!-- remove Category:FeaturePageIncomplete and change it to Category:FeatureReadyForWrangler -->

<!-- remove Category:FeaturePageIncomplete and change it to Category:FeatureReadyForWrangler -->

<!-- After review, the feature wrangler will move your page to Category:FeatureReadyForFesco... if it still needs more work it will move back to Category:FeaturePageIncomplete-->

<!-- After review, the feature wrangler will move your page to Category:FeatureReadyForFesco... if it still needs more work it will move back to Category:FeaturePageIncomplete-->

<!-- A pretty picture of the page category usage is at: https://fedoraproject.org/wiki/Features/Policy/Process -->

<!-- A pretty picture of the page category usage is at: https://fedoraproject.org/wiki/Features/Policy/Process -->

Provide a simple way of mounting almost the entire installed operating system read-only, atomically snapshot it, or share it between multiple hosts to save maintenance and space. Instead of spreading RPM package content all over the place in the filesystem, and artificially separate /bin from /usr/bin and /lib from /usr/lib, move all content to /usr and provide only symlinks in the root filesystem.

/usr on its own filesystem provides a lot of valuable options in custom setups. For historic reasons, we split-off more and more tools from /usr and put them in /. But, advanced features in today's systems can not really bootup with an empty /usr anymore. More and more fails in subtle ways in such setups.

Instead of moving more tools to /, we today already require /usr to be mounted from inside the initramfs, to be available before the real 'init' starts. The split of the root filesystem and /usr serves no purpose in Linux anymore and only complicates or prevents simple and more flexible setups.

There is no way to reliably bring up a modern system with an empty /usr, there are two alternatives to fix it: copy /usr back to the rootfs or use an initramfs which can hide the split-off from the system.

Historically /bin, /sbin, /lib had the purpose to contain the utilities to mount /usr. This role can now be taken by the initramfs. Because the initramfs knows, where to find the root partition (which includes /etc), it can parse /etc/fstab and other configuration files and mount /usr before it finally switches the root partition and executes /usr/bin/init. From this point on init mounts the remaining partitions in /etc/fstab and the system starts as usual.

The long-term plan is to clean up the mess and confusion the current split of / vs. /usr has created. All tools will move back to /usr where they belong, and the rootfs will only contain compat-symlinks into /usr. Almost the entire system installed by packages will reside in /usr. This will split all non-host specific data to /usr. /usr can then be seen as the Unix System Resources partition (/System), which defines the base operating system (e.g. F18 or RHEL-7).

This new /usr could be mounted read-only by default, while the rootfs is read-write and contains only empty mount points, compat-symlinks to /usr and the host-specific data like /etc, /root, /srv. Compared to today's setups, the rootfs will be very small. The new /usr could also easily be shared read-only across several systems, and it would contain almost the entire system. Such setups are more efficient, can optionally provide a lot more security, are more flexible, provide more sane options for custom setups, and are much simpler to setup and maintain.

This leaves us with the following well-defined directories, which compose the base of the system:

Simpler and cleaner overall file system layout, with full compatibility.

Clear separation of operating system and host specific resources.

Improve compatibility with other Unixes/linux, no confusion about tools install locations, no $PATH fiddling, all possible paths to a binary will always work. All binaries will be available on both /usr and / thus minimizing compatibility problems.

Improve compatibility with build systems such as GNU autotools who never have been aware of the /usr split in the first place

Minimize difference to other Unixes, such as Solaris, which already did the same move

Isolate the vendor-supplied mostly read-only operating system resources from the rest, thus allow snapshotting of the OS, and easy lightweight container OS duplication

[edit] What is currently broken with having /usr as a separate partition?

[edit] I don’t have /usr as a separate partition. What changes for me?

Nothing changes in functionality. All the old paths are reachable, because there a compat symlinks in place, which will not go away (at least not in the near future). All your scripts and binaries should work, like they did before. For the upgrade process to work, you will find /sbin, /bin, /lib and /lib64 mostly containing symbolic links. As soon, as these directories only contain symbolic links, the whole directory is replaced by only one symbolic link. These three or four toplevel symbolic links will stay there as long as the linux elf loader ABI is defined with “/lib/ld-linux.so.2” or their architecture specific counterpart like “/lib64/ld-linux-x86-64.so.2”, and as long as scripts use “#!/bin/sh”.

[edit] Why don’t you fix the /usr situation by putting all the relevant binaries in /bin /sbin /lib and /lib64?

Then you would share the toplevel directory hierarchy among all hosts. Hosts would need to mount /etc and /var for host-only versions. Especially /etc/fstab is not accessible, without adding information to the initramfs on how to mount it. For every host-only additional top level directory like /opt and /srv, you would have to have a mountpoint.

[edit] So, why don’t you just mount /usr from the initramfs and leave the files where they are?

Ok, so imagine you have a /usr mounted from a network location and you want to update a package. So maybe you mount the master copy of /usr on your master machine and update /usr with your package manager. Then you provide a new copy of the master /usr to the other machines, when they reboot. They all have the new updated /usr now. But what about /sbin /bin /lib and /lib64? They still have the old binaries. No glibc security update for them. So, every machine has to update these directories via rsync or such (rpm will not work with a readonly /usr). This doubles the maintenance to keep both parts of the system in sync.

[edit] You are doing it wrong! /bin and /sbin are there to rescue a broken /usr!

The most critical filesystem is /boot, because the kernel lives there. So the purpose of having /bin and /sbin for /usr repairing relied on _two_ working filesystems ( / and /boot). If either of them was broken, you were not able to rescue /usr. The role of the rescue system can easily be fulfilled by a rescue initramfs. So having the rescue initramfs in /boot, which contains the fsck utils, is in the same danger of becoming corrupted as the kernel. Now you only have to pull out your rescue CD, if /boot is corrupted and not if / is corrupted.

[edit] Then, let’s share /bin /sbin /lib /lib64 and /usr and mount them all from the initramfs!

Now, you get a feeling, that moving everything to /usr might make things easier....

[edit] Why don’t you move all /usr contents to / and forget about /usr?

Because this introduces a lot of new toplevel directories, which all have to be mount points then to be shared across other hosts.

[edit] Ok, but what about a root filesystem on the network and mounting local filesystems only?

Then you would share the toplevel directory hierarchy among all hosts. Hosts would need to mount /etc and /var for host-only versions. Especially /etc/fstab is not accessible, without adding information to the initramfs on how to mount it. For every host-only additional top level directory like /opt and /srv, you would have to have a mountpoint.