Friday, April 12, 2013

Upstart User Sessions in Ubuntu Raring

Overview

Ubuntu Raring now includes Upstart 1.8. Upstart 1.7 and 1.8 combined mark a major milestone since they bring the proven power of Upstart to the user as never before: not only is Upstart now managing the system, but it is also capable of managing the default Ubuntu user's desktop sessions too.

Why?

Question: Why reuse a system facility at the user level in this way?

The modern Linux desktop environment is a very dynamic one: users start and stop applications, switch workspaces, search the dash, adjust personal settings in the panel, connect to different networks, hot-plug USB devices and so on. All of these activities can be represented by "events".

Stepping back a second, recall that Upstart was written from the outset to take advantage of the dynamic nature of a modern Linux system. Long gone are the days when a system booted serially. A modern Linux system abounds with "events" from all sorts of different sources:

The user plugs or unplugs a peripheral device.

A process changes state.

A file gets created.

A file is deleted.

A D-Bus signal is emitted.

et cetera.

In recognition of this, Upstarts core concept is that of an event. At its heart, Upstart is an event-based process supervisor. So what better than to use a proven facility such as Upstart at the session level too? Upstarts design fits perfectly with the requirements for an init daemon but also fits the requirements for a session supervisor!

Not really - Upstart is not a big application: it was written to be small, fast, safe and reliable, all of which are highly desirable attributes for controlling sessions.

Aside: Upstart has in fact been able to run as a non-privileged user for a couple of years now (since version 1.3) although the facility was added initially for testing and then later for the init-checkconf(8) utility.

This cycle, the Ubuntu developers have gone to great lengths to squash down the default image making the overall system smaller, leaner and faster. By allowing Upstart to manage user sessions, even greater gains can be achieved going forwards since although we've now added a small number of extra (albeit small) user processes, these allow us to convert some of the historically "long-running" user applications to "on-demand" ones. Like the corresponding Upstart system jobs, by default these won't run at all: Upstart will start them if and when they need to run.

Initial targets are apps like update-notifier and update-manager. However, for the "S" cycle we'll be looking even more closely at what is running by default to see if we can start more of those processes on demand.

Some of the changes recently introduced to Upstart also mean that additional system services can also benefit further from on-demand startup. For example, the plan is for the whoopsie daemon to only start on-demand.

This will be an ongoing project, but the foundations have been laid and the concept proved.

Enabling

The default Ubuntu Desktop (*) in Raring is now fully capable of being managed by Upstart. However, since all the changes landed relatively late in the cycle, the decision was taken not to enable this feature by default.

(*) - we'd like to offer this facility to all the others desktop environments so please contact us if you wish to get involved in making the remaining Xsession scripts Upstart-aware.

However, the good news is two-fold:

Upstart User Sessions for the default desktop will be enabled by default for the "S" cycle.

If you wish to try out User Sessions (and I'd like to encourage as many folks as possible to) and you use the default Ubuntu desktop, you just need to change a single file, then logout and back in again:

To disable, simply comment-out "ubuntu" again by running the following:

sudo sed -i 's/^ubuntu/#ubuntu/' /etc/upstart-xsessions

Show me my Session

When you've enabled sessions and re-logged in, everything will look the same but you'll be running within an Upstart environment. You can see the session you're running in by looking at the value of the $UPSTART_SESSION environment variable:

What's interesting is that some of those jobs are "built-ins" from /usr/share/upstart/sessions/ whilst some are jobs I've created myself (thunderbird, emacs, term, procenv and mumble).

Note that if you're already familiar with Upstart, there is a subtle behavioural change here - since you are running within an Upstart session, "initctl list" shows jobs in that session. To see system jobs, you can run either of the following:

$ sudo initctl list
$ initctl --system list

Writing your first user job

Let's create a job that pops up xclock when it's started. Upstart 1.7 can read configuration files from multiple locations, but the main location for user jobs should be $XDG_CONFIG_HOME/upstart/. For most folk, this equates to $HOME/.config/upstart/:

And you should now see the wonderfully retro xclock running.
To stop the job simply run:

$ stop xclock

Note that since we specified a "start on" and "stop on" stanza in the xclock.conf job configuration file, this job will now run every time you login and stop just before you logout. If you don't want that to happen, you have a few choices:

Delete xclock.conf - the job is then gone.

Add "manual" anywhere after the start on line in xclock.conf such that the job will not be auto-started.

Create a new file called xclock.override in the same directory that simply contains "manual". Again, this will stop the file from being auto-started but allows you to leave the existing .conf file as-is.

Now, that is more useful - firefox will now start when we login and stop just before we logout!

But what if we want xclock to start after firefox starts and stop just before firefox stops? Simple, just change the start on and stop on stanzas in xclock.conf:

start on started firefox
stop on stopping firefox

exec xclock -update 1

The job events (events that are emitted by jobs in your "initctl list" output) are called starting, started, stopping and stopped but there are many more interesting ones.

Application Output

Error output from applications started from within a session (that is applications started from the Unity dash or launcher) still gets redirected to the usual $HOME/.xsession-errors file. However, all output from Upstart session jobs is redirected automatically to $XDG_CACHE_HOME/upstart/$job.log ,which equates to $HOME/.cache/upstart/$job.log by default.

Since gnome-session is now running as an Upstart job, it too gets it own log file. By each job having its own log file, you can now identify where messages are coming from (not always obvious when perusing $HOME/.xsession-errors).

We have included a logrotate job that automatically keeps the Upstart logs conveniently compressed and rotated. In fact, if you discover whilst testing a job that it is starting to use too much space, or if you simply want to shrink the amount of space your logs are taking, you can run this job whenever you wish like this:

$ start logrotate

The logrotate job (/usr/share/upstart/sessions/logrotate.conf) is rather interesting in that it changes its behavior based on whether it was started by the Session Init, or was invoked manually by a user. Study of this job could be instructive for those wishing to explore some of the possibilities provided by having Upstart manage your session.

upstart-monitor

To use the GUI, you'll first need to install it (as it is not part of the core upstart package):

$ sudo apt-get install upstart-monitor
$ upstart-monitor

If you have already enabled User Sessions, it will connect to your session. Otherwise, the default is to connect to the system Upstart (running as PID 1).

Auto-start an application on plugging a USB device

This is my favourite example of the power of user jobs. Let's create a job that auto-starts a VoIP client when you plug in your USB headset.

The skeleton of the job is simple. So, I'll create $HOME/.config/upstart/mumble.conf containing:

# start VoIP client of choice
exec mumble

It's a start, but it's not terribly impressive is it? However, we can now "start mumble" and "stop mumble" and Upstart will DTRT.

But what about the 'start on' condition? We need a way to tell Upstart when to start mumble. The simplest way to ascertain this for your particular USB headset is to fire up the upstart-monitor, plug in your headset and see what happens:

As you can see, I get quite a few (15!) events for my USB headset, partly because it is providing multiple sound devices (headphones and a microphone). But which event do I need? Luckily, the answer is simple for sounds devices: the sound-device-changed event. So, let's copy it...

The event containing SOUND_INITIALIZED='1' denotes that the "overall" headset device comprising multiple sound devices is "ready to use". So we have the correct type of event and we know that that event is only emitted when the device really is ready but we still need to identify the device uniquely. That might sound like an arduous task, but it really isn't so panic not! Since this is my workstation and since I know I've only got a single USB headset, all I have to do is select some unique value (or combination of values) from the variables in the above data.

I've added a stop on condition too such that Upstart will stop mumble just before the desktop session ends (in case I forget to stop it myself). And that's it. To try it out, simply plug in your USB headset!

An approximation of what is happening when the USB device is plugged in is:

The kernel detects the device and generates a uevent.

The upstart-udev-bridge, running at the system level, is monitoring the kernels netlink socket via libudev and therefore sees the event and emits a corresponding Upstart system event.

The users upstart-event-bridge sees the system event and proxies it down to the users Upstart session by emitting an Upstart user event tagged with a ":sys:" prefix.

The users Session Init compares the start on condition in mumble.conf with the Upstart event emitted by the upstart-event-bridge, finds that it matches and so starts the mumble job.

This example should give you a taste of the power you now have at your fingertips to harness system-level events in your own jobs :-)

A summary of well-known user-level and system-level upstart jobs is available in the upstart-events(7) manual page.

Restrictions

The above will not work if the device is already plugged at boot time. The problem is that although the upstart-udev-bridge will emit the correct set of events for your device at boot time, your session won't have started at that point in time so there will be no upstart-event-bridge to listen for them. So by the time your session starts, those events will have gone.

The File Bridge

Wouldn't it be great if you could create a job that would only start when a particular file got created? Well, now you can....

Now, xclock will get started when files get created, modified or deleted in the directory $HOME/var/mydir/.

Or maybe a file glob might be useful to you?

start on file FILE=~/.cache/myapp/*.crash
exec handle-my-buggy-app

Here, the "handle-my-buggy-app" application will be started to do something useful when your app creates ".crash" files.

Understanding how jobs interact

Upstart provides a tool called initctl2dot which generates graphviz visualisations of jobs. Let's run it:

$ initctl2dot -o - | dot -Tpng -o upstart.png

Here's the result on my system:

The rectangular nodes represent jobs.

The diamonds denote events.

Blue lines represent start on conditions.

Red lines represent stop on conditions.

Green lines show the events that jobs may emit.

There's quite a lot happening in that diagram partly as we've added some useful "well-known" events that you can use for your jobs. In particular, note the desktop-start and desktop-end events. These are not the first events emitted, but if your job specifies the following, it is guaranteed to run in a fully-functional desktop environment:

start on desktop-start
stop on desktop-end

If you want your job to start as early as possible and end as late as possible, but don't care about whether the desktop is usable when the job starts, you can specify:

start on startup
stop on session-end

Note: The desktop-start and desktop-end events are emitted by the job that actually starts the users graphical desktop. Currently, only the default Ubuntu desktop is emitting these events, but all desktop environments that support for Session Inits need to emit these events.

The Future

There are already a lot of "well-known" events your jobs can make use of, but we plan to introduce even more to enrich the available palette next cycle. Ideas for event sources include:

13 comments:

What this does make me think of is an issue I had. When I put Ubuntu on my new desktop, with an SSD, I wanted to test how quickly I could get to the desktop, and more importantly, get to the web. I turned on automatic log-in, and added my browser as a start-up item.

Problem was, I was logged in and my browser had loaded long before network manager had established a connection. All I had were tabs from my last session which had failed to load.

Seems to me that switching to upstart to handle the user-session will fix this little gem, and I'm sure there will be lots of extremely creative ways it can be used to.

A logind session is a mapping between a user login and a "seat" (keyboard/screen/mouse/etc combo). So loginds features are orthogonal and complementary to those provided by an Upstart Session Init. The term "session" is rather overloaded I'm afraid ;-)

Please be sure to have sane default events so that it becomes trivial to mount nfs shares in user sessions when a network comes up and to unmount them /before/ a connection is disconnected (due to logout and/or suspend).

Ever since Unity/lightdm, I have to remember to umount manually since the gdm hooks are being ignored.

PS: And report that the OpenID integration seems to be broken, as I have entered my https://launchpad.net/~ayers ID, and I was redirected to the OpenID login at Launchpad, yet according to the preview, it seems I'm still posting as anonymous. :-/