Thursday, 18 April 2013

Observing the initial LXC environment using procenv

If you're interested in seeing the initial environment inside an LXC container, here's how:
  1. Install procenv inside the container:

    $ sudo apt-get install -y procenv

  2. Shutdown the container:

    $ sudo shutdown -h now
  3. Boot the container (mine is called "raring" in this example) like this:

    $ sudo lxc-start -n raring --console-log /tmp/lxc-console.log \
        -- /usr/bin/procenv --file=/dev/console --exec \
        -- /sbin/init --debug
  4. View the /tmp/lxc-console.log logfile in the host environment.
Note that those two sets of non-option double-dashes are required; the first tells LXC to treat what follows as a command to run after the container starts (in this case procenv), and the second set tells procenv to treat what follows as a command to run after it has finished running (in this case Upstart (/sbin/init :-))!

Monday, 15 April 2013

Byobu Bling with Unicode Custom Indicators


In these heady times of almost daily advances in UI pizzazz, you could be forgiven for thinking the terminal is somewhat of a dead-zone when it comes to excitement. Oh how wrong you would be though...!

If you've never used ByobuDustin's awesome text-based window manage, sudo apt-get install it without delay to be bathed in pure terminal goodness. As shown below, byobu comes with a whole host of standard indicators, allowing the display of useful snippets of information.

The indicators are minimal out of terminal-width real estate necessity, but that's also their strength - they provide just the details you really care about in a tight ASCII format. And they're cute of course :-)

However, byobu has a well-kept secret: when running with the tmux backend, it supports Unicode indicators. So with the proviso that you are running "byobu-tmux" rather than "byobu-screen", you can squeeze down those indicators even further, saving precious onscreen characters and get extra bling at the same time! Intrigued? Then read on...


A few years ago (yes, it's taken that long to get around to writing this up! ;-), I was playing around with Unicode and ended up writing a very simple C program to display the time in "binary" using the Braille Unicode Block. Assuming you were only displaying hours and minutes, that meant you could display any time in 2 onscreen characters - nice and compact :) I wanted to use this program as a byobu custom indicator but at the time byobu was only using GNU screen and alas even the latest version of screen did not have full UTF-8 support (for its status line atleast). However, tmux was fast becoming the worthy successor to screen and of course byobu now supports both backends. I ended up rewriting the C program as a shell script to fit in with the other byobu indicators. That shell script became the time_binary indicator shipped with byobu.

Here it is displaying the time (21:04) in 24-hour format:


Admittedly, it's kinda terse and somewhat difficult to read but that's half the fun right? :-) time_binary is slightly mis-named as it can also display the time in hexadecimal and octal.

But why stop there? I now wanted all sorts of Unicode indicators showing the level of things like sound, wifi signal strength, left/right speaker volumes, memory used, system load, etc. However, since UIs are very personal, I wanted to be able to write short custom indicators for each of the above but allow the choice of unicode indicators to be changed with minimal trouble as desired.

Enter byobu-ulevel

What I ended up doing is trawling through some of those myriad Unicode symbols looking for suitable groups and creating a small utility called byobu-ulevel. This takes a numeric value and converts it into an appropriate "level" symbol. This all sounds a bit vague, so here's an example:

$ byobu-ulevel 100

The above example has converted the value 100 to its default 100% value using the default theme. And here's the default 30% value:

$ byobu-ulevel 30

Starting to make sense? If your data source doesn't provide values from 0% to 100%, fear not. Let's say your data source ranges from -123 to +456 and your value to display is 333.0:

$ byobu-ulevel -c 333 -m -123.45 -x +678.90

This utility is generally only useful if your data has some specifiable bounded range. However, there is a "cheat mode" - if you are happy for values to overflow/underflow silently, you can specify "-p":

$ byobu-ulevel -c 999 -x 100
ERROR: value (999) > maximum (100)
$ byobu-ulevel -c 999 -x 100 -p

Template for a Byobu Unicode Custom Notification

We can create a Unicode custom notification:

$ mkdir -p ~/.byobu/bin
$ cd ~/.byobu/bin
$ cat >5_utest<<EOT

# XXX: rather than hard-coding the value, put your logic here
# XXX: to determine an actual value from some data source.

printf "U[%s]\n" $(byobu-ulevel $value)
$ chmod 755 5_utest

In fact, as you'll see once you finish this article, the example notification above is really a template for most of your Unicode notifications.

Once you've activated the custom notifications in byobu (F9 → Toggle status notifications → check the "custom" option), and waited for a few seconds (5 in the example above), you'll see the following on your byobu status line:


I've added a "U" to denote my Unicode test indicator and delimited the actual indicator value glyph between square brackets, but of course you can use whatever ornamentation you wish.


So, we now have an indicator that, assuming you add in some appropriate logic to set "$value" dynamically, will show a 1-character wide bar whose height varies depending on the magnitude of $value.

But what if you don't like that bar much? No problem because byobu-ulevel comes with a set of built-in "themes" and since we haven't specified one, we get the default (vbars_8). Let's list the available themes by asking byobu-ulevel to display them:

$ byobu-ulevel -l

So now we know the theme names, let's take a look at them:

$ for theme in $(byobu-ulevel -l); do byobu-ulevel -l -t "$theme"; done
Listing theme 'circles_2'

○ ● 

Listing theme 'diamonds_2'

◇ ◆ 

Listing theme 'flags_2'

⚐ ⚑ 

Listing theme 'hearts_2'

♡ ♥ 

Listing theme 'squares_2'

◻ ◼ 

Listing theme 'squares_small_2'

◽ ◾ 

Listing theme 'stars_2'

☆ ★ 

Listing theme 'vdots_thick_4'

⣀ ⣤ ⣶ ⣿ 

Listing theme 'vdots_thin_4'

⢀ ⢠ ⢰ ⢸ 

Listing theme 'fractions_4'

¼ ½ ¾ ¹ 

Listing theme 'quadrants_4'

◔ ◑ ◕ ● 

Listing theme 'shades_4'

░ ▒ ▓ █ 

Listing theme 'circles_5'

◦ ○ ◎ ◉ ● 

Listing theme 'dice_6'

⚀ ⚁ ⚂ ⚃ ⚄ ⚅ 

Listing theme 'hbars_8'

▏ ▎ ▍ ▌ ▋ ▊ ▉ █ 

Listing theme 'vbars_8'

▁ ▂ ▃ ▄ ▅ ▆ ▇ █ 

Listing theme 'circle_number_10'

➀ ➁ ➂ ➃ ➄ ➅ ➆ ➇ ➈ ➉ 

Listing theme 'solid_numbers_a_10'

➊ ➋ ➌ ➍ ➎ ➏ ➐ ➑ ➒ ➓ 

Listing theme 'solid_numbers_b_10'

❶ ❷ ❸ ❹ ❺ ❻ ❼ ❽ ❾ ❿ 

So how do you use these themes? Easy, just choose the one you like the look of and specify it using "-t $theme". Let's try out the stars_2 theme:

$ byobu-ulevel -c 60 -t stars_2

All themes ending with "_2" contain two glyphs and are called "rating themes" (for obvious reasons I hope). The first glyph in a rating theme represents the "unused" part of the range and the second glyph represents the "used" part. So, here we see that 60% (equating to 3 solid stars) is "used" since the value is 60 and the default range is 0-100%. Obviously, we can't display half glyphs so for this theme for example the value provided is "rounded to the nearest star" :-)

By default rating themes are displayed left-to-right. However, you can change that using the "-r" option:

$ byobu-ulevel -c 60 -t stars_2 -r

Or you can invert the theme such that the first glyph will represent the "used" part and the second the "unused" using the "-i" option:

$ byobu-ulevel -c 60 -t stars_2 -i

But maybe 5 stars isn't enough for you? No problem, use the width option ("-w"):

$ byobu-ulevel -c 60 -t stars_2  -w 10

Themes with more than 2 glyph values are different to rating themes in that rather than using a combination of the "used" and "unused" glyphs to show the value, only a single glyph is selected. For example, here's our 60% again using the circle_number_10 theme:

$ byobu-ulevel -c 60 -t circle_number_10

Since I'm bound to have missed out your favourite Unicode glyphs, you can of course make your own theme trivially too. Let's create a rating theme:

$ byobu-ulevel -c 60 -u "· ☢"

And now a normal theme with, in this example, 10 values:

$ byobu-ulevel -c 60 -u "a b c d e f g h i j"

You could even use utfout to construct your theme:

$ byobu-ulevel -c 45 -u "$(utfout -a ' ' '\{α..ω}')"


But what about accessibility? What about users who cannot display some of these characters? byobu-ulevel provides for these folk too: either specify "-a" or set the BYOBU_A11Y environment variable to any value, and byobu-ulevel will display purely numeric (floating point) values rather than Unicode bling. You can specify the number of decimal places too with the "-e" option.

Multiple Values

Great, but byobu-ulevel only handles a single value - what if you want to display multiple values? For example a range of values over time? No problem because along with byobu-ulevel we have byobu-ugraph which takes a series of values and uses byobu-ulevel to display them. The canonical example being the load average display. Let's create a rolling load average showing 5 data points by creating ~/.byobu/bin/5_uload and making it executable:

awk '{print $1}' /proc/loadavg >> $file
printf "L[%s]\n" $(byobu-ugraph -p 5 -x 3 -f $file)

Leave it to run for a few iterations (to generate the 5 data points we've specified we want and we get something like this:


Like byobu-ulevel, byobu-ugraph assumes all data values range from 0 to 100 so assuming your system load is somewhat more manageable than that, you'll want to specify a "reasonable" maximum value as I have with "-x 3" to allow byobu-ugraph to scale the values appropriately.

Again, you can specify a theme to pass through to byobu-ulevel. Here's the same data using the solid_numbers_b_10 theme:


... And the same data again this time using the vdots_thick_4 theme:


Can console bliss get any better than this? Of course it can - we haven't introduced colours yet!

This part is still rather a work-in-progress as my current unreleased colour utility is rather basic. I'm hoping to expand it and potentially merge it into byobu-ulevel "Real Soon Now". However, my latest versions of byobu-ulevel and byobu-ugraph now accept a "-y" option to enable colour. This allow us to bling up that original load example rather trivially:

awk '{print $1}' /proc/loadavg >> $file
printf "L[%s]\n" $(byobu-ugraph -p 5 -x 3 -f $file -y)

All I've done is add that "-y" to produce:


The colour script currently takes current, minimum and maximum values but also allows thresholds to be specified along with colours for each threshold. By default, two thresholds are specified: "low" (0-15%) and "high" (85-100%). If the value is within the low threshold, it will be coloured red, if in the high threshold it gets coloured green, and yellow if in the middle. The threshold colours can also be inverted so that for example an indicator showing 95% disk usage will default to displaying in red when inverted (as that's a bad scenario), whilst an indicator showing 95% free memory will show in green when not inverted (as that's good :-)

Obligatory Screenshot

Here's an example of the sorts of indicators you can create with just a few lines of shell script for your custom indicators:

As you can see, it's pretty sparse - just the way I like it.

The indicators, from left to right (ignoring the "0:*" byobu window number) are:

  • Time (23:01) in binary ("T[...]")
  • Free disk ("D[...]")
  • Some sort of rating (you decide :-) using hearts ("R[...]")
  • Free memory ("M[...]")
  • Another rating this time using stars ("R[...]")
  • Load average with 5 values ("L[...]")
  • Wifi signal strengh ("W[...]")

Tips and Observations

  • Sometimes, you may find that an indicator doesn't appear. This appears to be an issue with either tmux or the terminal emulator you are using (gnome-terminal, rxvt, urxvt, xterm, etc). The solution seems to be to exit byobu, close the tab/window and restart byobu.
  • When writing custom indicators, ensure that the output ends with a newline character ('\n') - without it, byobu won't display the indicator.
  • Performance can be an issue if you specify a large (10+ say) number of values to byobu-ugraph or if you have more than a few byobu-ulevel and/or byobu-ugraph indicators on-screen at once.
  • byobu-ulevel and byobu-ugraph can run in a terminal as long as you don't enable colour mode (which outputs not colour escape sequences but tmux colour commands).
  • For extra fun, you could have your indicator script call byobu-ulevel specifying a random theme. Here's an indicator which will use a random theme to display the value (80) each time it is called:
  • # change this to by the value you want to display
    count=$(byobu-ulevel -l | wc -l)
    selected=$(((RANDOM % count)+1))
    theme=$(byobu-ulevel -l | sed -n ${selected}p)
    printf "X[%s]\n" $(byobu-ulevel -c $num -t $theme)

Planned Future Work

  • Finish the colour script and potentially merge with byobu-ulevel. New features that could be added to the colour script:
    • ability to specify arbitrary thresholds with arbitrary colours and attributes (bold, etc).
    • ability to invert the colour applied to a value from a colour theme.
  • Improve performance (probably by rewriting in C).
  • Improve accessibility support.
  • Finish another utility I've half-written that shows sound levels (with individual left and right speaker channels all in a single on-screen character!)


There are lots of other Unicode possibilities. Just browse through the wikipedia Unicode page for inspiration.
For example, I'd love it for example if someone would write a Unicode weather app that downloaded the current weather for my location and then displayed it using suitable glyphs like: ⁂ ※ ⁑   ⁕ ⁖  ⁜ ⁛ ☀ ☁ ☂ ☔  ☃.
Find an excuse to get involved!


With minimal effort, you can now add custom Unicode indicators to your byobu session saving you precious onscreen space whilst also offering a dash of visual spice.

Stateful re-exec for Upstart Sessions? You bet!

As of Upstart 1.6, restarting PID 1 is supported without losing state. So as soon as you get an update to either Upstart itself or one of its dependent libraries in Ubuntu, you'll be running the new version automatically, without the need for a reboot.

But what about User Sessions? Are they "second-class citizens"? Of course not! When PID 1 restarts, all the running Session Inits will automatically restart too. You won't notice this of course as your desktop session will continue to run and be fully functional but you too will be running the latest version of Upstart.

How does this work? When an instance of Upstart has restarted, it signals this fact over D-Bus. From the user perspective, the upstart-event-bridge, listening to the system instance of Upstart running as PID 1, will generate a ":sys:restarted" event when it detects the restart. This in turn will cause the session job /usr/share/upstart/sessions/re-exec.conf to run. And that job requests the Session Init restarts. Since all Session Inits will have the re-exec job registered, all Session Init instances will restart as soon as PID 1 does.

To convince yourself, let's create a session job that will start when your Session Init restarts. We'll do this by having the job start when the re-exec job has finished:

$ cat >$HOME/.config/upstart/restart.conf<<EOT
start on stopped re-exec
exec xclock

Now, let's force Upstart running as the system init daemon to restart:

$ sudo telinit u

You should now see xclock running!

Note that the job could have specified any of the following start on conditions...

  • start on starting re-exec
  • start on started re-exec
  • start on stopping re-exec
  • start on stopped re-exec
  • start on :sys:restarted
... however, I chose "start on stopped re-exec" because I want xclock to run just after the re-exec job has finished.

Friday, 12 April 2013

Upstart User Sessions in Ubuntu Raring


Ubuntu Raring now includes Upstart 1.8. Upstart 1.7 and 1.8 combined mark a major milestone since they bring the proven power of Upstart to the user as never before: not only is Upstart now managing the system, but it is also capable of managing the default Ubuntu user's desktop sessions too.


Question: Why reuse a system facility at the user level in this way?

The modern Linux desktop environment is a very dynamic one: users start and stop applications, switch workspaces, search the dash, adjust personal settings in the panel, connect to different networks, hot-plug USB devices and so on. All of these activities can be represented by "events".

Stepping back a second, recall that Upstart was written from the outset to take advantage of the dynamic nature of a modern Linux system. Long gone are the days when a system booted serially. A modern Linux system abounds with "events" from all sorts of different sources:
  • The user plugs or unplugs a peripheral device.
  • A process changes state.
  • A file gets created.
  • A file is deleted.
  • A D-Bus signal is emitted.
  • et cetera.
In recognition of this, Upstarts core concept is that of an event. At its heart, Upstart is an event-based process supervisor. So what better than to use a proven facility such as Upstart at the session level too? Upstarts design fits perfectly with the requirements for an init daemon but also fits the requirements for a session supervisor!

Question: Yes but isn't having Upstart manage users sessions overkill?

Not really - Upstart is not a big application: it was written to be small, fast, safe and reliable, all of which are highly desirable attributes for controlling sessions.

Aside: Upstart has in fact been able to run as a non-privileged user for a couple of years now (since version 1.3) although the facility was added initially for testing and then later for the init-checkconf(8) utility.

This cycle, the Ubuntu developers have gone to great lengths to squash down the default image making the overall system smaller, leaner and faster. By allowing Upstart to manage user sessions, even greater gains can be achieved going forwards since although we've now added a small number of extra (albeit small) user processes, these allow us to convert some of the historically "long-running" user applications to "on-demand" ones. Like the corresponding Upstart system jobs, by default these won't run at all: Upstart will start them if and when they need to run.

Initial targets are apps like update-notifier and update-manager. However, for the "S" cycle we'll be looking even more closely at what is running by default to see if we can start more of those processes on demand.

Some of the changes recently introduced to Upstart also mean that additional system services can also benefit further from on-demand startup. For example, the plan is for the whoopsie daemon to only start on-demand.

This will be an ongoing project, but the foundations have been laid and the concept proved.


The default Ubuntu Desktop (*) in Raring is now fully capable of being managed by Upstart. However, since all the changes landed relatively late in the cycle, the decision was taken not to enable this feature by default.

(*) - we'd like to offer this facility to all the others desktop environments so please contact us if you wish to get involved in making the remaining Xsession scripts Upstart-aware.

However, the good news is two-fold:
  • Upstart User Sessions for the default desktop will be enabled by default for the "S" cycle.
  • Enabling Upstart User Sessions for the default Ubuntu Raring desktop is extremely easy!
If you wish to try out User Sessions (and I'd like to encourage as many folks as possible to) and you use the default Ubuntu desktop, you just need to change a single file, then logout and back in again:

To enable, simply uncomment "ubuntu" in file /etc/upstart-xsessions like this:

sudo sed -i 's/^#ubuntu/ubuntu/' /etc/upstart-xsessions

To disable, simply comment-out "ubuntu" again by running the following:

sudo sed -i 's/^ubuntu/#ubuntu/' /etc/upstart-xsessions

Show me my Session

When you've enabled sessions and re-logged in, everything will look the same but you'll be running within an Upstart environment. You can see the session you're running in by looking at the value of the $UPSTART_SESSION environment variable:


To list all running sessions for your user:

$ initctl list-sessions
3066 unix:abstract=/com/ubuntu/upstart-session/1000/3066

The format of the list-sessions output in case you haven't guessed it is:


Proof that we are running Upstart:

$ ps -fp 3066
james     3066  2884  0 07:58 ?        00:00:00 init --user

Have I got any jobs running?


$ initctl list
xsession-init stop/waiting
dbus start/running, process 3304
firefox stop/waiting
procenv stop/waiting
term start/running, process 3289
gnome-session start/running, process 3343
logrotate stop/waiting
thunderbird start/running, process 3339
mumble stop/waiting
gnome-settings-daemon start/running, process 3309
emacs start/running, process 4324
re-exec stop/waiting
upstart-event-bridge start/running, process 3305

What's interesting is that some of those jobs are "built-ins" from /usr/share/upstart/sessions/ whilst some are jobs I've created myself (thunderbird, emacs, term, procenv and mumble).

Note that if you're already familiar with Upstart, there is a subtle behavioural change here - since you are running within an Upstart session, "initctl list" shows jobs in that session. To see system jobs, you can run either of the following:

$ sudo initctl list
$ initctl --system list

Writing your first user job

Let's create a job that pops up xclock when it's started. Upstart 1.7 can read configuration files from multiple locations, but the main location for user jobs should be $XDG_CONFIG_HOME/upstart/. For  most folk, this equates to $HOME/.config/upstart/:

$ conf_dir=${XDG_CONFIG_HOME:-$HOME/.config}/upstart
$ mkdir -p $conf_dir

(Note that if $conf_dir did not exist initially, you will need to logout and back in for jobs in that directory to take effect - a one-off task)

Now, create file $conf_dir/xclock.conf containing:

start on desktop-start
stop on desktop-end
exec xclock -update 1

That's it. But to be sure we have specified the syntax correctly, let's check that Upstart is happy with it first:

$ init-checkconf $conf_dir/xclock.conf
File /home/james/.config/upstart/xclock.conf: syntax ok

I'd recommend always using init-checkconf(8) to sanity-check your jobs. So, Upstart is happy with this job. Let's start it:

$ start xclock

And you should now see the wonderfully retro xclock running. To stop the job simply run:

$ stop xclock

Note that since we specified a "start on" and "stop on" stanza in the xclock.conf job configuration file, this job will now run every time you login and stop just before you logout. If you don't want that to happen, you have a few choices:
  • Delete xclock.conf - the job is then gone.
  • Add "manual" anywhere after the start on line in xclock.conf such that the job will not be auto-started.
  • Create a new file called xclock.override in the same directory that simply contains "manual". Again, this will stop the file from being auto-started but allows you to leave the existing .conf file as-is.
  • Delete (or simply comment out) the start on condition.

Let's create another job to start firefox which we shall call somewhat unimaginatively firefox.conf:

start on desktop-start
stop on desktop-end
exec firefox

Now, that is more useful - firefox will now start when we login and stop just before we logout!

But what if we want xclock to start after firefox starts and stop just before firefox stops? Simple, just change the start on and stop on stanzas in xclock.conf:

start on started firefox
stop on stopping firefox
exec xclock -update 1

The job events (events that are emitted by jobs in your "initctl list" output) are called starting, started, stopping and stopped but there are many more interesting ones.

Application Output

Error output from applications started from within a session (that is applications started from the Unity dash or launcher) still gets redirected to the usual $HOME/.xsession-errors file. However, all output from Upstart session jobs is redirected automatically to $XDG_CACHE_HOME/upstart/$job.log ,which equates to $HOME/.cache/upstart/$job.log by default.

Since gnome-session is now running as an Upstart job, it too gets it own log file. By each job having its own log file, you can now identify where messages are coming from (not always obvious when perusing $HOME/.xsession-errors).

We have included a logrotate job that automatically keeps the Upstart logs conveniently compressed and rotated. In fact, if you discover whilst testing a job that it is starting to use too much space, or if you simply want to shrink the amount of space your logs are taking, you can run this job whenever you wish like this:

$ start logrotate

The logrotate job (/usr/share/upstart/sessions/logrotate.conf) is rather interesting in that it changes its behavior based on whether it was started by the Session Init, or was invoked manually by a user. Study of this job could be instructive for those wishing to explore some of the possibilities provided by having Upstart manage your session.


To use the GUI, you'll first need to install it (as it is not part of the core upstart package):

$ sudo apt-get install upstart-monitor
$ upstart-monitor

If you have already enabled User Sessions, it will connect to your session. Otherwise, the default is to connect to the system Upstart (running as PID 1).

Auto-start an application on plugging a USB device

This is my favourite example of the power of user jobs. Let's create a job that auto-starts a VoIP client when you plug in your USB headset.

The skeleton of the job is simple. So, I'll create $HOME/.config/upstart/mumble.conf containing:

# start VoIP client of choice
exec mumble

It's a start, but it's not terribly impressive is it? However, we can now "start mumble" and "stop mumble" and Upstart will DTRT.

But what about the 'start on' condition? We need a way to tell Upstart when to start mumble. The simplest way to ascertain this for your particular USB headset is to fire up the upstart-monitor, plug in your headset and see what happens:

As you can see, I get quite a few (15!) events for my USB headset, partly because it is providing multiple sound devices (headphones and a microphone). But which event do I need? Luckily, the answer is simple for sounds devices: the sound-device-changed event. So, let's copy it...

... and paste it here:

:sys:sound-device-changed KERNEL='card2' DEVPATH='/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/sound/card2' SUBSYSTEM='sound' ACTION='change' ID_BUS='usb' ID_FOR_SEAT='sound-pci-0000_00_1d_0-usb-0_1_2_1_0' ID_ID='usb-Logitech_Logitech_USB_Headset-00-Headset' ID_MODEL='Logitech_USB_Headset' ID_MODEL_ENC='Logitech\x20USB\x20Headset' ID_MODEL_FROM_DATABASE='ClearChat Pro USB' ID_MODEL_ID='0a0b' ID_PATH='pci-0000:00:1d.0-usb-0:1.2:1.0' ID_PATH_TAG='pci-0000_00_1d_0-usb-0_1_2_1_0' ID_REVISION='1013' ID_SERIAL='Logitech_Logitech_USB_Headset' ID_TYPE='audio' ID_USB_DRIVER='snd-usb-audio' ID_USB_INTERFACES=':010100:010200:030000:' ID_USB_INTERFACE_NUM='00' ID_VENDOR='Logitech' ID_VENDOR_ENC='Logitech' ID_VENDOR_FROM_DATABASE='Logitech, Inc.' ID_VENDOR_ID='046d' SEQNUM='2803' SOUND_FORM_FACTOR='headset' SOUND_INITIALIZED='1' TAGS=':seat:' UDEV_LOG='3' USEC_INITIALIZED='795038608'

As you can see, that's a lot of information! The most important part of this event and the reason we chose the sound-device-changed event in the first place is highlighted below:

:sys:sound-device-changed KERNEL='card2' DEVPATH='/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/sound/card2' SUBSYSTEM='sound' ACTION='change' ID_BUS='usb' ID_FOR_SEAT='sound-pci-0000_00_1d_0-usb-0_1_2_1_0' ID_ID='usb-Logitech_Logitech_USB_Headset-00-Headset' ID_MODEL='Logitech_USB_Headset' ID_MODEL_ENC='Logitech\x20USB\x20Headset' ID_MODEL_FROM_DATABASE='ClearChat Pro USB' ID_MODEL_ID='0a0b' ID_PATH='pci-0000:00:1d.0-usb-0:1.2:1.0' ID_PATH_TAG='pci-0000_00_1d_0-usb-0_1_2_1_0' ID_REVISION='1013' ID_SERIAL='Logitech_Logitech_USB_Headset' ID_TYPE='audio' ID_USB_DRIVER='snd-usb-audio' ID_USB_INTERFACES=':010100:010200:030000:' ID_USB_INTERFACE_NUM='00' ID_VENDOR='Logitech' ID_VENDOR_ENC='Logitech' ID_VENDOR_FROM_DATABASE='Logitech, Inc.' ID_VENDOR_ID='046d' SEQNUM='2803' SOUND_FORM_FACTOR='headset' SOUND_INITIALIZED='1' TAGS=':seat:' UDEV_LOG='3' USEC_INITIALIZED='795038608'

The event containing SOUND_INITIALIZED='1' denotes that the "overall" headset device comprising multiple sound devices is "ready to use". So we have the correct type of event and we know that that event is only emitted when the device really is ready but we still need to identify the device uniquely. That might sound like an arduous task, but it really isn't so panic not! Since this is my workstation and since I know I've only got a single USB headset, all I have to do is select  some unique value (or  combination of values) from the variables in the above data.

:sys:sound-device-changed KERNEL='card2' DEVPATH='/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/sound/card2' SUBSYSTEM='sound' ACTION='change' ID_BUS='usb' ID_FOR_SEAT='sound-pci-0000_00_1d_0-usb-0_1_2_1_0' ID_ID='usb-Logitech_Logitech_USB_Headset-00-Headset' ID_MODEL='Logitech_USB_Headset' ID_MODEL_ENC='Logitech\x20USB\x20Headset' ID_MODEL_FROM_DATABASE='ClearChat Pro USB' ID_MODEL_ID='0a0b' ID_PATH='pci-0000:00:1d.0-usb-0:1.2:1.0' ID_PATH_TAG='pci-0000_00_1d_0-usb-0_1_2_1_0' ID_REVISION='1013' ID_SERIAL='Logitech_Logitech_USB_Headset' ID_TYPE='audio' ID_USB_DRIVER='snd-usb-audio' ID_USB_INTERFACES=':010100:010200:030000:' ID_USB_INTERFACE_NUM='00' ID_VENDOR='Logitech' ID_VENDOR_ENC='Logitech' ID_VENDOR_FROM_DATABASE='Logitech, Inc.' ID_VENDOR_ID='046d' SEQNUM='2803' SOUND_FORM_FACTOR='headset' SOUND_INITIALIZED='1' TAGS=':seat:' UDEV_LOG='3' USEC_INITIALIZED='795038608'

There are some variables you do not want to select on:

  • KERNEL and DEVPATH: these refer to the physical location of the device so don't select these as that location will differ depending on which USB port you plug your device into.
  • DEVNAME and DEVNUM are incremented every time a new device is plugged in (even if it's the same device you plugged in before).

What you are looking for ideally is one more more variables that are:

  • Describe your device clearly.
  • Will always have the same value whenever and wherever you plug the device.

Your mileage may vary (for example if you have multiple different USB headsets). However, for me, ID_MODEL is sufficient and conveniently human-readable ;-)

Back to our $HOME/.config/upstart/mumble.conf session job. I can now specify the start on condition:

start on :sys:sound-device-changed ID_MODEL="Logitech_USB_Headset" SOUND_INITIALIZED="1"
stop on desktop-end

# start VoIP client of choice
exec mumble

I've added a stop on condition too such that Upstart will stop mumble just before the desktop session ends (in case I forget to stop it myself). And that's it. To try it out, simply plug in your USB headset!

An approximation of what is happening when the USB device is plugged in is:
  • The kernel detects the device and generates a uevent.
  • The upstart-udev-bridge, running at the system level, is monitoring the kernels netlink socket via libudev and therefore sees the event and emits a corresponding Upstart system event.
  • The users upstart-event-bridge sees the system event and proxies it down to the users Upstart session by emitting an Upstart user event tagged with a ":sys:" prefix.
  • The users Session Init compares the start on condition in mumble.conf with the Upstart event emitted by the upstart-event-bridge, finds that it matches and so starts the mumble job.

This example should give you a taste of the power you now have at your fingertips to harness system-level events in your own jobs :-)

A summary of well-known user-level and system-level upstart jobs is available in the  upstart-events(7) manual page.


The above will not work if the device is already plugged at boot time. The problem is that although the upstart-udev-bridge will emit the correct set of events for your device at boot time, your session won't have started at that point in time so there will be no upstart-event-bridge to listen for them. So by the time your session starts, those events will have gone.

The File Bridge

Wouldn't it be great if you could create a job that would only start when a particular file got created? Well, now you can....

Highly contrived, but let's start xclock when the X session errors file gets modified:

start on file FILE=~/.xsession-errors EVENT=modify
exec xclock

How about watching a directory for any activity?

start on file FILE=~/var/mydir/
exec xclock

Now, xclock will get started when files get created, modified or deleted in the directory $HOME/var/mydir/.

Or maybe a file glob might be useful to you?

start on file FILE=~/.cache/myapp/*.crash
exec handle-my-buggy-app

Here, the "handle-my-buggy-app" application will be started to do something useful when your app creates ".crash" files.

Understanding how jobs interact

Upstart provides a tool called initctl2dot which generates graphviz visualisations of jobs. Let's run it:

$ initctl2dot -o - | dot -Tpng -o upstart.png

Here's the result on my system:

  • The rectangular nodes represent jobs.
  • The diamonds denote events.
  • Blue lines represent start on conditions.
  • Red lines represent stop on conditions.
  • Green lines show the events that jobs may emit.
There's quite a lot happening in that diagram partly as we've added some useful "well-known" events that you can use for your jobs. In particular, note the desktop-start and desktop-end events. These are not the first events emitted, but if your job specifies the following, it is guaranteed to run in a fully-functional desktop environment:

start on desktop-start
stop on desktop-end

If you want your job to start as early as possible and end as late as possible, but don't care about whether the desktop is usable when the job starts, you can specify:

start on startup
stop on session-end

Note: The desktop-start and desktop-end events are emitted by the job that actually starts the users graphical desktop. Currently, only the default Ubuntu desktop is emitting these events, but all desktop environments that support for Session Inits need to emit these events.

The Future

There are already a lot of "well-known" events your jobs can make use of, but we plan to introduce even more to enrich the available palette next cycle. Ideas for event sources include:

  • Suspend/Resume events
  • Improved sound device events (for example, start a job when headphones plugged)
  • network events (for example, start/stop jobs when you connect to an open wifi network)
  • D-Bus signals (upstart-dbus-bridge)
  • dconf/gsettings changes (upstart-dconf-bridge)
  • cron-like temporal events (upstart-time-bridge)

Further Information

Monday, 8 April 2013

Coverity static analysis for C, C++ and Java code

It's a well known principle of software engineering that the earlier bugs can be caught, the lower the overall cost. As such, testing needs to happen at every level. Once your project is at the coding stage, the earliest form of testing is on the code itself, not on the binaries the compiler produces.

We run a variety of tools over critical codebases such as Upstart and Whoopsie regularly to identify issues well before they "escape into the wild". These tools include Coverity Scan (see the list of projects already using it).

If you really care about your code and you are involved with a C, C++ or Java project, I'd strongly encourage you to take a look at this awesome tool. If you aren't directly involved in such projects, try contacting those running them and suggesting they use Coverity.

The Coverity Scan service is entirely free for OSS projects. You will need to register to obtain an account and then download the client analysis tool. Once setup, a particularly attractive feature is the ability to auto-upload the analysis data generated for your project using ESR's coverity-submit tool. This could for example be hooked into your upload or release process to ensure no code quality regressions. After you have uploaded the analysis data, you can browse through the results of the scan using the web interface in a variety of ways, including a view that shows the errors "inline" with markers added around the code Coverity has identified as problematic.

For those who have either never used static analysis tools, or have simply never used Coverity, don't fall into the trap of thinking that gcc -pedantic -Wall or even LLVM's scan-build should be "good enough for anyone" - it simply is not. Consider too Steckel's Rule to Success,

"Good enough is never good enough"

Coverity performs very deep analysis and its results may well surprise you... but rather that than unexpected surprises for your users.

Apologies if this post sounds like a bit of a sales pitch. It really isn't though: the Coverity service is free and what they are offering really is too good to ignore.

Note: I have no affiliation with Coverity - I'm just extremely impressed with their Scan tool! :-)