When it comes to systemd middleware, Lennart Poettering often takes the blame
and has sole authorship attributed. But there are many more developers
(git shows 593 authors in total) – missing their portion of berating,
thus unappreciated and unhappy.
Over the Winter Holidays I’ve run LWN's “who wrote” scripts to gather
more insight into systemd’s developer base.

Developers with most changesets

Developers with the most changesets

Lennart Poettering

Red Hat

7104

38.5%

Kay Sievers

Red Hat

3711

20.1%

Zbigniew Jędrzejewski-Szmek

hobbyist

1446

7.8%

Tom Gundersen

Red Hat

948

5.1%

Greg KH

Linux Foundation

624

3.4%

Michal Schmidt

Red Hat

369

2.0%

Thomas Hindoe Paaboel Andersen

GNOME

253

1.4%

David Herrmann

hobbyist

233

1.3%

Martin Pitt

Canonical

231

1.3%

Harald Hoyer

Red Hat

207

1.1%

Dave Reisner

Arch

148

0.8%

Lennart leads, but with less than 40% of all changesets. It seems he has written less than
half of systemd. Sorry, Lennart :)
Kay Sievers earned second place mostly by maintaining udev for past couple of years.
Majority of his commits were made while he was employed by Novell. Kay now works
at Red Hat.
Third most active developer, Zbigniew Jędrzejewski-Szmek, commited less than 8%
during the history of systemd. Nevertheless, Zbigniew steadily increases his productivity.
OpenHub’s graphs for systemd
shows he wrote 12% of changesets in 2014.
Clearly, systemd is driven by top five developers. There is a short drop afterwards.
Eleventh person on list contributed less than 1% of all changesets.
Red Hat as an affiliation appears few times. Further analysis is in the last section – ”Direct comitters”.

Developers with most changed lines

Developers with the most changed lines

Kay Sievers

565610

34.3%

Lennart Poettering

438513

26.6%

David Herrmann

94753

5.7%

Tom Gundersen

70541

4.3%

Greg KH

66951

4.1%

Zbigniew Jędrzejewski-Szmek

61506

3.7%

Michal Schmidt

16948

1.0%

Patrik Flykt

7386

0.4%

Per-line statistics looks a little bit different. Here Kay Sievers leads the pack
with one-third of all lines changed. It is again mostly caused by the long history
of udev, which was merged into systemd two and a half years ago – in April 2012.
Big individual counts by David Herrmann and Tom Gundersen can be attributed to non-init
components they developed in systemd project. David wrote virtual console implementation,
hacked on kdbus and logind. Tom created network configuration element of systemd.
Speaking of the networking, Patrik Flykt’s high position comes from refactoring ConnMan's
DHCP client library into systemd-networkd. Recently, the library was
utilised by NetworkManager 1.0.

Employers with the most hackers

Employers with the most hackers

Red Hat

46

7.1%

Novell

19

2.9%

Intel

14

2.2%

IBM

10

1.5%

Samsung

9

1.4%

Canonical

5

0.8%

Fujitsu

4

0.6%

ProFUSION

4

0.6%

Mandriva

3

0.5%

Dell

2

0.3%

Axis Communications

2

0.3%

Linux Foundation

2

0.3%

OLPC

2

0.3%

SGI

2

0.3%

HP

2

0.3%

While LWN’s scripts try to map each author to his parent company,
it’s less robust than other kinds of statistics. People change employers while
contributing to the same project. I do not see “hobbyists” in the table above,
while over 10% of systemd has been created by them. Some authors contribute
patches in their free-time, without relation to the employer.
Nevertheless, it is interesting to see more or less recognizable company names in the list.

Direct commiters

There are 26 people with commit access to systemd git.
Biggest group – 9 people – work for Red Hat. This is not surprising.
RH have choosen systemd as a foundation for their enterprise offering. They
had to build expertise and assure sustainable development. Sure way for that is to hire
people contributing to systemd.

Next groups come in threes. Three people without clear company ties – I call them
hobbyist. Three Debian/Ubuntu developers, with at
least one person employed by Canonical. There are also three names from Intel.

Two developers can be associated with GNOME project. There are single developers from openSUSE,
Mageia, Arch and CoreOS.
There are also single committers from companies: Pantheon Systems and Aldebaran Robotics.

Bear in mind, above only talks about people with direct commit access to systemd’s git.
There are companies contributing many patches through the mailinglist,
but without dedicated commiter. I can recall few names from Samsung,
which has contributed quite a number
of patches. Tangentailly, there were only 2 patches from Jolla, which produces mobile phones and tablets with systemd onboard.

Summary

Red Hat pays the bills for biggest number of systemd authors. But please note, some of them were hired by RH
specifically because of their earlier contributions to systemd. Community around
project contains representatives from other major distributions, some of them even
with direct commit access. Tom Gundersen, for example, did majority of his work as an Arch developer.

Some parts of systemd came from specific needs of external users.
For example: CoreOS sponsored development of systemd-networkd;
Pengutronix contributed watchdog support;
Pantheon improved scalability with thousand of units, etc.

Archived comments:

quest2014-12-30 22:03:35

Wow. What a long post!

First off. I didn't know Poettering wasnn't sole developer, but after his recent letter where he complained for people "trying to hire hitman to kill him", I think the other are somewhat lucky.

and second. "author" is only noun. It shouldn't be used as verb > http://www.thefreedictionary.com/author

zdz2014-12-30 23:15:11

Thanks, I've replaced "author" with synonyms.

Linuxhippy2014-12-30 23:56:46

As most of the time, Redhat is one of the few companies truly investing in open-source instead of just making bucks with it (canonical which e.g. has their desktop stuff under a license which makes it unuseable for anybody else)

Jristz2014-12-31 01:07:33

@quest English is a language not normed, in contrast with spanish, french, bulgarian, etc; therefor any complain on what is correct and what is not in english grammar is either correct and wrong at same time, the main objetive is simply to be underestandable for the english readers, since without a norm solid writted like the examples I give, everithing that is underestandable is valid, even using author as a verb.

lRem2014-12-31 01:15:57

> Google 0%

Impressive ;)

tg2014-12-31 02:24:00

Nice post.

Minor comments: Dave Reisner works for Google (but I don't know if he contributes on their time) and David Herrmann works (now) for Red Hat.

Out of interest, how does the changed lines statistics handle non-code changes? If stuff like hwdb/*.hwdb and src/libsystemd-terminal/unifont.hex is included it would skew the numbers a lot I guess.

Jristz2014-12-31 03:17:41

Dave Reiser is Arch developer, right, but they work indirectly to Google since the place where he work was buyed by google around 2009

Adam Williamson2014-12-31 05:50:20

It's perfectly fine to use 'author' as a verb, e.g. "this changed was authored by Bob". Perfectly good English. I don't really like *any* free dictionary sites, but m-w.com is the best of a bad bunch, and it lists the verb use of 'author':

http://www.merriam-webster.com/dictionary/author

really, though, if you want a good dictionary, pay for an OED subscription; it's worth it.

Jono Loves Bacon2015-01-01 23:05:32

>> Google 0%

>Impressive ;)

Why would you expect Google to contribute to this project?

zdz2015-01-02 13:39:01

There are 34 patches from Filipe Brandenburger, using his @google.com email address. It's hard to say if those were made during his paid hours.

SDSSS2015-01-02 18:06:05

Well, the top two developers (Poettering and Sievers) have been chided more than once for their attitude, in general, and their sloppiness, in particular.

Anonymous2015-01-08 00:55:31

Where are you getting the employer data from? I didn't think the LWN database was generally available, and relying on the email domain is not reliable.

Proper supervision of daemons requires knowledge which cannot be inferred automatically. Some aspects of service behaviour have to be described by administrator. Widely used upstart system manager has 3 type-like job properties: expect daemon, fork and stop. Their description starts with a warning:

This stanza is extremely important: read this section carefully!

The warning is no less true with systemd's Type=.

Wrong type specification impacts service monitoring, failure detection and dependencies handling. There are six service types in systemd. Three basic ones (simple, forking, oneshot) and three more sophisticated (dbus, idle, notify).

Failure to set proper type may not bite you immediately. Service may run for minute or two and then get killed mysteriously. Services depending on wrongly specified one won't be started, even if it appears to run fine; but systemd won't consider the service ready and will not start dependent units.

Type highlights

simple - default type; daemon has to stay in foreground. Ready: as soon as the binary is executed (which may be too soon, service may not be ready to serve requests yet).

forking - daemon must fork. Ready: after first process exits. If PIDFile= is defined, readiness is delayed until PID is written to the pidfile.

notify - most flexible and robust type; by defining simple communication protocol between daemon and systemd, precise information about state can be provided. Requires simple patching, for scripts you can use systemd-notify helper. Ready: when service says so.

Basic type determination

Suitable type can be guessed correctly in 75% cases by just running the daemon binary. If it stays in foreground, connected to the terminal, the type is “simple”. If it forks into the background, the type is “forking” (suprisingly!)

# /usr/sbin/daemon
Copyright 2014 Foo Baz Bar Corp.
Serving Requests…

→ Type=simple

# /usr/sbin/otherdaemon
#

→ Type=forking

If you need to run a program/script which does its jobs and exits (does not stay running), use oneshot. Please note: if you can choose how the daemon behaves, it generally better to go with "forking". This makes systemd declare "ready" after daemon is actually able to accept connections.

Failure modes

Most of the failures shown below happen after TimeoutStartSec=. By default it’s set to 90 seconds, so lets stick with this value.

Incorrectly selected Type=

simple

forking

oneshot

notify

Correct type

simple

✓

killed after 90 s1

stays in “activating” state forever2

killed after 90 s3

forking

killed almost immediately4

✓

stays in “activating” state forever

killed after 90 s

oneshot

becomes “failed” after executing5

killed after 90 s

✓

killed after 90 s

notify

probably works fine

killed after 90 s

stays in “activating” state forever

✓

Notes:

systemd expects process to fork and parent to exit. It waits until timeout.

it never gets to the “ready” state, so no units depending on it will start; start timeout is ignored for “oneshot” units.

systemd will wait for ready notification. Until timeout.

”main” process exits

daemons should not exit, but “oneshot” units should

I hope that explains why sometimes systemd kills your service after a minute. Of course you should read
man systemd.service, it contains much more details.