Introduction

As cdist is getting more popular, more people are using
cdist and some questions arrive more often from newbies than others.
One of them is why we require having Python 3.2 on the source host.
If you are also wondering about the motivation, or you're screaming because
you only have Python 2.x or Python 3.1 available in your distribution, this
article is for you.

Note: Cdist does *not* require Python on the target hosts!
Note: Cdist requires only ssh and a posix shell on the targets.

History of cdist

Because you may be one of the people who are screaming, I'm giving you
an overview about the whole development history of cdist, which will
make things more clear for you.

The logical consequence of my statement was to try out, whether it's
actually possible to write a CM tool completly in a very simple manner.
For instance in posix shell script. This led to the first commit to the
newly born cdist repository:

After a lot of discussions, design ideas, pictures on the whiteboard,
trying out different implementations, weighting up advantages of
each one, the first official release of cdist was put into public,
cdist 1.0.0:

tag 1.0.0
Date: Mon Mar 7 18:21:18 2011 +0100

Cdist 1.0.0 is completly implemented in posix shell and had almost all
features of the current cdist implementation. With one major drawback:
Performance. When running cdist 1.0.0 in parallel mode, the source host
easily became the bottleneck. A typical run of cdist 1.0.0 caused around
3500 - 5000 forks. Running in parallel mode with about 10-15 target hosts,
most time of a cdist run was spent in kernel space to handle the forks.

The logical sequence again was to search for the reason for the huge amount
of forks, which was easily detected: Every routine was a shell script on its
own, that required a new launch of the shell. Now, for some operations,
like working on the dependency tree, a lot of sub-routines were called, leading
to the huge number of forks.

We tried to minimise the number of forks by migrating from shell scripts to
shell functions, which was a big pain, when we realised that posix shell
does not have local variable support anymore. Posix states that you should
use shell scripts instead of shell functions, if you need distinct environments.
Which is exactly were we came from...

Thus we decided to switch to a different programming language for cdist's core,
but only for the core. For us it is very important to minimise the dependencies
on the target hosts: We do not want to install Ruby, Python, Java, libfoo or
anything that is not usually available on a freshly installed base system.
Cdist should be able to take over, as soon as the system is setup in a very
basic state.

The choice felt on Python, because it felt very mature and easy to use.
Additionally, Python 3 already provides a lot of functionality in its
base installation: Everything we were used to do in shell, could easily be
rewritten in plain Python 3. After exactly one year after
the initial commit, cdist 2.0.0, a complete rewrite in Python 3
was finished and released:

tag 2.0.0
Date: Fri Sep 16 12:11:28 2011 +0200

Now, why Python 3.2?

During the development of cdist 2.0, we had a lot of discussions
about clean design, pythonic ways of doing stuff (versus using the
shell approach in python) and which functions to use. In the beginning,
we were discussing about whether to settle for Python 2 or Python 3.
As we did not have any dependencies or old code that relies on Python 2,
the choice for the current stable tree, Python 3, was easy to make.

Python 3.2, in contrast to Python 3.1, includes the very good usable
argparse module,
as well as an enhanced variant of the
os.makedirs
method that supports the exist_ok parameter.

The argparse module is also available for Python 3.1, but not the
enhanced os.makedirs method. So we had to decide: Should we
integrate a simple workaround to support the previous Python 3 release,
Python 3.1, or shall we upset users with old Python installations?

To answer this question, we had a look at the current situation of
Python in various distributions.

There are distros which do not have Python 3 at all (Centos, Redhat, Slackware)

Python 3 is definitely needed, so requiring 3.1 or 3.2
does not make a difference

There are only two cases, which would make it interesting to support
Python 3.1: Debian Squeeze (currently stable) and NetBSD.

As I've been a long time Debian user, I understand this may be a bit
annoying, because it's unclear, when wheezy will become stable.
On the other hand, you can easily install python 3.2 from source to
anywhere, like you'd do in the situation, when you wouldn't have
python 3 at all.

Another point speaking against 3.1 support for Debian is the fact that
distributions should support the user and not developers should support
distributions that ship old software (there's nothing against supporting
old and new versions, though). That's by the way one of the reasons
why I moved away from Debian...

I am short on experience regarding NetBSD, but installing via source
shouldn't be an issue either.

To summarise: Support for Python 3.1 only makes sense for Debian Squeeze
and NetBSD at the moment. This requirement will soon [tm] be superseeded
and can easily be worked around by selecting one of many distributions
with more recent software packages. And that's the reason, why we settled
for Python 3.2.

Btw, who is we?

You mave have noticed that I am often referring to we in this article.
The second main developer for cdist is
Steven Armstrong, a sysadmin at
ETH Zurich and good friend of mine.
The discussions and development time we spent together was very valuable
for me as well for the whole cdist project and thus I wanted to use this
article to say