As presented at the November french meetup of saltstack users, we've published code to generate some statistics about a salstack infrastructure. We're using it, for the moment, to identify which parts of our infrastructure need attention. One of the tools we're using to monitor this distance is munin.

So far, we've handled configuration changes and services restart for apache, nginx, postfix and user configuration for iceweasel (debian's firefox) and chromium (adapting to firefox and chrome should be a breeze). Some credit goes to mtpettyp for his answer on askubuntu.

The code is also published as a gist on github. Feel free to comment and fork the gist. There is room for improvement, and don't forget that by disabling SSLv3 you might prevent some users with "legacy" browsers from accessing your services.

Last week I attended DebConf14 in Portland, Oregon. As usual the conference was a blur, with lots of talks, lots of new people, and lots of old friends.
The organizers tried to do something different this year, with a longer conference (9 days instead of a week) and some dedicated hack time, instead of a pre-DebConf "DebCamp" week.
That worked quite well for me, as it meant the schedule was not quite so full with talks, and even though I didn't really get any hacking done, it felt a bit more relaxed and allowed some more hallway track discussions.

On the talks side, the keynotes from Zack and Biella provided some interesting thoughts. Some nice progress was made on making package builds reproducible.

And as usual all talks were streamed live and recorded, and many are already available thanks to the awesome DebConf video team. Also for a change, and because I'm a sucker for punishment, I came back with more stuff to do.

The EP14 Pylint sprint team (more on this here and there) is proud to announce they just released Pylint 1.3 together with its companion Astroid 1.2. As usual, this includes several new features as well and bug fixes. You'll find below some structured list of the changes.

Packages are uploaded to pypi, debian/ubuntu packages should be soon provided by Logilab, until they get into the standard packaging system of your favorite distribution.

Please notice Pylint 1.3 will be the last release branch support python 2.5 and 2.6. Starting from 1.4, we will only support python greater or equal to 2.7. This will be the occasion to do some great cleanup in the code base. Notice this is only about the Pylint's runtime, you should still be able to run Pylint on your Python 2.5 code, through using Python 2.7 at least.

Emit 'undefined-variable' when using the Python 3 metaclass= argument. Also fix 'unused-import' false for that construction (#143)

Emit 'broad-except' and 'bare-except' even if the number of except handlers is different than 1. Fixes issue (#113)

Emit 'attribute-defined-outside-init' for all statements in the same module as the offended class, not just for the last assignment (#262, as well as a long standing output mangling problem in some edge cases)

Emit 'not-callable' when calling properties (#268)

Don't let ImportError propagate from the imports checker, leading to crash in some namespace package related cases (#203)

Claudiu killed maybe 10 bugs or so and did some heavy issues cleanup in the trackers. He also demonstrated some experimental support of python 3 style annotation to drive a better inference. Pretty exciting!
Torsten also killed several bugs, restored python 2.5 compat (though that will need a logilab-common release as well), introduced a new functional test framework that will replace the old one once all the existing tests will be backported. On wednesday, he did show us a near future feature they already have at Google: some kind of confidence level associated to messages so that you can filter out based on that.
Sylvain fixed a couple of bugs (including https://bitbucket.org/logilab/pylint/issue/58/ which was annoying all the numpy community), started some refactoring of the PyLinter class so it does a little bit fewer things (still way too many though) and attempted to improve the pylint note on both pylint and astroid, which went down recently "thanks" to the new checks like 'bad-continuation'.

Also, we merged the pylint-brain project into astroid to simplify things, so you should now submit your brain plugins directly to the astroid project. Hopefuly you'll be redirected there on attempt to use the old (removed) pylint-brain project on bitbucket.

And, the good news is that now both Torsten and Claudiu have new powers: they should be able to do some releases of pylint and astroid. To celebrate that and the end of the sprint, we published Pylint 1.3 together with Astroid 1.2. More on this here.

We've had a fairly enjoyable and productive first day in our little hidden room at EuroPython in Berlin ! Below are some noticeable things we've worked on and discussed about.

First, we discussed and agreed that while we should at some point cut the cord to the logilab.common package, it will take some time notably because of the usage logilab.common.configuration which would be somewhat costly to replace (and is working pretty well). There are some small steps we should do but basically we should mostly get back some pylint/astroid specific things from logilab.common to astroid or pylint. This should be partly done during the sprint, and remaining work will go to tickets in the tracker.

We also discussed about release management. The point is that we should release more often, so every pylint maintainers should be able to do that easily. Sylvain will write some document about the release procedure and ensure access are granted to the pylint and astroid projects on pypi. We shall release pylint 1.3 / astroid 1.2 soon, and those releases branches will be the last one supporting python < 2.7.

During this first day, we also had the opportunity to meet Carl Crowder, the guy behind http://landscape.io, as well as David Halter which is building the Jedi completion library (https://github.com/davidhalter/jedi). Landscape.io runs pylint on thousands of projects, and it would be nice if we could test beta release on some part of this panel. On the other hand, there are probably many code to share with the Jedi library like the parser and ast generation, as well as a static inference engine. That deserves a sprint on his own though, so we agreed that a nice first step would be to build a common library for import resolution without relying on the python interpreter for that, while handling most of the python dark import features like zip/egg import, .pth files and so one. Indeed that may be two nice future collaborations!

Last but not least, we got some actual work done:

Michal Nowikowski from Intel in Poland joined us to work on the ability to run pylint in different processes so it may drastically improve performance on multiple cores box.

Torsten did continue some work on various improvements of the functionnal test framework.

Sylvain did merge logilab.common.modutils module into astroid as it's mostly driven by astroid and pylint needs. Also fixed the annoying namespace package crash.

Claudiu keep up the good work he does daily at improving and fixing pylint :)

This monday (19th of may 2014), Thomas Hatch was in Paris for
dotScale 2014. After presenting SaltStack there (videos will be published at some
point), he spent the evening with members of the French SaltStack community
during a meetup set up by Logilab at IRILL.

Here is a list of what we talked about :

Since Salt seems to have pushed ZMQ
to its limits, SaltStack has been working on RAET
(Reliable Asynchronous Event Transport Protocol ), a transport layer based on UDP and
elliptic curve cryptography (Dan Berstein's CURVE-255-19) that works more like a
stack than a socket and has reliability built in. RAET will be released as an
optionnal beta feature in the next Salt release.

Folks from Dailymotion bumped into a bug that seems related to high
latency networks and the auth_timeout. Updating to the very latest
release should fix the issue.

Thomas told us about how a dedicated team at SaltStack handles pull
requests and another team works on triaging github issues to input them
into their internal SCRUM process. There are a lot of duplicate issues and old inactive
issues that need attention and clutter the issue tracker. Help will be welcome.

Continuous integration is based on Jenkins and spins up VMs to test pull request.
There is work in progress to test multiple clouds, various latencies and loads.

For the Docker integration, salt now keeps track of forwarded ports
and relevant information about the containers.

salt-virt bumped into problems with chroots and timeouts due to ZMQ.

Multi-master: the problem lies with syncronisation of data which is
sent to minions but also the data that is sent to the
masters. Possible solutions to be explored are : the use of gitfs,
there is no built-in solution for keys (salt-key has to be run on
all masters), mine.send should send the data at both masters,
for the jobs cache: one could use an external returner.

Thomas talked briefly about ioflo which should bring queuing,
data hierarchy and data pub-sub to Salt.

About the rolling release question: versions in Salt are definitely
not git snapshots, things get backported into previous versions. No
clear definition yet of length of LTS versions.

salt-cloud and libcloud : in the next release, libcloud will not be
a hard dependency. Some clouds didn't work in libcloud (for example
AWS), so these providers got implemented directly in salt-cloud or by using
third-party libraries (eg. python-boto).

Documentation: a sprint is planned next week. Reference
documentation will not be completly revamped, but tutorial content
will be added.

Boris Feld showed a demo of vagrant images orchestrated by salt and a web UI
to monitor a salt install.

Thanks again to Thomas Hatch for coming and meeting up with (part of) the
community here in France.