Since I’ve been designing the split-flap display as an open source project, I wanted to make sure that all of the different components were easily accessible and visible for someone new or just browsing the project. Today’s post continues the series on automatically rendering images to include in the project’s README, but this time we go beyond simple programmatic bindings to get what we want: the schematic!

"Wow, I bet someone had to manually click through the GUI to
export such a beautiful schematic!" Nope.

Unfortunately, KiCad’s schematic editor, Eeschema, doesn’t have nice Python bindings like its pcb-editing cousin Pcbnew (and probably won’t for quite some time). And there aren’t really any command line arguments to do this either. So we turn to the last resort: UI automation. That is, simulating interaction with the graphical user interface.

There are two main issues with automating the graphical user interface: the build system (Travis CI) is running on a headless machine with no display, and the script needs to somehow know where to click on screen.

As I mentioned in my last post, we can use X Virtual Framebuffer (Xvfb), which acts as a virtual display server, to solve the first problem. As long as Xvfb is running, we can launch Eeschema even when there’s no physical screen. This time, instead of using `xvfb-run` from a Bash script, I decided to use the xvfbwrapper Python library for additional flexibility. xvfbwrapper provides a Python context manager so you can easily run an Xvfb server while some other code executes.

from xvfbwrapper import Xvfb
with Xvfb(width=800, height=600, colordepth=24):
# Everything within this block now has access
# to an 800x600 24-bit virtual display
do_something_that_needs_a_display()

So how do we actually script and automate interactions with the GUI, such as opening menus, typing text, and clicking buttons? I looked into a number of different approaches, such as Sikuli, which allows you to write high level “visual code” using screenshots and image matching, or Java’s Robot class which lets you program the mouse and keyboard using Java, but the easiest option I found by far was the command-line program xdotool.

With xdotool, you can easily probe and interact with the window system from the command line. For instance, you can output a list of all named windows by running:xdotool search --name '.+' getwindowname %@

(This is an example of a chained command: the first part (search --name '.+') finds all windows whose name matches the regular expression ‘.+’ (any non-empty string) and places those window ids onto a stack. The second part runs the command getwindowname, with the argument %@ meaning “all window ids currently on the stack.”)

Going back to Eeschema, the option we want to automate (exporting the schematic) lives under the File → Plot → Plot menu. The trick to automating this is not to use the mouse to click (since then we’d need to know the coordinates on screen) but instead use keyboard shortcuts. Opening that menu from the keyboard just requires pressing “Alt+F” then “P” then “P”, which we can automate like this:

We can similarly write commands to fill out the correct information in the “Plot Schematic” dialog once it opens. To change radio button selections, we can “Tab” numerous times to move focus through the various options. This is a bit fragile, since it relies on there being a stable set of options in the same order to work (and might break if KiCad were to add a new Page Size option, for instance), but is about the best we can do without using more complex UI automation tools.

To make it easier to debug what’s happening in the X virtual display, we can use a screen-recording tool like recordmydesktop to save a screencast of the graphical automation. This is particularly helpful when running on Travis where you can’t actually see what’s going on as the script runs.

Since we’re writing in Python, we can use some syntactic sugar with Python context managers to make it really easy to wrap a section of code with Xvfb and video recording. As a first step, we’ll need a context manager for running a subprocess:

You can use that macro like so:
with recorded_xvfb('output_video.ogv', width=800, height=600, colordepth=24):
# This code runs with an Xvfb display available
# and is recorded to output_video.ogv
do_something_that_needs_a_display()

# Once the 'with' block exits, the X virtual display is
# no longer available, and the recording has stopped
run_non_recorded_things()

So, putting all of those elements together, we can use Xvfb to host the Eeschema GUI (even on a headless build machine), run recordmydesktop to save a video screencast to help understand and debug the visual interactions, and use xdotool to simulate key presses in order to click through Eeschema’s menus and dialogs. The code looks roughly something like this:

Hopefully this was a useful overview of how I used UI automation to export schematics from KiCad. If you have questions, leave a comment here or open an issue on on github and I’ll try to respond. In my next post in this series I’ll switch gears a bit and talk about how I programmatically generate the OpenSCAD 3d animation you see at the top of the project’s README!

Sunday, April 17, 2016

This is my second post in a series about the open source split-flap display I’ve been designing in my free time. I’ll hopefully write a bit more about the overall design process in the future, but for now wanted to start with some fairly technical posts about build automation on that project.

In my last post, I discussed how I scripted the export of 2d renderings of the custom PCB. In this post, I’ll cover how I hooked up that script and others to run automatically on every commit using Travis CI, with automated deployments to S3 to keep all the renderings in the README updated, like this one:

I'll talk about this particular animated OpenSCAD rendering in a future blog post

Why Travis?

Travis CI is a continuous build and test system, with Github integration and a matching free tier for open source projects. If you’ve ever seen one of these badges in a Github README, it’s probably using Travis:

That's the current build status, hopefully it's green!

The best thing about Travis though is that unlike many build systems (like Jenkins or Buildbot), nearly the entire build system configuration for Travis lives directly inside the repo itself (in a .travis.yml file). This has a few major advantages:

Reproducible (or at least reasonably well defined) build environment
Each Travis build starts off as a clean slate, and you’re responsible for defining and installing any extra dependencies on the machine yourself through code. This way you always end up with clearly documented dependencies, and that documentation can never go stale!

Enables different build/test configurations on each branch
One big problem with keeping your code separate from the build configuration (as is often the case with tools like Jenkins/Buildbot) is that the two need to stay in sync. Typically this is not a huge problem for slow, linear development, since occasional lock-step updates across repo and build system aren’t too painful.

The issues start when you have faster development with frequently changing build configurations or parallel development across branches. Now not only do you have to keep your build configuration in sync with changes in the source repo, but you also have to make it branch-aware and keep each branch’s build config in sync with the branches in the source repo! Travis avoids all of this because the .travis.yml file is naturally versioned alongside the source it’s building, and therefore just works in branches with no extra effort!

Build configuration changes can be tested!
Related to the previous point — since the .travis.yml file is checked in and versioned with the source code, changes to the source code that e.g. require new packages to be installed in the build environment can actually be fully tested as part of a feature branch or pull request before landing in `master`.

Travis with KiCad and OpenSCAD

The first step to automating my build was to install the right packages. The basic .travis.yml config looks like this:

Both KiCad (schematic/pcb software) and OpenSCAD (3d cad software) are under fairly active development, and their packages in the Ubuntu 14.04 are woefully out of date, so I use snapshot PPAs to install more modern versions of each (this necessitates the use of `sudo: true` above which allows for running `add-apt-repository` under sudo).

Each of the install scripts referenced above is pretty straightforward and looks roughly like this:

The only interesting part of that is the use of `xvfb-run`. Getting OpenSCAD exports working is slightly trickier than KiCad, since even OpenSCAD’s command-line interface requires a graphical environment to render images. The trick to make this work on a headless build machine is to use X virtual framebuffer (Xvfb), which lets you run a standalone X server detached from an actual display. So in the config above, I use the `xvfb-run` utility, which starts an Xvfb server, sets up the DISPLAY environment, runs the specified command, and then shuts everything down when the command completes; easy! (I’ll discuss the actual `generate_2d.py` and `generate_gif.py` script implementations in a future post)

From Travis to the README

Now that we’ve got Travis set up installing KiCad and OpenSCAD and exporting images from each on every commit, the next step is to actually get those renderings off the build machine and somewhere useful. To do that, I use Travis’s deploy tool to upload those build artifacts to S3.

The configuration is again pretty simple. Here’s what it takes to upload the entire “deploy” directory on the build machine to a publicly-readable directory named “latest” in my “splitflap-travis” S3 bucket:

Since the .travis.yml file is checked into the repo and public, putting your actual S3 credentials inside would be silly! But Travis allows you to encrypt your credentials using a secret that only their build machines know, so everything’s nice and secure despite being public.

One thing you may notice is the black bar at the bottom with the date and commit hash. I added that because Github’s image proxy caches extremely aggressively and I originally didn’t include the `cache_control: no-cache` line in my deployment config, so I needed some way to debug. It was pretty easy to add using ImageMagick, and now I can easily tell that the images in my README are showing the latest designs correctly:

If you do find yourself stuck with cached images on Github, you can manually evict them from the cache using an http PURGE request to the image url:
`$ curl -X PURGE https://camo.githubusercontent.com/xxxxxxxxxxxxx`

For the past few months I’ve been designing an open source split-flap display in my free time — the kind of retro electromechanical display that used to be in airports and train stations before LEDs and LCDs took over and makes that distinctive “tick tick tick tick” sound as the letters and numbers flip into place.

I designed the electronics in KiCad, and one of the things I wanted to do was include a nice picture of the current state of the custom PCB design in the project’s README file. Of course, I could generate a snapshot of the PCB manually whenever I made a change by using the “File→Export SVG file” menu option and then check that image into my git repo…

…but that gets tedious, is prone to human error, pollutes the git history with a bunch of old binary files, and isn’t very customizable.

For instance, the manual SVG export uses opaque colors which make it hard to see features that overlap, as well as using two different colors for items on the same layer (yellow and teal are both part of the front silkscreen layer below):

Functional rendering, but not exactly what I wanted.

Luckily, Pcbnew has built-in Python bindings which make it pretty straightforward to invoke certain features from standalone Python scripts. As a simple example, here’s how to plot a single layer to an SVG:

While it’s in theory possible to specify the colors to use when plotting, I ran into issues where certain items were always plotted in their default color. For instance, when I plot the front silkscreen layer with the following options, the footprints are plotted in teal rather than the specified color, red:

So instead of trying to get Pcbnew to output the exact SVG I wanted, I decided to export each layer as a separate monochrome SVG image and then post-process them to apply colors and merge them into a single output file. Since SVG images are just XML, it was easy to write a script, svg_processor.py, which allowed me to override the “fill” and “stroke” style attributes of the shapes, and then wrap all of the shapes in a <g> group tag to set the desired opacity.

(Note: the reason for wrapping in a group before applying opacity is that things like traces are rendered as a combination of multiple shapes, like a line + circle, so if you applied alpha=0.5 to each shape individually, a single trace would have varying degrees of opacity depending on how its subcomponents overlapped)

This allowed me to write a simple definition of the PCB layers to export and turn that into a nice, customizable rendering:

As a final step after processing and merging, I use Inkscape's command line interface to shrink the .svg canvas to fit the image and convert the vector .svg file into a raster .png image like you see above: