Docker containers for building deviation

I've just pushed a set of docker containers for building deviation to
hub.docker.com/u/mwmeyer/
. If you're not familiar with it, docker is a lightweight container system designed for running commands - and not systems - in virtual machines. It's perfectly reasonable - and even normal - to spin up a docker container to to do part of a build process, and then remove it when that's done. They don't have the overhead of running an operating system, and expect to run in the CLI, so they are easy to integrate into an existing development environment, as opposed to having to deal with a second machine.

See the description of mwmeyer/deviation description for some suggested uses for these. But the short version is that you'll want to run containers that mount your local source tree as a volume and build against that, as they don't include sources. This means the images never change.

There are four of them:

mwmeyer/deviation - has the tools recommended in the deviation README to build transmitter images and zips of the same.

mwmeyer/deviation-emulators - this adds the tools needed to build the windows emulators and zip files. The command "make release" in this container will build everything you'd want to upload.

mwmeyer/deviation-manual - This only has tools for building the manuals.

mwmeyer/deviation-sources - This is a data-only image that comes pre-populated with the sources for deviation and the manual. You probably don't want this unless you're going to be building in the cloud.

This is a different type of solution than the existing virtualbox image. Which you'd prefer will depend a lot on you and your preferred tools.

Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.

Yeah, I've been meaning to share those. Just haven't done it yet. I've attached a zip file with everything you need to create them except gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar.bz2, which you'll want to download from launchpad.net.

The one odd thing is the "deviate" script, which was designed as a simple "build a branch" command. You can safely ignore it if yu want.

As for the dfuse tool, they can be used to build those. You'll want to mount the sources on the container, and you'll have to make sure the libopencm3 sources as well as the bootloader are in it. The toolchain is the same one, though.

Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.

Well, this is my first docker project, so I'm not in much better shape than you are. For the record, I'm running it on a Mac.

However, your first command line should start "docker run -v ...", not "docker run deviation -v ...". That extra "deviation" becomes the image name, which makes it search for it in the docker.io/library instead of my account, hence the error message.

silpstream replied the topic: Docker containers for building deviation

Thanks so much for the files mwm!

I used Kitematic to link the volume and start up the container and then went to the cli ("docker cli" button on the bottom left of Kitematic) and did a "docker attach deviation" to get inside.

I then ran "make zip TARGET=devoXX" (replace XX with model number) from the src folder. This created a zip file with everything I needed to flash using the 0.8 version of uploader tool. Remember to run "make distclean" after to clean things up for the next build.

next make a directory where you want the builds to go on your host machine:
mkdir ~/devo_builds

now you can start the image in a couple ways:
As a developer where you already have a cloned deviation repo
docker run -it -v <path to git repo head>:/git -v ~/devo_builds:/release --name deviation_build deviationtx/deviation-docker
NOTE: <git repo head> is the directory above your repo (it should contain a subdirectory called 'deviation')

As a user without a cloned repo:
docker run -it -v ~/devo_builds:/release --name deviation_build deviationtx/deviation-docker

On subsequent runs, you can do:
docker start -i deviation_build

The scripts will 1st check whether there have been any updates to the build script recently. It will then update your deviation repo (or clone it if you didn't provide one). next it will show a menu that will let you choose what you want to build. Third it will fetch any additional files needed to run the build. Lastly it will build the selected targets and store the results in the ~/devo_builds dir on your host machine.

The build env will be cached, so the next time you run, it will be much faster.

The build script is still very crude. I will provide options to further configure the builds in the future. I also need to test building on Windows, and write some instructions, but my understanding is that I should be able to build a lightweight installer to automate all of that.

If this works, I plan to make it the only supported build system. This image should be nearly identical to the version used by our official and nightly builds, so there should be no cases where a build works in one place and not in another.

silpstream replied the topic: Docker containers for building deviation

PhracturedBlue wrote: now you can start the image in a couple ways:
As a developer where you already have a cloned deviation repo
docker run -it -v <path to git repo head>:/git -v ~/devo_builds:/release --name deviation_build deviationtx/deviation-docker
NOTE: <git repo head> is the directory above your repo (it should contain a subdirectory called 'deviation')

As a user without a cloned repo:
docker run -it -v ~/devo_builds:/release --name deviation_build deviationtx/deviation-docker

On subsequent runs, you can do:
docker start -i deviation

Very nice. Having the interface will go a long way to easing the build process. Especially if you are going to start adding in build time options later on.

Worked for me on "devo7e". "devo7e-256" throws up an error "No rule to make target `zip_devo7e-256'".

You have a typo error for subsequent runs above, should read as "docker start -i deviation_build", if not mwm's docker container gets started instead.

I've not looked into the scripts you have.

I'm wondering, for those of us developing on forked clones (most developers would be right?), I assume that it doesn't actually do any updating and will not switch me back to master branch. Developer forks are probably not a use case right?

PhracturedBlue replied the topic: Docker containers for building deviation

It should work on developer forks, and yes they are absolutely a use case. Ideally everyone will use these images (once they are fully baked). Note that you MUST use an external git repo and the '-v <...>:/git' option if you use a non-standard repo. It WILL run a 'git pull' 1st. Maybe I'll disable that if any modifications are detected.

The current proff of concept GUI is using 'dialog' which is very limited in its capability. I found something called 'snack' for python which allows building more complicated text-based GUIs. I'm going to rework the current script into a python script which will allow me to add some developer-centric options. I will take a look at the devo7e-256. Not sure why that wouldn't work.

PhracturedBlue replied the topic: Docker containers for building deviation

I updated the docker image today and replaced the shell based gui with a python based one. It doesn't actually work exactly as I want, but it is close, and will add a ot more flexibility for the future. You'll need a new image since I changed the pre-installed packages. Hopefully for the last time.

The issue I have is that I want the 'Update GIT' button to run a pull and return to the menu. This works fine on my linux machine, but for some reason in docker it hangs. I've temporarily changed the code to exit after an update to avoid the hang, which isn't ideal, but it will do for the time being until I figure out what is wrong in docker.

If you already have VirtualBox installed, make sure it is at the latest version before you begin AND unselect it from the list during docker install. If you don't have the latest VirtualBox, the install will pass, but initialization will fail. It took me a while to figure out what was going on

now click 'Docker Quickstart Terminal'
When you eventually get a prompt (this is a one-time setup), run:
docker pull deviationtx/deviation-docker

Note that the OSX version of docker uses the same technology that Windows does, running containers on a virtualbox image, and dealing with the installation of virtualbox if you need to. The instructions are virtually identical to the Windows ones.

I guess the question is - how hard is this compared to installing a build environment "by hand". For the Mac, I believe it's much easier, but never bothered to do that as we don't have instructions for it. For windows, it seems a bit easier, but again - I haven't done that. For linux, it fixes the issue of making sure you have the right build chain, which has gotten a bit painful to deal with.

Are these 32-bit or 64 bit Linux images? The ones I built are 64 bits and can't work on my BSD desktop.

Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.

Most of the time for a docker container to access outside world the simplest solution is to use networking type as 'host' - this will make the container inherit the network configuration - same interfaces, same IP settings and routes. This way there's no need to fiddle with SNAT rules, at least on Linux. This can be different on other OSes though.

silpstream replied the topic: Docker containers for building deviation

@PhracturedBlue. This is great! I got things working on OS X for devo7e, devo7e-256 and the devo7e emulator. Just 3 things:

1) I cannot update from git, I'm guessing it might be because I'm using my own repository and am using ssh keys, maybe?

Updating build.py
Warning: Permanently added the RSA host key for IP address '192.30.252.128' to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.

2) Any chance you can give us the ability to get out of the snack gui and into command line within the container. It is useful to be able to operate from command line within the build environment when developing.
3) Could we possibly also specify the git branch prior to pull. Makes life easier when testing code in a branch. Of course if (2) comes true, then we should be okay.

PhracturedBlue replied the topic: Docker containers for building deviation

I'm out of space on the current gui page. I'll need to think about how to add more options on a 2nd page. In developer mode, I assumed you would manage your git repo yourself, and just use the docker image for building. so just turn off auto-update and setup the repo as you want before you run it. I think switching branches is probably too much to ask of the little gui, but adding a shell option should be reasonably easy to to, so I'll look into that