Allegro Planet Remote Builds

Some of you may recall discussions about Allegro Planet offering automated builds of your Allegro games. I wanted to share my very promising progress on the project so far.

Progress

Support for project-specific GitHub webhooks has been developed and is in Allegro Planet. Webhook calls from GitHub are collected in Allegro Planet on a per-project basis. (These will be used eventually to trigger builds).

Click here for an image demonstrating how you would add a webhook to connect your Allegro Planet Game <-> GitHub Repo.

Plans

Move the built file from the tmp/ directory into a publically accessible location.

Create a Build class that keeps status of an ongoing build, and notifies the user/subscriber of build events.

Add a "watch build" type of page that allows users to follow the progress (and possible error messages) of their builds.

Move work done on RemoteGithubRepoBuilder into a background job. Have it create a Build object and update that object during during build process.

Investigate Rails' ActionCable to notify subscribers.

Attach new builds to a project (ensure no filename collisions), and add a download button to the Game's project page 🎉

Problems

Early versions will likely have very limited build options and may impose restrictions on your directory structure and Makefile. Since Makefiles can arbitrarily run any commands on the system, there will need to be safeguards to ensure that the system remains secure. This may include a requirement for container builds, or might introduce an optional .allegro-planet file in your repo that include a limited scope of build settings for your project.

Project build dependencies will likely need to be included in the project's .allegro-planet file and may be restricted to allegro and allegro_flare for the time being.

Early versions will likely only support Linux builds with the clang/gcc compiler until cross-compilation can be solved.

Remote builds initially adds support for GitHub webhooks, but a more open-ended build system allowing cloning repos from arbitrary locations could be pretty cool, too.

Builds should ideally be handled completely on the Allegro Planet (or docker on the Allegro Planet) server. It's possible there will be some pressure to have builds run on remote systems and I would like to avoid that if possible.

GitHub allows the remote server to post ongoing status directly back to your repo. This feature would reduce build<->check iteration time substantially but would require users to generate a token and/or integration permissions on a repo and would add an additional time dependency on development.

Ways That You Can Help

Cross-platform building is still unexplored and a bit of an unknown. Ideally, the system could cross-compile binaries with the llvm compiler and not rely on any third party OSs to produce the builds. This also means that there are dlls and other dependencies needed for distribution, and that needs to be accounted for as well. None of this has been explored with much depth yet.

(botvm is the name of my VM, botsnapshot is the name of a snapshot with the initial state I want it to be in before running the arbitrary command)

Then I run the binary over ssh into that VM. There are solutions which are faster (but the above seems plenty fast, the bot still replies in less than a second) and more secure (there are ways to break out of virtual box). But just saying, it should not be very hard. And with Virtual Box you can also choose to run commands inside of Windows or OSX

I use docker for everything now - my build server, my blog and the TINS website all run on docker now. And on the build server most of the build jobs run in a separate docker container as well. It's the perfect way to keep dependency chains isolated and avoid conflicts. Docker ftw!

Theoretically it would be possible to set up a docker build environment for cross compilation, but I haven't tried this yet. I still do my windows builds on a fully fledged windows VM.

Remember a docker environment is linux (even if it's supported on Win10 and OS X with docker machine, that's just a thinly veiled VM running linux).

What's the configuration of your build server? Are you on a cloud-hosted service?

The build server is an old laptop running under my desk. It uses Jenkins to define and schedule the jobs (you can set it up to track git commits).

My website is hosted on a VPS, which basically gives you a complete linux VM to play with. I can install anything, the sky is the limit (well, as long as it fits in 20G disk space and 1G mem). My provider is Cloudvps, a dutch provider.

Quote:

Would I essentially still need to setup a complex cross-compile environment? E.g. no native dockers for emulating other OSs?

That is correct. The advantage of using docker, though, is that you can separate the maintenance of the build environment from the maintenance of the server, and you can track the maintenance in git.I think cross-compilation for windows targets should be possible, but I haven't gotten it working yet myself.