Continuous integration testing of Arch User Repository packages

I maintain a couple of ArchLinux user-contributed packages on the Arch User Repository (AUR), and over time I’ve built out a bit of infrastructure around that to make that maintenance easier (and hopefully the results better). The core of it is automated building of packages in Continuous Integration, which catches a number of issues which otherwise would be more difficult.

This write-up will go through the entire packaging process to make it easily reproducible.

Contributing a package

Packages are created by cloning an empty git repository with the desired package name. I do it in a slightly different setup compared to the wiki that’s linked just above, as:

Clone an empty repo to create a new AUR package

Shell

1

git clonessh+git://aur@aur.archlinux.org/<PACKAGENAME>.git

Add your PKGBUILD and any other required files, run mksrcinfo, and git commit, and push… If everything went well, your package is now visible in the AUR search.

Next time that repository is cloned, it will contain the code, and changes (i.e. package updates) can be pushed just as well too.

Keeping track of packages

As more packages are contributed, it is increasingly hard to keep track of them as separate repositories. One way to improve on this, is creating a “meta” repository (or repo), where all the contributed packages are linked as git submodules.

This organization is achieved by creating your meta-repo, and add your package as a submodule:

Adding AUR package as a submodule to a git repo

Shell

1

git submodule add ssh+git://aur@aur.archlinux.org/<PACKAGENAME>.git

Then you’d make package updates in that submodule, and the meta repo would contain all your packages as a collection.

My packages’ meta repo that show this arrangement is on Github at imrehg/aur.

Continuous integration testing

What we can do with this setup now, is to automatically check out, build, analyze, and test (including installation) of the all the packages. I’ve set that up as CircleCI build jobs for each of the packages: each of them built and installed in a clean Arch Linux environment.

The clean Arch Linux environment is provided by a Docker image, that I’ve created for this purpose, archlinux-makepkg-docker. That image builds on an upstream Arch Linux image, and sets a few things up:

updates the image with the latest base build system

creates a “builder” user that can run sudo

installs two packages from scratch that are sometimes needed for working with AUR packages: “package-query” and “yaourt”

The sample CircleCI “config.yml” here is set up to build an AUR package called “my-package”:

it pulls the Arch Linux Docker image mentioned earlier

updates any outdated OS package

checks out meta repo that we are working from

updates the submodule configuration to be able to pull the required submodule without authentication. the “ssh+git://” setup requires the maintainer’s SSH credentials, while switching to “https://” the CI environment is allowed to check the package’s code out (and won’t be able to push back upstream, which is safer)

runs “namcap” on the PKGBUILD to catch any obvious issues

builds and installs the package (including dependencies)

As “my-package” is set up above, it does not have any line specific in to that package in the build steps. The specifics are set up using CircleCI variables (CIRCLE_JOB) and YAMLMerge Key Language-Independent Types (the “foo: &foo” and “<< : *foo” section). Thus if there’s “another-package”, it’s easy to clone the “my-package” section as it is, naming that “another-package”, and adding a new build job to the end of the file called “another-package”. With this “templating” when the build steps need to be modified, they can be updated in the header, and all the packages will pick that up.

Workflows are also useful, as jobs can be made dependent on each other, if they are related, such as my “gnushogi” and “xshogi” packages, or likely any AUR package that requires other AUR packages that need to be built.

Workflow with dependency

YAML

1

2

3

4

5

6

7

8

9

10

...

workflows:

version: 2

build:

jobs:

<otherjobs>

-gnushogi

- xshogi:

requires:

-gnushogi

This would result in a dependency in the jobs as:

Jobs in the CircleCI workflow

The workflows also allow for jobs to give files to each other. E.g. as above “xshogi” depends on “gnushogi” to be installed, I could build all the required dependencies again in “xshogi”, but it was already built, I could just pass on the created package from the earlier job to the next, using CircleCI workspaces.

Sample file passing between jobs with workspaces

YAML

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

gnushogi:

<<: *defaults

steps:

- run:

<<: *updatepackage

-checkout

- run:

<<: *gitupdate

- run:

<<: *pkgbuildtest

- run:

<<: *buildtest

- persist_to_workspace:

root: gnushogi

paths: gnushogi-*.pkg.tar.xz

xshogi:

<<: *defaults

steps:

<<: *defaults

steps:

- run:

<<: *updatepackage

-checkout

- run:

<<: *gitupdate

- run:

<<: *pkgbuildtest

- attach_workspace:

at: /tmp/workspace

- run:

name: Installing gnushogi

command: sudo pacman -U --noconfirm /tmp/workspace/gnushogi*.pkg.*

- run:

<<: *buildtest

The meta repo is now ready to go with such “.circleci/config.yml”, and on each push, it will build all the packages defined in the job list. You can check how the results look for my AUR packages in CircleCI’s build job view (one entry by build job, ie. package-per-push) or workflow view (one entry per push, aggregating all jobs).

Last build workflows

One of the advantages of this setup, is that if a build fails on any of the package (e.g. a source file is no longer available) it’s easy to see, and I can catch a number of out-of-date packages sooner than someone reports it on AUR.

That applet just uses the Date & Time and Webhooks recipes. The webhook points to the Trigger URL provided by the “Build Settings / Build Triggers” section on Docker Hub for the image, and it’s a POST request with payload of:

Docker Hub build trigger payload

JavaScript

1

{"docker_tag":"latest"}

Docker HUB Build Settings / Build Triggers settings

Keeping the image fresh like this shortens the build time when running the jobs on CircleCI (fewer packages need to be updated), which especially important as free users have limited CPU time available each month.

Not many packages which use other AUR packages, which likely need more setup here.

Update workflow

As an aside, the process to update any given package with this setup as follows:

Update the “PKGBUILD” for the package, quite often it’s just the version number

Update the checksums easily with “updpkgsums” (part of “pacman” so it should be always available)

Build the package

If everything goes well, update the required “.SRCINFO” with “mksrcinfo” (part of “pkgbuild-introspection”)

git add, commit (signed if you can:), and push to AUR

Clean up the package directory (“git clean -d -f && rm -rf src”)

Going back up in the folder hierarchy to the meta repo git add and commit the changes to the submodules

Push to github, and enjoy the build!

Future

Many things can be improved on this setup (one day), here are some ideas

It should be possible to publish the build artifacts to somewhere (say S3) and set it up as a custom Arch Linux package repository, thus can be reused without everyone needing to build from scratch every time.

If that publishing would happen, I’m guessing it would be good to also sign the built packages, which might be a bit trickier to set up safely, but would make package distribution nicer and more robust.

In my list of packages there are not that many that depend on other AUR packages. Other packages with more AUR dependencies might need even more custom setup than shown above, besides the templated sections, to make them speedy and logical.

In the package testing steps, probably should run “namcap” on the finished package too, to catch other issues (e.g. dependencies required but not included).

What’s your experience with maintaining AUR packages, or with CircleCI? Have any feedback on how to make this above even more useful?

Join the conversation

Using your example I created a configuration which only builds the PKGBUILDs I maintain from the AUR with the yay AUR helper. This way I at least know if something is wrong with an uploaded build. It is easy for me to maintain the circleci configuration because it doesn’t have any git submodules. https://github.com/dmp1ce/PKGBUILD-circleci-testing

Great job! You are maintaining quite a few packages! The no-submodule setup is cool too, indeed it’s optional, though I wonder when the tests are run, since package update doesn’t result in new code thus testing. Do you manually trigger tests or use scheduled builds? Actually, based this idea, just set up a nightly schedule for my AUR builds, that should be a good improvement to catch broken packages earlier. Thanks for the feedback, and let me know if you have any further ideas! :D