Knowledge

Adventures in Debian packaging

About a year ago, work started on packaging SCAPE tools. Jpylyzer was the first SCAPE tool that was turned into a Debian package. Some time later, the OPF set up a couple of machine images at Amazon Web Services, which can be used to create packages repeatedly using a virtual machine. Even though I’ve used the Amazon service a couple of times myself, I really know next to nothing about Debian packages, and it’s safe to say that the underlying build process has been more or less a complete mystery to me.

To get a better understanding of the process for building Debian packages, I had a try at packaging jpylyzer on my local machine (which runs on Linux Mint 14). Some time ago Dave Tarrant and Rui Castro wrote a nice step-by-step guide on building Debian packages on the OPF Wiki, so I tried to follow the instructions there. While working on this, I made some notes, mainly to remind myself of what I was doing. Then I realised that some of this might be useful to others as well, so I decided to turn it into a blog post.

Objectives

The objectives of this exercise were:

to get more more familiar with the packaging process myself;

to provide some input on how useful the guide on the OPF Wiki is from the perspective of someone who is largely ignorant of the packaging procedure;

to identify any problems in jpylyzer‘s packaging procedure.

I did two experiments: first, I did a very limited test where I tried to create a template directory structure using debhelper, which would be the first step when starting from scratch. Since for jpylyzer all the files in the debian directory already exist, I then moved on to building jpylyzer using the existing files.

Test 1: creating the directory structure from scratch

For this, I first installed all the required packages listed in the Pre-Requisites section of the guide using:

Subsequently I followed the instructions in the Getting Started section. For this I simply created an empty directory:

mkdir debtest_1.0.0

And then:

cd debtest_1.0.0

Then I ran dh_make:

dh_make

This resulted in an error message, telling me that the package name and its version number should be separated by a dash (‘-‘) instead of an underscore (‘_’), or, alternatively, that the -p flag should be used. So I changed the directory name:

mv debtest_1.0.0 debtest-1.0.0

Re-running dh_make, it now accepted the directory name, but it complained about a missing tarball (which I purposefully didn’t make in this test). However, as dh_make offered the suggestion to use the –createorig option (which creates a tarball) I tried this:

dh_make --createorig

This resulted in the creation of a debian directory with file templates, and an (empty) tarball debtest_1.0.0.orig.tar.gz which was created in the parent (debtest) directory.

So, apart from the dash/underscore mix-up this is all pretty straightforward.

From there I tried to build jpylyzer directly, using the command given in the guide’sBuilding your package section:

dpkg-buildpackage -tc

Missing changelog

The above command resulted in an error message about a missing changelog file in the debian folder. The changelog section in the OPF guide does mention an OPF-hosted GitHub 2 Changelog service, which is supposed to be callable from the rules file. But I don’t see any reference to it in jpylyzer’s rules file, so I don’t really know how this is supposed to work! To to keep going I simply grabbed the default changelog that was created by debhelper in an earlier experiment. After this I ran the command again.

The pymakespec and pyinstaller commands above are most likely shell scripts that launch the Makespec.py and pyinstaller.py scripts that are both part of PyInstaller (these are used for building an executable from the source code). However, neither the shell scripts nor any references to them are included in jpylyzer‘s repository (my best guess is that they exist only on a specific machine instance – perhaps the Amazon virtual machines?), so the makefile simply won’t work.

I was able to fix this by changing the references to the shell scripts to this (using PyInstaller 1.5):

Note here that PyInstaller has no default installation location, and the file paths will vary from machine to machine!

After making these changes I was able to run dpkg-buildpackage without any problems:

dpkg-buildpackage -tc

Result: the following files were created in the repo’s parent directory:

jpylyzer_1.9.0_amd64.changes

jpylyzer_1.9.0_amd64.deb

jpylyzer_1.9.0.dsc

jpylyzer_1.9.0.tar.gz

Tarball schmarball

One thing that confused me at first: the Getting Started section in the OPF guide mentions the need for building a native package before starting the Debian packaging:

If you have got here and you don’t have any already packaged code (a tar ball with makefile etc) then you will need to build a native package.

So, I initially thought I would need to create a tarball of my repo first. As it turns out this is not the case: the tarball is created automatically once you run dpkg-buildpackage. So this is one thing less to worry about!

I still need to give these errors and warnings an in-depth look. At least one error is related to the bogus changelog file I used. Some others (e.g. unstripped-binary-or-object) appear to be related to the build process of the binaries.

Conclusions

Using the Building Your Debian Package guide on the OPF Wiki I was able to create a rudimentary skeleton structure for Debian packaging. I was also able to build a Debian package for jpylyzer. The exercise revealed some problems with the Debian setup for jpylyzer. The most important ones are:

It’s unclear how jpylyzer‘s changelog file is supposed to be generated. Perhaps there’s a dependency on some external service (the OPF Github 2 Changelog service?), but I cannot find any documentation on how to make this work!

The makefile calls PyInstaller in a non-standard an undocumented way. This is easy to fix locally if you are familiar with PyInstaller, but not so otherwise. Also, the interfaces of versions 1.5 and 2 of PyInstaller are different, and depending of what version you are running this may require additional changes to the makefile.

Even though I was able to build a Debian package for jpylyzer, it still ended up with some lintian errors.

I also came across a few minor errors in the OPF guide. I left a short comment on this here (scroll to bottom). Overall, I found the guide really helpful, and it provides an accessible and relatively painless introduction to the packaging process.

Share this page

Latest news

The iPres Working Group invites you to provide feedback on the Future of iPres, the international digital preservation conference. We look forward to hearing from you! As you may know, in September 2018 the iPres Steering Committee approved the convening of the iPres Working Group and at iPres 2019, we will share the outcome of our […]

Upcoming Event

Born-digital material introduces new challenges around trust and authenticity, The ARCHANGEL Project is investigating the use of blockchain to verify that documents stored in digital archives have not been altered or modified. This webinar will introduce blockchain technology, explain the project and give a demo of the software we have developed. Speakers Mark Bell, The […]