And some more kernel packaging bits - I finally found a site with an explanation of "the debian way" of packaging kernels - http://users.wowway.com/~zlinuxman/Kernel.htm - which seems up to date. After a first pass through this, I think my next step is getting everything going using make-kpkg and looking at putting a boot config file modification into /etc/kernel/postinst.d. If I use latest firmware, and with the bootc patch applied to the kernel, this can just be the normal vmlinuz kernel.

Just under 600 packages left to build before parity is reached with Debian armhf. The counts are going a little slower because we've ran into some difficulties with updates from Debian Wheezy that temporarily gave the autobuild systems some fits. Also, we seem to be getting a lot of updates from Debian Wheezy which the autobuilders then have to rebuild to bring the Raspbian packages up to date. This rebuilding cuts into the resources available to compile new packages. I suspect the updates are more frequent as Debian is looking to freeze Wheezy into the next stable release in late June or July. Many developers are trying to get their packages into the Debian Wheezy before the freeze occurs.

Something that occurred today is that the build queue has now finally been depleted. All the packages that can be built, have been. However, there are about 160 packages where the builds failed and these packages in turn are keeping other packages from building because of dependencies. So to close the gap between Debian Wheezy armhf and and Raspbian armhf we'll need to carefully start prioritizing and examining the issues that are preventing these remaining packages from building.

Also, the build systems will continue to be busy as we update each day the new packages that are being updated in Debian Wheezy. At this time, usually 60 to 90 source packages are updated each day which then need to be rebuilt in Raspbian. Once Debian Wheezy freezes, this daily update of packages should be much less.

Finally, there are issues with the edos-debcheck application that autobuilders use to figure out the dependencies between what binary packages are available for installation and the source packages that are dependent upon those binary packages for building purposes. Plugwash has been after me to look into the issue, but I haven't had much of an opportunity the last few days. Hopefully once that issue is fixed dozens of more unbuilt packages will be built.

Now that Debian main is almost completed porting across will you be turning your attention to porting the Debian non-free repositories? If some of the build machines are available then this would be a very good idea. There are a lot of packages that would be really useful to have available on the Pi and provide a much better user experience going forward.

john.mills wrote:Now that Debian main is almost completed porting across will you be turning your attention to porting the Debian non-free repositories? If some of the build machines are available then this would be a very good idea. There are a lot of packages that would be really useful to have available on the Pi and provide a much better user experience going forward.

We'll be looking into this soon. I'll need plugwash's help as he's much more familiar with the ins & outs of the Debian repository and the configuration changes on our side to accommodate multiple repositories in our infrastructure. At a minimum, we'll get the 'all' architecture packages which include a lot of USB device firmware which will make many USB devices usable under Raspbian (ie. USB wifi dongles and such). How soon we'll be able to devote the build systems to the actual architecture dependent packages is more of an open question. I do have additional hardware on order, but it won't get here until the last week of June at the earliest.

One of the tricky things is we have to make the configuration changes on an infrastructure that now exceeds 80GB of compiled armv6 code. Although it's backed up, I still want to tread very lightly and make sure we don't break things that could take days, or longer, to fix.

Actually, we use Freescale iMX53 Quick Start Boards to build Raspbian. These are 1GHz, 1GB, ARM devices with SATA interfaces. The largest Debian packages barely build on these systems. Anything less and Raspbian would be largely impossible to build.

I have three additional with SATA drives on order to supplement the five already used to build Raspbian. Unfortunately, these devices seem to be in short supply.

To be honest, I'd had envisaged a more "conventional" cross-compilation environment - by which I mean a number of x86 boxes cranking out cross-compiled arm binaries (which is a part of what I do at work some days...) rather than using these, which are almost-but-not-quite native builds, I guess.

I imagine that makes a lot of things easier for configure, though must leave you with occasional really subtle host/target differences to resolve.

I know there are a lot of packages out in the wild that will not build at all, or at least not without a lot of manual intervention, in my "traditional" cross-compilation environment, so this way seems like a win in that regard!

Must be some costs associated with maintaining the build-farm though - you ok for funding on this?
--
Ian

CPU wise and price wise they are comparable to the beagleXM but unlike the beagleXM they have 1GB of ram, SATA and native ethernet (the ethernet on the beagleXM is USB based).

which are almost-but-not-quite native builds, I guess.

They are native builds. Everything inside the build chroot on the autobuilers is from raspbian.

I imagine that makes a lot of things easier for configure, though must leave you with occasional really subtle host/target differences to resolve.

It's not generally a huge issue. Occasionally a package makes inappropriate assumptions but that can happen on any architecture. Having a build machines support more features than the port requires is not exactly unusual.

Must be some costs associated with maintaining the build-farm though -

There are obviously capital costs in building and expanding the farm but on an ongoing basis it doesn't need much, just a bit of power (indeed I bet that the repo host probablly draws more than all the autobuilders put together :/)

Currently the biggest ongoing cost by far is paying for the public repo server. To avoid really nasty surprises we needed a host with unlimited traffic and that is relatively expensive.

Please do... I am sure that there are many members on here (including myself) who would be more than willing to make a donation to this project... especially if it would help to move things along at all!

The Second one is about the build system. Is it possible to benefit from emulation on x86 ?
I was used to build arm binaries using scratchbox and I just find this (http://www.stgraber.org/2012/02/03/ever ... u-precise/). This was just for simple binaries not the whole system.

Replacing gcc with a linaro version would be a risky move and not one I plan to make. Hopefully in time improvements from gcc will filter up to upstream gcc and then down to the debian gcc packages. Anything recent will probablly have to wait for wheezy+1 though.

Optimisation levels are generally at the packagers discretion, most debian packages use -O2 but if you beleive there are cases where -O3 is appropriate and isn't being used please take that up with the debian maintainers of said package in the first instance. In extreme cases where a significant gain can be found from -O3 and the debian maintainer is uncooperative we may do this ourselves but I'd rather avoid it if possible.

The Second one is about the build system. Is it possible to benefit from emulation on x86 ?
I was used to build arm binaries using scratchbox and I just find this (http://www.stgraber.org/2012/02/03/ever ... u-precise/). This was just for simple binaries not the whole system.

I set up a vm with qemu-user chroots and tried building apt in it, the result was that the build took slightly longer than it did on an IMX board (admittedly the host was only a pentium G).

The main case where emulation would seem useful is building stuff that needs massive ammounts of ram. Currently some packages cause our autobuilers to swap for days.

I have not investigated cross compilers yet, it is probablly possible to build them from our gcc source packages but all the debian documentation on doing so seems to be out of date and so I don't know how many problems you would run into. Furthermore debian packages are not required to support cross-compilng so this is unlikely to be an option for autobuilding.

Time for another package update. The current binary package counts now stand at:

Debian armhf 34816
Raspbian armhf 34737

As of June 11th, Raspbian is hovering at just under 80 binary packages under Debian Wheezy. Out of 34,816 packages, a gap of 80 packages is not bad at all. Quite frankly, I'm surprised we have been able to get this close. I'll have to check to make sure there are not orphaned binary package in the Raspbian repository that may be artificially increasing the Raspbian package count.

I'm not certain we'll narrow this gap by much as many of these packages have build problems that are unlikely to be resolved, unless someone really needs one of the packages in question to be built and has the time to sort through what the issues might be.

An up to date list of the packages that aren't building in Raspbian can be found here:

mpthompson wrote:
As of June 11th, Raspbian is hovering at just under 80 binary packages under Debian Wheezy. Out of 34,816 packages, a gap of 80 packages is not bad at all. Quite frankly, I'm surprised we have been able to get this close. I'll have to check to make sure there are not orphaned binary package in the Raspbian repository that may be artificially increasing the Raspbian package count.

Note that due to the way debian testing works it is likely that we have some stuff that debian wheezy armhf does not. In particular I beleive this applies to haskell packages since there has been a haskell transition going on for a LONG time.

Damn ! It's 1:08 AM in the morning and it took me more than 2 and half hours to read through all the 23 pages of this post. Big ups to plugwash and mpthompson for such an amazing contribution to the RPi community. I've sent $25 your way as a small token of my gratitude.

Zinahe wrote:Damn ! It's 1:08 AM in the morning and it took me more than 2 and half hours to read through all the 23 pages of this post. Big ups to plugwash and mpthompson for such an amazing contribution to the RPi community. I've sent $25 your way as a small token of my gratitude.

Keep up the good work.

Thank you very much for the donation. We'll do our best to keep things moving forward.

Today I found myself reading the first few pages of this thread so I could understand how I did some of the things early on. Amazing what is forgotten in just a few months time.