Bug Description

We're currently only able to process about three core files per minute with each retracer. We can fire up more retracers and drop some core files at random when we're approaching a high load, both of which mitigate this problem to varying degrees. However, we should also address the problem that our current retracing code is taking far too long to process each core file:

[4:40pm] jjo: ev: basically afaics we are losing the producer|consumer rate battle
[4:40pm] mthaddon: ev: basically the rabbitmq queues are fairly consistently increasing in size, but the server itself is very lightly loaded - should we be firing up more retracers, or is there some way of making existing retracers do more work?
[4:40pm] jjo: ev: while finfolk doesnt get squeezed for load
[4:41pm] ev: the retracers process in serial, so bringing up more of them would be advisable
[4:41pm] jjo: ev: FTR I had fired up to ~7 retracers in parallel , which hover'd loadavg~=8 -> got ~3 oopsen per min
[4:41pm] jjo: ev: but still I couldn't manage to get more than ~3/min

I followed up with James this afternoon. He doesn't want to drop core dumps from the queue at random as the first solution to this, but he's okay with a proper RED-like solution if we ultimately need one.

He also suggested that a cache of extracted debs would be a good idea. I had tried this over the weekend and ultimately avoided it because I was concerned about packages with Replaces fields. However, he pointed out that's largely an academic exercise. Library packages wont be using replaces on the paths to the libraries or their debugging symbols.

We'll need to ensure that the sandboxes created with --sandbox-dir are per-release:

[2:56pm] pitti: one scenario where this would fail is:
[2:56pm] pitti: - retrace a bug on package foo version 1
[2:56pm] pitti: - retrace the next bug with foo version 2
[2:56pm] pitti: (unpacks v2 over v1)
[2:56pm] pitti: - retrace the next bug with foo version 1
[2:56pm] pitti: -> does not change any file, as both versions are considered unpacked
[2:56pm] pitti: so you'd gdb against v2 when you want v1
[2:56pm] pitti: now, we do not actually support his yet
[2:57pm] pitti: as apport always takes the latest available version of a package
[2:57pm] pitti: but you will at least need to take care to have a per-release sandbox

One way in which we can improve this is by keeping very large caches. Currently, the caches are purged every so often as we run out of disk space. We can reduce the amount of disk space we need by moving to a copy-on-write approach using aufs.

Is this still a problem? Last I spoke to Tom, I vaguely recall him saying he did not think we needed to add an aufs layer to the cache to speed things up (by letting us keep the caches longer) and keep space down.