Search

LUG Community Blogs

One problem with the previous test is that I hit a limit of 20 parallel appliances and mistakenly thought that I’d hit a memory limit. In fact libvirt out of the box limits the number of client connections to 20. You can adjust libvirt’s limit by editing /etc/libvirt/libvirtd.conf, but easier for us is to simply eliminate libvirt from the equation by doing:

export LIBGUESTFS_ATTACH_METHOD=appliance

which causes libguestfs to run qemu directly. In my first test I reached 48 parallel launches before I killed the program (because that’s a lot of parallelism and there seemed no end in sight). Scalability of the libguestfs / qemu combination was excellent again:

I wrote the Perl script below to find out how many libguestfs appliances we can start in parallel. The results are surprising (-ly good):

What’s happening here is that we’re booting up a KVM guest with 500 MB of memory, booting the Linux kernel, booting a minimal userspace, then shutting the whole lot down. And then doing that in parallel with 1, 2, .. 20 threads.

The test fails at 21 threads because there isn’t enough free memory, so each qemu instance is allocating around 660 MB of RAM. This is wrong: It failed because libvirt out of the box limits the maximum number of clients to 20. See next part in this series.

Up to 4 parallel launches, you can clearly see the effect of better utilization of the parallelism of the CPU — the total elapsed time hardly moves, even though we’re doing up to 4 times more work.

My blog has been on Posterous for some years now and it's been awesome. The best thing about it is that you just email a bunch of stuff, with whatever attachments in whatever format are relevant, and they just work.

Sadly they're shutting down following their talent acquisition by Twitter. That's a real shame. Now I have to find an alternative.

Well, that's another FOSDEM over
with. In general this year seemed the same as the last couple of years
but slightly bigger than usual (although it seems that way every year).
The (newish) K building was in full swing with dozens of project stalls
and dev rooms. The usual suspects - virtualisation / cloud,
configuration management and MySQL rooms had nearly as many people
trying to get in to the rooms as they did sitting down.

I think some of the main dev rooms have reached the level of popularity
that forces you to either arrive early, get a seat and not move for the
rest of the day or accept a very high level of probability that you
won't get to see the talks you want. I know a few of us had trouble
cherry picking sessions across tracks - which obviously means we have
excellent taste in topics. I wonder if having the same talks on both
days would make it easier to move around as a visitor - you'd attempt to
catch it the first time and if that fails, come back tomorrow. I realise
however that this puts even more of a burden on speakers that graciously
give their own time in both the preparation and performing of their
talks. It does seem that scaling the rooms is the problem of the day
once again.

I'd like to say a big thank you to all the organisers, speakers and
other attendees for making it another enjoyable couple of days. See you
next year.

To clarify, what is the memory overhead, or how many guests can you cram onto a single host, memory being the typical limiting factor when you virtualize.

This was the question someone asked at work today. I don’t know the answer either, but the small program I wrote (below) aims to find out. If you believe the numbers below from qemu 1.2.2 running on Fedora 18, then the overhead is around 150 MB per qemu process that cannot be shared, plus around 200 MB per host (that is, shared between all qemu processes).

The number to pay attention to is “Anonymous memory” since that is what cannot be shared between guests (except if you have KSM and your guests are such that KSM can be effective).

There are some known shortcomings with my testing methodology that I summarise below. You may be able to see others.

We’re testing a libguestfsappliance. A libguestfs appliance does not have the full range of normal qemu devices that a real guest would have, and so the overhead of a real guest is likely to be higher. The main difference is probably lack of a video device (so no video RAM is evident).

This uses virtio-scsi. Real guests use IDE, virtio-blk, etc which may have quite different characteristics.

This guest has one user network device (ie. SLIRP) which could be quite different from a real network device.

During the test, the guest only runs for a few seconds. A normal, long-running guest would experience qemu memory growth or even memory leaks. You could fix this relatively easily by adding some libguestfs busy-work after the launch.

The guest does not do any significant writes, so during the test qemu won’t be storing any cached or in-flight data blocks.

It only accounts for memory used by qemu in userspace, not memory used by the host kernel on behalf of qemu.

The effectiveness or otherwise of KSM is not tested. It’s likely that KSM depends heavily on your workload, so it wouldn’t be fair to publish any KSM figures.

The script uses /proc/PID/maps but it would be better to use smaps so that we can see how much of the file-backed copy-on-write segments have actually been copied. Currently the script overestimates these by assuming that (eg) all the data pages from a library would be dirtied by qemu.

Another interesting question would be whether qemu is getting better or worse over time.

It's been a while since I've attended a Puppet Camp but considering the
quality of the last one (organised by Patrick Debois) and the fact it was
being held in the lovely city of Ghent again I thought it'd be a wise
investment to scrape together the time off.

The quality of the talks seemed quite high and considering the number of
newer users present the content level was well pitched. A couple of
deeper talks for the more experienced members would have been nice but
we mostly made our own in the open sessions. Facter, writing MCollective
plugins, off-line and bulk catalogue compilation and the murky corners of
our production puppets all came under discussion - in some cases quite
fruitfully.

The wireless was a point of annoyance and amusement (depending on the
person and the time of day). We had 20 users for an audience of ten
times that - the attitudes covered the gamut from "I only need to check
my mail once a day" to "I have my own tethering" and all the way to
"This is my brute force script I run in a loop". You can tell when most
of us lost our access based on the twitter hash tag.

I was a little surprised at the number of Puppet Camps there will be
this year - 27 was the number mentioned. I think a lot of the more
experienced members of the community value the camps and confs as a
chance to catch up with each other and the PuppetLabs people and I'd
hate to see us sticking to our own local camps and losing the cross
pollination of ideas, plans and pains.