VPS Comparison between Slicehost and Prgmr

For the last year I've been using a 256MB
VPS
from Slicehost to host a low
traffic site running Django. I've also used the slice as a development
platform thanks to screen and vim.

Two weeks ago I came across a mention of another Xen based VPS provider
called Prgmr. What intrigued me was that I could get a 1GB instance
for the $20/month I was paying Slicehost for a 256MB slice. 4 times the
memory, more than double disk capacity, and 60% more bandwidth. Was this too
good to be true?

I decided to rent a 256MB instance from Prgmr (sets you back $8/month) and
compare it against my trusty old slice with the same memory configuration.

As we can see they are both Running Debian GNU/Linux 5.0. As of this writing
Slicehost gives you a 2.6.24 kernel while Prgmr provides a more recent
2.6.26 kernel. Both systems use the x86_64 architecture. Slicehost uses
a dual core Opteron with 1MB L2 cache for each core while Prgmr uses a
quad core Opteron with half the L2 cache (512KB) for each core. This
difference is uninteresting as our individual performance is dependant
on how many of these CPUs the system has in total compared to how many
VPS instances the system hosts. The output from /proc/cpuinfo only tells us
about the amount of cores available to our VPS instance, not how many there
are in total.

In reality one VPS instance does not get access to these cores
directly. One are given one or more virtual CPUs (VCPUs). Slicehost gives
us 4 VCPUs while Prgmr gives us 1 VCPU. How does this relate to
performance? Truly 4 is better than 1? According to Luke Crawford
(from Prgmr):

Giving you more VCPUs increases your peak performance (that is, you get
more CPU when nobody else wants it) but it decreases your worst-case
performance (that is, when everyone is fighting over the CPUs, the
more VCPUs you have, the more context switches you have) and so I've
chosen better worst-case performance here.

This means that theoretically Prgmr should give you more stable performance
and Slicehost should be more performant, except for when
your VPS neighbors get really busy.

Django test suite

Since most of my work on my VPS involves working with Django, I decided to
benchmark the two providers by timing their execution of the Django test
suite.

Unneeded services were disabled and the instances was rebooted before
starting the tests.
All tests were run against revision 10108 of the Django trunk, using
the sqlite3 database engine. The following script was used for running
the tests:

The test suite is executed 10 times with pauses of 5 minutes in between,
then we wait 12 hours before running the tests 10 more times.
Note that this approach is highly unscientific. One can expect increased
performance when none of the neighboring instances are doing any work --
since these VPS providers use credit based scheduling.
A better approach would therefore be to run the tests once per hour for a
week. Sadly I can't afford to strip down my VPS for a week of testing.
You should therefore take these results with a grain of salt.

Lets first look at how the test suite run times varies over time:

We see that the 10 first test runs if fairly equal for the two providers.
Prgmr is generally a bit slower and tend to vary more in execution time than
Slicehost. But after a 12 hour sleep something strange happens. Prgmr
becomes more stable and significantly faster than Slicehost. My take is
that we suddenly got the CPU all to our self on Prgmr. These results
indicate that we should perform these benchmarks over a longer time period.

With the skewed metrics in mind, lets take a look at the mean execution
time for these two services:

Prgmr "wins" thanks to its sprint after the 12 hour pause. The differences
is not extraordinary large though. But what if we adjust the run times
according to price? We multiply the Prgmr runtimes with 8/20 to find the
theoretical performance we could get if we spent as much money as we
did on the Slicehost rental:

With price in mind, Prgmr gives us almost 3 times more performance than
Slicehost. Note that this is just a theoretical exercise.
The 1GB Prgmr offering should be benchmarked against the 256MB
Slicehost offering for finding what you really get for your $20.

Network

Average ping time was measured using Just Ping
to find the network latency from several locations around the globe:

The results shows that Slicehost has the lowest latency from the eastern
parts of the US and Europe. Prgmr on the other hand has lower latency from
the western parts of the US, Asia, and Australia. This is largely due to
the geographical locations of the two providers.
Slicehost is placed in Saint Louis, Missouri and Prgmr is located in
San Jose, California.

Stability

I've been a Slicehost customer for some time and have never had to restart my
slice or had it gone down due to system failure. Before updating to Debian 5.0
and changing kernels it had an uptime of over 330 days.

As I'm a brand new Prgrm customer I don't have any data to compare the two
services.

Community

Slicehost has an active community with their forums, IRC, and Wiki. In
addition they have several articles describing how to setup various types
of systems.

Prgmr on the other hand has none of these community building platforms
(except for a fairly empty Wiki). If you want to use Prgmr you are
pretty much on your own. This is highlighted on their web page with
the phrase:

We don't assume you are stupid.

Support

Prgmr's tagline is also apparent in their support software. While Slicehost
provides a flashed out control panel (SliceManager) for monitoring and
administering your
VPS, Prgmr only provides console access through a SSH gateway. Much of
what's automated over at Slicehost (like provisioning a new instance) is
handled manually at Prgmr.

I've only had one encounter with the Slicehost support team. Upgrading a
kernel took about one hour, from when I sent a support request to
the VPS was rebooted with the new kernel. The only encounter with the
support staff at Prgmr I've had was when I ordered my instance. That
process took 3 hours, from placing the order to being able to log in.

Conclusion

We've seen that it's hard to benchmark VPS instances as factors out
of our control largely influences the results. All in all I think
the two offerings is on par performance wise, but more testing
needs to be done to see how the load of other customers influences
our performance.

I rarely use the SliceManager, and so far Prgmr's console have served
me well. Slicehost's network latency is a bit lower for me here in
Norway, but not large enough to exclude Prgmr as a worthy contender.

My only concern is the stability of Prgmr, as my Slicehost experience
has been stellar. I think I'll keep both VPSs for the time being
and switch to using solely Prgmr if my long term experience proves
to be as good as what I've seen so far.