Last week, I moved my blog to Amazon EC2 in order to have more control of it (blogged here). I chose a Micro instance running Ubuntu Linux because Micro is the least expensive (2 cents per hour) and from my previous experience hosting various websites and services on Micro instances, I felt it was more than adequate to support my blog.

To my surprise, the results were disappointing. I noticed that when accessing my blog, it would sometimes pause for several seconds before serving the requested page. I spent several hours trying to figure out what was going on. When I looked in the Apache logs, I realized that the CPU spiked when my blog was crawled by one of several search engine bots. When a full crawl wasn’t happening, my blog was lightning fast but when one of the search engine bots crawled my site, my blog was nearly unresponsive until the bot finished. Obviously this was not acceptable.

While running top, I quickly discovered that when my site was being crawled, the CPU would spike, but the spike was not the typical user or system CPU. It was showing up as 97%st “CPU steal time”. Here’s a screenshot:

During the first 5 to 10 seconds of the bot crawling my site, everything was super fast, but then things would come to a near halt.

Steal time is a metric that only has meaning in a virtualized computing environment. It represents the amount of CPU that is taken away from a virtual machine to serve other purposes.

Amazon’s website describes Micro instances as follows: “Instances of this family provide a small amount of consistent CPU resources and allow you to burst CPU capacity when additional cycles are available. They are well suited for lower throughput applications and web sites that consume significant compute cycles periodically“.

What I’ve discovered is that when you tax the CPU more than a few seconds, you get throttled back almost to a standstill. I put together a brief video where I demonstrate this happening. In this video, I run a simple script that makes repeated calls to sysbench to tax the CPU for 2 seconds. After it runs for a few times, you’ll see an immediate and severe reduction in performance and you’ll see the steal time go to over 97% leaving very little left to run my job.

Advance to the 1:20 mark if you want to skip to the benchmark

I still think Micro instances are awesome, especially considering the cost, but I’ve now learned that they are not right for all applications. A simple 10-page website with only a few thousand hits a day will do just fine since it will only need CPU in short bursts. These burst are executed very fast, so it is a good fit for this type of application. However, if your application needs sustained good CPU performance, you’ll need to upgrade to a Small instance or higher.

The nice side effect of this whole experience is that my blog is very optimized. Pre-cached pages, minified HTML/JS/CSS and a well tuned DB makes a speedy blog!

Editorial: I think the extreme CPU throttling really limits the usefulness of Micro instances. The next step up in the Amazon EC2 world is a small instance but the price jump to a small instance is substantial. Originally I saw micro instances as a “foot in the door” to Amazon EC2. It was a way for someone to try the model and then grow into larger instances as the demand grows. It seems to me that having a model that basically shuts down the CPU after a few seconds of heavy use gives a lot of new users a bad experience, especially considering that Amazon does not clearly explain the model. I think Amazon needs to either modify the micro instance to not throttle CPU as severely, or offer a “tiny” instance that provides a good entry into the EC2 world.

SHAMELESS PLUG – 10/27/2011

As I continued to work with Amazon EC2, S3, Beanstalk, RDS, etc., I quickly saw a need for a tool to help visualize the various AWS assets. I teamed up with James Ward and Jon Rose and we’ve created, “Greg’s Toolkit” for desktop, iPad and Android tablets. I encourage you to check it out at http://www.gregstoolkit.com.

I played for a (short) while with a windows server micro instance. It took hours to install sql server express – jammed at 100% cpu all the time – and eventually I gave up. I didn’t realize it was throttled so badly!

http://gregsramblings.com Greg Wilson

I had the same experience while installing ColdFusion – got about 50% through the install…then hung for a few minutes. At the time, I didn’t know what was going on…but now I do

Greg

johans

Greg – have you looked at Rackspace CloudServer? How do you think the smallest (256MB) would compare with EC2 micro instance? Its about $11/month and my understanding is you do not have the CPU capping as with EC2 – fee comparisons here:

Be interested in your thoughts as I’m a novice with these cloud hosting options.

http://gregsramblings.com Greg Wilson

Hey Johans – I’ve been playing with Rackspace CloudServer the past couple of days. The CPU for the low-end instance is far more consistent from my I can see so far. I’m going to be blogging soon comparing RackSpace CloudServer, EC2, Linode and others.

Thanks!

Greg

johans

Look forward to your post.

I signed up for EC2 but after 4 days, despite verification via phone/PIN I am still stuck in a loop of Account Pending -> Click here to verify -> you are already signed up click here to use -> Account Pending.

So I went looking and that’s when I saw Rackspace CloudServers.

http://www.kadimi.com/ Nabil

2 seconds it a lot of time for a WP blog…

Q. How does a Micro instance compare in compute power to a Standard Small instance?
At steady state, Micro instances receive a fraction of the compute resources that Small instances do. Therefore, if your application has compute-intensive or steady state needs we recommend using a Small instance (or larger, depending on your needs). However, Micro instances can periodically burst up to 2 ECUs (for short periods of time). This is double the number of ECUs available from a Standard Small instance. Therefore, if you have a relatively low throughput application or web site with an occasional need to consume significant compute cycles, we recommend using Micro instances.

http://gregsramblings.com Greg Wilson

Yes, small instances are slower than the burst speed of a micro instance, but they handle a sustained load far better.

The burst speed is quite amazing – I just with that when it throttled down, it would leave enough CPU left over to serve up basic HTML pages from Apache. Throttling is severe.

Thanks!

Greg

http://infiniteshelter.com piousbox

C’mon you cannot actually recommend micro instances other than to a beginner who has no idea about cloud computing. “if you have… an occasional need to consume significant compute cycles” – a micro instance will literally come to a halt if you need to compute anything.

Just to let others know – my EC2 account setup was eventually completed. Seems it can take a few days – my expectation was it was instant as I already had an active account for S3.

I setup/installed Node – it took about 20 minutes to compile due to the limited CPU. So one has to be patient.

Your instructions/video really helped a lot. A few things I came across:

After setting up the Elastic IP I needed to use the IP in place of the public DNS when connecting.

When connecting via ssh I also had a permission problem with my key pair file – “WARNING: UNPROTECTED PRIVATE KEY FILE!” I solved that by setting permission using chmod 700 keyFileName.pem

@Nabil – thanks that information is interesting.

http://gregsramblings.com Greg Wilson

Johans – that’s weird that it took so long to get your account active. It took about 6 hours on mine.

One nice thing about EC2 is that you can change the instance type of a given instance. You could start it up as a small instance to get everything installed, and then downsize it to a micro instance. I’ve used micro instances for development and then upsized to small or larger when it’s production time. The flexibility is great.

I had the same issues on the pem file permissions.

Thanks!

Greg

http://twitter.com/magoon andy magoon

I think there is a misconception that a “small blog” running on WordPress can be easily hosted on the smallest virtual server offerings. In my experience WordPress demands quite a lot more.

http://gregsramblings.com Greg Wilson

I’ve seen the same…however… with the wp super cache, you can basically cache your entire site as static HTML pages…so when people access pages on my blog (including this page), they are actually hitting static content…so no PHP or MySQL is being used other than logging.

I think WP should ship with caching built in…seems like an obvious requirements because without caching, it is a bit of a resource consumer

Greg

http://www.sentientmatter.com Nick Daugherty

Agreed. Just trying to configure a WordPress site (that has NO traffic) very frequently causes throttling on my micro instance. Sigh. Guess I’ll have to investigate Rackspace.

http://kasperbergholt.org Kasper Bergholt

Hi Greg!

Very interesting. This was new information to me — and makes me reconsider hosting even small WordPress projects on micro instances. I’m looking forward to your comparative tests!

Thanks,

Kasper

http://kasperbergholt.org Kasper Bergholt

Btw — what’s the blog running on now?

http://gregsramblings.com Greg Wilson

It’s currently on a c1.medium (high CPU) but it is WAY WAY overkill for my blog. I’m consolidating some other sites on this new instance to save money across the board.

The c1.medium is very fast. If I wasn’t combining with other stuff, I’d go with m1.small. If you pay the reserve price, it gets affordable.

The awesome thing is that you can change instance types with a simple command line command.

I’m also wanting to try two micro instances with a load balancer for fun.

Greg

http://bradleyproctor.com Brad Proctor

Thanks, glad I read this. I’m currently on a 256mb rackspace cloud instance. I have 8 low traffic sites, two of them wordpress. I’ve done quite a bit to reduce the amount of memory used, but I’m often bumping the limit so I was considering moving to a Amazon EC2 Micro instance since for only a little more per month I would get more than double the memory, but from what you said, it doesn’t sound like a good move. I’ll think I’ll stick with Rackspace for now.

http://gregsramblings.com Greg Wilson

It really depends on whether a search bot scan will trigger the throttling. If all request are under 4 or 5 seconds, it will be dramatically faster than rackspace because the “burst cpu” is awesome. However, if you hit throttling, it will go to near zero.

What’s taking so much memory?

will

Great research. This will save people many hours of confusion.

http://faisal.altlimit.com Lasiaf

Thanks for the article. I was actually thinking on how cheap micro instance was specially with 3yr plans. I’m currently using fibervolt.com
They are the cheapest vps I found. Its been reliable so far.

http://www.tech-flex.com Poul Wittig

Hi Greg:

You can overcome Amazon’s CPU throttling by throttling your applications on your own, with a tool such as cpulimit (aptitude install cpulimit). That way you can prevent processes from ever reaching the “cap” in which Amazon’s throttling kicks in, resulting in a severely incapacitated server.

At least that’s what I did for my Micro-instance, and it has proven to work out great

Kind regards
Poul Wittig

http://gregsramblings.com Greg Wilson

Hey Poul… This is a great idea. I’ll try it and do some testing. Maybe a good “formula” can be found to make the micro instances more useful.

Thanks!

Greg

http://www.tech-flex.com Poul Wittig

Although keep in mind it is more of a hack and is not a fail-free solution, since you obviously can’t limit every single process on your server (in my case it was only teamspeak3, mysql, & apache processes I chose to limit).

As an example, I came across my server still weekly entering that incapacitated state, despite having limited the processes which I thought were causing the Amazon CPU throttle. It turns out a default Ubuntu has a weekly cron by the name man-db, which will would (afaik) re-index available packages in the sources which spikes the CPU for a long amount of time. This job caused my server to enter that “incapacitated” state, and I’ve had to force a reboot on the instance since even after 10 hours it didn’t recover. My solution was to “sudo aptitude purge apt-xapian-index”, and have since then not experienced any Amazon-throttling occurring.

http://www.tech-flex.com Poul Wittig

Oh! Forgot to mention CloudFlare! I started using it recently, and it has saved my server many GB’s of traffic each month, by filtering out botnet’s and other bad stuff, but also by providing some additional cache (even CDN’ing it to speed my blog up for visitors far away from US-east) . A nice addition to remove some strain from your server (and it’s actually free).

Haha not affiliated with them btw, just really impresssed so far

http://gregsramblings.com Greg Wilson

CloudFlare looks very interesting. I do block some things with .htaccess rules (mod rewrite) but there’s looks more comprehensive.

Alex Athanasopoulos

I ran into this problem when I typed “sudo apt-get install ec2-api-tools” on an ubuntu micro instance. Ironic, isn’t it? I wanted to try out the EC2 tools but the installer hangs on the micro instance. I ended up killing (stopping) the instance twice. In order to go past the install, I converted the instance to Small, did the install, and then converted back to Micro It was really fast. You can change the instance type from the AWS console and simply restart the instance.

http://gregsramblings.com Greg Wilson

Yeah – good plan. I had to do a full ColdFusion install on a micro instance so I converted it to a c1.medium (high cpu), did the install then went back to micro. Worked great and maybe cost me 3 cents

Greg

http://www.patricksalami.com Patrick Salami

Good idea about the CPU throttling. You could also try playing with the settings in your apache2.conf file. For example, you could try limiting the number of client threads that are spawned. That would reduce the number of concurrent clients you can serve but it may also reduce your overall CPU consumption during peaks. In addition, you could also try hosting at least your static files (.css, .js, images, video, audio, static .html files, etc) in S3. There is no hourly charge for S3; you only pay for the storage and bandwidth costs, which should both be negligible. You could even set up CloudFront to serve the static content even faster.

Ben

Greg, thanks a million for sharing this. I found your video very easy to understand. You probably saved me 37.8 hours of work!

http://www.merchantservices.cc Jon Perry

Greg, this was a great article for me. I had been considering moving one of my sites to EC2, but now I’ll try it with a throw away domain name that I can measure and monitor it with.

Again, thank you so much for the post.

By the way, where are you hosting this site? Still on EC2? It’s pretty fast.

Jon

Hayden

Ahh! Maybe this is why when I use apachebench it slows down A LOT after the first few seconds of requests. Did you upgrade to small instance? or found a work around?

Hayden

Was listening to video while I posted. lol Just noticed at the end of vid you said you were able to upgrade to small instance with just one command! How is this done? Cant find blog post. Thanks

Adam Burr

Hi Greg,

Two thoughts from me…

1. An alternative solution: The file called robots.txt on your website allows you to throttle back search engines. You can add commands to instruct search engines to pause between requests and not hit your site too hard. Most search engines will obey these instructions.

2. Do you know if you can upgrade a reserved micro instance to small? What happens about the fact that you have paid an advance reserved fee?

I just started an t1.micro instance with Windows 2008 R2 64-bit, IIS and a few WordPress 3.3 sites. It’s working fairly well, actually! I don’t have a lot of traffic but generally the CPU is fairly low utilization, but there are occasional spike (probably from me doing something via RDP).

I have not found anyone readily on the net using this kind of instance with Windows 2008 R2, 64-bit! I’d love to learn tips on how to optimize and secure this setup! Comments?

http://freemusicformormons.com/lds-ward-choir-music roger pack

bluehost does the same thing (limits your cpu to like nothing if you spike). Recommend linode or digital ocean or maybe others over it

http://www.picapol.com/ Steven T Habel

Thank you so much for sharing. FINALLY realized my CPU was the culprit when google analytics was crawling the pages. Lightbulb midway through reading your post

Reece

Thank you! I was baffled about why my micro instance was performing so poorly at strange intervals. Has this changed in the last few years or with t2.micro instances?