Search

You Should Use EBS Boot Instances on Amazon EC2

If you are just getting started with Amazon EC2, then use EBS boot
instances and stop reading this article. Forget that you ever heard
about instance-store and accept my apology that I just mentioned
it. Once you are completely comfortable with using EBS boot instances on
EC2, you may (or may not) want to come back here and read why you made
a good decision.

EC2 experts may find that there are specific cases, few and far
between, where instance-store might make sense, but they don’t attempt
to use instance-store without understanding and accounting for all the
serious drawbacks and dangers that go with making this choice. For
example, experts using instance-store don’t mind losing all of the
data on the instance as they have designed the system so that the data
is stored elsewhere and so that a new instance can easily and
automatically be rebuilt from scratch.

One of the challenges for beginners is that many of the benefits of
EBS boot don’t necessarily seem like something you’ll need to use
right away. Then they get down the road and into situations where
they realize that they would have been much better off if they had
gone with EBS boot in the first place and may find it takes some work
to make the transition.

Big benefits of EBS boot instances

Here are some of the reasons I use and recommend EBS boot instances.
None of these benefits are available with instance-store, so even a
single one of these can be an overriding factor for choosing EBS boot.

EBS boot instances store the root file system on an EBS volume
which is persistent storage. If the instance hardware fails, the
EBS volume remains accessible. It is also possible to request the
EBS volume to persist beyond the termination of an EC2 instance.
When an instance-store instance fails or is shut down, all of the
data on the root disk is lost forever and can never be retrieved.
Read more about protecting EC2 instances from accidental
termination and loss of data.

EBS boot instances can be stopped and restarted at will. The
“stopped” state suspends the hourly instance billing charges,
preserving the information on the EBS volumes. The stopped
instance can be started again a few minutes later or months later,
restoring state just as if the instance was rebooted. An
instance-store instance can only be left running with full charges
or terminated, which causes you to lose all data on the disk. Read
more about how stop/start of an EBS boot instance is similar to
and different from a simple reboot.

When something goes wrong with an EBS boot instance so that it
can’t be booted, or you lose access through ssh (e.g., lost keys,
bad ssh config change), you can still view and modify or fix the
EBS root volume by attaching it to another running instance. With
an instance-store instance, everything on the root disk is lost and
cannot be recovered. Read more about fixing files on the root EBS
volume of an EC2 instance.

EBS boot instances can be run with a root EBS volume size from the
default specified by the AMI (often 8GB) up to 1,000GB (1TB). The
instance-store AMIs have a max root disk size of 10GB with no way
to increase it. Read more about increasing the root disk size of
an EBS boot AMI.

It is possible to change the instance type for a running EBS boot
instance without needing to start a new instance. For example, you
can scale up from an m1.large to an m1.xlarge and then a few hours
or days later, scale back down. An instance-store instance is
stuck with the type on which it was originally run. Read more
about changing the instance type of a running EBS boot
instance.

You can easily replace the hardware for your instance if you are
running an EBS boot instance. This is extremely valuable if your
instance is having problems that you suspect may be related to the
underlying hardware. An instance-store instance is bound to the
hardware it started on and cannot be moved. Read more about using
stop/start to replace EC2 hardware.

EBS boot AMIs are simpler and faster to create than instance-store
AMIs. In fact, you can trigger the creation of an EBS boot AMI
from a running instance in one command, API call, or console click.
You need to copy sensitive AWS credentials to the instance when
creating an instance-store AMI.

Amazon has stated that EBS boot AMIs boot up faster than S3 based
AMIs (instance-store). In my recent experience, the difference is
negligible, especially when testing popular AMIs that are likely to
be cached, but we might as well chalk this up as another benefit.

The t1.micro instance type released recently by Amazon only
supports EBS boot instances. This move is like a sign from Amazon
that you really don’t want to run the legacy instance-store
instances.

Some versions of Windows (Server 2008) only run on EBS boot instances. I believe this may be related to the disk size limitations of instance-store, but I don’t use Windows, so am not an expert in that area.

Possible benefits of instance-store instances

EBS volumes and EBS boot instances aren’t perfect. Running an
instance-store instance might be preferable in some very specific
cases where you don’t care so much about losing the data you are
storing on the root disk.

I’m going to list some of the possible benefits of instance-store, but
each of these may not be as beneficial as they appear at first glance
and you must remember that running instance-store loses all of the
above benefits.

There is a negigible cost savings with instance-store, as there is
no charge for an EBS volume nor the I/O transactions. Note
however, that the cost for an 8GB root EBS volume is only around 80
cents per month (depending on the region) and you get about a
billion I/O requests for a dollar, billed by the penny increment.
The cost savings to run instance-store is generally not worth the
increased risk of losing your valuable data, especially when you
compare it as a percentage of the cost of running the instance
hours in the first place (tens or hundreds of dollars per month).

There may be some applications that perform somewhat better when
running against ephemeral or instance-store disks than against EBS
volumes. Some high end users have also reported inconsistency
in the performance they see from EBS volumes at high I/O transaction
rates for long periods of time.
EBS volumes perform better for some applications and are
generally good enough for most applications. If you don’t care
about losing your application data, and you have tested to see that
instance-store performs better than EBS volumes, then you could
either run instance-store instances, or you could run EBS boot
instances and drop your data on the ephemeral disks available to
all instances above t1.micro. It is also becoming increasingly popular
to run multiple EBS volumes in a mdadm RAID-0 or LVM striping
configuration to improve performance and smooth out some
experiences of performance volatility.

There are a couple historical failure modes that have happened with
EBS volumes that could not happen for instance-store disks. You
may hear people say this is a reason not to use EBS volumes, but
EBS volumes are still far more reliable, persistent, and protected
against failure than instance-store. The fact that it is possible
for EBS to fail is not a reason to use the less persistent
instance-store. It is a reason to create regular snapshots of your
EBS volumes to improve reliability and act as backups.

If you are creating a “paid AMI” you can only do it as an
instance-store AMI, not EBS boot. Only two of the thousands of
people reading this article are creating paid AMIs and they already
know this fact.

Why did I write this article?

I regularly provide consulting services in public communities like
serverfault, the ##aws IRC channel on Freenode, the
ec2ubuntu Google group, and Amazon’s EC2 forum.
It’s a rare week that passes where I am not telling somebody that they
would not have the problem they’re having if they had been using EBS
boot instances.

The saddest response I can give to a plea for help is that the
customer’s valuable data has been lost because they were running
instance-store instead of EBS boot and they did not have real-time
streaming backups.

Historical background

When Amazon launched EC2 in 2006, only instance-store AMIs were
available (though they weren’t called instance-store at the time as
they were the only kind). For years, Amazon customers learned to work
with the server limitations of risking the loss of all data at any
point in time.

In August of 2008, Amazon introduced the concept of EBS volumes, and
there was much rejoicing. Data could finally be stored on persistent
disks even though the root disk remained on ephemeral storage
(another name for instance-store).

In December of 2009, Amazon released the ability to launch instances
with EBS volumes as the root disk, or EBS boot instances. Now, the
entire server could be persistent and all of the above benefits were
realized.

Notes

Just because EBS boot volumes are “persistent” does not mean that they
do not ever fail. Amazon has released figures about their failure
rates, which are proportional to the number of blocks modified since
the last EBS snapshot was created. Regular snapshots improve the
intrinsic reliability of your EBS volumes in addition to acting as
backups. I also recommend creating regular off-site backups (outside
of Amazon) to eliminate your AWS account as a single point of
failuire.

Categories:

Tags:

23 Comments

It seems that Eric Hammond doesn't use (or know about?) RightScale or a similar service, and so he's focused on benefits that aren't necessary if you have a proper cloud architecture under RightScale (or a similar tool). The best comparison I can give is this: Eric Hammond is telling us about the necessity of backing up our data onto thumb drives instead of just storing files on a single computer--while completely ignoring the fact that Dropbox exists. He's right, but he's missing the bigger picture and the best practices.

A proper cloud architecture has seemless backups off EBS (or instance stores) and extremely quick disaster recovery. The "major advantages" that Eric cites for EBS-backed instances are all related to either (1) having proper backups, (2) scaling, or (3) saving money on running instances. The first is best done much differently than just leaving a bunch of EBS volumes hanging around (RightScale has a great solution for this). The second is best done through a proper scaling system (RightScale has a great for solution for this, too). The third is best done by backing up to *S3* or EBS snapshots, not leaving EBS instances around in a "stopped" status (Guess what--RightScale for this too!)

He also omits/understates three biggest reason for using instance-backed AMIs instead of EBS-backed AMIs: (1) your server doesn't need EBS to launch, which means that the various problems that Amazon has had with EBS over the years won't affect your server launches, (2) EBS-backed AMIs are quite a bit more expensive than using ephemeral storage when you get to any decent amount of space ($100/month for 1TB of EBS, which is less than the ephemeral storge on a c1.xlarge...the 8GB in the piece is definitely a straw man), and (3) EBS volumes are limited to 1TB of space, so you have to stripe them to get bigger volume sizes (huge pain in the butt).

Don't get me wrong--EBS is fabulous. But not for backing instances. It's fabulous for storing massive amounts of data, getting quick snapshot backups, and quick restores for disaster recovery...all of which is best run through RightScale.

Thanks for your feedback. I've been a fan of RightScale for years and should point out that RightScale is also a long term sponsor of this tech blog: http://alestic.com/sponsors

I am deliberately biased towards EBS boot instances in this article, because I want anybody who chooses instance-store after reading it to know what they are giving up and to have good solutions in place for handling the deficiencies of instance-store. I'm calling these people "experts" because it is not yet a common practice to create the infrastructure necessary to treat production servers and storage as temporary objects and to be able to recreate server configurations and data from external sources.

You can either be an expert because you have developed your own solutions for managing the ephemeral nature of instance-store, or you can depend on the expertise of other people by using third party products, RightScale included.

That said, I'm an expert in working with instance-store (did it for years and invented some of the tools used to automate their configuration) but I still use EBS boot for just about everything. All the EC2 instances run by my company are built through automated processes, have live replication and backups, can be replaced at a moments notice; and all are EBS boot.

Benefits 1-7 of EBS boot instances are only relevant to servers that need to easily persist state. The exceptions in that list are the points about AMI size limits, and the fixed size of the root device. Neither one of those seems particularly important however.

#8 I think is fairly important, and I think that EBS boot makes sense for anyone needing to create their own AMIs.

As you already mentioned, #9 is really stretching it as there's no significant difference in practice between popular instance-store AMIs and EBS volumes. #10 on the other hand seems like pure speculation. Inferring that instance-store is a deprecated option from the fact that t1.micro only supports EBS boot seems like a giant stretch.

A more reasonable assumption would be that t1.micro has no instance storage in order to minimize I/O impact to the instance store hard drive for the other larger instances that are running on the same server. Therefore the set of EBS only instance types is likely to remain t1.micro.

Going back to the point about stateful servers though, instance-store brings the pain of relying on persistent, stateful servers forward. Yes, it's difficult to deal with, but you'll need to deal with this at some point anyway. Instance-store wil also force one to think about places where truly stateless servers would work just fine. For example, most web servers and application servers have no need to persist their state. Doing so only invites one to create unique snowflake machines with an accumulation of small tweaks that is not documented anywhere.

EBS-boot instances are definitely more user friendly, because they closely emulate the physical machines booting of persistent hard drives that people are used to managing in their own data centers. That's not necessarily a good thing when moving to AWS.

The fact that you must use EBS boot instances to run on t1.micro is a factual benefit of EBS boot, not speculation. That this might indicate Amazon deprecates instance-store may be considered speculation.

For most applications it is just fine to build that sort of infrastructure on EBS boot instead of instance-store.

I can appreciate the point that using non-persistent storage forces you to learn a better way of building infrastructure on EC2. I was forced to learn that lesson on EC2 in the years before EBS was released. I'm not sure all EC2 users need to learn that lesson in such a harsh way.

Yes, I should remember to lock my apartment door living in the city. No, I don't think apartment managers should teach their tenants a lesson by stealing things when they forget to lock their doors.

My AWS experience started a few months ago, when I became the sysadmin for a small company with existing instances and a manual process for replacing most of them. We use instance-store on almost everything, and I've made that farily autotomated, but now that I'm familiar with aws, I could see where ebs root might be useful.

One thing I noticed that you left out in this post, though, is a mention of auto scaling. We use the amazon elb/autoscaler for our app and api servers, and those AMI's are all instance-store. I have some magic boot scripts that name and configure them properly, and even create an (or use an existing) ebs volume for storing in-flight data that we'd like to persist. My question: does ebs make sense for auto-scaled applications? To me, it seems like since these systems constantly are brought up and taken down, that instance store might work just as well as ebs, since the autoscaler terminates an unhealthy instance (unless I'm mistaken, and it just stops/reboots an ebs).

I have been using AWS for the past 4+ years and my recommendation to use instance store or EBS backed Instances have always been use case driven. We did some benchmark where we have seen that instance store had better I/O performance than a EBS backed Instance. And off late the Instance store images boot up time is almost equal to the EBS backed ones. So areas in infrastructure architecture which are completely stateless can go with instance store and Instances would automatically get configured on boot.

I don't see a necessity to have larger root volumes - at least for web applications. We design the root volumes to have only the OS and the application which does not require larger volume sizes. Of course if there are any layers in the architecture with persistent data then additional EBS volumes will be attached. System and application log files will be configured to use the ephemeral store since all the Instances come with larger ephemeral stores - logs can grow pretty fast and they enjoy the benefit of better I/O from ephemeral store.

Coming to Windows, we prefer using the EBS backed ones specifically because the image bundling process takes so much time with instance store ones. And of course for any R&D purposes EBS backed ones are preferred because you have the ability to stop and start them.

As I said earlier, it is primarily driven on the use case basis and one should not just go with a specific type with any over sight.

I agree that there are some situations where instance-store instances might be preferable to EBS boot. I agree that many of the EBS boot benefits might not be that important to many users.

However, it is important to understand *all* of the EBS boot benefits, to realize that you are giving up *all* of them if you are running instance-store, and to have an architecture design to account for the drawbacks of instance-store.

As mentioned in the article, I agree with the approach of only putting the OS and applications on the root disk and keeping data on separate disks (generally EBS volumes). When you do this, there isn't much benefit to using instance-store for the root disk as very little activity is happening on it, so it might as well be EBS boot so you can get the above benefits.

I don't put application logs on ephemeral store because I can't afford to lose them. They are valuable to the business. I've asked Amazon for a syslogd service where we can stream logs and let them worry about storage and growth, but until then, they go on EBS volumes, get snapshotted, and get backed up offsite.

I don't put system logs on ephemeral store because it's nice to be able to look at the system logs if the instance kernel crashes. If you're running instance-store when that happens, you might have no idea why the instance suddenly became unreachable.

Once you've reached the point where you can read this article and argue for instance-store in a particular case with an explanation of how you are accounting for losing each of the listed benefits of EBS boot... then I name you "expert" as described in the second paragraph and you are free to move to the next level.

Auto Scaling and spot instances are cases where a lot of the EBS boot benefits don't seem to apply since Amazon may start and terminate them autonomously. On the other hand, are you losing anything if you just always use EBS boot for consistency across your architecture?

Given the nature of Auto Scaling, you probably want the instances to boot up quickly with your desired software and configuration. The easiest way to do this is to create an AMI that boots with very little need to upgrade its configuration after coming up.

It's a lot easier to create EBS boot AMIs for Auto Scaling or spot instances than it is to create instance-store AMIs (point #8 above).

Some people are looking for the equivalent of VPS or dedicated hosting on EC2. They might not ever need to worry about the complexities of building auto scalable infrastructure or dealing with servers that can go away at any minute. Instances that mostly stick around and fail once in a blue moon, backed by EBS, are probably sufficient.

For anyone that wants to take advantage of the E in EC2, the lessons that come from dealing with instance-store instances are not optional. To me it feels that forcing yourself to deal with instance-store, fully aware of the pitfalls, is the best ways to learn those lessons early. Perhaps this way of learning is too harsh and risky. I haven't found a better way though to teach these lessons.

As far as using EBS boot instances even for stateless servers, it might certainly be fine, but I would feel more comfortable without the additional dependency. The EBS outage from not that long ago comes to mind. I would feel much more comfortable from eliminating EBS completely from an architecture. To quote Heroku's post mortem "block storage is not a cloud-friendly technology".

Thanks for this article; it's going to be useful to give people this link.

One comment - don't you think you're being a little too kind on the matter of EC2 I/O performance with saying only: "There may be some applications that perform somewhat better when running against ephemeral or instance-store disks than against EBS volumes. Some high end users have also reported inconsistency in the performance they see from EBS volumes at high I/O transaction rates for long periods of time. EBS volumes perform better for some applications and are generally good enough for most applications."

My personal experience has been that while instance-store disk performance is relatively poor, EBS is *much* worse - and highly variable, to boot. For me, bringing users to EC2 that are used to our internal vSphere deployment, I find myself having to warn them about how bad disk I/O performance is, especially on EBS, the storage type that maps most closely to what they use on vSphere.

This isn't meant to be a vSphere-is-better-than-EC2 comment - to get okay disk performance with vSphere requires substantial capital and operational investments - we just happen to have made those investments, so it's relevant to my users. My point is more that any discussion of disk on EC2 should probably start with an acknowledgement, in bold letters, that "EC2 disk I/O is poor across the board; choose from the lesser of these evils."

Yes, I am being a little too kind to EBS volumes. This is deliberate. I'm recommending that people without experience and understanding of the complexities of EC2 should use EBS first.

I believe it is better for a tiny segment of new users to eventually run into performance problems and learn to solve them, than a large segment of users to run into complete data loss and have no way to solve it.

I have a hard time believing that EBS boot are unusable or bad for the majority of EC2 users.

People have been shouting "EBS performs poorly/inconsistently" from the mountain tops. A few of these are people are running some high end applications that actually do need to eek out the last drop of performance even at the cost of building more complicated systems. Most of these are probably just repeating what they heard because that's what humans do.

Some performance issues people are experiencing while running on EBS may not be the fault of EBS, but could be basic application design issues. Disk I/O in general is slow and should be avoided when possible. Sure, EBS (and instance-store/ephemeral) might be slower than physical hardware, but many of the same principles apply.

In fact, in the first paragraph Adrian writes: "The conclusion we reached is a rule of thumb for EBS - If you sustain less than 100 iops (input+output per second) long term average it works fine. Short term bursts can be 1000 iops. By short term I mean less than a minute, long term more than 10 minutes. YMMV."

That post was written 10 months ago and Adrian raises a great question in this tweet related to my article: "do you think EBS has improved in the last 6 months? AWS have been working hard on it."

Things move fast and keep improving in this market. We should challenge and test our assumptions regularly.

EBS vs. instance-store is a property of the AMI which determines the type of the instance. Wherever you see a list of AMIs, it should indicate "ebs" or "instance-store". Other names for instance-store AMIs might include "ephemeral" or some mention of "S3" since S3 is where the instance-store AMI is stored before it is run on an instance. After an instance-store AMI is run, there is no S3 backing for the instance as the root disk is then ephemeral (disappears when the instance terminates or fails).

This posting and discussion reminds me of the old saw about how to get replies by posting wrong (or misleading) information... but Eric Hammond is intentionally overstating the case for EBS to counter the historical EC2 bias towards instance-store.

EBS boot instances are very useful for persistent systems (including those that are only active intermittently) and instance-store/ephemeral instances are for flexible scaling like N machines of the same type.

One HUGE negative issue with using EBS backed windows 2008 instances is as follows: Say your instance (I1) is degraded and you want to launch a new Win2008 instance (I2). However, because I1 has an EBS volume, it should be easy to launch I2 with I1's EBS volume, right? Wrong!

This means that when I1 is degraded, you need to launch I2 with it's own root volume, attach I1's root volume, pull any data off of it that you need, etc. Simply a royal pain - which makes for a lot of manual work.

Oh - by the way - though Amazon says in the documentation that you can attach an EBS as the root volume for windows, this does NOT work.

So I have gone back to instance stores. All machine level configuration stuff is baked into an AMI. All my app stuff is stored in S3 - which the instance pulls down and customizes upon launch. Simple.

I don't quite follow how losing your data when the instance goes down is better than having to copy it from the old EBS volume you attach to the new instance, but it certainly does sound simpler as you say ;-)

Thanks for the clarification. It sounds like you have tackled the challenges of running on instance-store, and part of your solution is to use attached EBS volumes. You still risk losing things that the OS and apps store on the boot volume at run time (logs?), but with care your approach makes the best of the situation.

Automatic instance launch and configuration qualifies you for "expert" status as defined in this article.

In fact, I've been using EBS volumes for my EC2 instances and I love them.

However, when it comes to autoscaling, I'm having a difficult figuring out how to use EBS volumes here. Interestingly, instance-store might be the better solution here since I will have less troubles keeping data consistent among the EC2 instances.

Would really love to hear the author's opinions on autoscaling and EBS vs. instance store.

By default the EBS root volume will be created when an instance is started with Auto Scaling and it will be deleted when Auto Scaling terminates the instance. This is similar to instance-store with Auto Scaling so there is no additional "trouble keeping data consistent" when you use EBS boot with Auto Scaling.

With Auto Scaling you lose everything when an instance is terminated either way, so you have to make sure you are saving any important data elsewhere.

I use EBS boot with Auto Scaling as it's just simpler to use the same AMIs for different purposes and instance-store doesn't offer much additional value. Plus it's much easier to build custom EBS boot AMIs, which you may end up doing more frequently so that your Auto Scaling instances come up faster.

Did you know that Amazon includes status messages about the health of availability zones in the output of the ec2-describe-availability-zones command, the associated API call, and the AWS console? Right…

The source for ec2-conssitent-snapshot has historically been available here: ec2-consistent-snapshot on Launchpad.net using Bazaar For your convenience, it is now also available here: ec2-consistent-snapshot on GitHub using Git You are…