Thursday, January 20, 2011

How To Use Amazon EC2 as Your Desktop

Amazon's Elastic Computing Cloud (EC2) is a service allowing you to rent servers in Amazon datacenters by the hour. Using EC2 for your desktop makes sense if you need a powerful machine with a great Internet connection, colocated with your production servers. There are many benefits to using EC2 for your desktop:

In this article, I describe how I use EC2 as my Linux development desktop. I provide detailed instructions for every step of the setup process. This guide assumes that your client machine is Windows.

Monthly Cost

Amazon offers 32-bit and 64-bit Linux and Windows servers with various amounts of memory and processing power. I use a 32-bit Linux server with 1.7 GB of memory and 1 EC2 Compute Unit (equivalent to a 1.2 GHz Xeon cpu). This server type is called a “Small Instance” or ‘m1.small’. The server boots from a 16 GB chunk of disk space on Amazon Elastic Block Store (EBS). I also rent a static IP address (Elastic IP address) for my server.

Since Amazon charges by the hour, I tend to start up my server when I start my work-day and shut it down in the evening. It takes about 60 seconds to start up and log in. Amazon charges a small fee for my static IP address when it's not attached to a running server. I still save about USD $50 per month by turning the server off on nights and weekends. Here is my average monthly cost breakdown:

$19.00 for 200 Small Instance hours at $0.095/hour

$1.76 for 16 GB EBS at $0.11/GB-mo

$0.03 for 300 MB data transfer in at $0.10/GB

$0.05 for 300 MB data transfer out at $0.15/GB

$5.46 for 544 non-attached Elastic IP hours at $0.01/hour

Total Monthly Charges: $26.26

This works out to about $315 per year for a development machine. Amazon offers discounts for long-term commitments (Reserved Instances), but they are not economical if you're using the machine only during the workday. I would pay $409/year with a 1-year reserved instance and $298/year with a 3-year reserved instance.

Tools Used

FreeNX Solves the Network Latency Problem

X-Windows programs send many sequential requests to the X server to update the screen. When a program is used over the network, each request is delayed by network latency and the whole program feels sluggish. FreeNX solves this problem with a local caching proxy server that can respond to all X requests locally. Your programs run fast and you experience only one round-trip of network latency.

EC2 API Tools

Amazon provides a suite of command-line tools for managing EC2 instances. Use these tools to set up your new instance. Also use them every day from handy 'start-dev' and 'stop-dev' scripts to start and stop your instance.

PuTTY

Use PuTTY to log into your newly created instance, create your user account, and set up FreeNX.

Ubuntu 10.04LTS

This guide assumes that you are running Ubuntu 10.04LTS on your instance. The Ubuntu team has pledged to support this version with bug fixes and security updates until April 2015. They also provide a system image that can boot from EBS.

Set Up Your Instance

Choose EC2 Region

EC2 is available in five regions: US West, US East, EU, South-east Asia, and North-east Asia. You should choose the region that has the lowest latency for you. Use www.cloudping.info or traceroute to measure your latency to each region. Due to routing inefficiencies, the AWS region nearest you may not have the lowest latency; try them all.

US West 204.236.128.1

US East 216.182.224.1

Europe 79.125.0.1

SE Asia 175.41.128.1

NE Asia 175.41.192.1

For an up-to-date list of addresses, go to the EC2 Developer Forum and look for an announcement titled "Amazon EC2 Public IP Ranges".

EC2 prices are slightly cheaper in the US East (Northern Virginia) region. You don't have to choose the lowest latency region if US East is good enough for you. Emacs and Gnome Terminal are very usable over FreeNX with 160ms of network latency. Eclipse is best with 100ms or less.

Click “Download X.509 Certificate” and save the file as Documents\aws\certificate.pem

Create a file Documents\aws\ec2env.bat with the following code. Use the correct URL for the region you chose. You may need to use PROGRA~2 in JAVA_HOME if your OS is 64-bit.

set PATH=%PATH%;"c:\PROGRA~1\ec2-api-tools\bin"
set EC2_HOME="c:\PROGRA~1\ec2-api-tools"
set JAVA_HOME="c:\PROGRA~1\Java\jre6"
set EC2_PRIVATE_KEY=%USERPROFILE%\Documents\aws\private-key.pem
set EC2_CERT=%USERPROFILE%\Documents\aws\certificate.pem
set EC2_URL=https://ec2.us-west-1.amazonaws.com/

Open a Command Prompt and run ec2env.bat to load the settings

Run ec2-describe-regions to test the settings

Create SSH Key Pair

An SSH key pair allows you to log into your new instance and create a user account for yourself. EC2 loads the public key into the new server image. You will use the private key on your client.

ec2-add-keypair sshkeypair

Copy and paste the lines from “BEGIN RSA PRIVATE KEY” through “END RSA PRIVATE KEY” into the file Documents\aws\sshkeypair.pem. To copy text from the command prompt, first right-click and choose Mark, then highlight the desired text and press Enter.

Run PuTTYgen from the Start Menu

On the Conversions menu, choose “Import Key” and open sshkeypair.pem

Change the comment to “sshkeypair“

Enter a good passphrase

Click the “Save private key” button and save it as Documents\aws\sshkeypair.ppk

Close PuTTY Key Generator

Delete sshkeypair.pem for safety

Double-click on sshkeypair.ppk and enter the passphrase. This will start the PuTTY Authentication Agent (pageant) and load the private key.

Create Your Instance

Choose your AMI (Amazon Machine Image)

US West: ami-cb97c68e

US East: ami-714ba518

Europe: ami-a94d67dd

SE Asia: ami-2c354b7e

NE Asia: ami-5c0fa45d

These images are published by Canonical Ltd, the commercial sponsor of Ubuntu Linux. They are all 32-bit Ubuntu 10.04 LTS images that boot from EBS. To find updated AMIs, go to http://alestic.com/ and click the tabs under the title “Ubuntu and Debian AMIs for Amazon EC2”.

Create and start your instance with the following command. Replace ami-FFFFFFFF with the proper AMI identifier. The command may take 30 seconds or more to complete.ec2-run-instances --group default --key sshkeypair --block-device-mapping "/dev/sda1=:16:off" --instance-initiated-shutdown-behavior stop --disable-api-termination ami-FFFFFFFF What the options mean:

“--group default” puts this server in the default security group

“--key sshkeypair” loads your public key into the new server so you can log in

“--block-device-mapping "/dev/sda1=:16:off"” maps /dev/sda1 to a new 16 GB EBS volume. EC2 will not automatically delete the volume when you terminate your instance.

“--instance-initiated-shutdown-behavior stop” configures EC2 to stop your instance when you shutdown the OS, preserving the instance ID and config for easy restarting. Without this, your instance would be terminated (deleted) each time you shut down.

“--disable-api-termination” prevents anyone from accidentally deleting your instance. If you want to delete your instance, you must first run this command to re-enable termination:ec2-modify-instance-attribute --disable-api-termination false i-FFFFFFFF Then you can terminate your instance, delete your EBS volume, and release your elastic IP address using the command-line tools or the AWS Management Console

Your instance has a public IP address. Look it up and ping it:ec2-describe-instances

The public IP address changes every time you stop and start your instance. Run these commands to get a static IP address and point it at your instance. Replace 1.2.3.4 with the newly allocated address. Also replace i-FFFFFFFF with your instance ID. Remember that Amazon charges for addresses that are not associated with running instances.ec2-allocate-addressec2-associate-address 1.2.3.4 -i i-FFFFFFFF

This would be a good time to make a DNS A-record pointing to the static IP address. A name like leonhard.restbackup.com is much easier to remember than 50.18.62.58.

Start/Stop Scripts

Create the file Documents\aws\start-leonhard.restbackup.com.bat with the following code. Replace 1.2.3.4 and i-FFFFFFFF with the appropriate values.

About the Author

Comments

Great post. I was wondering what the responsiveness of using EC2 as your desktop is like? Could you watch a YouTube video, for example?

James.

Michael Leonhard said...

James> The responsiveness is great for Eclipse and other software that primarily draws text and lines. Web browsing has some lag, as the rendered pages are downloaded through the ssh tunnel. YouTube is streaming uncompressed video to the screen, so it gets about 1 fps. I just run my browser locally. YouTube works great on Windows!

Don't forget that if you like Windows, you can do the same thing with Windows and log in with RDP. Personally if I want to run GUI apps, I like Windows, not just because the GUI is better developed, but it's dead easy to log into into a Windows machine with RDP if you're running Windows, MacOS, or Linux.

Overall, AWS is awesome, even though it would be nice to have finer grained steps in the instances you can rent. Some other providers, like Softlayer, let you pick an arbitrary number of CPUs, arbitrary amount of RAM and all that, but, starting out, I find that much choice to be intimidating.

I thought Amazon now offered a totally free low-resource instance on EC2. That would bring the desktop cost to very little. Unless of course, using an instance as a desktop consumes more resources than the free instance allows.

Jimbo said...

Why would you be this technical and setup this much...then still use windows for you primary machine..what a waste.

Michael Leonhard said...

Paul A Houle> Yeah, I used Windows on EC2 for the first time this week and found that it works very well. I wish there was an RDP server for Linux that had good performance like FreeNX.

Jimbo> I got used to a dual-machine setup at Amazon. There I had a Linux desktop and a Windows laptop. Having a good Linux dev machine is great. Having one colocated with your upstream dependencies is better. Also, Windows 7 rocks.

Could you use one of these services to run a game server? (#BFBC2 or #Left4Dead2 for instance)

Shane said...

One of the cons you list is a lack of cheap 64 bit server type. I use the t1.micro for this desktop use scenario with 64bit. It's only 2 cents an hour, unless I start it as a Spot request (which has many disadvantages for desktop use) then it's even cheaper. Moving up to the next 64bit server type, though, is a big jump in price. The lack of RAM, but the dual core nature of the t1.micro seems to work well enough for me, though.

I don't think this solution is for everyone. For me...I've been doing it for a while now and it helped me migrate my world to the mac.

I run small instance mostly....the costs are a business expense....and justified with the type of work I do. Having a public IP address on my centralized workstation helps a lot.

Plus the ability to scale up and down based upon doing some harvesting or processing is nice.

I don't experience any latency...unless the Internet connect I'm on is slow.

I also backup the AMI and all the files + data to S3 and simultaneously to the Rackspace cloud for redundancy. And cover my ass.

I think Michael took some time and laid out a great plan for people to follow. If you think its dumb....I'd move on. But I think your limiting your perspective on whats possible out there because it definitely has its purpose for some.

I think its funny how people get hung up on the costs? If you run a professional shop. Its cost of doing business. I have a central place to host all my files, projects, give places for my devs and clients to access my resources. I don't worry about losing a workstation. Keeps me machine independent.

Thanks Michael. Appreciate your work here.

Steve S said...

>The responsiveness is great for Eclipse

I used to use FreeNX on windows to connect to an ubuntu desktop running at my office, primarily for Eclipse. It was pretty unresponsive for me. I always thought that was because the window was drawn by the JRE or something like that. My ctrl, shift and alt keys would get stuck sometimes, or not even register. Sometimes sections of a window would blank out and I needed to hide and restore the window to redraw it. Sometimes text would blank out.

In order to improve my experience, I ended up configuring a separate development environment at home and checking out a copy of my repositories from my remote machines.

I always thought that FreeNX felt slower than it should. I know my connection to work is not in the same league as Amazon EC2, but there is more than enough bandwidth to spare for one desktop session, since I would more than likely be the only person using the connection after hours.

When I asked around on IRC, I heard that the general consensus was that this was the level of performance that I should expect, and if I wanted a smoother experience I should stick to running things locally.

Do you have any tips I could try to improve the responsiveness of Eclipse over FreeNX? Thanks.

I believe I have followed the following steps, but then I enter the following,bash-3.2$ ec2-run-instances --group default --key sshkeypair --block-device-mapping "/dev/sda1=:16:off" --instance-initiated-shutdown-behavior stop --disable-api-termination ami-714ba518

but get the following response,Client.InvalidAMIID.NotFound: The AMI ID 'ami-714ba518' does not exist

Am I doing anything obviously wrong?

Michael Leonhard said...

stephen> You can get that error if you run the command against one region and specify an AMI that only exists in another region. Try setting EC2_URL=https://ec2.us-east-1.amazonaws.com/ and try again.

I've followed all these steps, and can log into NX and I get the gnome-terminal. Am I supposed to be seeing a GUI, or is the purpose to kick off the GUI such as Eclipse or Thunderbird from the command line?

Michael Leonhard said...

Adam> Yes, you can start graphical programs from the command line. Add '&' to the end of your command to run it in the background.

Stefan said...

Thanks! Keep up the good work!

David said...

Thanks for the great guide. Until I read this article I didn't realize that it could be done - the key fact I was missing is the EBS device which gives your dev machine the state you need (you know, for your ~/.emacs), which you don't have for arbitrary production EC2 instances. I'm going to try this out!

45 comments:

<<As of July 2011, the FreeNX PPA package is missing the nxsetup script. The script can be downloaded from here by issuing:wget https://bugs.launchpad.net/freenx-server/+bug/576359/+attachment/1378450/+files/nxsetup.tar.gzUnpack the nxsetup script:tar -xvf nxsetup.tar.gzCopy the setup script to /usr/lib/nx:sudo cp nxsetup /usr/lib/nx/nxsetup>>

Excellent post!!! In this competitive market, customer relationship management plays a significant role in determining a business success. That too, cloud based CRM product offer more flexibility to business owners to main strong relationship with the consumers. Salesforce Training Institutes in Chennai

I have been following you for a couple of months now but this is my first time commenting on a blog post. Thank you for sharing your knowledge and experience with us. Keep up the good work. Already bookmarked for future reference.

Wow amazing i saw the article with execution models you had posted. It was such informative. Really its a wonderful article. Thank you for sharing and please keep update like this type of article because i want to learn more relevant to this topic.

Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care and we take your comments to heart.As always, we appreciate your confidence and trust in us.

This blog is having the general information. Got a creative work and this is very different one.We have to develop our creativity mind.This blog helps for this. Thank you for this blog. This is very interesting and useful.

Thanks for posting useful information.You have provided an nice article, Thank you very much for this one. And i hope this will be useful for many people.. and i am waiting for your next post keep on updating these kinds of knowledgeable things...Really it was an awesome article...very interesting to read..please sharing like this information......Web Development Company

Everyone wants to get unique place in the IT industry’s for that you need to upgrade your skills, your blog helps me improvise my skill set to get good career, keep sharing your thoughts with us.Aws Online Training

Nice it seems to be good post... It will get readers engagement on the article since readers engagement plays an vital role in every blog.i am expecting more updated posts from your hands.iOS App Development Company

Besant Technologies has done creative work in Training through its Software Testing Course. We offer short-term modular and comprehensive programs to Software Testing career aspirants and working professionals. Candidates can choose between full-time and part-time Software Testing Course options based on the individual’s time constraints. So join us besant technologies. Amazon Web Services Training in Bangalore | DataScience Training in Bangalore |