Launching TurnKey Hub into private beta: cloud deployment simplified

Towards the end of last year we decided it was time to start working on an idea we've been toying with for a while. Mapping out the feature set was fun, and a lot of the current and future features are based on feedback we received from you guys and gals, as well as many related questions and comments from around the net.

Today after a last round of internal testing we are pleased to announce that we've launched TurnKey Hub into private beta:

We've just sent out the first batch of invites. If you've already requested an invite, you should receive one as we roll them out. Please be patient. We initially have limited capacity so it's first come, first served.

What is this TurnKey Hub?

The short version: TurnKey cloud deployment: simplified.

The slightly longer version: An easy way to launch and manage TurnKey Linux appliances in the Amazon EC2 cloud (more clouds and VPS providers on the way).

A simple Amazon EC2 console optimized for TurnKey

Launch TurnKey appliances in the click of a button.

TurnKey optimized firewall templates.

Configure custom passwords on launch.

Authenticate with personal SSH key in addition to EC2 keypairs.

Automatically setup EBS devices and Elastic IP's on launch.

Easier management with descriptive labels for all assets.

Unified interface for all regions and all your EC2 accounts.

And that's just the tip of the iceberg. There's much more in development...

Upcoming features

More clouds: Support all clouds and VPS providers.

Backup: Automatic encrypted appliance backups.

Migration: Automatically restore backups anywhere.

You decide: Suggest features and help us prioritize.

In other words this is just the first modest step in a much more ambitious plan to continue making TurnKey easier to use, as Liraz recently explained:

"Imagine being able to develop your site on a locally running appliance (e.g., running in VirtualBox or VMWare). Then, when you're ready you can automatically migrate your appliance, with all your customizations to a cloud hosting provider of your choice."

So once you receive your invite, take the Hub for a spin, let us know what you think. We'd like to know how to make this better. What new features you'd like to see implemented. That sort of thing.

Comments

Thanks Turnkey Gang for a very very easy to use Virtual Appliance deployer!

In less than 30 minutes, I created my Amazon EC2 account, setup a Media Wiki and was off and running.

I'm sure there is more to work out - ie. something tells me that when I terminate the host, the data goes with it. Or is the state preserved / snap shotted.

When does one need to use S3?

All that can be worked out later. I'm impressed! I must say it makes me nervous to put my Amazon keys into your hub as that gives one access to run up a fierce bill.. What makes Hub Secure enough to do this?

Currently, if you want your data to survive instance termination you should store it in an EBS volume. But it's not automatic.. yet. The next major feature we'll add into the Hub is automatic backup and migration which will take care of that usage scenario and many others.

Regarding security, that's a good question. I think it would make a good idea to dedicate a blog post to this but I'll cover the main points right now:

Security background: We, the developers, come from a computer security background. You can Google us to find the full details. For example, Alon, which is the lead developer for the Hub founded a computer security training company. Before that he worked as a penetration tester, and before that he was responsible for network security in the Israeli air force. My background is similar. Security is a big priority for us and we are not naive regarding the risks.

High-level language and webapp framework: The Hub is implemented in a high-level language (Python) using the Django framework. which contains a range of security mechanisms you can use to protect from all sorts of common web application attacks (e.g,. SQL injection, XSS, XSRF, etc.). Using a high-level language wipes out a large class of security vulnerabilities (e.g., issues with out-of-bound pointers, etc.)

Encryption: All of your network traffic to and from the Hub is encrypted.

Isolation: We've setup the Hub to run in a separate server from the rest of our network infrastructure. It's isolated. If the CMS that runs this web site is compromised, or our mail server, etc, that doesn't effect the Hub.

Security audits: We've run extensive security audits on the Hub prior to launch including manual code inspection AND a gauntlet of automatic tools.

Reputation: We have a reputation to maintain and understand the consequences of failure. If the Hub's security fails that will be a huge setback for the viability of the service. This is a risk we take very seriously.

The precautions we're taking, coupled with our background mean that it's far more likely in my opinion that your access keys / credit card / bank account credentials will be stolen via malware running on your PC (for example). Nothing is perfect but if you already accept the risk of doing any sort of commerce on the Internet you shouldn't loose any sleep over the Hub storing your AWS keys.

Note that Amazon allows you to create a separate set of Amazon keys you can use just for the Hub if you believe this reduces your risk. You can then revoke the keys at any time.

There are pros and cons either way. The Hub isn't a generic web application that it would make sense to create multiple instances of. It's a complex piece of infrastructure that is tightly integrated with the TurnKey Amazon Machine Images. If I've learned anything doing TurnKey is that open source is harder than it looks. It's not magic and it seems to work best when you have a repeatable, generic need and it's both easy and desirable for outsiders on the same side of a problem to jump in and collaborate.

I'm not yet convinced that's the case with the Hub. As we add more features it's going to grow more complex and become more tightly integrated with the TurnKey images.

But eventually once things have settled down maybe we can extract some generically useful pieces out of the Hub and open source that. Needs more thought...

Open source cloud systems like Eucalyptus and OpenNebula are compatible with the Amazon API so using the TurnKey Hub on these systems should not be very hard. And having ready to run appliances in your own cloud is a real benefit.

I've been using your Drupal app for my blog for about a year now and I love it. I've upgraded and customized it as I've gone along, and it kinda helped me ease into the Linux world (I'm a reform-ing MS server guy).

I currently run the Drupal appliance on my own ESX 4 box using a DSL connection, but now would like to migrate it to EC2.

Do you have anything in place for this yet or do you know of a 3rd party way of doing it effectively?

We're developing a secure backup and migration tool that will allow you to seamlessly do exactly what you want. The back-end is working "in the lab" but we haven't yet finished developing the front-end for it. This is the next major feature we will be integrating into the Hub though I can't give you a hard schedule for when it will be available so if you need this done right away you may want to look into migrating your Drupal instance manually. That can be a pain but that's how everybody does it right now. Basically you'll need to transfer your theme, contrib modules and database. The "Backup and migrate" Drupal module can help you with the database. You might want to experiment doing this between two VMs in your ESX setup first...

Bulk copying directories can have unexpected results. That might work too mind you, but I'd test it carefully first. When you copy your theme and modules you know exactly what you're in for. What I've done with this website (which is also in Drupal) is sync the VM devel box's themes and modules to the production box. Initially you also sync the database but after your site is up and running you'll usually want to sync the database from the production site to the development VM and not vice versa.

then register your new AMI (1 command) ec2-register in order to be able to launch your instance

This way you have a complete snapshot of your current system

But if your data makes the created image too big (10G) you could just bundle/upload/register your base machine then later FTP (or use the firefox add-on called S3Fox) S3Fox is a companion to the Firefox AWS mgmt tool called ElasticFox... both of which you should learn more about as they are great tools for working with AWS's cloud.

Using S3Fox uploading your created image to AWS S3 is a simple click of the mouse (after you've cofigured S3Fox with your AWS Credential and specified a S3 storage "bucket" name to store your image into.

Now the above will give you what is called an S3 backed image... where your root file system is limited to 10G ... but you can mount other storage.

However, today most people are creating and using what are called EBS back AMIs. Elastic Block Storage (EBS) AMI's boot faster and can be started/stopped (ie paused) without terminating the instance. With EBS backed AMI you aren't limited by root file system and your data is persistent on the EBS volume which by the way you can make snapshots of whenever you make significant changes.

Growth in private clouds based on canonical/ubuntu's UEC/eucalyptus based cloud architecture is on fire.

I've signed up and used Canonical's Landscape application and it works well and it does let you manage both your private UEC cloud as well as AWS VMs.... but I think their pricing of it makes it prohibitive (at least to me) as there is a subscription cost for every machine/vm managed. Which can climb in $ rapidly.

I'd like to see an alternative Open Source application that provides an intelligently integrated combination of something like the core capabilities of:

ebox - for network, security, ddns, printer, AD, etc. mgmt/reporting

ebox already integrates a great deal of this from various GPL projects

integrated with -- chef/puppet/cfegine - some provisioning mgmt tool with a Gui front-end?

I think in a cloud environment, people would be willing to pay a decent subscription for this type of cloud manager application.

Second.. But don't you just hate it when you throw an idea out there and you have no clue if it was dropped, or if someone is making a business plan this very moment to make millions.. er billions off your excellent idea..

Hello! I have been trying to launch a Drupal 6 appliance to Amazon EC2 via the Hub and I keep getting the message that Amazon is over capacity. Got it last night and just now too. Anyone know what's going on?

Amazon regions are split into zones. When you launch an instance in the AWS console and you don't specify a specific zone (which is the default), Amazon will launch your instance in the "best" zone available.

But, that comes with a cost as some assets, such as EBS volumes, are zone specific. So if you have an EBS volume in us-east-1a but your instance is launched in us-east-1b, then you can't attach the volume to your instance. For this reason, you can force a specific zone.

When launching instances via the Hub, it will force the first zone (e.g., us-east-1a), and when you create EBS volumes they are created in the first zone (e.g., us-east-1a) as well. This was a design decision to keep the user interface an usage as simple as possible. There is a downside though - at certain peak times Amazon is over capacity and you won't be able to launch new instances until they have provisioned more servers.

Bottom line, you can either wait a couple of minutes or specify a different region.

Up until now there hasn't been too many capacity issues, but if the persists (yes, I know, its really annoying) then we'll have to see what we can do about it.

I just deployed an update to the Hub which should fix the insufficient capacity issues.

As I explained above, prior to this update the Hub would default to and force the first zone (e.g. us-east-1a) in each region to simplify the user experience with use of EBS volumes. This update changes that. There is no global, default, hardcoded zone anymore.

The Hub will now use a default_zone for each region according to your prior use of the region. For example, if you have an EBS volume in us-west-1b, when you launch an instance in us-west-1, us-west-1b will be forced so you can attach the volume.

Another example, if you already have an instance registered in the Hub in eu-west-1a, when you create an EBS volume in eu-west-1, it will automatically be created in the correct zone.

And finally, if you don't have any instances or volumes in a region, the Hub will request that Amazon choose the best zone when launching an instance in that region. The zone Amazon chooses will become the default zone for that region, until such time that the instance and (if applicable) volumes are deleted from the Hub.

Closing note, if you don't know what zones are, ignore this post - the Hub will do what is best, you don't need to worry about it :)