Scalability has been the final frontier for web applications for far too long. For web developers like myself, the “building the app” part of offering a web service is the easy part. The challenge comes when you try to scale it.

FeedLounge is a great example. By all accounts, we built a fantastic web feed reading experience. Unfortunately, we didn’t succeed in scaling it. Sure, we learned a lot and I can think of a couple dozen things we would do differently if we could go back and do it again, but scaling would still be our number one issue.

Google App Engine changes that.

Sure, Big Table isn’t a drop-in replacement for a relational database like MySQL or PostgreSQL, but writing GQL isn’t any harder than writing SQL. Basically, you need to re-think how you do the data layer of your application, and the reward is scalability that would have cost you months to develop and significant hardware costs.

GAE not only offers a scalability solution, it also revolutionizes deployment. When you roll your own scalable infrastructure, you get rewarded with a variety of deployment challenges. You need to update code on a variety of machines, upgrade database schemas, etc.

At Crowd Favorite we are web developers, not system admins. We understand how to build something so it can scale, but actually doing the server config and managing them on a day to day basis is not our core competency. This service is designed for us.

The Campfire was a very well produced event. The speakers, as you can see in the video, did a very good job. Questions I had during the presentation were often answered within the next couple of minutes – always the sign of a well designed talk.

As an attendee of the Campfire, I got a chance to ask the GAE team some questions after the presentation. Most of the questions that I was asking had very good answers, and all had very reasonable answers.

A couple of things I asked about:

How do you bring the data from your live application back into your local development environment?

There isn’t a data export/import built-in yet, though it seems like something they’d be open to building. There is a feature to create an XML representation of any data object, so I expect a library that handles the production data export/local development data import situation to be one of the first tools that is built and well supported by the development community.

Is there a CRON equivalent?

No. The solution that seems best to me is to use another machine with CRON to send HTTP requests to your GAE application. If you want a pure “cloud” solution, you could use Amazon EC2 for this.

How long can a request live?

There is a 10 second execution limit for all requests. This includes requesting and submitting data to external services. If it takes 8.5 seconds to download a tweet stream from Twitter, you’ve only got 1.5 seconds to do something with it. This is very reasonable. In thinking through the things I normally do in a long executing script, they could pretty much all b broken out into more atomic events that could each be done in individual requests.

How much will it cost if you go over your quotas?

There wasn’t an answer for this yet, and it’s definitely something you’d want to know when you’re trying to evaluate GAE vs. alternative solutions. I’ll say this though, whatever they decide to charge, it will likely be a fraction of the hardware, bandwidth and human costs of scaling to that degree on your own. I believe they will take this on a case by case basis – they are building a platform they want people to use, and I think it’s in their best interest to do what they can to support successful apps built on GAE.

What the GAE team has already done in just ~2 years is very impressive, but they’ll also be the first ones to tell you they have a long list of things they’d like to improve and enhance. I’ve got no doubt they are going to do just that.

I’d obviously like to see PHP as a supported language as soon as possible, but I don’t know when it’s realistic to expect to see support for new languages. Balancing the building of platform features against support for other languages is a tough choice.

While I’m not going to run out and rewrite all of my PHP products in Python, I’ve already mentally re-architected a component of MyFreeBusy that we probably will rewrite in Python so that it can leverage GAE to scale.

Another idea that I’ve had for a service that has been backburnered for a while is probably going to be written in Python instead of PHP so that we can roll it out on GAE.

Once we complete these internal projects, I’m sure we’ll start building GAE apps for clients as well. I’m excited about the possibilities that this opens for us.

Besides my own intended usage, the general innovation that this is going to allow is going to be fascinating. ISVs have long been able to create great software that can be downloaded and installed on a per-user or small team basis, but Google App Engine promises to let us bring that craftsmanship to internet scale.

Great writeup Alex. I have been feeling the pain of scaling as well lately and it’s only getting worse. I am tied to PHP currently but S3 has already helped a lot, and EC2 soon but, I am exited to see how much more GAE will go. Python is starting to look better that’s for sure. Good luck with your projects!

[…] feature along will be worth it to a certain class of application developers. Says my office mate Alex King: At Crowd Favorite we are web developers, not system admins. We understand how to build something […]

[…] “At Crowd Favorite we are web developers, not system admins. We understand how to build something so it can scale, but actually doing the server config and managing them on a day to day basis is not our core competency.” – Alex King […]

[…] Google App Engine | alexking.org Scalability has been the final frontier for web applications for far too long. For web developers like myself, the â€œbuilding the appâ€ part of offering a web service is the easy part. The challenge comes when you try to scale it. (tags: google scalability appengine alexking googleappengine) […]

[…] cloud platforms, like Google’s App Engine, to assume the burden of scaling. Developers, then, may focus on their core competency – building applications – and let Google worry about the challenge of […]