Meta

When you launch an EBS instance on EC2, you get a root disk size that is the same as what the machine image was created with. For example, the Ubuntu AMIs by default are 8GB. Typically, the best approach is to leave the root volume this size and add additional volumes provisioned with the size needed for particular data (mysql data files or whatever).

However, sometimes, you just need a bigger root disk than 8GB. It IS possible to launch such an instance. Use this command:

TL;DR: The real problem is there isn’t enough capacity. The us-east1 region is too big and during an outage there simply aren’t enough resources to allow users to recover their sites.

In part 1 I discussed how the bug-rate on AWS doesn’t seem to be getting better. A few bugs aren’t necessarily a big deal. For one, they’re expected given the trailblazing that AWS is doing and the incredibly hard & complex problems they’re solving. This is a recipe for bad things to happen sometimes. To account for this the true selling point in AWS has always been “if something goes wrong, just get more instances from somewhere else and keep on running”.

In the early days (pre-EBS), good AWS architecture dictated that you had to be prepared for any instance to disappear at any time for any reason. Had important data on it? Better have four instances with copies of that data in different AZs & regions along with offsite backups, because your data could disappear at any time. Done properly, you simply started a new instance to replace it, copied in your data, and went about your merry way.

What has unfortunately happened is that nearly all customers are centralized onto us-east-1. This has many consequences to the architecture model described above.

Traffic Load

A very common thread in all of the us-east-1 outages over the last two years is that any time there is trouble, the API & management console becomes overloaded. Every user will be trying to move and/or restore their services. All at once. And the API/console has shown to be extremely dependent on east-1. PagerDuty went so far as to move to another region to de-correlate east1 failures from their own failures.

Competition for Resources

Once again, by virtue of us-east-1 being the largest region, whenever there is an outage every customer will start trying to provision new capacity in other AZs. But there is seldom enough capacity. Inevitably in each outage there is an entry in the status updates that says “We’re adding more disks to expand EBS capacity”, or “We’re bring more systems online to make more instances available”, and so forth. You can’t really blame Amazon for this one: They can’t keep the prices they have and always be running below 50% capacity. But when lots of instances fail, or lots of disks fill up, or lots of IP addresses get allocated, there just aren’t enough left.

This is a painful side effect of forcing everyone to be centralized into the us-east1 region. us-west has us-west1 & us-west2 because the datacenters are too far apart to maintain a low-latency connection to put them into the same regional designation. us-east has a dozen or more datacenters, and thanks to them being so close, Amazon has been able to call them all ‘us-east’ instead of ‘us-east1′ and ‘us-east2′.

But what happens when a bug affects multiple AZs in a region? Suddenly, having all the AZs in a single region becomes a liability. Too many people are affected at once and they have nowhere to go. And all those organizations that have architected under the assumption that they can “just launch more instances somewhere else” are left with few options.

P.S. I know things are sounding a little negative, but stay tuned. My goal here is first to identify what are the truly dangerous issues facing AWS, and then to describe the best ways to deal with them as well as why I still think AWS is the absolute best cloud provider available.

TL;DR: There are many very real problem with AWS that are being ignored. #1 is that they appear to have no plan for dealing with their ongoing software bugs.

There has been a surprising amount of talk about the most recent AWS outage (Oct 22, 2012). In truth, I was busy coding and didn’t even hear about the outage until it was almost over (my company runs partially on AWS, but in another region). From what I read on the Amazon status site, the scope sounded pretty limited; I didn’t see it as a major event. But the talk since then says otherwise.

Amazon has now released their complete post-mortem and in reading I was struck by several hidden truths that I think many people will miss. I was an early closed beta tester of AWS (when you basically needed a personal invite to get in) and have done over 30 top-to-bottom builds of complete production app stacks while I was with RoundHouse. So I hope to provide some additional insight into what makes the last few AWS outages especially interesting.

What you should really worry about Part 1 – The bugs aren’t getting better

Widespread issues occurred because of (by my reading) 6 different software bugs. That’s a lot. This fact can be spun as both a positive and a negative. Here’s what we would assume are the positives:

The bugs can (and will) be fixed. Once they are, they won’t happen again.

By virtue of AWS’s advancing age, they have “shaken out” more bugs than any of their lesser competitors. Similar bugs surely lie dormant in every other cloud provider’s code as well. No provider is immune and every single one will experience downtime because of it. In this regard AWS is ahead of the game.

But this line of thinking fails to address the underlying negatives:

Any codebase is always growing & evolving and that means new bugs. The alarming part is that each outage seems to have some underlying bugs discovered. The incidence of Amazon letting new bugs through seems to be disconcertingly high. It does no good to fix the old ones if other areas of the service have secretly added twice as many new bugs. If things were really being “fixed” we would expect new bugs to show up less often, not more often. After a certain point, we have to start assuming that the demonstrated error rate will continue.

When bugs like this happen, they often can’t be seen coming. The impact and scope can’t be anticipated, but it is typically very large.

So far, we have not yet heard of any sort of comprehensive plan intended to address this issue. Ironically, AWS is consistently dropping the ball on their ‘root-cause’ analysis but failing the “Five Whys” test. They’re stopping at about the 2nd or 3rd why without addressing the true root cause.

In the case of AWS, anecdotal evidence suggests they have not yet succeeded in building a development protocol that is accountable to the bugs it produces. They’re letting too many through. Yes, these bugs are very difficult to detect and test for, but they need to have a higher standard or outages like this will continue to occur.

Our environment is using Ubuntu 12.04 LTS & Chef with the nginx::source & nginx::passenger_module recipes from the opscode cookbook. It turns out there were two root causes here that needed to be resolved:

Even though the config explicitly stated to use version 3.0.12, the 3.0.17 passenger gem was also getting installed. Some things were going to one place, some to another.

SOLUTION: I figured it’d just be easier to stick with the latest release. So I changed the setting to just use 3.0.17 and uninstalled the old version

The PassengerLoggingAgent was failing to be installed (but was failing silently.

SOLUTION: It turned out that we were missing some libraries. Building the passenger package manually showed the details:

Luckily, my company has premium AWS support and a quick 10 minute chat got the answer I needed. You must use the --network-attachment param, which takes the place of --group, --private-ip-address, and --subnet

I recently returned from RailsConf 2012 in Austin. I had a terrific time, learned a ton, and most importantly: I was inspired. I was inspired to return to making more contributions to Open Source and to help introduce others (kids and adults) to the computer world.

RailsConf had three double-sided whiteboards that (as far as I could tell) with nothing but “We’re hiring” ads on them. I also stopped by a job fair where there were more companies hiring than applicants. And finally, when all the attendees were asked if they worked for a company that was hiring Rails developers, at least two-thirds raised their hands!

So my first idea to get wings is a new mentorship program I’m starting in my town. This is specifically a locally-oriented program. I’m currently searching for 1-3 high school or college students who are interested in learning how to be the kind of well-rounded & highly-knowledgeable developers that these businesses are looking for.

Over about the next four months, we’ll cover the types of topics not typically covered at university, such as: git, TDD, setting up your own servers, and a complete understanding of the Rails stack. This isn’t an internship (they’re doing their own work, not work for some other company), and it’s not a class (I’m not getting paid). It’s just a chance to get some young new developers off on the right foot. There will be a ton of material to cover, and there definitely will not be a speed limit! Along the way, I hope to have each student come up with their own web app idea and fully implement it, start to finish. In the end, I hope they’ll wind up with something even more impressive than a portfolio to be able to show potential employers.

I’m pretty excited and hope to find some equally excited students. I’ll be periodically offering updates on how the program is going. I also hope to open source my lesson plan as well for use by others, so check back for more!

It’s still very new (give me a break, SES was announced yesterday!!), but I plan to get the functionality down solid and continue to support it. Of course feedback, help, and patches would be greatly appreciated!

I searched long and hard for an idea on how I could celebrate #whyday. In the end, I decided to launch a new website service that I (and I’m hoping a few others) have a need for: StatusBench.com

Status Bench is a service that provides an easy-to-use status or health dashboard. It performs the same function as http://www.google.com/appsstatus or http://status.37signals.com. This allows you to have an independent site that is not part of your normal infrastructure. When and if your site has downtime or outages, your customers will be able to rely on your status page to get updates about what is going on.

StatusBench.com screenshot

The current featureset is small, but solid:

Set up multiple status sites under your account

Use your own domain name: ‘http://status.YOUR-DOMAIN.com’

Each site can have multiple services underneath it for different parts of your app that may need independent updates

Each service can be in the states of: ‘Normal’, ‘Degraded’, ‘Offline’

Post multiple event updates about each service (mini-blog style)

I plan to add a lot of awesome new things in the future, so please look out for:

RSS Feed

History of updates / status over time

API to automatically change service status

E-mail subscriptions

SMS alerts

Twitter / Facebook / OpenID integration

Put your logo on the status page

Style your status page to your liking

Of course, any suggestions or ideas are welcome. E-mail: support ~~AT~~ statusbench.com for whatever you need and I’ll help as much as I can.

This is a first draft of the service and I intend to expand it significantly (release-early and all that). I’ll probably be adding premium options at some point as well. But for now, I hope you enjoy the service and find good uses for it.

After the analysis of several people (including myself) about the inclusion of GPL’ed WordPress code, I think the debate over Thesis has mostly subsided with Chris on the losing end. However, the reason the debate was so huge has a lot more to do with what people thought Thesis was doing and why they felt it should (or should not) be subject to the GPL. Although it turns out to be a poor test case, the fact remains that there are several heavily grey areas in the GPL, especially when dealing with dynamic, object-oriented code. Linux module developers have been dealing with these issues for a long time and there still remains a lot of questions. As a disclaimer once again:

This post now has nothing to do with Thesis. I’ll talk totally in abstract and simple examples to try and explore the GPL

I am a developer, not a lawyer, and I intend to look at these issues from a technical perspective.

The GPL is a license. It deals with copyright law. It defines who is allowed to copy & distribute the software (everyone).

The GPL doesn’t prevent you from charging for the software, but anyone you sell it to inherits the copyright and can then copy & distribute it as they see fit.

The GPL has been tested in court a few times. The biggest issue is that these cases have dealt really with embedded systems where they should have been providing the source, but weren’t. See Harald Welte vs. Sitecom, gpl-violations.org vs. D-Link, and BusyBox vs. Monsoon Multimedia.

The GPL stood up well in these tests. However, they did little to answer the fundamental question of what constitutes a derivative work of source code when dealing with dynamic & scripted code.

What is a derivative work?

Basically, anytime you copy and modify something that is copyrighted, you are producing a derivative work. Let’s start with what US copyright law defines as a derivative work: 17 U.S.C. § 101 says ”

“derivative work” is a work based upon one or more pre-existing works, such as a translation, musical arrangement, dramatization, fictionalization, motion picture version, sound recording, art reproduction, abridgment, condensation, or any other form in which a work may be recast, transformed, or adapted.”

It seems clear and obvious that source code can produce a derivative work, the key being is it “recast, transformed, or adapted“? Please note that these are legal terms, not computer coding terms (which don’t mean the same thing). US Copyright Office Circular 14: Derivative Works goes on to say:

A typical example of a derivative work received for registration in the Copyright Office is one that is primarily a new work but incorporates some previously published material. This previously published material makes the work a derivative work under the copyright law. To be copyrightable, a derivative work must be different enough from the original to be regarded as a “new work” or must contain a substantial amount of new material. Making minor changes or additions of little substance to a preexisting work will not qualify the work as a new version for copyright purposes. The new material must be original and copyrightable in itself. Titles, short phrases, and format, for example, are not copyrightable.

So what’s clear here is that 1) You must have incorporated the original material in some way and 2) The original material must be significant (short lines of code, common tidbits, etc don’t count). This is where the Abstraction/Filtration/Comparison test comes in.

Now let’s look at what the GPL says is a derivative work. My examples will be with regards to writing plug-ins, themes, and extensions. This is where the Thesis debate was originally derived and where there still exists a lot of great area. From the GPL FAQ‘s:

If the program dynamically links plug-ins, and they make function calls to each other and share data structures, we believe they form a single program, which must be treated as an extension of both the main program and the plug-ins. This means the plug-ins must be released under the GPL or a GPL-compatible free software license, and that the terms of the GPL must be followed when those plug-ins are distributed.

If the program dynamically links plug-ins, but the communication between them is limited to invoking the ‘main’ function of the plug-in with some options and waiting for it to return, that is a borderline case.

Even the GPL writers acknowledge how grey this area is. As we’ll see, I think their interpretation already over-reaches the legal limits of copyrights. It’s also important to read the FSLC’s opinion with regard to WordPress Themes:

The PHP elements, taken together, are clearly derivative of WordPress code. The template is loaded via the include() function. Its contents are combined with the WordPress code in memory to be processed by PHP along with (and completely indistinguishable from) the rest of WordPress. The PHP code consists largely of calls to WordPress functions and sparse, minimal logic to control which WordPress functions are accessed and how many times they will be called. They are derivative of WordPress because every part of them is determined by the content of the WordPress functions they call. As works of authorship, they are designed only to be combined with WordPress into a larger work.

Some Simple Test Cases

So let’s look at some simple code examples and see where things break down. I’ll provide three pieces of code:

Able has written:

[sourcecode language=”ruby”]
#Able has released this code under the GPL
class Foo
def perform_work
puts "Able’s Foo has performed work"
end
end
[/sourcecode]

Baker has written:

[sourcecode language=”ruby”]
#Baker has licensed this privately and very strictly
class Foo
def perform_work
puts "Baker’s Foo has performed work"
end
end
[/sourcecode]

First a sticking point: WordPress calls include() to include a theme. My example has the supposedly derivative work calling the dependency. I don’t believe the flipflop is relevant because the GPL FAQs use the same answer for both scenarios. I’ve done it this way for simplicity and ease-of-understanding.

So is Charlie’s work derivative of Able’s? It’s dependent on either Able’s or Baker’s to be sure. but derivative?
The key argument for themes is “They are derivative of WordPress because every part of them is determined by the content of the WordPress functions they call”. This is almost entirely the definition of inspiration. What seems to matter is “Did Charlie refer to Able’s code or Baker’s code when developing his own code?”. However, I believe this argument breaks down as well.

Inspiration

When you’re talking about a creative work, inspiration is a key factor. Fan fiction runs up against this wall. Fan fiction is derivative because someone else’s intellectual property (their characters, settings, plot lines, etc) have been used to create your work. Fan fic has been consistently found to be derivative and subject to copyright claims. But there is a very big difference between Charlie’s work and a piece of fan fiction. When you sell a fan-fic book, the original author’s characters are actually incorporated into what you’re selling. However, Charlie’s work does not significantly incorporate any bits of Able’s code, at least at the time of distribution. The key is that the in-memory combination occurs with the end user, who is not distributing ANY code and therefore is not required by the GPL to do anything. What you incorporate into your distribution is really the issue, not inspiration.

Distribution

This becomes a major distinction. Charlie is not selling or distributing Able’s source, only his own. The incorporation must happen at the time of distribution. With non-compiled source code, the incorporation happens with the end user when they run the software. So here is a more apt analogy: Monopoly the board game. Let’s say I create my own board for a board game, with its own set of rules and no shared ideas (beyond those common to all board games, such as having a big card board piece on which you play). Now I wish to sell this board and only the board. In order to use my board game, I instruct you the end user to use the pieces from a Monopoly game because they work very well with my game. My game was even designed with using these exact pieces in mind (I was inspired by Monopoly pieces, although other pieces COULD work…that’s totally up to the people playing). However, you must obtain the Monopoly game on your own. My board game will not work without Monopoly pieces, so it is dependent but not derivative. The two pieces are combined at someone’s kitchen table to produce the final product. So it is clear that having distributed only my own original work, regardless of where my inspiration came from, no copyright is inherited from the original product.

Interoperability

So Charlie’s code contains no code from Able. Charlie wrote it all himself. Copyright law seems to stop right there. Having failed a test for substantial similarity and a lack of distribution of anything under the GPL, the GPL can exert no copyright over Charlie’s code. In fact, this is a copyright issue that has been settled many times. In Lewis Galoob Toys, Inc. v. Nintendo of America, Inc., or the GameGenie case, the court decided that interoperability does not create a derivative work. Much like our coding examples and WordPress themes, the GameGenie wouldn’t work without a Nintendo. It was once again clearly dependent, but not derivative. This argument becomes even clearer when we refer only to binaries.

Summary

I think in the end it’s clear that the GPL over-reaches in its determination of what constitutes a derivative work. This is where the “infectious” moniker comes from. It tries to ‘claim’ copyright over completely original works that it has no authority over. My examples deal with the very precise cases of dynamic scripting (non-compiled) code, distributed via source only, where no shared code exists between the GPL’ed and non-GPL’ed product, so please be cautious about reading into the too much with regards to things like Linux Kernel Modules or the like. Finally, I’m really enjoying the overall debate (I do love the GPL), and the great attitude everyone I’ve talked to (on both sides) has taken. Let’s continue that, for the good of all OSS.

Introduction

There has come to be a huge debate regarding whether the Thesis WordPress theme can be premium licensed when the WordPress code itself is released under the GPL. The GPL requires that any ‘derivative work’ must also be licensed under the GPL, so the raw question is whether or not Thesis is a derivative work.

First some disclaimers:

My blog is hosted on wordpress.com. I even pay them a little for some premium features. But I otherwise have no affiliation with either WordPress or Thesis

I’m not a lawyer, I’m a developer. My views here are my own and are based on TECHNICAL knowledge and experience with the GPL, not on the law (which astute observers will note often does not reflect real life)

So is Thesis a derivative work? Wordpress and the Software Freedom Law Center think so. But their claim is based on

“The template is loaded via the include() function. Its contents are combined with the WordPress code in memory to be processed by PHP along with (and completely indistinguishable from) the rest of WordPress. The PHP code consists largely of calls to WordPress functions and sparse, minimal logic to control which WordPress functions are accessed and how many times they will be called. They are derivative of WordPress because every part of them is determined by the content of the WordPress functions they call.”

This seems extremely far reaching. My viewpoint is based on Why the GPL does not apply to premium wordpress themes. The long and short is that SFLC’s opinion could be applied to any software that runs on Linux. Meaning you could never have a closed-source software product running on the linux kernel (“Oh, your code calls fork()? GPL!”). It is commonly accepted that simply integrating with an existing product does not produce a derivative work. If your code is totally your own, the GPL has no say over how you license it. This is actually an argument about fair-use far outside just the GPL and has been settled on many different topics including OEM car parts, Nintendo, iPod connectors and other questions of being allowed to build something that interoperates with someone else’s product. Per the GPL itself a derivative work is: “a work containing the Program or a portion of it, either verbatim or with modifications” so you must copy actual lines of code from their source to be a derivative work. Simply calling WordPress functions doesn’t cut it.

The Question

The problem is then simply one of analysis: Does Thesis contain code from from WordPress? I wrote a quick script to find out. Here are the basics:

The script takes every line of WordPress source and puts it into an in-memory hash.

Every line is lower cased and has all whitespace removed to prevent missing matches from simple indentation or capitalization changes

This hash is then checked against every line in the Thesis source

It checks only PHP files (for simplicity…avoiding images and such)

It excludes lines less than 20 characters long: This could cause it to miss matches, but also helped to filter a lot of stuff like ‘<?php‘ lines

It will fail to find code lines that have been modified

So the short of it is that the script can easily detect wholesale copying, but can’t prove that code wasn’t copied and then modified. However, I think it serves its purpose. See the bottom of the post for the code (GPL’ed) and instructions on running it yourself. The results are extensive because many small lines are similar, although many are insignificant. For example:

This is a line you would see in any PHP code where you need to check if the download parameter was in the GET request. It shows up as a match, but is irrelevant. In the end, some common sense and technical knowledge must be applied to know if the results are significant.

The Results

My conclusion is that Thesis does contain GPL licensed code from WordPress. There were several examples that fit, so I’ve chosen the strongest one here that is sufficient to show that the code has been reused. ONE OF the functions in question is:

Not every line is sequential, but these two functions match pretty closely. I feel comfortable that this section of code is very clearly in the GPL, and so I am posting a portion of it here:

WordPress/Thesis diff

A section from the start of the function in WP was removed. But then the rest of it is nearly identical. Where the lines do not match exactly, their differences are insignificant and clearly show that the original method was copied as a template.

So what does this mean? I’d say it’s clear that Thesis uses GPL’ed code from WordPress and is therefore subject to the GPL as well. This makes the whole issue of whether “calling functions” or “running in memory with” requires the code to be subject to the GPL completely irrelevent. Whether or not this means the ENTIRE Thesis codebase must be GPL’ed I can’t say.

The Code

The Perl script for doing this comparison is available at: http://gist.github.com/477060. You must have a subdirectory with the WordPress source called ‘wordpress’ and a subdirectory with the Thesis source (which is not freely available, so you must have paid for it) in ‘thesis_17′. I used the Thesis 1.7 source code and WordPress 3.0 (latest) for my analysis.

Closing

I don’t have any cruel feelings towards Chris Pearson and this isn’t about who is a jerk or any of the other flaming going on. But as developers, it’s critical that you are very careful when using and re-selling code to follow the license agreements that we all adhere to. Thesis clearly has a ways to go in this regard.

I encourage constructive comments, forking of my code, and additional analysis. If you’re interested in how the GPL over-reaches in cases where there is no copied code, please read about What is a derivative work?