Eureka! Netflix makes Amazon more reliable with open source software

Netflix built cloud load balancing tool called Eureka, gives it away.

Netflix is a big user of Amazon's cloud infrastructure services—such a big user, in fact, that it has built multiple software programs that make Amazon resources more reliable. Recently, Netflix described its "Chaos Monkey," which randomly takes Amazon virtual machines offline to help engineers identify network weaknesses. Netflix released its Chaos Monkey as open source, and this week did the same with another project, known as Eureka.

Released on GitHub and described in the Netflix Tech Blog yesterday, Eureka fills a gap in Amazon Web Services by providing load balancing and failover for middle-tier servers, Netflix explained.

"In AWS cloud, because of its inherent nature, servers come and go," Netflix's Karthikeyan Ranganathan wrote. "Unlike the traditional load balancers which work with servers with well known IP addresses and host names, in AWS load balancing requires much more sophistication in registering and de-registering servers with the load balancer on the fly. Since AWS does not yet provide a middle tier load balancer, Eureka fills a big gap in that area."

Amazon offers its own Elastic Load Balancer, but this is for services exposed to end-user Web traffic, as opposed to mid-tier services intended to be private. Netflix thinks Eureka will be useful for Amazon customers who have services they don't want exposed to traffic from the outside world.

Eureka is a REST-based service with a Java client. Besides mid-tier load balancing, Netflix uses Eureka with its Cassandra database (for taking instances offline for maintenance) and with its Memcached-based services. Netflix isn't done releasing open source software. The company said to stay tuned for Eureka integration with Asgard, a Netflix tool for managing cloud deployments, and for "more sophisticated Netflix mid-tier load balancing solutions."

It wouldn't matter if it was. They released it under Apache so there likely wasn't a legal obligation for them to do so to begin with. Software development/management would be so much better if this sort of thing were done more often by corporate developers.

In any case, I guess this slightly makes up for the refusal to support their streaming services on Linux (Android and Chrome OS excluded) as far as PR with the OSS community goes.

So why does Netflix release these things for free? Were they community/open-sourced developed to begin with?

Besides the other listed benfits, there isn't much of a downside. Competition may find this software useful, but its hardly the center of their service.

While that may be so, most companies (as far as I know) keep their tools solely to themselves. I don't really watch much TV/movies, but this sort of thing is making me consider re-subscribing just to support their approach to business.

So why does Netflix release these things for free? Were they community/open-sourced developed to begin with?

They gain goodwill, and probably extensions that they can upstream with.

And they don't lose anything. These tools are nice to have, but just make AWS a bit more resilient. So, they're being a good citizen of software.

I'm sure Amazon helps them out as well. After all it is business. Keep in mind the better AWS is the better and more reliable Netflix's service can be. Netflix wants to keep getting bigger and better but you need a solid foundation for that to happen especially when you are starting to push thresholds.

So why does Netflix release these things for free? Were they community/open-sourced developed to begin with?

They gain goodwill, and probably extensions that they can upstream with.

And they don't lose anything. These tools are nice to have, but just make AWS a bit more resilient. So, they're being a good citizen of software.

I'm sure Amazon helps them out as well. After all it is business. Keep in mind the better AWS is the better and more reliable Netflix's service can be. Netflix wants to keep getting bigger and better but you need a solid foundation for that to happen especially when you are starting to push thresholds.

Basically this. If they use AWS for portions of their service, and they come up with a tool to improve load balancing on AWS, it's in their interest to get as many people using it as possible so that AWS in general becomes more robust, and thus whatever portions of their service run on AWS become more robust.

About release the software for free. I'm also guessing they want prevent other customers of amazon to suck down resources that would cause AWS to slow down. So the benefit to them is the system stays robust. Rather than having to wait for one suck ass client who doesn't have programing ability to do the same slow down AWS under heavy load to figure out the answer.

So why does Netflix release these things for free? Were they community/open-sourced developed to begin with?

Besides the other listed benfits, there isn't much of a downside. Competition may find this software useful, but its hardly the center of their service.

Typically technology is just a tool, not a product. While advanced tools could give an edge, it is not a crutch one wants to use to maintain their advantage. Unless of course your job is making tools, but software is a different sort of tool as it is virtually free to copy.

So why does Netflix release these things for free? Were they community/open-sourced developed to begin with?

They gain goodwill, and probably extensions that they can upstream with.

And they don't lose anything. These tools are nice to have, but just make AWS a bit more resilient. So, they're being a good citizen of software.

I'm sure Amazon helps them out as well. After all it is business. Keep in mind the better AWS is the better and more reliable Netflix's service can be. Netflix wants to keep getting bigger and better but you need a solid foundation for that to happen especially when you are starting to push thresholds.

Basically this. If they use AWS for portions of their service, and they come up with a tool to improve load balancing on AWS, it's in their interest to get as many people using it as possible so that AWS in general becomes more robust, and thus whatever portions of their service run on AWS become more robust.

Also, if they *don't* release it as open source, there's a much higher chance that someone else (like Amazon) will release a different implementation to do the same thing. If the competing implementation catches on, Netflix is left maintaining a non-standard infrastructure by themselves while their competitors leverage the economies of going with the standard.

If it's not your profitable sekrit sauce, it makes a lot of sense to release it open source like this.

It wouldn't matter if it was. They released it under Apache so there likely wasn't a legal obligation for them to do so to begin with. Software development/management would be so much better if this sort of thing were done more often by corporate developers.

In any case, I guess this slightly makes up for the refusal to support their streaming services on Linux (Android and Chrome OS excluded) as far as PR with the OSS community goes.

From what I understand, this is a Hollywood limitation, not a Netflix limitation. They would like to stream it to everybody on every device, but the Studios insist that it uses DRM that is not compatible with the idea of open source software.

So why does Netflix release these things for free? Were they community/open-sourced developed to begin with?

They gain goodwill

Particularly among the developer community. Putting out really solid code for the world to use and learn from is an effective recruiting tool to reel in top talent. Smart developers want to work with other smart developers, and you can learn more about an organization's software development practices by reading their code than by talking through a few hours of interviews. We've been using their Curator library to interface with Zookeeper, and it just works.

Interesting post you link to. I find the phrase "peer pressure from social coding" an interesting concept -- I guess it works at technology companies, but I can't imagine some places I've worked being very happy hearing, "Yeah, well, the internet doesn't like our coding style, so we've had to refactor and rework the entire code base, meaning Project X will be delayed by six months. Mind you, our nerdy peers will love us. Sorry about the marketing campaign you had planned, but it's our nerd cred on the line".

I find the phrase "peer pressure from social coding" an interesting concept ... but I can't imagine some places I've worked being very happy hearing, "Yeah, well, the internet doesn't like our coding style, so we've had to refactor and rework the entire code base, meaning Project X will be delayed by six months.

Solid code is pretty universal. Code sucks or it doesn't.

The scenario in the quote implies that management is caught by surprise by the need to refactor, which means someone is not communicating need-to-know information up the food chain. There is always tension between the "incur technical debt now + fix it later" camp and the "do it right the first time" camp. Management should be the person(s) deciding which approach is appropriate in a given situation. If management finds out only just before release that the code is a bunch if crap (necessitating six months of rewrites), then someone is not doing his or her job.

I, as a developer, don't know which is worse. This:

Quote:

we've had to refactor and rework the entire code base, meaning Project X will be delayed by six months.

Or this:

Quote:

we've had to refactor and rework the entire code base, meaning Project X 2.0 will be delayed by EIGHT months.

Technical debt will be paid down, now or later. And if you pay later, you also have to deal with the interest. Management, not the developer, has the job of deciding between delay now and delay later.

I find the phrase "peer pressure from social coding" an interesting concept ... but I can't imagine some places I've worked being very happy hearing, "Yeah, well, the internet doesn't like our coding style, so we've had to refactor and rework the entire code base, meaning Project X will be delayed by six months.

Solid code is pretty universal. Code sucks or it doesn't.

The scenario in the quote implies that management is caught by surprise by the need to refactor, which means someone is not communicating need-to-know information up the food chain. There is always tension between the "incur technical debt now + fix it later" camp and the "do it right the first time" camp. Management should be the person(s) deciding which approach is appropriate in a given situation. If management finds out only just before release that the code is a bunch if crap (necessitating six months of rewrites), then someone is not doing his or her job.

I, as a developer, don't know which is worse. This:

Quote:

we've had to refactor and rework the entire code base, meaning Project X will be delayed by six months.

Or this:

Quote:

we've had to refactor and rework the entire code base, meaning Project X 2.0 will be delayed by EIGHT months.

Technical debt will be paid down, now or later. And if you pay later, you also have to deal with the interest. Management, not the developer, has the job of deciding between delay now and delay later.

So why does Netflix release these things for free? Were they community/open-sourced developed to begin with?

They gain goodwill, and probably extensions that they can upstream with.

And they don't lose anything. These tools are nice to have, but just make AWS a bit more resilient. So, they're being a good citizen of software.

Plus there's the normal potential benefits of releasing it open source, most notably community improvements to the existing software.

I hear this said many times in regards to open source, yet I also hear the counter-argument that since open source is not "owned" by one entity, many companies avoid it since they don't know if it'll be maintained nor would it have a "help desk" they could call for issues.

That's an advantage of releasing a product as open source though. You don't have to set up a help desk or guarantee to support it in the future. You just dump the code out there and let people figure it out for themselves. If Netflix had tried to productize it, they would have to charge an arm and a leg to cover the support costs given the very small number of users for this software, and frankly anybody who wants to use it is going to have to have experts in the field anyway who would probably prefer to just take ownership of the code since they're going to have to modify it anyway to suit their needs.

Plus, I doubt the documentation is very good on this project, so access to the source can be a big help. Sometimes the best documentation is just reading the source yourself.

Here's to hoping this Netflix/Amazon relationship one day leads to a Amazon Video / Netflix mashup that can take on the content industry...

I have been hoping that for some time now. All of the video content I consume now comes from Netflix or Amazon Prime so a melding of the two would be fantastic.

Ditto. Amazon could buy Netflix; that'd be just fine by me.

We're using AWS pretty heavily at my new shop, so Curator, Eureka and Chaos Monkey (in particular) are all very interesting to me.

Deridex wrote:

Technical debt will be paid down, now or later. And if you pay later, you also have to deal with the interest. Management, not the developer, has the job of deciding between delay now and delay later.

Amen. At our technology "state of the union" this morning, we decide to bite the bullet and get to working on X 2.0 in parallel with on-going maintenance on 1.x - and with a migration path to support legacy consumers of 1.x on 2.0 if necessary. The debt, as you say, will be paid down sooner or later.