Topics

Featured in Development

Peter Alvaro talks about the reasons one should engage in language design and why many of us would (or should) do something so perverse as to design a language that no one will ever use. He shares some of the extreme and sometimes obnoxious opinions that guided his design process.

Featured in AI, ML & Data Engineering

Today on The InfoQ Podcast, Wes talks with Katharine Jarmul about privacy and fairness in machine learning algorithms. Jarul discusses what’s meant by Ethical Machine Learning and some things to consider when working towards achieving fairness. Jarmul is the co-founder at KIProtect a machine learning security and privacy firm based in Germany and is one of the three keynote speakers at QCon.ai.

Featured in Culture & Methods

Organizations struggle to scale their agility. While every organization is different, common patterns explain the major challenges that most organizations face: organizational design, trying to copy others, “one-size-fits-all” scaling, scaling in siloes, and neglecting engineering practices. This article explains why, what to do about it, and how the three leading scaling frameworks compare.

How Heroku Manages High Availability - QCon London Talk summary

For a PaaS solution, high availability is quite a challenge, as not only hardware, but also software infrastructure services have to be available all the time. In his QCON London talk (PDF slides download), Mark McGranaghan, explained the patterns used at Heroku to provide a PaaS system which is always available.

Mark described, that in his view, those patterns fall into two categories

Architectural, being certain design concepts.

Socio-Technial, describing how people interact with the software.

According to Mark, the human factor is often underestimated, when it comes to how HA applications are deployed and operated. From the architectural side a lot of concepts have been taken from Erlang, as will become clear on some examples.

Architecture Aspects

When deployed to Heroku, applications get a lot of services from Heroku and do not need to take care of them themselves. Routing HTTP requests for example is one service that is taken care of by Heroku. In fact the application couldn't achieve it on its own, because applications running on Heroku are running in multiple instances and monitored by supervisors which can restart them. So the application does not know where it is running, but the Heroku routing mesh is aware of the location of each instance of the application and can direct the HTTP traffic accordingly. The routing mesh itself consists of many cloud instances that are also supervised and scaled or restarted if required. This follows the error kernel concept of Erlang. All services are supervised and also these supervisors are supervised. Everything is allowed to fail and will be restarted. This concept eases error handling for both Heroku and the applications running on Heroku.
Because everything is allowed to fail, applications should be designed in a graceful degrading way. Mark gave a few examples on how that looks in practice:

Reliable Messaging

In a traditional Message broker architecture, there is again a single point of failure: the endpoint where messages are sent to has to be available. Heroku uses a pattern called publish one / subscribe many, where each sender knows multiple possible message brokers. Receivers just need to subscribe to all of them. This change makes the messaging system reliable to individual broker failures.
The messages themselves are simple key value pairs, which can be easily extended, allowing upgrades of systems without downtime. Incompatible changes can be performed by using new attributes which are only used by instances of the new version, while the remaining old ones still can work with new messages passed to them.

Service unavailability

Whenever an application depends on a service to read data, it should be possible to graceful degrade when that service is not available.
Whenever data shall be written to a service, the data can be buffered when the receiving service is not available. The writing will automatically retransmit the write request when the service becomes available again. For simplicity reasons Heroku is moving most services to directly buffer before trying to send the data across. This pattern is called write de-syncronizing.
What is interesting is that they still use PostgreSQL for this intermediate buffering, not their own distributed storage called Doozer. According to Mark, the reason is that the PostgreSQL limitations are pretty well understood and there is good operational tooling and experience, which are missing from Doozer. As we covered November 2011, Heroku offers PostgreSQL even as a service to non Heroku applications.
That led to the second part of the talk, the "socio technical" aspect.

Socio-Technical Aspects

Because humans interact with the system, it is important to make it as easy as possible. The most common reasons for system problems are human interactions. In a system which is designed for change, these human interactions of course involve change a lot.

Deployments and Roll Outs

Using custom tooling, Heroku can deploy in an incremental mode. Each deployment runs through various stages. But unlike traditional stages, which are separate environments, these are subsets of the production infrastructure. Those subsets are like "internal", "beta" and "all". The same concept follows for roll out of new features. Heroku uses feature flags to enable and disable features in productions in a similar way as deployments are made. This allows also decoupling deployment and feature roll out: First a new deployment is made, delivering the same functionality, with new features being hidden behind feature flags. If it its proven ok, features can be turned of gradually, allowing much easier localization of problem points.

Visibility

Especially in a system with many moving parts, monitoring is essential. Heroku continues to invest heavily in monitoring and alerting because the visibility they provide are so valuable for production systems. Problems are detected using asserts. Examples of such asserts are

99 Percentile Latency < 50

Active connections > 10

Mark gave two examples where those asserts could have warned them about an outage when they would have been in place already at the time.

Feedback cascades

A typical problem in a distributed system is that problems in a downstream component affect everything which is invoking it. While the architecture already took care of unavailable components with graceful degradation or write de-syncronizing, there is a concept needed to avoid overloading subsystems. Heroku uses "Flow-Control" for that. Certain code paths can be limited to a maximum number of invocations. Those rates are made configurable during runtime to adjust for changed code or hardware.