How Twitter's back-end stays up

From the Boing Boing Shop

Follow Us

Raffi Krikorian is a smart engineer and technical manager; I've know him for decades, but didn't realize that he was currently Vice President for Platform Engineering at Twitter. This interview with him at InfoQ gives a really fascinating flavor for how Twitter's reliability engineering is baked in at the systems level:

So Decider is one of our runtime configuration mechanisms at Twitter. What I mean by that is that we can turn off features and software in Twitter without doing a deploy. So every single service at Twitter is constantly looking to the Decider system as to what are the current runtime values of Twitter right now. How that practically maps is I could say the discover homepage, for example, has a Decider value that wraps it, and that Decider value tells discover whether it's on or off right now.

So I can deploy discover into Twitter and have it deployed in the state that Decider says it should be off. So this point we don't get an inconsistent state. The discover, for example, or any feature at Twitter runs across many machines. It doesn't run on one machine, so you don't want to get in the inconsistent state where some of the machines have the feature and some of them don't. So we can deploy it off using Decider and then when is on all the machines that we want it to be on, we can turn it on atomically across the data center by flipping a Decider switch.

This also gives us the ability to do a percentage-based control. So I can say actually now that it's on all of the machines, I only want 50% of users to get it. I can actually make that decision as opposed to it being a side effect of the way that things are being deployed in Twitter. So this allows us to really have a runtime control over Twitter without having to push code. Pushing code is actually a dangerous thing, like the highest correlation to failure in a system like ours, not just Twitter but any big system, is software development error. So this way we can actually deploy software in a relatively safe way because it's off. Turn it on really slowly, purposely, make sure it's good and then ramp it up as fast as I want.

Before being convicted of felony securities fraud, smirking cartoon villain pharma-douche-bro Martin Shkreli had to be tried in front of a jury and this presented a unique problem because everyone hates Martin Shkreli, and thus more than 100 jurors were dismissed from the pool during pre-trial questioning. Here are some of the statements that led […]

Stanford’s Center for Research on Education Outcomes released this study in 2015, comparing the outcomes for students enrolled in online charter schools with comparable students (controlled for grade level, gender, race/ethnicity, free lunch eligibility, English language status, special ed status and historical state achievement test scores) in brick-and-mortar classrooms.

The World Wealth and Inequality project’s latest white-paper, co-authored by Thomas “Capital in the 21st Century” Piketty, painstaking pieces together fragmentary data-sources to build up a detailed picture of wealth inequality in Russia in the pre-revolutionary period; during phases of the Soviet era; on the eve of the collapse of the USSR; and ever since.

The Pry.Me Bottle Opener holds tens of thousands of times its own weight, and you can pick one up now from the Boing Boing Store.This remarkable keychain is considerably smaller than any of your keys, but don’t let that fool you: it can easily open any bottle, and could even tow a trailer full of […]

Guaranteeing your privacy online goes way beyond checking the “Do Not Track” option in your browser’s settings. To ensure that your internet activity is totally hidden from Internet Service Providers, advertisers, and other prying eyes, take a look at Windscribe’s VPN protection. It usually costs $7.50 per month, but you can get a 3-year subscription […]

This project management bundle will help you get organized and learn how to lead a team to success. You can pay what you want for these five courses when you pick them up from the Boing Boing Store.To help you become an invaluable asset for your company, this bundle includes a curated collection of professional […]