Links for 2018-10-31

A network outage caused a split-brain scenario, and their failover system allowed writes to occur in both regional databases. Once the outage was repaired it was impossible to reconcile writes in an automated fashion as a result. Embarrassingly, this exact scenario was called out in their previous blog post about their Raft-based failover system at https://githubengineering.com/mysql-high-availability-at-github/ — “In a data center isolation scenario, and assuming a master is in the isolated DC, apps in that DC are still able to write to the master. This may result in state inconsistency once network is brought back up. We are working to mitigate this split-brain by implementing a reliable STONITH from within the very isolated DC. As before, some time will pass before bringing down the master, and there could be a short period of split-brain. The operational cost of avoiding split-brains altogether is very high.” Failover is hard.

Assuming this all works out, the image in this tweet is also a valid ZIP archive, containing a multipart RAR archive, containing the complete works of Shakespeare. This technique also survives twitter’s thumbnailer :P

John Lilly, a Silicon Valley-based venture capitalist with Greylock Partners and the former C.E.O. of Mozilla, said he tries to help his 13-year-old son understand that he is being manipulated by those who built the technology. “I try to tell him somebody wrote code to make you feel this way — I’m trying to help him understand how things are made, the values that are going into things and what people are doing to create that feeling,” Mr. Lilly said. “And he’s like, ‘I just want to spend my 20 bucks to get my Fortnite skins.’”

In apps marketed for children 5 and under in the Google Play store, there were pop-up ads with disturbing imagery. There were ads that no child could reasonably be expected to close out of, and which, when triggered, would send a player into more ads. Dancing treasure chests would give young players points for watching video ads, potentially endlessly. The vast majority of ads were not marked at all. Characters in children’s games gently pressured the kids to make purchases, a practice known as host-selling, banned in children’s TV programs in 1974 by the Federal Trade Commission. At other times an onscreen character would cry if the child did not buy something. “The first word that comes to mind is furious,” said Dr. Radesky, an assistant professor of developmental behavioral pediatrics at the University of Michigan Medical School. “I’m a researcher. I want to stay objective. We started this study really just trying to look at distraction. My frustrated response is about all the surprising, potentially deceptive stuff we found.”

One Comment

Boggle. So GitHub had a replication topology which did not actually increase reliability because you couldn’t use it for failover; considered 954 writes so crucial that they degraded their entire system for a day in case of the loss of those writes; did an entire restore from backup (why? that is never explained) which took many hours and made everything else take longer due to the increased backlog; and in the process were handing out outdated data to users in any case, so it appeared to users that far more than a mere 954 writes had been dropped.

This… does not seem like good site design, at all. (Also: MySQL replication sucks far worse than I knew, if this is how it handles a not very complicated short-lived split-brain scenario.)