Tuesday, July 29, 2014

This post was inspired by a terrific new and important paper that speaks to the role of Saharan dust, the iron and other mineral micronutrients it carries to the ocean, and how this results in a new explanation of the power and potency of ocean photosynthesis in regulating global CO2. Once again we find that the living ocean is not merely a chemical test tube but rather a complex living ecology. But sometimes the chemistry shines through as well or I should say the bio-chemistry.

In short the authors show that when Sahara dust arrives in the Bahamas cyano-bacteria, what we used to call blue-green algae, bloom. As they bloom their photosynthesis removes CO2 from the water making the pH locally rise, alleviating ocean acidification. That blooming rise of ocean pH to a slightly more alkaline state results in what the Bahamanian’s have long called “Ocean Whitings” where the ocean becomes white like milk.

A one hectare site outside Doha, Qatar, will soon host the SFP AS Pilot Plant. It will contain a unique combination of promising environmental technologies carefully integrated in a system to maximize beneficial synergies. A wide specter of leading experts and technology developers are taking part in the effort to realize the project. A cornerstone of the pilot is greenhouses utilizing seawater to provide cool and humid growing conditions for vegetables. The greenhouses will also produce freshwater themselves. The greenhouses will be coupled with a state of the art parabolic trough solar collector with a thermal desalination unit supported by PV-technology. The pilot will also allow for cultivation of algae in a system of photobioreactors and open pond cultivation systems.

An important part of the pilot is to demonstrate the potential for cultivating desert land and making it green. Outdoor vertical evaporators will create sheltered and humid environments for cultivation of plants. Additionally, the pilot will contain outdoor hydroponic raceways for cultivation of halophytes – plants tolerant of irrigation with salty water. The facilities in the pilot plant will be supported by on-site laboratories, scientists and professional growers.

I have this feeling that the Sahara Forest Project team meant "wide spectrum", not "wide specter", yes? Or maybe they wanted to say "broad coalition"?

The John Muir Wilderness Area is the most spectacular of all the central Sierra Nevada wilderness areas, and is also the location of the world famous John Muir Trail, surely the most amazing backpacking trail in the lower 48 states. Our trip didn't actually take us on the John Muir Trail, but we came very close to it, and there were many hikers on our trail headed to and from the JMT/PCT (in this area they are the same trail).

Although our hike was short, the elevation gain and overall altitude was substantial: the Onion Valley trailhead is at 9,200 feet, and our campsite was at 10,600 feet; with the various ups and downs of the trail we found that Deb's FitBit registered an astonishing 180 staircases at the completion of the first day's hike.

So we were good and tired at the end of the first day!

We were lucky enough to have clear skies, fair weather, and a new moon, giving us near perfect star gazing and an active evening debate about whether or not we were seeing Iridium Flashers.

On the second day we took a cross-country scramble to nearby Bench Lake, a gorgeous hidden lake which is about 300 feet above Matlock Lake. Bench Lake was beautiful and secluded, and the views were marvelous.

On our third day, we took a trip up and over Kearsarge Pass. At 11,800 feet, this pass was the highest I've been on foot in many years, perhaps decades. Sitting at the pass is an astonishing spectacle, as you can see, simultaneously, more than 60 miles to the east, across the Owens Valley and beyond, and more than 25 miles to the west, down through Kings Canyon National Park and beyond.

The trail to the pass is well-maintained and pleasant (it has to be, as it is a major mule train pack trail), but the conditions at the pass are not suited for long periods of relaxation; as another hiker at the pass commented, "above 10,000 feet in the Sierras there are only two situations that apply: either the sun is out and you are too hot, or there are clouds and you are too cold."

Indeed it is true.

Some of our party extended the hike by dropping down to visit the Kearsarge Lakes and then returning via the pass, while I made it a shorter day by returning straight to Matlock Lake where I had time for an afternoon dip in the lake.

Mountain yellow-legged frogs in the California Sierra Nevada are disappearing at an alarming rate, primarily due to a virulent fungus. UCSB scientists Cheryl Briggs and Roland Knapp are racing to understand why some populations of frogs succumb and others survive, with the aim of not only saving the frogs but also gaining knowledge of how and why organisms develop resistance to virulent pathogen attack.

A final decision on the critical habitat proposal is expected to be made early next year, but the proposals were met with opposition in the Eastern Sierra, where residents said there is a fear that the designations will close off backcountry access (or at least appear to) and negatively impact the local, tourist-based economy.
Some of those fears were validated during the public comment process when it was brought to light that the California Department of Fish and Wildlife had been removing trout from backcountry lakes for several years in an effort to protect the frogs and prevent an endangered species or critical habitat listing.

With fishing season kicking off today, anglers seeking solitude in the remote reaches of the Sierra Nevada are being advised that several lakes that were once thriving fisheries have been cleared of all trout to protect the frogs, which are eaten by the fish in their tadpole form.

Sure enough, there was not a fish to be found in the lakes we visited. As the Inyo Register observed:

In the Independence area, the DFW has removed trout from Bench, Matlock and Slim lake but the higher-elevation waters continue to produce trout. "We have no plans to remove fish from any of those."

It's not clear which higher-elevation waters they mean.

But we definitely saw lots of tadpoles and frogs, so that part of the program is certainly working!

Although we might have wished for a bit more solitude, overall this was a near perfect trip for us: the weather was perfect, the scenery was glorious, everything went just as you would hope a backpacking trip would go.

If you're looking for a great place to go backpacking, and you haven't yet tried the Kearsarge Pass trail out of the Onion Valley Trailhead, you should put it on your list.

Now other, similar systems are starting to emerge, demonstrating that one way to get attention in the world is to hire some Google engineers who worked on-or-near Spanner and/or F1 and/or Paxos, and build something yourself.

Under development for the past two years, Impala is a means of instantly analyzing the massive amounts of data stored in Hadoop, and it’s based on a sweeping Google database known as F1. Google only revealed F1 this past May, with a presentation delivered at a conference in Arizona, and it has yet to release a full paper describing the technology. Two years ago, Cloudera hired away one of the main Google engineers behind the project, a database guru named Marcel Kornacker.

Raft is a consensus algorithm for managing a replicated log. It produces a result equivalent to (multi-)Paxos, and it is as efficient as Paxos, but its structure is different from Paxos; this makes Raft more understandable than Paxos and also provides a better foundation for building practical systems.

Ark is an implementation of a consensus algorithm (also known as elections) similar to Paxos and Raft that we are working on to handle replica set elections and failovers in TokuMX. It has many similarities to Raft, but also has some big differences.

Ark was designed from first principles, improving on the election algorithm used by TokuMX, to fix deficiencies in MongoDB’s consensus algorithms that can cause data loss. It ultimately has many similar- ities with Raft, but diverges in a few ways, mainly to support other features like chained replication and unacknowledged writes.

But if anyone is up for the challenge of rebuilding Spanner—one of the most impressive systems in the history of computing—it’s the CockroachDB team. Many of them were engineers at Google, though none of them worked on Spanner.

Cockroach is a distributed key/value datastore which supports ACID transactional semantics and versioned values as first-class features. The primary design goal is global consistency and survivability, hence the name. Cockroach aims to tolerate disk, machine, rack, and even datacenter failures with minimal latency disruption and no manual intervention. Cockroach nodes are symmetric; a design goal is one binary with minimal configuration and no required auxiliary services.

Single mutations to ranges are mediated via an instance of a distributed consensus algorithm to ensure consistency. We've chosen to use the Raft consensus algorithm. All consensus state is stored in RocksDB.

Friday, July 18, 2014

Yes, I'm asking a wave to tell me where it was born. Can you do that? Crazily enough, you can. Waves do have birthplaces. Once upon a time, one of the world's greatest oceanographers asked this very question.

Munk's experiment was not easy to carry out:

From a beach you can't see an old set of swells go by. They aren't that noticeable. Walter and his team had highly sensitive measuring devices that could spot swells that were very subtle, rising for a mile or two, then subsiding, with the peak being only a tenth of a millimeter high.

But what a fascinating result:

The swells they were tracking, when they reached Yakutat, Alaska, had indeed traveled halfway around the world. Working the data backward, Walter figured that the storm that had generated those swells had taken place two weeks earlier, in a remote patch of ocean near a bunch of snowy volcanic islands — Heard Island and the McDonald Islands, about 2,500 miles southwest of Perth, Australia.

Neat article, and neat to learn about Professor Munk, who I hadn't known of previously.

Wednesday, July 16, 2014

Eveans and his team analyze more than 7,000 securities worldwide but only buy names that offer payouts no less than double the yield of the overall stock market — as well as reasonable valuation and competitive advantages that will keep earnings growing over time.

Sounds like a pleasant article to read, no?

Well, it turns out that the companies that they are recommending you invest in are:

Cigarette companies (Philip Morris)

Oil companies (Vanguard Resources, Williams Partners)

Leveraged buyout specialists (KKR)

I guess the good news is that they didn't include any arms dealers or pesticide manufacturers.

Forking, on the other hand, is an operation which is used by a certain git workflow, made popular by GitHub, called the Fork and Pull Workflow:

The fork & pull model lets anyone fork an existing repository and push changes to their personal fork without requiring access be granted to the source repository. The changes must then be pulled into the source repository by the project maintainer. This model reduces the amount of friction for new contributors and is popular with open source projects because it allows people to work independently without upfront coordination.

The difference between forking and cloning is really a difference in intent and purpose:

The forked repository is mostly static. It exists in order to allow you to publish work for code review purposes. You don't do active development in your forked repository (in fact, you can't; because it doesn't exist on your computer, it exists on GitHub's server in the cloud).

The cloned repository is your active repo. It is where you do all your work. But other people generally don't have access to your personal cloned repo, because it's on your laptop. So that's why you have the forked repo, so you can push changes to it for others to see and review

In this workflow, you both fork and clone: first you fork the repo that you are interested in, so that you have a separate repo that is clearly associated with your GitHub account.

Then, you clone that repo, and do your work. When and if you wish, you may push to your forked repo.

One thing that's sort of interesting is that you never directly update your forked repo from the original ("upstream") repo after the original "fork" operation. Subsequent to that, updates to your forked repo are indirect: you pull from upstream into your cloned repo, to bring it up to date, then (if you wish), you push those changes into your forked repo.

When a repository is cloned, it has a default remote called origin that points to your fork on GitHub, not the original repository it was forked from. To keep track of the original repository, you need to add another remote named upstream.

In Stash, clicking the ‘Fork’ button on a repository creates a copy that is tracked by Stash and modified independently of the original repository, insulating code from the original repository to unwanted changes or errors.

if you want to make changes to any of its cookbooks, you will need to fork the repository, which creates an editable copy of the entire repository (including all of its branches and commits) in your own source control management (e.g. GitHub) account. Later, if you want to contribute back to the original project, you can make a pull request to the owner of the original cookbook repository so that your submitted changes can be merged into the main branch.

Since we’re starting fresh with this data center, we wanted to come up with our own naming scheme to address the common problems we’ve seen elsewhere. We gleaned ideas from numerous sources such as data published by large-scale companies, various RFC‘s on the topic, and blog/forum posts a’plenty. Taking all of that into account, we’ve developed some best practices that should work for most small-to-medium businesses naming their own hardware.

A common question that comes up when people start tuning queries is “why doesn’t this query use the index I expect?”. There are a few myths surrounding when database optimizers will use an index. A common one I’ve heard is that an index will be used when accessing 5% or less of the rows in a table. This isn’t the case however - the basic decision on whether or not to use an index comes down to its cost.

This paper identifies three basic assumptions made by the cost based optimizer in the estimation of cardinalities of the results of relational operations on the base and intermediate row sources and ultimately the query result set.

Booger Hunt. George Lucas avoiding tax penalties. Monkey Island and dependency charts. The Lost Patrol. A file drawer full of crazy ideas. This is the story about life at Lucasfilm Games - as told by the people who lived it.

While the field of PCG is mostly based on AI methods, we want to set it apart from the more “mainstream” use of game-based tasks to test AI algorithms, where AI is most often used to learn to play a game.

Thursday, July 10, 2014

I have to admit to at least a bit of sadness that Arjen Robben won't be in the final. I have this weird love/hate relationship with Robben: he is just phenomenally skillful, and he runs like a maniac for the full 90 minutes (I said to a friend: "he's like Michael Bradley, but with the perfect first touch").

But then he channels his inner Leonardo DiCaprio, and I just throw up my arms.

Barring the few thousand overjoyed Germans there was an atmosphere of stunned, disbelieving horror in that stadium that has possibly never before been experienced in sport. It was as though Germany had gathered 60,000 4-year-olds together and briskly announced that there is no such thing as Santa Claus.

As I mentioned, however, the Elo system discounts lopsided victories. Since it was the lopsidedness of the scoreline that made Tuesday’s match such an outlier, that somewhat defeats our purpose of placing the result in historical context.

This should be regarded as one of the most historic defeats football as seen: the hosts, pre-tournament favourites and the most successful side in the history of the World Cup humbled 1-7 in their own country, in the semi-final. Everyone is wise after the event, and many will suggest Germany were always likely to win, but in reality, with the bookmakers had Germany and Brazil at exactly the same odds to triumph. This was considered 50:50, and expected to be a tight, tense game...

Further down this week’s road we will turn our thoughts to the brilliance of this Germany side, and how they have shown the rest of the world the right path to youth development. But first there is much more angst to seep out of Brazil. Social equilibrium always appeared dependent on the team’s ability to go on winning games. Scolari’s promise to bestow a sixth world title on his people was meant to calm the nation’s nerves. It reads now like a rhetorical leap off a cliff.

Over the next 90 minutes, in perhaps the most surprising, jaw-dropping result in World Cup history, Brazil were demolished 7-1 by a rampant Germany side, as a combination of woeful organisation, shoddy defending, individual mistakes and incisive attacking (the Europeans deserve some credit, after all) sent the tournament hosts out of the competition with their tails firmly between their legs.

This was scarcely believable stuff, even as it happened in front of the world’s eyes. To put it in some type of context, this was Brazil’s first competitive defeat on home soil since 1975—a 3-1 loss to Peru that also happened in Belo Horizonte’s Estadio Mineirao. It was the first time they had conceded four goals since a 4-2 loss to Hungary in the 1954 World Cup.

The first time Brazil had ever conceded seven goals in a World Cup match. It has only conceded more once in any fixture, an 8-4 loss to Yugoslavia in a friendly in 1934.

Various publications have attempted to frame this in historical terms by comparing events of similar magnitude.

I have one to offer.

It happened in 1940, which was a long time ago (75 years ago!). There are probably very few people alive who remember this game, and certainly it was 25 years before my time (all I knew about Sammy Baugh came from a dog-eared, flimsy paperback book that I used to read at night before I went to bed): 1940 NFL Championship Game

The game was played at Griffith Stadium in Washington, D.C. on December 8, 1940. The Chicago Bears defeated the Washington Redskins, 73–0, the most one-sided victory in NFL history. The game was broadcast on radio by Mutual Broadcasting System, the first NFL title game broadcast nationwide.

...

the Chicago Bears played perfect football for a greater percentage of the official hour than any team before or since. In the championship game, as an underdog to the team which had just beaten them, the Bears made an eleven-touchdown pile and used it as a pedestal to raise the NFL to view in all corners of the country.

It's not a great comparison, because it was just the United States.

The 2014 Brazil-Germany semi-final, my friends, was the most shocking sporting event that has been played in

the entire world

I've really enjoyed this World Cup, and I hope you did, too.

Next week, I promise, I'll get back to All Those Other Things That Matter To Me.

Straightforwardly, it is a tale of a financiers and real-estate developers in Ireland during the great real estate speculation that preceded the credit crash of 2008. Together, Tristram St. Lawrence and Desmond Hickey purchase land for redevelopment:

He drove in the direction of the castle but pulled in at the old cement factory, which was located a few hundred yards shy of my gateposts and on the other side of the road. Only in Ireland would the acreage flanking a white sand beach be zoned for industrial use.

...

'this is me next project. I'm developing it for residential an commercial use.' He pointed out through the passenger window. 'There'll be an apartment block here,' we rolled along, 'an another here, an two over there. Eight blocks in total, ranging in height from three to eight storeys. We're looking at the guts of 400 residential units, with about 12,000 square metres of office an commercial space at ground level, to include a hotel.'

The tale of the chaos, ecstasy, and frenzy of those days is marvelously handled. Kilroy's light touch spirits us along through the absurdity of it all:

'You can't build an apartment complex in Ireland without a hotel.'

'Don't be ridiculous. Of course you can.'

'Ah see,' he said, 'you can an you can't. No investor will touch you unless you qualify for Section 23-type reliefs.'

'Section 23?'

'Tax write-offs. So we have to build either a hotel or a multi-storey car park or a hospital or a student residence. None a which are needed, but the way I see it, if you build a hotel, then at least you have a bar.'

At a slightly deeper level, The Devil I Know is about the power of greed and the tenacity of addiction. Tristram is an alcoholic, struggling with his addiction:

Engine rise, engine fall, engine rise, engine fall as Hickey turned memories over in his head. He swilled back another snifter and smiled at some recollection. Ahhhhh. He had entered the first stage of inebriation, which is perfect, just perfect: it is heaven. That mellow, impeccable stage when nothing can harm you. The log fire is crackling away on the inside and you are safe in your warm little snug. I'd have given my right arm to feel that way again. But I couldn't give my life.

But now Tristram is struggling with a new addiction as well.

My name, the zeros, my name, the zeros -- my eyes cranked up their shuttling. Money disrupts the cognitive process. It gums electrodes to your skull and scrambles your brain. That document was a test, I see now, of my character. A test I failed. Tristram St Lawrence I wrote at the bottom of the page. Everyone has a price.

As it turns out, as Kilroy observes, as Tristram sees as he enters this new phase of his life, he's not the only one struggling with this new addiction. The entire world, it seems, is on the verge of succumbing:

I looked at him. He believed it. All of them around the boardroom table had believed it too. They believed that the land had changed, and that they, the Golden Circle, were the agents of this change, that somehow, by linking hands around a table, or through the appliance of their balls, they had managed to perform alchemy upon Irish soil. Hickey grinned as he contemplated the open road stretching before us. Every light ahead had turned to green.

At yet a deeper level, something more mystical and meaningful is going on. This story is populated by angels and demons, by the living but also by the dead, by visions, by puzzles, by riddles. There are exultations of the highest joy, and there are nightmares of the deepest horror. What is real, what is fantasy?

To some extent, Kilroy is playing and dancing and spinning a tale, but at another level I think she is drawing a parallel, a metaphor. Real estate speculation requires a certain suspension of disbelief: you have to see a house where there isn't one; you have to see a family where only dirt exists; you have to see an alternate future instead of an old cement factory. This listening to the voices and visions in your head might be delusional and insane, but on the other hand it might be the creative process at work. Where is the line?

In the end, I think Kilroy sees herself as an observer, a journalist, a recorder of a strange time, a bizarre occurrence that came and went, and perhaps we will never come to understand it.

The historian squinted at the setting sun. I was stricken by an overwhelming sense of things coming to an end, of the torch being passed on, or not passed on, just extinguished. 'It's getting late,' he told me, barely telling me at all. 'It is time to leave the garden.'

I found myself at a loss and looked around frantically. Quite what I was searching for, exactly, I still do not know, and I possibly never will know, but I felt certain that I was forgetting something, that I was leaving some critical belonging behind, some vital possession without which everything, everything, everything would go awry. I appealed to the historian. 'Now, you mean?' I asked him, panic surging up my through. Doom, doom went my heart. 'Do you mean we're leaving now?'

'Yes, now, I'm afraid.'

I was afraid too. Afraid and unprepared. I glanced up. The sky was rapidly dimming.

He guided me to the exit -- or was it the entrance, and if so, the entrance to what? -- and he extended a crooked hand when we reached the crooked stile.

I, too, don't know what this is the entrance to, but I'm entranced to let Kilroy guide me there.

Kilroy is a wonderfully fluid, gifted, and skilled writer. Her writing is trememdously literate, weaving in references to Joyce, Faust, Shakespeare, the Bible and so much more. But her story is never ponderous and sluggish. The book was a joy to read.

If you're looking for a fun summer read, put Claire Kilroy's The Devil I Know on your list.

Friday, July 4, 2014

By now I’ve studied nearly every aspect of Messi’s game, down to a touch-by-touch level: his shooting and scoring production; where he shoots from; how often he sets up his own shots; what kind of kicks he uses to make those shots; his ability to take on defenders; how accurate his passes are; the kind of passes he makes; how often he creates scoring chances; how often those chances lead to goals; even how his defensive playmaking compares to other high-volume shooters.

And that’s just the stuff that made it into this article. I arrived at a conclusion that I wasn’t really expecting or prepared for: Lionel Messi is impossible.

It’s not possible to shoot more efficiently from outside the penalty area than many players shoot inside it. It’s not possible to lead the world in weak-kick goals and long-range goals. It’s not possible to score on unassisted plays as well as the best players in the world score on assisted ones. It’s not possible to lead the world’s forwards both in taking on defenders and in dishing the ball to others. And it’s certainly not possible to do most of these things by insanely wide margins.

But what about those two uncredited stories? Newspaper byline rules could be stringent and arbitrary, and these were different times. Maybe the Post-Dispatch withheld credit from McSkimming because he was on vacation. Maybe the stories were too short to merit a byline. But McSkimming had joined the paper in 1922. Wouldn’t he have wanted credit for his work, and wouldn’t the paper have wanted to showcase the fact that its star soccer writer was on the scene?

Then came an inspired USA fightback. Substitute Julian Green, on for Bedoya, scored a consolation goal and his side rallied. Those final 13 minutes were extremely impressive, with the goal seemingly giving the USA another gear and simultaneously making Belgium even more tired.

Bradley, Jones and Cameron, so frustrating in the first half because of their collective insistence on running high up the pitch and leaving space in behind, were now perfect for the situation – a desperate fightback. Witsel and Marouane Fellaini looked exhausted, and Wilmots’ refusal to bring on another midfielder, or even strengthen his defence, was remarkable. He left his side open to constant attacks, and few other sides have exited the competition with such an impressive late rally as this from the United States.

Of course, if three or four Swiss players were watching Messi, it means that there was space elsewhere on the pitch for Argentina to exploit. But Argentina had not exploited that space, because it seemed that all 10 of their players were watching Messi, too. All of their attacking play was directed through the No. 10, to the exclusion of any other ideas. Every time we recover the ball we try to pass to him, as he is the best player we have in the team and he will score goals.

There's no question that MLS went out to buy as many US National Team players as they could last year. The reasoning behind it is simple; Bring the hordes of USMNT supporters that pack international events over to MLS. Jury is out on this reasoning. Michael Bradley, one of the new signings that's making more than all but four whole teams, played in front of just 12k fans at Crew Stadium last week.

Six figure salaries among USMNT players, even historically is not rare. Going back to 2006, each World Cup squad has only had one MLS player making under the six figure mark: Robbie Findley in 2010, and Clint Dempsey in 2006 before he left for England.

As the league gets stable, there's some risk that American stars will have to choose between getting guaranteed starts in a country they know well for the same kind of money they would make in a foreign land where they might not prosper. For the leading soccer leagues - England, Germany, Italy, Spain - the interest in national team players staying at home is not so important for the health of the national team, as those players that do stay home (and this includes the vast majority of them) face elite competition when they play for their native clubs. For countries with only okay, but well-heeled leagues - Mexico, Russia, and now, maybe, the United States - the fact that the best players can make the best money if they stay at home arguably retards progression, rather than encouraging it.

Etched into the pale caliche soil, some are hundreds of feet long and depict human and animal shapes; others show intricate spiral and zigzag patterns. Only Peru's famed Nazca Lines are believed to hold a larger trove of land drawings.

Most consider the geoglyphs a mystery. But Figueroa thinks he's deciphered their meaning: The glyphs, he says, interweave with the region's natural features to tell the creation story of the ancestors of the Chemehuevi and other Uto-Aztecan speaking peoples of the Southwest. Figueroa also believes that the lost homeland of the Aztecs – la cuna de Aztlan – is here, amid the ancient washes and sun-seared mountainsides of the Lower Colorado River. He's self-published a book about it: Ancient Footprints of the Colorado River: La Cuna de Aztlan.

"Between 1928 and 1932," the film explains, "Western Union and AT&T Long Lines built two of the most advanced telecommunications buildings in the world, at 60 Hudson Street and 32 Avenue of the Americas in Lower Manhattan. Nearly a century later, they remain among the world's finest Art Deco towers—and cornerstones of global communication. Urban Giants is a 9-minute filmic portrait of their birth and ongoing life, combining never-before-seen-construction footage, archival photographs and films, interviews with architectural and technology historians, and stunning contemporary cinematography."

Akamai’s globally-distributed Intelligent Platform allows us to gather massive amounts of information on many metrics, including connection speeds, attack traffic, network connectivity/availability issues, and IPv6 growth/transition progress, as well as traffic patterns across leading Web properties and digital media providers. Each quarter, Akamai publishes the State of the Internet Report.

In this post I'll show the design of a system that beats the CAP theorem by preventing the complexity it normally causes. But I won't stop there. The CAP theorem is a result about the degree to which data systems can be fault-tolerant to machine failure. Yet there's a form of fault-tolerance that's much more important than machine fault-tolerance: human fault-tolerance. If there's any certainty in software development, it's that developers aren't perfect and bugs will inevitably reach production. Our data systems must be resilient to buggy programs that write bad data, and the system I'm going to show is as human fault-tolerant as you can get.

The Lambda Architecture is an approach to building stream processing applications on top of MapReduce and Storm or similar systems. This has proven to be a surprisingly popular idea, with a dedicated website and an upcoming book. Since I’ve been involved in building out the real-time data processing infrastructure at LinkedIn using Kafka and Samza, I often get asked about the Lambda Architecture. I thought I would describe my thoughts and experiences.

Boot parameter nohz=off disables dynamic ticks. Dynamic ticks were introduced by Linux developers to keep computers a bit cooler, and to prevent battery drain in laptops. They have little benefit for hosting servers that have significant work load most of the time, and it looks like they conflict with our CPU scheduler.

How serious are these errors? Of course, everybody overrates their own products to some extent; it would be hard to maintain enthusiasm otherwise. Still, the more extreme forms of misjudgment are fraught with danger.

Ah, I thought—this is precisely where linear algebra can come to the rescue! Just like in CLEVER or PageRank, we can begin by giving everyone in the community an equal number of “morality starting credits.” Then we can apply an iterative update rule, where each person A can gain morality credits by cooperating with each other person B, and A gains more credits the more credits B has already. We apply the rule over and over, until the number of morality credits per person converges to an equilibrium. (Or, of course, we can shortcut the process by simply finding the principal eigenvector of the “cooperation matrix,” using whatever algorithm we like.) We then have our objective measure of morality for each individual, solving a 2400-year-old open problem in philosophy.