EOF - The Limits of Scale

Linux is like limestone; you can build anything with it. So, while you find
limestone in everything from lipstick to pyramids, you find Linux in
everything from picture frames to Google.

What brings this analogy to mind is the matter of
scale, long regarded as a
virtue in the tech world. Getting to scale and staying there are both
considered Good Things. But, as with other Good Things, is it possible to
have too much? At what point do the biggest things we make with Linux risk
turning into pyramids—that is, durable landmarks that are also dead?

These questions came up for me back in January, when two things happened.
One was Larry Page replacing Eric Schmidt as Google's CEO. The other was
mysterious account deletions at Flickr. Without Linux, there would be no
Google or Flickr.

In Google's case, I saw the writing on the wall at the Techonomy conference
in Lake Tahoe, August 2010. On stage was Eric Schmidt, amid four other
panelists. In the Q&A, Eric said, “If we look at enough of your messaging
and your location, and use artificial intelligence, we can predict where
you are going to go....Show us 14 photos of yourself and we can
identify who you are.” He added:

I would make a stronger point—that the only way to meet this set of
challenges that we are facing is by much greater transparency and no
anonymity. And the reason is that in a world of asymmetric threats, true
anonymity is too dangerous....One of the errors that the Internet made a
long time ago is that there was not an accurate and non-revocable identity
management service....You need a name service for humans.....governments
are going to require it at some point.

I wanted to freeze time and say “Eric, no! Stop, big guy! Better to say
nothing than this kind of stuff!” But I just sat and winced. Two months
later in an interview with The Atlantic at the Washington Ideas Forum, Eric
said, “We don't need you to type at all. We know where you are. We know
where you've been. We can more or less know what you're thinking
about.”
Spoken like an eyeball on a pyramid.

At this point, it was just a matter of time before one of the founders would
return, Steve Jobs-like (and hopefully not Jerry Yang-like) to bring the
company back in alignment with Original Principles. That happened in
January, followed quickly by a Bloomberg Businessweek cover story titled
“Larry Page's Google 3.0”. Said the writers, “The unstated goal is to save
the search giant from the ossification that can paralyze large
corporations. It won't be easy, because Google is a tech conglomerate, an
assemblage of parts that sometimes work at cross-purposes.” The piece goes
on to profile a half-dozen “star deputies”. Of them, it says,
“Together,
their mandate is to help the company move more quickly and
effectively—to keep it from becoming yet another once-dominant tech
company that sees its mantle of innovation stolen away by upstarts.” Good
luck with that.

Flickr's first pyramid moment was a report that photographer Deepa Praveen
had her entire Pro account (the kind people pay for) deleted without
explanation. The story broke first in Thomas Hawk's blog, and then the
action moved to my own blog, with a post titled “What if Flickr
fails?”
That one racked up 107 comments, including a pair from Yahoo executives.
(Flickr belongs to Yahoo.) Nowhere was there anything to relieve fears that
an account deletion might come at any time, to anybody, with no chance of
recovering whatever was lost. (My own exposure is about 50,000 photos.)

Then Mirco Wilhelm, another Flickr Pro photographer, had his 3,400 photos
deleted, in what Flickr eventually admitted was its own error. These were
later restored, with much apologizing by Flickr. Still, one had to wonder
how much of the problem had to do with Flickr's size. According to the most
recent reports at this writing, Flickr hosts more than 5,000,000,000 photos
for 51,000,000 registered users, with new photos arriving at more than
3,000 per minute.

One of the best talks on Linux deployment was one given by Cal Henderson at
the March 2006 O'Reilly Emerging Technology Conference. It was an all-day
tutorial about “launching and scaling new Web services”. I remember being
highly impressed at how well Linux allowed a fast-growing pile of digital
goods to expand, while still providing near-instantaneous service to
everybody who wanted it. I also remember wondering what would happen after
Cal left—which he did in 2009.

Nature's idea is to take its course. It's as much Linux's nature to start
something as it is to grow to the limits of viability. It may help to
remember that limestone is made from the corpses of once-living things.
Without abundant endings, we wouldn't have beginnings.

Doc Searls is Senior Editor of Linux Journal. He is
also a fellow with the
Berkman Center for Internet and Society at Harvard University and the
Center
for Information Technology and Society at UC Santa Barbara.

Comments

Comment viewing options

Doing large scale grid work, we regularlly press various limits of Linux and other systems. If you're in a situation where you're pushing various limits Meat Slicer like open file descriptors and network sockets, here is how to ensure that the limits are large enough.