Posted
by
timothy
on Friday October 05, 2012 @06:08PM
from the yet-they-keep-most-recent-instead-of-top-stories dept.

pacopico writes "Much has been made about Facebook hitting 1 billion users. But Businessweek has the inside story detailing how the site actually copes with this many people and the software Facebook has invented that pushes the limits of computer science. The story quotes database guru Mike Stonebraker saying, 'I think Facebook has the hardest information technology problem on the planet.' To keep Facebooking moving fast, Mark Zuckerberg apparently instituted a program called Boot Camp in which engineers spend six-weeks learning every bit of Facebook's code."

Facebook's source code is PHP. Which is then compiled into C++ (complete with all assets) and then compiled into a native binary. Linking is a huge problem as it produces a huge (multi-gigabyte) executable that is run directly.

Deployment is another issue - I believe they use a form of Bittorrent to do it, and naturally, the scripts that update from one executable to another don't work completely across the entire server farm - so those failed deploym

a) that's a terrible idea, and b) the fact that it's even possible (if it is, sounds like business magazine bs to me) speaks volumes. I only work for Red Hat, we're pretty cool but we're hardly the biggest fish out there, and you can imagine the chaos if we tried that...I'm sure others can apply it to their com

Oh yeah. I worked on an embedded project that had custom kernel code as well as over 2 million lines in system libraries. No one could possibly know every single line of that. The project I was in charge of there maybe had 200,000 lines of code, and I often had to rely on comments to remember what goes where! I had the unfortunate aspect of being the only team on an embedded processor and had to fix cross platform issues with the system libraries too. It was a lot of work.

Bfd, huh? You should drop that Zuckerberg guy a line and let him know they can just fire 3,500 of some of the finest IT staff and programmers in the world.

"I think Facebook has the hardest information technology problem on the planet," says Mike Stonebraker, a computer scientist and longtime professor at the University of California at Berkeley. "A company like Google certainly does innovative stuff, but Facebook solves the harder problem."

Each day, Facebook processes 2.7 billion "Likes," 300 million photo uploads, 2.5 billion status updates and check-ins, and countless other bits of data, and uses that mass of transactions to guesstimate which ads to serve up.

And let's not forget, it's constantly figuring out which of those items to show each of those 1 billion, variously-connect people.

Maybe you could just handle all that high frequency trading for the large exchanges in your off-hours too, for extra cash.

"Mark Zuckerberg apparently instituted a program called Boot Camp in which engineers spend six-weeks learning every bit of Facebook's code."

Ah that's Zuckerberg's secret sauce apparently, plenty of overtime for six-weeks so that a new engineer can learn every bit of Facebook's code. This way, they can push the limits of computer science [wikipedia.org] (or disregard them completely) and ignore the lessons from the Mythical Man Month [wikipedia.org].

I cringe to think that many business people will actually take BusinessWeeks' article seriously.

Really? You think keeping track of some people's dinner plans is the hardest IT problem on the planet? How about YouTube storing and serving truly ludicrous amounts of video. Web search? Watson?

Facebook is utterly trivial compared to many problems out there.

While I happen to agree with you, none of those other problems get daily front-page attention on slashdot. While facebook is one of the least interesting problems in computer science, it has been a staple of slashdot discussion ever since facebook became a staple of everyday conversation (or perhaps ever since the creator of facebook surpassed cmdrtaco in net worth).

Actually, Facebook's problem isn't trivial in any sense of the word. The complexity and joins of various database tables must be insane. With YouTube it's all about raw bandwidth, which actually is a fairly easy problem to solve especially since 99% of that data is static. You just physically distribute it and throw money / resources at the problem. As far as database structure, any CS student should be able to reproduce the bulk of it in a single day. You have videos associated with users, and comments associated with videos, etc. The gist of it is straightforward.

Now let's talk about Facebook. There is no compartmentalization of the data. You've heard the "six degrees of separation", whereby any two people on the planet can be socially connected to one another in at most 6 steps. Well, with Facebook, the average degree of separation between any two people is 3.74. What that means is everyone is very closely networked, all the data is dynamic (or more specifically, the data the users really care about is the dynamic and most recent data), and since many people (myself included) open up their information to "friends of friends", there is a tremendous amount of data that any one person can potentially have access to. Even Google searches don't have this problem, because the bulk of the common search terms can be preprocessed for easy retrieval, and having data that's an hour or two old isn't a huge issue.

So you have this massive database (1 billion users, each with many different types of associated data - posts, images, videos, things they've liked, things they've shared, etc, etc), and each of those 1 billion users has an entirely different set of friends from which recent (basically real-time) data must be polled - over, and over, and over again, all day long. Now, throw in the very complex privacy rules, as to which types of posts can be seen by which types of friends, groups, block lists, etc, and the problem becomes very, very complex. Sure, most of us could bang out something with that core functionality without too much difficulty, but to make it work nearly real-time for 1 billion users at once? That's an incredible undertaking.

And your opinion carries any weight, why? You realize that Facebook can pretty easily mine their massive data to link these duplicate Farmville accounts to the real accounts, right? This is pretty basic data analysis that companies like Facebook, Google, etc. can do. And their 1 billion active users is after taking out all the fake and duplicate accounts.

You think so? One person in six on this earth, including infants and the elderly in developing countries without regular internet access has an active facebook account do they? Facebook's numbers have never been properly audited, its not in their best interests to do so. The more users they can claim, the better for them. I would agree with possibly a couple hundred million, but I have a really hard time believing much more than that.

It's actually 1 in 7 now;)
I used to work for an ISP back when there lots of them, and we used to offer one month free for new members. Most people quit after the first month, but that didn't stop us advertising how many customers we had in our database. I'd be willing to bet a lot of money FB is doing the same thing.
I have 3 FB accounts myself, one I use, one I use for signing up to all those crap services which only let you use an FB account (hello Spotify) and like to spam your status, and another for

Lets not forget product accounts, accounts to market products. Then redundant accounts, people changed names. Try and forgotten accounts because it is so hard to erase your data. Of course there is always Facebooks rather crappy and falling share price
'hmm' 1 billion users, that'll pump up the share price. So Facebook should have been honest with the investing public and declared how many active user accounts, not presented information to the public with an intent to deceive non-inside investors (those as

It's made infinitely easier by being asynchronous and 99% reads. There are no timing issues. If a post is delayed to someones screen by a minute or two, nobody dies.

It's not terribly difficult to make numerous (near infinite) read-only replica's of a database which are within tens of milliseconds of the primary; so that takes care of 99% of their problems.

Handling their write load is harder but keep in mind the vast majority of their accounts are idle; and again asynchronous writes make it much much easier. They can shove everything through a message queue and put heavy-weight sharding of the data behind that.

I think handling 100 Million banking customers in 2000 was infinitely harder than Facebook has it from a technical standpoint.

Nah. You simply put one users data in one place (well more than one place for redundancy, but two or 3 not lots of places).To build a page you can ask each machine processing that persons data. You ask the machines processing their fiends data for that data, and build the page. Arrange your network so that groups of machines are in subnets, and place the users data based on the connectivity onto machines in the subnet. So more connected us

Shrug. It's a hard problem, but it's hardly a unique problem. I work on Google ads backend. Superficially this is a very different system from Facebook, which is all frontend... and yet every problem you describe there is one that we have had to solve at similar scale. (Yes we have an order of magnitude fewer users for example, but our users have an order of magnitude more data, and it is easier to shard many small users than a smaller number of goliath users.)

Facebook's problem IS trivial compared to the problems that deserve solving on this planet. Facebook does not solve a single real business problem (except their own). Technically, what they do may be a challenge but it doesn't contribute to anything except let some kids looking for attention share stuff that nobody cares about. If you ask me, this is a waste of resources, technically, intellectually, and energy-wise.

1bn users accessing a DB which, Yes, polls and manages a large amount of data, YES. However, you just slammed YouTube for doing pretty much the same thing but only exponentially better. Lets not forget massive amounts of comment management, video relevancy tools, algorithms that automatically scour video clips for copyright infringement and convert text to speech, etc so on an so fourth.

Unlike many other databases, errors can be tolerated in facebook. If a post gets lost or a connection or two dropped it really doesn't cost Facebook anything--and it's unlikely to be noticed. And downtime and retries are tolerated by the users.

Try running a real-time, financial system like credit card authorization & processing (which probably has more than 1 billion users), needs to balance at the end of the day and has response requirements measuring under 250 ms.

Facebook isn't just about status updates. They have a whole robust API they use to interact with apps and other websites. It hosts music, events, photos, videos, app data, along with tons of user data with timeline. You can share anything from a cat video to a milestone of you losing weight. Serving up all that data in quick and well-presented manner to millions of people around the world is very difficult.

As much as I hate Facebook, and I believe the number of true Facebook profiles are less then 250M, "To Caesar What Is Caesar's".
Just because you think the added value Facebook creates is not rocket science, Facebook not only does use high tech software architecture but also creates software technology and delivers some as open source.
I would recommend you read http://royal.pingdom.com/2010/06/18/the-software-behind-facebook/ [pingdom.com]
And trust me when your scale goes above 10 digit numbers nothing is trivial

PHP has proven to be the best web development kit. It's only persistent failure is the legacy growth of inconsistent api calls. For the rest, it's turing complete, does scale well, and most of all is the best tuned hammer for the job. It delivers.

In effect, PHP is a huge C api with its own C like language constructs, a layer of abstraction which takes away the mundane and gets you building web sites.

Now C is hailed for its great power, and not made fun of because of its ability to make real crappy, insecure code.PHP however is not hailed for its great power, and made fun of because of its ability to make real crappy, insecure code.

It's all a matter of perspective. The problem is low level programmers who can't live with the fact people make a billion dollar without obsessing over pointers or garbage collection.

Yes, and developers agree [hammerprinciple.com] with that sentiment. PHP has inertia behind it: tons of cheap webhosts and lots of libraries and existing codebases. As a programming language, there are definitely better out there: Python, Ruby, etc.

Good post AC. I think you are on to something with your last sentence too. Technical research on the web is a nightmare, because you have to parse the motivation behind the opinions and filter on that too. Low level programmers... obsessing over pointers or garbage collection... indeed. These people can come across as enormously well-informed but their opinions are often worthless outside their tiny, unknowable silos.

Oh yes, please tell me all about the computer geniuses that wrote the PHP scripts that power facebook!

Well, I know PHP bashing is all the rage, so how about the computer geniuses at Facebook that wrote HipHop, their PHP-to-binary compiler?

I think it is a pretty cool technical thing (and according to their stats it dropped their CPU usage by some significant figure) - and even better, they open sourced it. Like they do with a lot of their stuff [facebook.com].

I'm kinda disappointed...
I am truly interested in how Facebook scales and was hoping there would be actual Computer Science related material in the article...
Any Facebook employees care to comment? What do you guys do to scale stuff? How about./'ers from other companies that have to deal with scaling?
Hell, how do porn sites scale?
I've done the traditional Distributed Systems courses in University but I really wanted to know how it's done in the real world by AWS, Facebook etc...

Shops anymore tend to scale by throwing RAM and bandwidth at everything... It drives developers crazy because management cares little about what kind of mess they force their developers to ignore due to due-dates. And of course, the only casualties are the developers who were never given a fair shake to start with.
Wanna know how something scales? Continual tweaking, and yes, more RAM and bandwidth.
It's the only way to scale things anymore.

I don't recommend reading it. There is absolutely nothing in this article about the actual engineering problems behind scaling for this number of users and how these problems are solved. In fact, there is nothing technical at all in this article except for some vague descriptions of the "bootcamp".

It's actually a rather impressive setup. Some Facebook architects gave a talk in EE380 at Stanford a few years back.
Originally, Facebook's architecture assumed that most "friends" would be regionally local, reflecting Facebook's college-campus origin. That's not how it worked out after some growth. So they have to assemble pages across regions and data centers. There's caching, but there's also active cache invalidation, which they can do because they control both sides of the cache. There's extensive inter-process communication, and it's not HTTP. There's a lot of PHP for the user-facing stuff, but it's compiled with their in-house compiler, not interpreted.

Facebook's purpose is banal, but the technology behind it is non-trivial.

You misinterpreted the heading - Facebook has the hardest information technology problem on the planet.That information technology problem has nothing to do with servers and storage.

The hardest information technology problem on the planet is: How do the Facebook exec's stop the company going the way of Silicon Graphics (NYSE: SGI) - oh wait, no, (DELISTED by NYSE because the share price couldn't stay above $1: SGI); since the company creates no real value, and has done nothing but drop it's price since IPO.

If you believe there are actually 1 Billion, completely unique users on Facebook, then I need to ask that each of you turn over your Internet Licenses, power down your computers, and find a new hobby. You are just to dangerous to be allowed on the Internet without adult supervision.

Facebook actually thinks these are one billion distinct humanoids? Zuck is stoopider than his investors look. As anyone who ever posted a signup sheet in a college dorm for IM softball can attest, at least a third of people who sign up for anything, anywhere, ever, are fake.

Facebook is the worst performing and most opaque large scale site with the worst interface that I use regularly.

Browsing photos, the most basic Facebook activity is still a pain and buggy as hell on a slowish connection, and they keep changing the damn interface just when you figured out the previous unintuitive change. The mobile website sucks, their Android app sucks, I don't know what the new iOS app is like. The interface has gone from simplicity to being clutte

engineers it takes to keep such massive infrastructure up and running. If all it takes today is 2000 people, to manage the data of a billion people, then I really can't see a very __large__ need for software developers in the future.

I would be interested in learning more about the software and hardware side of Facebook. But after 15 seconds of scrolling I hadn't seen any... just a lot of tedious "gotta do this" journalism... and gave up. LOOOOOONG BOOOORING