How Facebook Ships Code

I’m fascinated by the way Facebook operates. It’s a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried). These are notes gathered from talking with many friends at Facebook about how the company develops and releases software.

Seems like others are also interested in Facebook… The company’s developer-driven culture is coming under greater public scrutiny and other companies are grappling with if/how to implement developer-driven culture. The company is pretty secretive about its internal processes, though. Facebook’s Engineering team releases public Notes on new features and some internal systems, but these are mostly “what” kinds of articles, not “how”… So it’s not easy for outsiders to see how Facebook is able to innovate and optimize their service so much more effectively than other companies. In my own attempt as an outsider to understand more about how Facebook operates, I assembled these observations over a period of months. Out of respect for the privacy of my sources, I’ve removed all names and mention of specific features/products. And I’ve also waited for over six months to publish these notes, so they’re surely a bit out-of-date. I hope that releasing these notes will help shed some light on how Facebook has managed to push decision-making “down” in its organization without descending into chaos… It’s hard to argue with Facebook’s results or the coherence of Facebook’s product offerings. I think and hope that many consumer internet companies can learn from Facebook’s example.

HUGE thanks to the many folks who helped put together this view inside of Facebook. Thanks are also due to folks like epriest and fryfrog who have written up corrections and edits.

Notes:

as of June 2010, the company has nearly 2000 employees, up from roughly 1100 employees 10 months ago. Nearly doubling staff in under a year!

the two largest teams are Engineering and Ops, with roughly 400-500 team members each. Between the two they make up about 50% of the company.

product manager to engineer ratio is roughly 1-to-7 or 1-to-10

all engineers go through 4 to 6 week “Boot Camp” training where they learn the Facebook system by fixing bugs and listening to lectures given by more senior/tenured engineers. estimate 10% of each boot camp’s trainee class don’t make it and are counseled out of the organization.

after boot camp, all engineers get access to live DB (comes with standard lecture about “with great power comes great responsibility” and a clear list of “fire-able offenses”, e.g., sharing private user data)

[EDIT thx fryfrog] “There are also very good safe guards in place to prevent anyone at the company from doing the horrible sorts of things you can imagine people have the power to do being on the inside. If you have to “become” someone who is asking for support, this is logged along with a reason and closely reviewed. Straying here is not tolerated, period.”

any engineer can modify any part of FB’s code base and check-in at-will

very engineering driven culture. “product managers are essentially useless here.” is a quote from an engineer. engineers can modify specs mid-process, re-order work projects, and inject new feature ideas anytime. [EDITORIAL] The author of this blog post is a product manager, so this sentiment really caught my attention. As you’ll see in the rest of these notes, though, it’s apparent that Facebook’s culture has really embraced product management practices so it’s not as though the role of product management is somehow ignored or omitted. Rather, the culture of the company seems to be set so that *everyone* feels responsibility for the product.

during monthly cross-team meetings, the engineers are the ones who present progress reports. product marketing and product management attend these meetings, but if they are particularly outspoken, there is actually feedback to the leadership that “product spoke too much at the last meeting.” they really want engineers to publicly own products and be the main point of contact for the things they built.

resourcing for projects is purely voluntary.Engineers decide which ones sound interesting to work on. a PM lobbies group of engineers, tries to get them excited about their ideas. Engineer talks to their manager, says “I’d like to work on these 5 things this week.” Engineering Manager mostly leaves engineers’ preferences alone, may sometimes ask that certain tasks get done first.

Engineers handle entire feature themselves — front end javascript, backend database code, and everything in between. If they want help from a Designer (there are a limited staff of dedicated designers available), they need to get a Designer interested enough in their project to take it on. Same for Architect help. But in general, expectation is that engineers will handle everything they need themselves.

arguments about whether or not a feature idea is worth doing or not generally get resolved by just spending a week implementing it and then testing it on a sample of users, e.g., 1% of Nevada users.

engineers generally want to work on infrastructure, scalability and “hard problems” — that’s where all the prestige is. can be hard to get engineers excited about working on front-end projects and user interfaces. this is the opposite of what you find in some consumer businesses where everyone wants to work on stuff that customers touch so you can point to a particular user experience and say “I built that.” At facebook, the back-end stuff like news feed algorithms, ad-targeting algorithms, memcache optimizations, etc. are the juicy projects that engineers want.

commits that affect certain high-priority features (e.g., news feed) get code reviewed before merge. News Feed is important enough that Zuckerberg reviews any changes to it, but that’s an exceptional case.

[CORRECTION — thx epriest] “There is mandatory code review for all changes (i.e., by one or more engineers). I think the article is just saying that Zuck doesn’t look at every change personally.”

[CORRECTION thx fryfrog] “All changes are reviewed by at least one person, and the system is easy for anyone else to look at and review your code even if you don’t invite them to. It would take intentionally malicious behavior to get un-reviewed code in.”

no QA at all, zero. engineers responsible for testing, bug fixes, and post-launch maintenance of their own work. there are some unit-testing and integration-testing frameworks available, but only sporadically used.

[CORRECTION thx fryfrog] “I would also add that we do have QA, just not an official QA group. Every employee at an office or connected via VPN is using a version of the site that includes all the changes that are next in line to go out. This version is updated frequently and is usually 1-12 hours ahead of what the world sees. All employees are strongly encouraged to report any bugs they see and these are very quickly actioned upon.”

re: surprise at lack of QA or automated unit tests — “most engineers are capable of writing bug-free code. it’s just that they don’t have an incentive to do so at most companies. when there’s a QA department, it’s easy to just throw it over to them to find the errors.” [EDIT: please note that this was subjective opinion, I chose to include it in this post because of the stark contrast that this draws with standard development practice at other companies]

[CORRECTION thx epriest] “We have automated testing, including “push-blocking” tests which must pass before the release goes out. We absolutely do not believe “most engineers are capable of writing bug-free code”, much less that this is a reasonable notion to base a business upon.”

re: surprise at lack of PM influence/control — product managers have a lot of independence and freedom. The key to being influential is to have really good relationships with engineering managers. Need to be technical enough not to suggest stupid ideas. Aside from that, there’s no need to ask for any permission or pass any reviews when establishing roadmaps/backlogs. There are relatively few PMs, but they all feel like they have responsibility for a really important and personally-interesting area of the company.

by default all code commits get packaged into weekly releases (tuesdays)

with extra effort, changes can go out same day

tuesday code releases require all engineers who committed code in that week’s release candidate to be on-site

engineers must be present in a specific IRC channel for “roll call” before the release begins or else suffer a public “shaming”

ops team runs code releases by gradually rolling code outthere are 9 concentric levels for rolling out new code

e.g., new tuesday release is rolled out to 6 servers (level 1), ops team then observes those 6 servers and make sure that they are behaving correctly before rolling forward to the next level.

if a release is causing any issues (e.g., throwing errors, etc.) then push is halted. the engineer who committed the offending changeset is paged to fix the problem. and then the release starts over again at level 1.

so a release may go thru levels repeatedly: 1-2-3-fix. back to 1. 1-2-3-4-5-fix. back to 1. 1-2-3-4-5-6-7-8-9.

ops team is really well-trained, well-respected, and very business-aware. their server metrics go beyond the usual error logs, load & memory utilization stats — also include user behavior. E.g., if a new release changes the percentage of users who engage with Facebook features, the ops team will see that in their metrics and may stop a release for that reason so they can investigate.

during the release process, ops team uses an IRC-based paging system that can ping individual engineers via Facebook, email, IRC, IM, and SMS if needed to get their attention. not responding to ops team results in public shaming.

once code has rolled out to level 9 and is stable, then done with weekly push.

if a feature doesn’t get coded in time for a particular weekly push, it’s not that big a deal (unless there are hard external dependencies) — features will just generally get shipped whenever they’re completed.

getting svn-blamed, publicly shamed, or slipping projects too often will result in an engineer getting fired. “it’s a very high performance culture”. people that aren’t productive or aren’t super talented really stick out. Managers will literally take poor performers aside within 6 months of hiring and say “this just isn’t working out, you’re not a good culture fit”. this actually applies at every level of the company, even C-level and VP-level hires have been quickly dismissed if they aren’t super productive.

[CORRECTION, thx epriest] “People do not get called out for introducing bugs. They only get called out if they ask for changes to go out with the release but aren’t around to support them in case something goes wrong (and haven’t found someone to cover for you).”

[CORRECTION, thx epriest] “Getting blamed will NOT get you fired. We are extremely forgiving in this respect, and most of the senior engineers have pushed at least one horrible thing, myself included. As far as I know, no one has ever been fired for making mistakes of this nature.”

[CORRECTION, thx fryfrog] “I also don’t know of anyone who has been fired for making mistakes like are mentioned in the article. I know of people who have inadvertently taken down the site. They work hard to fix what ever caused the problem and everyone learns from it. The public shaming is far more effective than fear of being fired, in my opinion.”

It’ll be super interesting to see how Facebook’s development culture evolves over time — and especially to see if the culture can continue scaling as the company grows into the thousands-of-employees.

What do you think? Would “developer-driven culture” work at your company?

@nithin sounds like you haven’t experienced any good product managers. they are often the only people in the organisation that can simultaneously understand the complexities of the technical effort, the needs of the market, and the nuances of the user experience, whilst providing balance to some of the egos that can quickly torpedo a viable business.

I agree with thepiedpipes. Project managers are very useful to a developer if they are good at what they do. If you need answers to something they go get it, if you don’t want to talk to some annoying account exec just refer them to the PM. Personally they make great “meat shields” and “go getters”. They have a tough job. They do everything in a project that no one else wants to do. Having a good PM, I can concentrate on the quality of my code and I can trust that everything else is taken care of. But, I’ve also had bad PMs and a bad PM just makes the project worse that if there weren’t one at all.

Well-well look. I already told you: I deal with the god damn customers so the engineers don’t have to. I have people skills; I am good at dealing with people. Can’t you understand that? What the hell is wrong with you people?

Hambone, are you that culturally illiterate that you did not pick up on the Office Space reference? Crawl back under your bowl of rice and or noodles and continue banging out code and let the “grown folks” continue this conversation.

Completely different skill sets. PMs are task masters, organisational gurus and often scrum masters. Product managers should own the vision for the product, steer its features backlog, have the right blend of UX/tech and business knowledge, and communicate the road map for the product outside the business and within.

@jason That’s probably because you aren’t one and don’t know what a good manager is able to do.

I’m a product manager and in that capacity I put together specs, coordinate designers, developers and marketing teams to make sure they execute the product vision and attain the financial goals set by management, put together the roadmap, get dev excited about new ideas, download every possible app out there and sign-up to any possible website to make sure I’m in tune with the market, etc… A developer can not spend its time analyzing the financials of a product to see if its profitable, nor deal with UI issues, needs to stay focus on what they do best, i.e. code. We stay focus on what we do best, i.e. sending emails, putting together useless Powerpoint and building business models. I agree that product managers are pretty useless if they can’t talk the language of developers (or designers as a matter of fact) and that business school is not necessarily the best training ground for that. Rather be passionate about geekish stuff. Regarding code, I always thought that a PM should be able to read, not necessarily write.

The cubicle is the result of culture driven work environment. Intel used it so that engineers could have privacy yet communicate if they needed to rapidly. Its interesting that other companies simply employ cubicles because of the successful nature of intel, not necessarily because of what it actually does to improve the work environment or improve the operations of the company.

Thanks for the clarifications re: mandatory code review, automated tests, and emphasis on bug fixing. Sounds like Facebook’s practices are actually more defensive around code checkins than my notes may indicate.

The thing I still find remarkable is how all of those practices (and it seems like the rest of the culture) centers on engineers…

You know, designers do actually have a purpose. Especially on a site like Facebook.

I hate to say this, because Facebook’s approach seems to generally work pretty well. Most engineers (read: developers) have little to no grasp of the user experience. And in terms of Facebook causing widespread user experience panic every time they shuffle the layout, I’m not surprised to learn that FB allows engineers to design the user experience most of the time. No wonder the layout this time around especially sucks…engineers are designing in a linear fashion.

You know that, most of the layout changes are to do with a backend process, or to do with improving the performance metrics of teh system as a whole; holistic application development cannot be undertaken by designers as they know what they are asking for 😉

Ok, so… I’m a noob because I believe that engineers make terrible front-end designers? Engineers think logically and end-users generally don’t. So to have a group of people (engineers) changing the layout to fit back-end needs, rather than the needs of the people who’ll actually be using it is absurd. Especially when you’re talking about 400 million users.

It’s pretty rare to find an engineer that has a decent grasp of the complexities and subtleties of Ux and UI design but only the best engineers will not kid themselves that they do and actually respect people that do (equally so inversely).

FB’s Ux/UI is flawed on many levels, breaks a bunch of industry standards and rules of thumb but that does not render it useless by any means, it just will prevent a percentage of users from accessing certain features/functionality easily. If these are non-critical areas then it is less of an issue than say “privacy settings” for example.

If the UX is so bad, I wonder why so many sites try to copy Facebook’s, and why I know so many UXler who love FB.

And: There are designers (and all should be able to do UX also, I do not see why these fields should be separated), if the developers think they need any. In most cases a general template is used however, and the developers can copy the structure and do the rest rather easily.

It’s easy to say that people love this and people love that, it’s another thing to demonstrate it with a solid used-centered process. We all have opinions about what we love and what we hate, but “Your opinion, while interesting is often irrelevant”. What’s relevant is what UX practitioners can demonstrate through the data collected from real users of the product. I’ve been in this battle many times and it’s what I consider to be the biggest problem with UI/UX design:

Since everyone uses a computer, everyone thinks they are able to design a UI and have an opinion on the UI. That’s the core of the problem here. If a civil engineer comes to you and makes a recommendation about a structure, most likely people will not debate, on the basis of his/her expertise (civil engineers may debate between them, but Joe and Jane from let’s say social studies won’t). When it comes to Human Factors Engineering/UX design, while there are specialists out there with the appropriate training who can make recommendations based on an established process, we all think we can have our say on the basis that we all used a UI at some point in our life. That’s a very dangerous approach to follow.

So why so many sites try to copy Facebook? Because they don’t know better. They think if millions of people use it, it most be good, so let’s do it. WRONG! It does not mean millions of people use it that it’s a good design. Millions of people may have been forced to adapt to a bad design (humans have great abilities to adapt over time, even the worst design may not be too bad for some users because they have adapted to it).

If you are carefully reading people’s comments on facebook about their UI when major UX overhauls are pushed through, you’ll read a lot of things like “OH NO, FACEBOOK UI CHANGES AGAIN #$%*&&#$” and after a few days, people painfully adapt to the new UI. Not the best approach to push a new UI through if you ask me. That being said, not every aspect of their UI is like that. There are definitely good aspects.

I agree with you on that one. (see comment I posted today below). I mean, I have met a few developers who can do UX design, but the reality is, most of them can’t. They just don’t think in a user-centered manner, they think from the structure of the code. So basically, if you want a UI that shows you the structure of the code (or the organization of your data structures), developers are good to produce that. The problem is that users don’t think that way. Thinking from a user-centered point of view is something different.

Now, as for designers, yes I agree with the one of the post that says designers should be able to do the UX (after all they are designers). The problem here again is that many of them never heard of UX in their design training. While they should not be UX specialist, they should have at least a minimum training in UX.

At the end of the day, it all comes down to how capable is someone (in any position) to think from a user-centered point of view. History shows us that certain people are capable of doing this naturally (these are the very good designers, engineers, developers, trade workers, etc… but are rare). Most often though, people don’t naturally think that way. I always use the example of an electrician when I teach my UI/UX class (nothing against BTW). If you hire an electrician to wire your house, you really want to hire the one who will think about how people in the house are going to move around and where switches, receptacles, and three-way switches should be located. That will lead to well-designed electrical system configurations that are usable by the people live in the house (the primary users). Do these people exist? Sure they do, in a very limited quantity. The experience most of us have when we move into our houses is that some switches are behind doors, some three-way switches are missing to allow you to turn off the lights at the end of the hallway, and a few receptacles are in usable spots. Yet, from a wiring point of view, everything is working… That does not mean electrician are not good at what they do. It just shows that their natural tendencies may not be to wire a house from a user-centered point of view.

Olivier,
Rather than being so long winded, why not just give example sites with a good interface. I think Facebook has a wonderful interface, one of the best out there, and that is why I use it everyday.

I most areas responsability is shifting towards engineers generally with the introduction of agile methodologies, and the more successful companies who are able to hire more talented, well rounded engineers are generally able to carry this off more effectively.

I think such practices become harder to maintain as the work force grows.

If all the engineers write bug free code or at least caught in the release process where they are paged to fix their issues or face public shaming or at worst getting fired then why are boot campers only dedicated to fixing bugs and listening to lectures? Wouldn’t that mean the boot campers have nothing to do but listen to lectures?

Interesting notes given what a hair-pulling nightmare their public APIs are to integrate with. Stuff changes regularly and gets dropped on apparent whims with no prior warning so it makes perfect sense that the development is run from the bottom up with no overarching vision to ensure that not just the sexy features du jour are getting attention.

whoa. look at the amazing work environment. all that focus on product and how powerful engineers are. it really shows in how profitable facebook is. their balance sheet is a glowing example of the success of this company.

Even with the corrections (re: the FB employee) I find the cultural implications at Facebook very interesting. I work in a company the requires intense engineering development, but still utilizes the cubicle model. I find that this is a major drain on productivity and slows our release schedule for a variety reasons (less camaraderie, less face time, lowered team cohesion, etc) . I’m sure that the FB culture isn’t right for everyone, but I’m sure there is a nice midpoint between cubiclism and full-open culture.

[…] Even with the corrections (re: the FB employee) I find the cultural implications at Facebook very interesting. I work in a company the requires intense engineering development, but still utilizes the cubicle model. I find that this is a major drain on productivity and slows our release schedule for a variety reasons (less camaraderie, less face time, lowered team cohesion, etc) . I’m sure that the FB culture isn’t right for everyone, but I’m sure there is a nice midpoint between cubiclism and full-open culture. ~ Jack Amplify’d from framethink.wordpress.com […]

How accurate is the “How Facebook Ships Code” article written by ‘yeeguy’?…

Thanks a ton for the corrections and additional material. I think it’d be fantastic for more companies to learn about how Facebook’s development process works — “developer-driven culture” is really the ultimate expression of pushing decisions “do…

[…] January 17, 2011 by Brian McKay Leave a Comment I'm fascinated by the way Facebook operates. It's a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried). These are notes gathered from talking with many friends at Facebook about how the company develops and releases software. It's been over six months since I assembled these observations and I'm sure Facebook has continuously evolved its software development practices in the meanti … Read More […]

I think techniques like these may be very successful only in organizations where developers are working on consumer end-products that they themselves use (like Facebook). In that environment, the fundamental role that a product manager plays of identifying, prioritizing and translating the needs of end-users is more easily obviated. I can’t see this working on software projects where the problem domain is much more complex or esoteric.

Good point John. The other item that’s overlooked is the fact that product managers are less important when you have lots of money to burn on “experiments” and the BEST engineers who are capable of understanding what users want and what features to build/prioritize. Facebook has that luxury today, as did Google a couple years ago. When the quality of engineering talent goes down, the importance of a talented product team goes up. Historically, you can see that shift at companies like Yahoo, and you can see it starting to happen at Google now (guys like Max Levchin and Brad Horowitz are undoubtably ‘product’ leaders who are hired when engineers can’t experiment their way through things). Lastly, it’s undeniable that Zuck is a ‘product’ guy just as he is an ‘engineer’ – this is common with most of the major web 2.0 companies.

Great insight, thank you. I am surprised most of all about the lack of unit testing and formal QA department. I’d have to work in such an environment to give a fair opinion, but it sounds like the wild west for quality. Their stability would seem to prove me wrong though.

I think that this is a great set of notes — extremely interesting. However, I think that the title is off, as is the notion that Facebook “ships and releases” software. Facebook does neither of those things: they publish a service and an API for other developers to use, they don’t ship any software.
Shipping software is a completely different matter than publishing a service, and demands a different development process. They’re still a brilliant organization filled with smart people, but it’s clear to see why their process wouldn’t work for a company that ships code. For example, if a nasty little bug makes it way into production during a weekly release at Facebook, they can page the engineer and get it fixed (downtime will be small and isolated). If this happens with shipped software, running in customer environments, a lot of people might get screwed and you’ll have to test and release an update that will cost your customers time and money to apply (that doesn’t make customers very happy). You definitely need more thorough testing and verification processes when the responsible engineer can get on the server and dig in to the problem immediately to fix it.
As a model for service delivery however, I think that organizations can learn a lot from Facebook’s developer-driven culture. If you make engineers directly responsible for their own stuff (including problems they cause), they’ll be pretty motivated to avoid public shame.

[…] I'm fascinated by the way Facebook operates. It's a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried). These are notes gathered from talking with many friends at Facebook about how the company develops and releases software. It's been over six months since I assembled these observations and I'm sure Facebook has continuously evolved its software development practices in the meanti … Read More […]

[…] How Facebook Ships Code – “I’m fascinated by the way Facebook operates. It’s a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried). These are notes gathered from talking with many friends at Facebook about how the company develops and releases software….” […]

I guess FB engineers should be not too happy about the article as it might look they are a bunch of crazy wild-west coders. Do not believe they do not have a discipline or that PMs or MKT do not play a role, might it be different from other companies standards.
Engineers involvement can be good or bad. I see some positive aspects like being responsible for their code and features, fact that empowers them, but things like that they decide the products features just give me the frights.
As a final comment, I think you are looking for a silver bullet that reading SW-Eng savvy people like Tom deMarco in “Peopleware” you can conclude does not exist. SWEng practices are there for a reason and a SW process is always needed, be it a formal RUP process or an Agile one. BTW, Agile does not mean “do what I want”. It is a formal process like any other, with its disciplines and musts.
Problem is many people out there just cover their anarchy and lack of process stating ” I am agile”.
Honestly do not think the process described is magic or a novelty, although has attractive bits. Hope it works fine for them.

This article is obviously written by a developer frustrated with pm’s, with sources of developers who get a huge hard on by the power and responsiblity they THINK they have. This is just not a realistic scenario.

[…] purpose or not, there is no denying that Facebook is a high traffic, technically challenging site. These rumors on how Facebook handles development and deployment are of course to take with a grain of salt, but […]

From my experience, this model always works, BUT:
-you need a team of excellent engineers (all of them, no exceptions)
-you need to convince management that it is the way to go.

So, in the real world, it is seldom used because:
-great engineers cost more money (and they just write code right? they’re all the same)
-management rarely has an engineering background to understand why this model is worth it

It was a disaster so much so that the organisation is now refocusing very strongly on a product management culture.

To me reading this suggests Facebook have too many developers and too little focus on quality and doing right by their users. No surprise that they frequently have users up in arms about UI and privacy.

You need balance in an organisation to get the best out of your team. Putting all the emphasis on developers does not give a balance of skills.

Sounds hideous to my mind. Similarly working for ferrari must be boring. Striving for perfection in the mundane. I prefer concept level and cant quite get excited about this but judging by all the comments I may be alone 🙂 and happy about it.

I wonder how User Experience/Usability people work within an organization like that. Typically, the fact that the engineers have the final words would probably turn off many UX practitioners who may feel that the work/studies they are doing and the findings they are proposing does not get listen to.

I think UX need to get people excited too. So what if engineering own the front end. UX would need to get the engineers excited about making features more usable. So the engineer that is coding a new interaction could use a UX mindset.

UX is a state of mind anyway, the UX team at Facebook could probably be a good catalyst role.

This will not work on all organizations, however, some of the ideas presented here are already well practiced in a number of places – I for example have been doing most of these for the last 12 years at different organizations. I got to a point where I did 63 releases in one year (medium and mayor, small where par-for-the-course)

I like that the systems guys are business oriented and well respected. In most organizations that is not the case. I have always struggled with that and solved it in couple of places but not in all places I have worked for.

I am not sure about the lack of QA. Maybe I am old school about that. I do agree that developers are capable of writing solid code and they should totally and completely unit test the heck out of the code they write – take ownership of what you do for goodness sakes – but I still feel that a formal QA step is of great importance.

[…] Camp” training where they learn the Facebook system by fixing bugs. After boot camp, all engineers get access to live DB and any engineer can modify any part of Facebook’s code base and check-in at-will so that […]

[…] Camp” training where they learn the Facebook system by fixing bugs. After boot camp, all engineers get access to live DB and any engineer can modify any part of Facebook’s code base and check-in at-will so that […]

[…] I'm fascinated by the way Facebook operates. It's a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried). These are notes gathered from talking with many friends at Facebook about how the company develops and releases software. Seems like others are also interested in Facebook… The company's developer-driven culture is coming under greater public scrutiny and other companies are … Read More […]

[…] Camp” training where they learn the Facebook system by fixing bugs.After boot camp, all engineers get access to live DB and any engineer can modify any part of Facebook’s code base and check-in at-will so that […]

[…] Manager Yee Lee talked to a bunch of friends who work at Facebook, and his resulting post on how Facebook ships code makes it sound like the biggest startup in the world. Some of the points in the original post were […]

[…] Manager Yee Lee talked to a bunch of friends who work at Facebook, and his resulting post on how Facebook ships code makes it sound like the biggest startup in the world. Some of the points in the original post were […]

I’m curious to understand the function of architecture at FB. To level set, I run enterprise Architecture at my company and I’m constantly seeking clarity on how architecture and architects enable the business and engineering. The article briefly alludes to architecture in FB as an advisory function akin to the designer function being a support function when engineering needs them. So…what do architects get called upon by the engineers to help them out with.

Traditionally, the word “architect” is viewed very negatively by engineers – something I’m keenly aware of. How is this at FB?

This will work when the following has been taken care of:
– business strategy & missions clear
– architects have defined the strategic technology direction to be taken, and identified key technologies that need to be used
– the company is able to hire & retain highly competent developers over a period of time
– focus & accountability areas are defined for business, architects, developers

sounds like something that would work great until the big buyout/ipo, at which point people would stop caring, the threat of being fired wouldn’t carry much weight and i suspect things would go to utter and complete hell.

[…] I'm fascinated by the way Facebook operates. It's a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried). These are notes gathered from talking with many friends at Facebook about how the company develops and releases software. Seems like others are also interested in Facebook… The company's developer-driven culture is coming under greater public scrutiny and other companies are … Read More […]

What does Facebook use to package and deploy PHP code? We currently use git for deployment at my company but we’ve come to the conclusion that using a packaged deployment system would be much better, especially for our cloud-based systems.

[…] I'm fascinated by the way Facebook operates. It's a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried). These are notes gathered from talking with many friends at Facebook about how the company develops and releases software. Seems like others are also interested in Facebook… The company's developer-driven culture is coming under greater public scrutiny and other companies are … Read More […]

So it all comes to this:
Security is not important.
Privacy of the user is not important.
Is is fun to program.

A developer driven development cycle is not the way to create a useful application. They teach this a long time ago already.
Thank you for sharing this information. Facebook development is not the way to create applications.

If the Engineers don’t think that their managers do much, and that the Engineers have a lot of freedom, and that the managers aren’t very influential, then my guess is that the managers are really excellent managers! Their job is to keep their team as productive as possible. Believe me, I am sure that the managers are doing TONS of work behind the scenes to let the developers work in the best possible environment and to feel the best about what they do. Those managers are keeping things together!

I do not think so, In my opinion it would hardly work anywhere else. Why ? it all about business strategy, from what i understood, facebook need creativity, and that culture also instill a sense of leadership in each employee, which kind of ensures that only good ideas transform into code. Whereas in most businesses, programming must be aligned with business goals( which comes down from the top, to make sure everything is well coordinated) which somehow kills creativity. what do u guys think?

I remember when I couldn’t create an event in facebook because the state dropdown was empty and yet it was “required”…that remained for days and really sucked! As a developer I am amazed that facebook prospers as it does, it really doesn’t seem so special – beyond having a massive user base. I don’t even find it particularly intutive but I am glad they exist and give the web more press!

facebook doesn’t strike me as a terribly complex system except for it’s scalability. I don’t think this process would work if they were building an office suite or something like that.

It’s also the worst development environment I’ve ever seen or even heard of.

I have an EXE file I built in turbo pascal in 1983 and that same binary will still run on the lastest version of windows on the latest hardware.

I wrote a facebook app and within 3 months it wasn’t working anymore because facebook doesn’t understand the concept of backward compatibility. This doesn’t strike me as a good feature of their culture being so entitled that they can break their api willy nilly without any regard for the work they’re creating for their VOLUNTEER application developers.

When the fad wears out they’re going to find that people aren’t so willing to cater to their every api change just because “It’s facebook.”

I hope this article isn’t accurate because it makes it sound like a real cackhanded way to run things. I mean I get that it sounds like Mark has made his website and is idealistic about the way he wants to keep the power in the devs hands but come on!

People have specialities, there is toooooo much to learn for a single dev. Making everyone do everything from db through to UI? I’m a developer, but as a regular user of Facebook I’m kind of disappointed at this haphazard way of doing new features. I mean apart from the fact that there aren’t that many new big new features, but I kind of expected there to be some real deep knowledge going into this. Like psychology professors thinking about person to person interactions. And dedicated people that take care of the scaling problems that arise when you are developing for 500 million users.

I cant believe this is really the way that Facebook is run, there must be some broader picture that isn’t being shown here. Are you seriously saying that Facebook dev division is 500 people just doing what they want? I really want to believe that the best minds in their fields are coming together to produce the website that shapes the lives of so many people.

Even if they have really high quality developers on the team they just simply cannot gain the depth of knowledge to offer the best if they do the whole gamut. We’ve all heard Seth’s theories of 10,000 hours before you are an expert. Either all the developers are over 60 or they just simply are not experts in what they do.

I can only guess that the real picture of this is that anyone can seed a feature idea but then teams take over various sub aspects of that feature and apply their various expertise to say, fleshing out the UI design, or optimising the caching, until really its just a nice notion that some developer “invented” a feature, just the same as I’m sure 99% of the code that Mark originally wrote has been replaced by now.

I don’t know if you heard, but most of facebook is written in php. Not exactly the language of geniuses. From what I’ve read, there’s a group of heavy hitters that wrote the php compiler and the javascript reinterpreter thing, but they pride themselves on being wide and short rather than thin and tall if you get what I mean. These guys aren’t writing compilers, or 3d rendering engines, they’re moving text from the database to the screen.

[…] they should know a bit about the science of pricing by nowHow Facebook Ships Code http://bit.ly/enT9nP “with great power comes great responsibility” Powered by Fresh From This entry was posted in […]

[…] that exposed user data, and the user IDs that rogue apps associated with personal information, every damn engineer at Facebook has access to the database? That’s right, all one thousand engineers can look at […]

[…] I'm fascinated by the way Facebook operates. It's a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried). These are notes gathered from talking with many friends at Facebook about how the company develops and releases software. Seems like others are also interested in Facebook… The company's developer-driven culture is coming under greater public scrutiny and other companies are … Read More […]

The inmates are running the asylum!
This goes a long ways to explaining the missteps and lack of real user experience that Facebook has.
I have over 31 years of experience building commercial software products and I can tell you what’s described here is a recipe for disaster in the long run.
This sort of cowboy culture can work early in the development conceptual stage of product development but it’s no way to run a business.
Developers are not your typical users and are generally terrible at building usable products.

With so many corrections one must ask itself: Based on what information did the author come up with this?

“Oh, they have QA and UnitTests bot nobody uses them”. Bam, some other guy writes the exact opposite: “No, we use UnitTests all over the place.”

“If you make many mistakes and get blamed now and then, by time you will eventually get fired” — “Oh no, nobody here gets fired for mistakes!”

Seriously, WTF?! Do you actually know *anything* about the internals of FB or did you just want to write something about FB, because, well, to be cool and stuff? There’s nothing wrong with comments that add information and clearify some points, but those totally contradicted your statements, rendering this whole article useless.

Just a big game of bullshit bingo going on here, IMHO.

PS: No, I don’t know anything myself about FB’s internals. But at least I’m not going around and writing articles with made up details.

Emm… In this case, “developer driven” sounds like a happy title for a sloppy development process covered by a *very good* operations process. It’d be really interesting to get an idea of the ratio of bounced pushes to successful full-production releases.

Thanks for a very interesting article. It’s great to know how exactly developers supporting such a huge number of users work. I really enjoyed reading the article. As for some topics, I thought it would be more interesting to see them in detail, like how much super-productive they are supposed to be, how is a public shaming specifically, etc.
Now I wonder how/if the environment have changed since then and if it’s working as expected.

FAcebook has to be a very complicated system because it looks really simple… That said think about the amount of functionality it provides, AND that it processes probably bilions of information a day. Thing is a lot of this stuff is going behind the scenes. Have You noticed that news feed only feeds information with only the info about friends You have recently interacted… There is a lot of initiative going up there – search, events, privacy model, platform, internal data warehouse, analytics… soon probably geo features and even more mobile support.
Do not forget that they have introduced a payment system (it NEEDS coding, security…) and many other stuff. And they claim it’s only 50 devs. That’s small amount. Very agile though. And very smart, but still it sounds like to organization is very RIGID in its agility.

HAve You noticed that the only part of information You can actually edit is Your profile (which only You can edit obviously) and small objects like events. This very smart of them (think about propagation of changes…) and makes the problem of advanced rights management and access managment go away.

I truly admire FB for its model (true also for google and android). And even though I love MS (.net rocks) it can’t be that agile in the consumer world (but is very good in business which will probably be it’s target market or at least where the money is.)

Describing Facebook as not having QA is inaccurate; the operations team is “Quality Assurance”. No similated (test) environment will tell you how live users will react to most changes; better to have your operations team measure what happens.

Some features (like, e.g., search ranking improvements at Google) may be testable via replaying logs, which the dev should do anyway in developing the feature.

For the rest, the operations team having great user behavior metrics, live feedback on A/B tests, the ability to incremental rollout, and the ability to patch errors “instantly” on the live site, is much better than any traditional test team for Facebook.

Facebook isn’t shipped Office 95 on a gold CD. The old methodology of large internal test orgs simulating the user experience, rather than just measuring the actual user experience, no longer makes sense.

Very useful post: we have been moving in this direction for testing at PayScale, but it is great to see we aren’t the only ones.

First of all, very impressed with the process. Particularly with the OPS metrics/monitoring. Looks like they truly achieve development / release agility, particularly with a huge and complex service like Facebook.

[…] simply did not care about privacy. I am not sure, but if you read Skype Manager Yee Lee’s post on “How Facebook Ships Code” one could argue that the way in which the organization is set up certainly lends itself to some of […]

This article and one first referenced(concerning Mahalo) were INCREDIBLE! Thanks so much for writing it. A developer-driven culture is exactly what my employer needs to become and here is why:

1. I’m on a project where the business analyst (who owns the project) announced the product on May 3, 2010, but never got completed requirements done until Jan 21, 2011. Our 1st designer quit for personal reasons and our 2nd designer is grudingly moving along…not liking to make frequent UI designs.

I’m the single developer/architect for a product that may have 10,000 plus users, all the while doing maintenance on other projects. Priority is based on the squeaky wheel theory.

We have two QA teams at our firm. The fitst QA team has only found 1 bug in all my projects they’ve tested. The 2nd QA team has tested two other projects : they had one project for 6 weeks and found 0 bugs and another project they had for 4 weeks and found 1.

In a developer-driven culture we’d have saved on the QA and designers. These two articles really opened my eyes. Now to find such places in Chicago.

Very interesting article. Seems like a very unique culture that they have developed that would definately succeed in motivating staff. Especially letting them choose which projects they want to work on!

[…] or developer-centric organizations is being actively discuss lately thanks to the likes of Facebook. Some people, mostly developers, love it for its freedom and lack of heavy processes. I love it too […]

[…] Around the same time that I wrote my last post “How to make Agile work for product development… It’s all about the people” a colleague sent me a link to an interesting and insightful post by @yeeguy on “How Facebook Ships Code” […]

I really don’t find anything unique here, this is process people follow in every Start-up until they fail and then they get back to normal and standard process… Face-book fluke in success is just prolonged and it wont take them much time to change as well.

[…] on to the video above there is this article. Which could use some editing, but is very informative. https://framethink.wordpress.com/… Some pretty damn disruptive and clever concepts in there:1) No dedicated QA, for realz2) Any […]

Here’s a post which contains a link to a great detailed case study of Facebook with comments from Don Reinertsen, Alistair Cockburn, Jeff Sutherland, Yishan Wong (former Director of Engineering at Facebook).

For one, FB is not a mission critical application. If you update your friend’s wall but it doesn’t show up in half an hour (occasionally), nobody would notice. Think about that if you are submitting an order online…

The only reason why FB can let dev push out code freely is because they can afford to. Many businesses/applications do not have that luxury.

[…] How Facebook Ships Code – Facebook has over 650 million users and over 60 000 servers. Integration, testing and deploying a new version of the application must a daunting task. This article explains how they do it. […]

[…] no shortage of articles about Facebook these days, but I suggest you read this interesting take on How Facebook Ships Code. According to the author, over 50 percent of Facebook employees reside in the Engineering or […]

Nice topic…you may be interested to know about Social India Conference 2011 in Bangalore,India which is organized by Akshay Patra Foundation to raise funds for a non-profit cause. The event brings together world’s well known social media speakers at the event…visithttp://www.socialindiaconference.in for more details

[…] features, live, at any time. From Google they borrow a free-ranging engineering-led culture, but with almost zero central coordination. “[E]ngineers can modify specs mid-process, re-order work projects, and inject new feature ideas […]

[…] no shortage of articles about Facebook these days, but I suggest you read this interesting take on How Facebook Ships Code. According to the author, over 50 percent of Facebook employees reside in the Engineering or […]

[…] On the desktop, Facebook is a machine designed to make itself better. They hoover up data about how you’re using every piece of the service. They A/B (or A/X) test every single part of the user experience. Based on all that feedback, they tweak and tweak and tweak. Then, they do it all over again. Disputes can be settled by simply testing alternatives on a small base of users: may the best data win. And they can do all this quickly because on the web, Facebook can change the code any time they damn well please (although mostly once a week on Tuesdays). […]

[…] On the desktop, Facebook is a machine designed to make itself better. They hoover up data about how you’re using every piece of the service. They A/B (or A/X) test every single part of the user experience. Based on all that feedback, they tweak and tweak and tweak. Then, they do it all over again. Disputes can be settled by simply testing alternatives on a small base of users: may the best data win. And they can do all this quickly because on the web, Facebook can change the code any time they damn well please (although mostly once a week on Tuesdays). […]

[…] On the desktop, Facebook is a machine designed to make itself better. They hoover up data about how you’re using every piece of the service. They A/B (or A/X) test every single part of the user experience. Based on all that feedback, they tweak and tweak and tweak. Then, they do it all over again. Disputes can be settled by simply testing alternatives on a small base of users: may the best data win. And they can do all this quickly because on the web, Facebook can change the code any time they damn well please (although mostly once a week on Tuesdays). […]

[…] On the desktop, Facebook is a machine designed to make itself better. They hoover up data about how you’re using every piece of the service. They A/B (or A/X) test every single part of the user experience. Based on all that feedback, they tweak and tweak and tweak. Then, they do it all over again. Disputes can be settled by simply testing alternatives on a small base of users: may the best data win. And they can do all this quickly because on the web, Facebook can change the code any time they damn well please (although mostly once a week on Tuesdays). […]

[…] On the desktop, Facebook is a machine designed to make itself better. They hoover up data about how you’re using every piece of the service. They A/B (or A/X) test every single part of the user experience. Based on all that feedback, they tweak and tweak and tweak. Then, they do it all over again. Disputes can be settled by simply testing alternatives on a small base of users: may the best data win. And they can do all this quickly because on the web, Facebook can change the code any time they damn well please (although mostly once a week on Tuesdays). […]

[…] On the desktop, Facebook is a machine designed to make itself better. They hoover up data about how you’re using every piece of the service. They A/B (or A/X) test every single part of the user experience. Based on all that feedback, they tweak and tweak and tweak. Then, they do it all over again. Disputes can be settled by simply testing alternatives on a small base of users: may the best data win. And they can do all this quickly because on the web, Facebook can change the code any time they damn well please (although mostly once a week on Tuesdays). […]

[…] process, especially early on in their growth. Wrote up this blog post on that topic in Jan 2011:https://framethink.wordpress.com/…I'm sure their PD process has evolved since then, especially now that they've gone public […]

[…] become part of the culture. For example, at Facebook engineers can change anybody else’s code without asking. And with Microsoft integrating Yammer into Office 365, collaborating on writing will become the […]

[…] continuous deployment process. FB developers are responsible for their own QA, and part of that is to push code out to a limited set of servers, see results and then push it out to more and do this each day if not more often. With Apple taking as least a week of review, rolling back a […]