How Facebook Survived 34 Intense Days Of "Lockdown" To Build Graph Search

It’s been merely two weeks since Facebook launched its Graph Search tool in limited beta, but already Loren Cheng has had his hands full. Cheng is a product manager on Facebook’s Graph Search team; Cheng’s team-within-the-team specifically worked on natural language problems: that is, ensuring that when people type ordinary English into the search bar, Graph Search is smart enough to return the proper results.

"One of the difficulties," Cheng tells Fast Company, "is the ambiguity of the English." Say you have a user who enters the query "photos of engineers in Mumbai." Is that engineers who live in Mumbai, or engineers who happened to be in Mumbai at the moment a photograph was taken? Graph Search is smart enough to make the educated guess that you want the first set, but at the top of the screen it will nonetheless offer you the second option, with clarifying language.

A neon sign announces when a Facebook team is on "lockdown," a blinders-on, launch-is-imminent, no-sleep-till-breakthrough phase of development.

Cheng can't share exact data on how Graph Search was being used just yet, except to say that some of the most popular queries are often some of the simplest. Cheng speaks of welcoming testers into Facebook’s user experience lab. "The subject is sitting on the other side, and we’re itching for them to type complex queries," ones that will really test the capabilities of the product, he says. Unmarried friends of friends who like disco dancing. Movies liked by people whose music I like and who like my liked books.

Cheng and his team will wait with bated breath, on the other side of the window. "And they’ll type in: ‘photos of friends,’" he says.

"We had a guy who worked for like two or three days straight and didn’t sleep. He didn’t come into work the next day and we were a little worried. He was just sleeping at home."

Cheng also told us more about a Facebook tradition: the change of pace and practice characterizing the run-up to a product launch, a phase Facebook terms "lockdown."

LOREN CHENG: It’s a time at Facebook when we say, "Hey, we’ve got this goal, it’s pretty near term—we’re gonna double down and focus on it." It lends focus. You make decisions faster. You shut down other meetings. At the beginning of our lockdown, the team got together and had a massive five-hour meeting where nobody was allowed to leave, where we figured out exactly what we were going to tackle. We filled three whiteboards full of ideas, and eventually I said, "Look, we have a very limited amount of time. If you would fall on your sword for something to be in there, what would it be?" It ended up being our "sword list," and all tasks had to relate to something on the sword list.

LOREN CHENG

A five-hour meeting? Who was in charge of keeping everyone in the room?

It was pretty clear who didn’t let anybody leave: That would be me. We had hundreds if not thousands of tasks to do, and I knew people had different concepts of what they wanted to do. Everyone was like, "Whoa, what? We don’t even have hour-long meetings at Facebook!" The nice thing was that after the five hours, we wrote up that list on a whiteboard and put it in our area. We told all the other teams: If you want something and it’s not on this list, we’re not gonna do it. It was a great way of messaging. It helped to have that physical thing, the whiteboard. Everyone could walk by and say, I know what they’re focused on. The meeting was excruciating—five hours!—but we walked out of it with this sense of focus.

How did you sense it was time for an unusual, epic meeting?

I was hearing conversations in the work area. I was sometimes seeing someone make a decision that I thought we’d already made, and then two weeks later, they were revisiting that decision. People were having a lot of these circle conversations, and I think everyone else was feeling the same thing, that we needed to figure out how to finalize some of these decisions.

Does lockdown translate into longer hours for your team?

Do people work longer hours? Probably. The end’s in sight, you’re tightening the focus, you’re spending a little more time: Those are the hallmarks of lockdown. There’s some craziness. We had a guy who worked for like two or three days straight and didn’t sleep. He didn’t come into work the next day and we were a little worried. He was just sleeping at home. We have a hammock we take naps on, and people occasionally take sleeping bags to work if they’re excited about something and just want to jam through it.

Did you personally sleep over during lockdown on Graph Search?

Yeah, there was a night or two. In Building 18, there’s a conference room that no one uses with two incredibly comfy couches. I went to take a nap "for 20 minutes," set my alarm, and woke up four hours later.

When you are in love with something, you can be completely focused. For most of these FB people, FB is their love. For me, its my business. I literally have to force myself to sleep, to eat, to reconnect with my family or I will sleep at my desk most nights.

Those of you who haven't loved their jobs in awhile, I can see why you don't understand. If you currently have a J.O.B. and can't wait for 5pm to roll around, you don't understand.

As for me, my work is my life. I have a wife and kids who I love very much, but they understand that when daddy is working on an important project he needs to be left alone. I donate 4 hours per day to my family when they are home, and 16 hours to my work when they are in bed or busy. 4 hours to sleep (except for naps).

I completely get the long nights and sleeping in your office. I believe more Americans should find the one task they are made to do and fall in love with that task every day!

After reading this article, I still don't know how the FB employees survived 34 days of lockdown. I have a vague idea what "lockdown" is at the company, I know about the 5-hour meeting that kicked it off, the potential longer hours that people worked...but *nothing* substantive that directly relates to article's title!

I would quit if I were made to work such long hours. Especially when you consider that your work is making the top-ranks richer, and killing you slowly. Some would be very proud of this achievement...I think it's sad to slave people that way.

This kind of practice is not uncommon in the software development business. I once had to endure 90 days of 14-18 hour work days 7 days a week. I got super sick and was ready to hurt people, but it was what was expected if I were to keep my job.

I've always found my best work comes with being rested, having time to iterate of my work, and not being in meetings. When I hear engineers made Facebook in sleepless, prison-style 'lockdown' I think they made a very engineer-like quantitative decision: X number of hours in X amount of time. Where's the creativity and delight? FB is color blind and blue. Sad.

This just seems very expert system, in some ways the "easy" AI. When they say "is intelligent enough to know you mean engineers who live in Mumbai" - they will have programmed the response, as [person detail] that live in [location] - will be a typical question and will have a set response. This does not strike me as very impressive AI, although it will be quite nice to be able to ask in English. By the 80/20 rule most of the searches to Graph Search on facebook will be very simple types. I get the impression most facebook users don't use the site in an intelligent way. Will it be worth doing the more complex ones? Perhaps. They do sound fun for engineers to solve. But they may just be opening a can of worms AI people have been trying to solve for decades. It will be much harder than they think as the number of meanings a sentence can have rapidly goes up, and the only way of solving it is by cross referencing a massive amount of general background knowledge, knowledge that we as humans take for granted. We don't realise just how much processing goes on when we recognise something. The brain is a massively capable parallel processing machine. I think they may be taking on more than they can chew, but by all means have fun with it! I wish them luck and slightly envious I am not working on it!