That's a total of more than 21 PB of configured storage capacity! This is larger than the previously known Yahoo!'s cluster of 14 PB. Here are the cluster statistics from the HDFS cluster at Facebook:

Hadoop started at Yahoo! and full marks to Yahoo! for developing such critical infrastructure technology in the open. I started working with Hadoop when I joined Yahoo! in 2006. Hadoop was in its infancy at that time and I was fortunate to be part of the core set of Hadoop engineers at Yahoo!. Many thanks to Doug Cutting for creating Hadoop and Eric14 for convincing the executing management at Yahoo! to develop Hadoop as open source software.

Facebook engineers work closely with the Hadoop engineering team at Yahoo! to push Hadoop to greater scalability and performance. Facebook has many Hadoop clusters, the largest among them is the one that is used for Datawarehousing. Here are some statistics that describe a few characteristics of the Facebook's Datawarehousing Hadoop cluster:

Here are two pictorial representations of the rate of growth of the Hadoop cluster:

Details about our Hadoop configuration

I have fielded many questions from developers and system administrators about the Hadoop configuration that is deployed in the Facebook Hadoop Datawarehouse. Some of these questions are from Linux kernel developers who would like to make Linux swapping work better with Hadoop workload; other questions are from JVM developers who may attempt to make Hadoop run faster for processes with large heap size; yet others are from GPU architects who would like to port a Hadoop workload to run on GPUs. To enable this type of outside research, here are the details about the Facebook's Hadoop warehouse configurations. I hope this open sharing of infrastructure details from Facebook jumpstarts the research community to design ways and means to optimize systems for Hadoop usage.

people who are concerned about carbon foot print here is my answer, the scenario would have been worse, the number of servers needing to serve such huge task is humengous and hadoop optimizies the resources.

@Naren: we have one admin person who manages the hdfs cluster. He is a person responsible for deploying new software, monitoring health, reporting and categorization of issues that arise as part of operations, etc.etc. Then maybe another virtual person(s) who spends a few hours every week to gather all failed machines/disks and send them to a repair facility.

1. we use LZO for map outputs (less CPU) but use GZIP for reduce outputs (lesser disk space).

2. we have 12 spindles.

3. our map or reduce computations are very CPU heavy and the cluster is bottlenecked on CPU (rather than IOPs). The 1 GB per task is just the default. Most jobs (via Hive) are allowed to set their own JVM heap size.

hi I have 4 machines Suse-Linux11 , I need to set up a 4 node hadoop cluster I have RAM 16GB [16 cores] per machines.I need to know how may maps and reduces should I configure? Also Can I have multiple clusters on same 4 machines by just changing the port numbers and other directories and running hadoop with separate user.?

What is your backup plan for the Hadoop cluster? does backup of hadoop cluster makes sense for you? if so do you quiesce the hive before backup? and how is new/modified data detected (as the data sizes are so huge)?

@toni: we use custom scripts to configure and deploy software on hadoop machines.

@The Hive cluster is a pure warehouse. That means that if you backup the 20+ TB of new data that comes in every day, all other data can b derived from that stream. So, we have processes to replicate data ascross data centers and as long as we can copy the source data to multiple data centers, we have a good story on backup (including DR).

@Jeff: we focussed on job-pipleline latencies. That means a certain pipeline (bunch of hive jobs) have to finish within a certain time. Regarding ur other question: we have had cases when a rack fails. A rack has 20 machines. When this happens, we see that HDFS re-replicates the data and this re-replication finishes in about an hour, i.e. our mean-time-to-recover from a failed rack is about 1 hour. However, jobs continue to run normally during this period.

Dhrub,Your comment >> A rack has 20 machines. When this happens, we see that HDFS re-replicates the data and this re-replication finishes in about an hour, i.e. our mean-time-to-recover from a failed rack is about 1 hour.

Unlike facebook, where you have 2000 machine (with 20 m/c per rack,so I am assuming you have 100 racks), the re-replication takes about an hour. For relatively small clusters - say 60 nodes (i.e. 3 racks with 20 nodes each) when a rack fails the re-replication can overwhelm Top-of-rack switch and the re-replication duration can be larger. Does the re-replication rate-limited ? Any suggestions and/or possible performance numbers for recovery time for such failures?

Hello,what a amazing news is this! The Datawarehouse Hadoop cluster at Facebook has become the largest known Hadoop storage cluster in the world is really a excellent information.I love it.Thanks a lot Used Pallet Racks

Hey this shows the scope of HADOOP.What do you think programmers?Its time to learn Hadoop online.I am looking for online hadoop live tutorial means online course.and one of my friend suggested me WIZIQ for online learning having course id 21308.and they are giving free demo for any course.So what do u think?I am thinking to take such course and make myself scope in such field.Wanna learn HADOOP then do check once WIZIQ.Thank You.

@Shruti: Ya Hadoop has great scope now a days.As you can get an idea from this blog too.And I took this course from WIZIQ and now I am doing job as HADOOP developer.wanna tell you that this course is awsme as tutor is cloudera certified and he knows where we lag and where we make mistakes.Thanks to him as I got a job only because of that tutor.And yes WIZIQ is very supportive and Very responsive.Just close your eyes and click enroll button.:)Hope it will be helpful for you.