Why Facebook thinks Blu-ray discs are perfect for the data center

Cheap storage with a 50-year life span? Facebook spreads the gospel of Blu-ray.

Facebook's hardware guru thinks Blu-ray discs might have a brighter future in the data center than in consumers' homes.

We wrote on Wednesday about how Facebook has developed a prototype storage system that uses 10,000 Blu-ray discs to hold a petabyte of data. After that story posted we were able to talk to Frank Frankovsky, VP of hardware design and supply chain operations at Facebook, to find out just why he's so excited about the project.

While the Blu-ray storage system is just a prototype, Facebook hopes to get it in production sometime this year and share the design with the Open Compute Project community to spur adoption elsewhere. If Facebook and others start using Blu-ray discs for long-term archival storage, Blu-ray manufacturers will see a new market opportunity and pursue it, Frankovsky said.

"I think that the media suppliers, especially after all of the community excitement around it with Open Compute, they see a huge opportunity here," Frankovsky said. "Economies of scale could take over really quickly, and they could start producing those discs for the Open Compute community at much lower cost than they do today because, believe it or not, this is one of those areas where really high-capacity Blu-ray discs are in relatively low demand on the consumer side and in relatively high demand on the data center side."

Facebook VP Jay Parikh discusses cold storage and Blu-rays.

Facebook intends to use Blu-rays for "cold storage," data that can't be thrown out but may not be accessed for many years, if at all. The best near-term use case is backups of photos and videos, but the discs could also be used for any data that Facebook is required by law to retain for a certain number of years.

Facebook's cold storage today is entirely on spinning disks. The prototype Blu-ray system is estimated to be 50 percent cheaper than the disk-based cold storage, and 80 percent more energy efficient.

The discs are housed in a fancy rack that holds 24 magazines, with each magazine holding 36 cartridges, and each cartridge holding 12 Blu-ray discs, for a total of 10,368. A robot lives and works inside the rack.

"We have a robotic picker that will go to a specified magazine and then locate a cartridge, it will unlock that cartridge, removing the drawer, and a picker will go down and is able to select a specified disc in that 12-disc arrangement," Facebook's Giovanni Coglitore said in a video demo posted yesterday.

When the robot isn't working, the rack consumes virtually no power, he said.

"Each disc is certified for 50 years of operation; you can actually get some discs that are certified for 1,000 years of reliability," Coglitore said. "Because the media is separate from the drives, if you ever have a drive issue, you simply replace the drive, and you won't have to replace the data within a disc. From a reliability and operational standpoint it's quite elegant and efficient."

Facebook is careful in how it's rolling out the optical storage system to production.

As Facebook does with other new technologies, "we'll start it out in what we call shadow testing," Frankovsky said. "Until it's proven, we'll take relatively small quantities, and we'll just mirror data from what's in production and shadow that data to the optical rack."

While Blu-ray can't match the performance of hard disk drives, that isn't as important for cold storage. Frankovsky also said the Blu-ray system will be "far superior to tape" because of its durability and performance.

It could also provide benefits in recovery from failure. Frankovsky said Facebook uses erasure encoding, which "distributes a file across multiple physical devices so that in any event of a failure of a physical device you can always recreate the file."

With disk drives, "you need to have a relatively aggressive erasure encoding environment where you have a lot more physical spinning disks, so you can sustain multiple failures and be able to recreate the file in any situation," Frankovsky said. Because the predictive annual failure rate for optical discs is lower, "it's quite possible that you won't need to over-provision how much optical capacity you have to be able to get the same level of protection with the complete file rebuild. There's kind of a double goodness there. Bit for bit, it's lower cost, and if we can over-provision less with optical, you can move that needle even further on the cost-saving side."

Separately, Facebook has also considered the use of substandard flash memory for cold storage. The idea is to take "partially good NAND flash which otherwise would have been sold off as thumb drives or potentially even scrap," Frankovsky said.

Flash storage is often thrown out by manufacturers when just a few cells have gone bad, he said. With smart enough software algorithms, "you can be cognizant of where the weak or bad cells are and write around those cells."

The bad flash initiative isn't as far along as the Blu-ray project. But both have progressed pretty quickly since Facebook revealed their existence last year, Frankovsky said. They could end up being important tools as data storage needs grow.

IDC predicts that by 2020 the entire "digital universe" will grow to 40,000 exabytes, or 40 trillion gigabytes.

"A large portion of that is going to be warm to cold data, and we need something better than tape and disk to store it," Frankovsky said.

Promoted Comments

I did this when I was low on HDD space many years ago at university, but with a fast internet connection.The fact that DVDs (as it was then) were so much cheaper made it affordable to back everything up.200GB HDD, or half the price for 200 DVD-Rs with 4x the capacity.

The problem I had was that after not that much time, there were various errors and other issues either meaning the whole disc was useless, or individual files (mostly, ahem, DVD/film rip backups) couldn't be accessed.

Given that they are probably going to select BluRays which have been made to a high standard, rather than the lowest cost ones, the coatings/etc will probably have guarantees from testing.They can also potentially do things like having parity files on the disc in case one part of it gets damaged or scratched, in order to restore the files on that disc, as well as also having multiple copies distributing the data.

It makes perfect sense since they would have a storage system built specifically to optimally store the Blurays, rather than is the case with consumers who might move about the container they put their discs in, plus having multiple backups reduces failure chances, and you can do on-disc parity as well for bad sectors.Plus the whole thing with needing people onboard to make the discs is likely related to having quality discs available to purchase, rather than cheap consumer level discs/coatings. That's where you get your comfort from. Not all discs are created equal.

Mechanical drive storage might be "safer", but that's because once you see a first failure, you can then respond to it, while Bluray failures would be hidden until you try and access the data, but the key period of damage would be when the discs are being handled/used, and that time will be minimal.

In an industry where most formats and standards usually die (or at least go on life support) within 10 years and therefore aren't build to last forever, I wonder how Blu-ray discs can be certified for 1000 years or even 50 years ?

You can make assumptions about the decay rate of your components, but how do you test that your guess is correct for 50 years on a format that isn't even 10 years old ?

I understand that blu-ray makers can try to give a rating, but as a consumer, do you trust that enough to build your infrastructure around it ?

I did this when I was low on HDD space many years ago at university, but with a fast internet connection.The fact that DVDs (as it was then) were so much cheaper made it affordable to back everything up.200GB HDD, or half the price for 200 DVD-Rs with 4x the capacity.

The problem I had was that after not that much time, there were various errors and other issues either meaning the whole disc was useless, or individual files (mostly, ahem, DVD/film rip backups) couldn't be accessed.

Given that they are probably going to select BluRays which have been made to a high standard, rather than the lowest cost ones, the coatings/etc will probably have guarantees from testing.They can also potentially do things like having parity files on the disc in case one part of it gets damaged or scratched, in order to restore the files on that disc, as well as also having multiple copies distributing the data.

It makes perfect sense since they would have a storage system built specifically to optimally store the Blurays, rather than is the case with consumers who might move about the container they put their discs in, plus having multiple backups reduces failure chances, and you can do on-disc parity as well for bad sectors.Plus the whole thing with needing people onboard to make the discs is likely related to having quality discs available to purchase, rather than cheap consumer level discs/coatings. That's where you get your comfort from. Not all discs are created equal.

Mechanical drive storage might be "safer", but that's because once you see a first failure, you can then respond to it, while Bluray failures would be hidden until you try and access the data, but the key period of damage would be when the discs are being handled/used, and that time will be minimal.

I dunno, I'm on board with this. I wasn't yet in the workforce at this time, but I'd be surprised if the same discussion didn't go on years ago when companies first started looking at 7,200 RPM SATA replacing 10k/15k RPM SCSI. I'm sure there were those that just couldn't get on board with the reliability of SATA drives in an enterprise environment, but at least in most of the shops I've worked in, it's all SATA. Redundant commodity hardware is the name of the game when you're working on the scale that Facebook is.

And seriously folks, we need to stop comparing the reliability and life expectancy of consumer recordable media to what companies could get from an OEM in bulk if the need and specify reliability. Consumer grade recordable media has crappy reliability because that's the status quo.. give consumers crappy quality writable disks and charge cheap enough, people won't expect wonders out of them. Facebook wouldn't dare use the current design of consumer disks in their datacenters. And for those that insist on comparing the price of consumer writable disks to the cost of tape, you'd better believe Facebook will be getting them exponentially cheaper than you can find from Newegg or Amazon.

I still have my social media tablet from year 1025 certified for 1000 years of data storage. It has "Angus taketh thou crossbow up thine butt,," chisled into it. I guess things don't really change all that much.

I remember doing the same basic thing with CD-ROMs decades ago, even to the point of playing around with the concept of RAID striped across multiple optical disks. What are they doing here that people seem to be perceiving as novel?

I have spent the last 11 years of my career in the Storage side of the business. I wrote a whitepaper a few years ago for a company and in doing the research found that BD and DVD, although cheaper for a backup or two quickly went through the roof at scale. The crossover point was It was between 5-10TB in 2007. I just don't see the economics of this move.

And seriously folks, we need to stop comparing the reliability and life expectancy of consumer recordable media to what companies could get from an OEM in bulk if the need and specify reliability. Consumer grade recordable media has crappy reliability because that's the status quo.. give consumers crappy quality writable disks and charge cheap enough, people won't expect wonders out of them. Facebook wouldn't dare use the current design of consumer disks in their datacenters. And for those that insist on comparing the price of consumer writable disks to the cost of tape, you'd better believe Facebook will be getting them exponentially cheaper than you can find from Newegg or Amazon.

Well, you may be right about those differences, but be reminded that DVDs were never intended and therefore never certified as long-term storage. They were designed as a relatively cheap replacement for VHS tapes, but not for data preservation. DVDs have a relatively simple error correction logic which can be defeated by even light scratches. It's not as bad as in GDROMs (yeah, those good old days), but it's still pretty easy to lose huge amounts of data.

BluRays, on the other hand, have been certified for long-term data preservation, because they were designed with a much more complex ECC model with a much more random distribution of data on the surface and more space reserved for ECC data, so in most cases a single scratch won't be able to damage even one sector. Also the standard coating on the disc is really really tough to scratch, even with cheap discs.

Call me cynical, but I see this more as an excuse for them to retain data on their product (the users) forever. "Sorry, we can't erase what we have on you, its on Blu-ray."

Once this is live you are correct if all your pictures go into a cold storage then there is really no way to "delete" your account, and no way to have your personal information removed from the system without taking out their entire cold storage array with it.

Call me cynical, but I see this more as an excuse for them to retain data on their product (the users) forever. "Sorry, we can't erase what we have on you, its on Blu-ray."

Once this is live you are correct if all your pictures go into a cold storage then there is really no way to "delete" your account, and no way to have your personal information removed from the system without taking out their entire cold storage array with it.

Well, maybe they encrypt the data using a key that matches your account, which gets thrown out once you delete your sccount.... Nah, who would ever do that...

A lot of these comments about FB keeping data archived seem to ignore that your data is already being archived. They're not developing a new system to store your data long term - they're thinking about replacing their existing one.

As for ease of deleting data, I assure you that deleting from a BD is no more difficult than deleting from tape.

I have spent the last 11 years of my career in the Storage side of the business. I wrote a whitepaper a few years ago for a company and in doing the research found that BD and DVD, although cheaper for a backup or two quickly went through the roof at scale. The crossover point was It was between 5-10TB in 2007. I just don't see the economics of this move.

That may have been the case a few years ago, but I think the people dealing with some of the biggest data in the world will have put in the work to know what they're getting into.

I too was expecting that, but I'm sure we'll see some kind of return fire from the LTO backers. Frankly, I'm really wondering why FB has chosen this relatively new (and untested) route when (as I'm sure the Tape crowd have told the entire world now) LTO-6 holds 2500GB native and robotic tape changers are commonplace. The overall cost/risk benefit had to be substantial in BRay's favor, lets just hope that they contribute this idea to the Open Compute project and we can see with our own eyes.

Lastly, although I loathe most of what FB does and stands for, I applaud them for pushing the hardware envelope in new and unexpected ways.

It'd be funny to see the performance of their sql queries on their blue-ray archives. "Oh! You want to find your earliest posts? Please wait 4-6 weeks while our disks load."

4-6 weeks would be enough to read through the whole petabyte. This is for cold storage, so probably mostly idle. That means seconds for the robot to swap in the correct disc, a fraction of a second to read the data.

I know a lot of tech people are waiting with bated breath for the death of optical, but not I. For the better part of a decade I've been burning my.. ahem.. "downloads" to Taiyo Yuden DVD-Rs for archival and viewing. I have literally hundreds of disks organized in CD books and have not had any failures.

Call me cynical, but I see this more as an excuse for them to retain data on their product (the users) forever. "Sorry, we can't erase what we have on you, its on Blu-ray."

I'm rather disappointed Ars didn't seem to ask that question. Even if Facebook didn't give an answer, at least let us known in the article that the question was asked.

You already can't really delete things from Facebook. They remove the link to the file(s) but the data is never really erased. (Although I guess they may chose to not copy erased data when they need to migrate to replacement storage.)

Fifty years? Sure thing, pal, whatever you say or whatever they pitch you in a meeting. It's a piece of plastic with a thin coating of metal on it. The thing about digital media, about digitizing everything in our lives, is that the media it's on isn't going to last, and the media certainly hasn't been around fifty years, or a thousand years. I don't buy a word of it.

Backup to every media you can find, in multiple sets, stored apart from one another in hella-safe storage somewhere at the bottom of a cave... WITH a device that can read the media. And even then your data is probably not all that safe. Now, carving 1s and 0s into clay tablets -- THAT HAS PROMISE FOR THE LONG TERM!

My experience with optical media (CD-R, DVD±R, no BD-R yet) is good. Most of my media is good quality (mostly TDK for CDs, mostly Verbatim [MCC] for DVDs) and I'm yet to have lost any data on the hundreds of discs I have, as far as I know.

The oldest CD-R I have was written 17 years ago. When I scanned it a year or two ago it was perfectly readable, with error levels (C1, C2) that are good also for a newly written disc.

In general, it seems error levels may increase over time, but so far it seems gradual, so predictable, and it's far from close to being critical. I've read data off discs with much much higher error levels (not part of my archive), and it was fine, just slower reading. If one of my discs got anywhere close to that state I'd have time to move the data to a new disc.

If the BD-R media cost $0.50 then it has price parity with non-volume purchases of LTO5 media (well, with no volume discount on the LTO5 media). The LTO media I saw was rated for 30 years. I wonder what conditions you'd have to store BD-R media in to get the full 50 years of life.

I hope this Facebook project will lead to advancement in BD media, and optical in general. Currently there just isn't any other archival/nearline medium (in the consumer space, at least). HDDs couple the mechanical parts and electronics to the media. SSDs use flash with limited data retention, and are completely electronics based.

The move from floppies to CDs was monumental. From CDs to DVDs underwhelming. DVDs to BDs was slightly worse.

My experience with optical media (CD-R, DVD±R, no BD-R yet) is good. Most of my media is good quality (mostly TDK for CDs, mostly Verbatim [MCC] for DVDs) and I'm yet to have lost any data on the hundreds of discs I have, as far as I know.

The oldest CD-R I have was written 17 years ago. When I scanned it a year or two ago it was perfectly readable, with error levels that are good also for a newly written disc.

In general, it seems error levels may increase over time, but so far it seems gradual, so predictable, and it's far from close to being critical. I've read data off discs with much much higher error levels (not part of my archive), and it was fine, just slower reading. If one of my discs got anywhere close to that state I'd have time to move the data to a new disc.

I had a first gen phillips cd rw and have used cd's as a medium extensively due to playing a lot of music in bands and bars and what have you. obviously cds are useful as a musician. prior to cdr I used DAT. I've been using CDRs for a long time and had a computer with multiple hard drives to work as a cd toaster for a long long time.

you need to handle your cds with extreme care. I've lost countless cdrs due to degradation despite being cased and unused. I've burned hundreds upon hundreds of cds but always also had multiple copies on multiple disks and finally now have my important data archived on multiple tapes. I would still keep CDR masters for copy purposes, nearly all of my master disks from the late 90s are dead. 75-90% while all of my DAT recordings are intact. I've managed to keep redundant harddrives running and data copied off before a disk failure can happen., knock on wood.

Magnetic tape, despite it's large volume capacity, does not archive well over long durations of time. The plastic degrades over time due to age and the mechanical wear of flexing, stretching, and it suffers from Sticky-shed Syndrome. Also, magnetic tape is very slow if you want to retrieve random spots of information on the fly, due to its one dimensional, linear nature. A disk, such as Blu-ray, is predominately two dimensional (three if you count layering), facilitating much faster on demand access to stored information.

These are a lot of the reasons that I no longer own any VCR or compact cassette tapes in my media collection.