Update:YouTube: The Platform. YouTube adds a new rich set of APIs in order to become your video platform leader--all for free. Upload, edit, watch, search, and comment on video from your own site without visiting YouTube. Compose your site internally from APIs because you'll need to expose them later anyway.

YouTube grew incredibly fast, to over 100 million video views per day, with only a handful of people responsible for scaling the site. How did they manage to deliver all that video to all those users? And how have they evolved since being acquired by Google?

Web Servers

Application server talks to various databases and other informations sources to get all the data and formats the html page.

Can usually scale web tier by adding more machines.

The Python web code is usually NOT the bottleneck, it spends most of its time blocked on RPCs.

Python allows rapid flexible development and deployment. This is critical given the competition they face.

Usually less than 100 ms page service times.

Use psyco, a dynamic python->C compiler that uses a JIT compiler approach to optimize inner loops.

For high CPU intensive activities like encryption, they use C extensions.

Some pre-generated cached HTML for expensive to render blocks.

Row level caching in the database.

Fully formed Python objects are cached.

Some data are calculated and sent to each application so the values are cached in local memory. This is an underused strategy. The fastest cache is in your application server and it doesn't take much time to send precalculated data to all your servers. Just have an agent that watches for changes, precalculates, and sends.

Video Serving

Costs include bandwidth, hardware, and power consumption.

Each video hosted by a mini-cluster. Each video is served by more than one machine.

Using a a cluster means:- More disks serving content which means more speed.- Headroom. If a machine goes down others can take over.- There are online backups.

Servers use the lighttpd web server for video:- Apache had too much overhead.- Uses epoll to wait on multiple fds.- Switched from single process to multiple process configuration to handle more connections.

Most popular content is moved to a CDN (content delivery network):- CDNs replicate content in multiple places. There's a better chance of content being closer to the user, with fewer hops, and content will run over a more friendly network.- CDN machines mostly serve out of memory because the content is so popular there's little thrashing of content into and out of memory.

Less popular content (1-20 views per day) uses YouTube servers in various colo sites.- There's a long tail effect. A video may have a few plays, but lots of videos are being played. Random disks blocks are being accessed.- Caching doesn't do a lot of good in this scenario, so spending money on more cache may not make sense. This is a very interesting point. If you have a long tail product caching won't always be your performance savior. - Tune RAID controller and pay attention to other lower level issues to help.- Tune memory on each machine so there's not too much and not too little.

Serving Video Key Points

Keep it simple and cheap.

Keep a simple network path. Not too many devices between content and users. Routers, switches, and other appliances may not be able to keep up with so much load.

Use commodity hardware. More expensive hardware gets the more expensive everything else gets too (support contracts). You are also less likely find help on the net.

Use simple common tools. They use most tools build into Linux and layer on top of those.

Handle random seeks well (SATA, tweaks).

Serving Thumbnails

Surprisingly difficult to do efficiently.

There are a like 4 thumbnails for each video so there are a lot more thumbnails than videos.

Thumbnails are hosted on just a few machines.

Saw problems associated with serving a lot of small objects:- Lots of disk seeks and problems with inode caches and page caches at OS level.- Ran into per directory file limit. Ext3 in particular. Moved to a more hierarchical structure. Recent improvements in the 2.6 kernel may improve Ext3 large directory handling up to 100 times, yet storing lots of files in a file system is still not a good idea.- A high number of requests/sec as web pages can display 60 thumbnails on page.- Under such high loads Apache performed badly.- Used squid (reverse proxy) in front of Apache. This worked for a while, but as load increased performance eventually decreased. Went from 300 requests/second to 20.- Tried using lighttpd but with a single threaded it stalled. Run into problems with multiprocesses mode because they would each keep a separate cache.- With so many images setting up a new machine took over 24 hours.- Rebooting machine took 6-10 hours for cache to warm up to not go to disk.

To solve all their problems they started using Google's BigTable, a distributed data store:- Avoids small file problem because it clumps files together.- Fast, fault tolerant. Assumes its working on a unreliable network.- Lower latency because it uses a distributed multilevel cache. This cache works across different collocation sites.- For more information on BigTable take a look at Google Architecture, GoogleTalk Architecture, and BigTable.

Databases

The Early Years- Use MySQL to store meta data like users, tags, and descriptions.- Served data off a monolithic RAID 10 Volume with 10 disks. - Living off credit cards so they leased hardware. When they needed more hardware to handle load it took a few days to order and get delivered. - They went through a common evolution: single server, went to a single master with multiple read slaves, then partitioned the database, and then settled on a sharding approach.- Suffered from replica lag. The master is multi-threaded and runs on a large machine so it can handle a lot of work. Slaves are single threaded and usually run on lesser machines and replication is asynchronous, so the slaves can lag significantly behind the master. - Updates cause cache misses which goes to disk where slow I/O causes slow replication.- Using a replicating architecture you need to spend a lot of money for incremental bits of write performance.- One of their solutions was prioritize traffic by splitting the data into two clusters: a video watch pool and a general cluster. The idea is that people want to watch video so that function should get the most resources. The social networking features of YouTube are less important so they can be routed to a less capable cluster.

The later years:- Went to database partitioning.- Split into shards with users assigned to different shards.- Spreads writes and reads.- Much better cache locality which means less IO.- Resulted in a 30% hardware reduction.- Reduced replica lag to 0.- Can now scale database almost arbitrarily.

Data Center Strategy

Used manage hosting providers at first. Living off credit cards so it was the only way.

Managed hosting can't scale with you. You can't control hardware or make favorable networking agreements.

So they went to a colocation arrangement. Now they can customize everything and negotiate their own contracts.

Use 5 or 6 data centers plus the CDN.

Videos come out of any data center. Not closest match or anything. If a video is popular enough it will move into the CDN.

Video bandwidth dependent, not really latency dependent. Can come from any colo.

For images latency matters, especially when you have 60 images on a page.

Images are replicated to different data centers using BigTable. Codelooks at different metrics to know who is closest.

Lessons Learned

Stall for time. Creative and risky tricks can help you cope in the short term while you work out longer term solutions.

Prioritize. Know what's essential to your service and prioritize your resources and efforts around those priorities.

Pick your battles. Don't be afraid to outsource some essential services. YouTube uses a CDN to distribute their most popular content. Creating their own network would have taken too long and cost too much. You may have similar opportunities in your system. Take a look at Software as a Service for more ideas.

Keep it simple! Simplicity allows you to rearchitect more quickly so you can respond to problems. It's true that nobody really knows what simplicity is, but if you aren't afraid to make changes then that's a good sign simplicity is happening.

Shard. Sharding helps to isolate and constrain storage, CPU, memory, and IO. It's not just about getting more writes performance.

You succeed as a team. Have a good cross discipline team that understands the whole system and what's underneath the system. People who can set up printers, machines, install networks, and so on. With a good team all things are possible.

Reader Comments (56)

The real meat of this is skipped over in a couple of lines: keep the popular content on a CDN. In other words, throw money at Akamai and let Akamai worry about it.

That is, of course, the right answer, but whether you're using Python or not hardly matters. Without Akamai's services, YouTube could never have kept up with the demand given the infrastructure described above.

- Living off credit cards so they leased hardware. When they needed more hardware to handle load it took a few days to order and get delivered.

How did this work exactly? When we looked into this, we found that because we were a new startup, we had no credit (I guess not 'found', its pretty obvious), so hardware leasing companies would only lease to us if one of us personally backed the loans. Given that startup risk was high and the bill was large, we ended up buying hardware and putting it on various low-intro-APR CC's, etc. All the big h/w vendors were like "unless we can see your last N years of tax returns etc., we're not leasing to you." Made it seem like leasing wasn't a real option for 'living off credit cards' startups.

be sure to accept private venture capital financing from Sequoia, who was also the largest shareholder of Google and controlled its board. Sequoia used its influence to force Google to massively overpay for Youtube, and the sequoia partners made an instant 500 million dollars in profit.

A very good article yet I have not learned anything new. The current YouTube architecture is already applied to one of our customer youtube-like, a romanian website called http://www.trilulilu.ro/ . The only thing that it wasn't yet implemented by our customer is database sharding, no need for now as the total MySQL database is under 250MB and the MySQL server handles at east more than 650 qps.

I think it's interesting to note that they used mod_fastcgi. I mean, I use it because I'm forced to on my shared host, but I've always heard of tons of problems when trying to scale big with it (even though that's what it's designed for). I guess if done right, FastCGI can be a great asset to a server farm.

Would love to hear more stories like this, Flickr, Twitter, MySpace, Meebo... Clients are still brain washed by big enterprise players thinking they need BEA Portal Server or the likes to achieve a robust, scalable enterprise solutions. It's a battle to convince them not to invest their money on something that's way to expensive, takes forever to deploy and cost a fortune (and takes forever) to make it do what you want it to do from the user experience perspective. I keep saying, "MySpace is registering 350,000 users a day and they aren't using Aqualogic - lets save that extra cash for some killer AJAX UI, widgets and an API that's actually useful.

Can you say anything about Unladen Swallow. It seems that Google's idea of a JIC for Python went away after making massive improvements (5x gains by 3rd quarter 2009), all because it was too complex a problem to be solved. Is this true?

I saw the video on YouTube's scalability a couple of months back and it was really impressive, especially since they were one of the earliest massively scalable services to implement databases sharding. The deal though is that the video is kinda old and even in the video he was talking about the need of a database that could do "parallel queries" which would imply something like migrating to Oracle.

Do we know what technologies YouTube uses now? Is it still running on Python and MySQL with a modified lightpd stack (I know lightpd actually integrated a couple of YouTube's patches into their server). I have a feeling a lot of their stuff may have migrated to some Google technologies since their buyout a few years back.

They're pretty secretive about what technology they use now. Google won't even comment on YouTube's profitability (of if it's even profitable for them at all right now).

Want to see the document mentioned in the google video which is 'Seattle Conference on Scalability: YouTube Scalability'Does anybody know where I can find those document? Book Store link in high scalability is wonderful, of course.