Best Practices for Performance on DigitalOcean Spaces

DigitalOcean's object storage solution, Spaces, is designed to be scalable and cost-effective to use for storing large amounts of data.

Here are some recommendations on how to get the best performance from Spaces based on your use case and application architecture.

Use a Content Delivery Network (CDN)

When Should I Do This?

A CDN, or content delivery network, caches your assets in geographically distributed locations to make downloads and page loads faster for your users.

You should use a CDN in front of Spaces if:

Your use case is mostly serving GET requests from the Internet, especially for frequent requests to small files.

For example, web-facing applications and media servers are especially likely to see significant performance improvements with the addition of a CDN.

How Does This Improve Performance?

By integrating a CDN with Spaces, you can distribute content to your users with low latency and a higher data transfer rate than you could get by serving your content directly from Spaces.

A CDN will fetch requested files from Spaces and cache them closer to your end users. By serving future requests for the same files from the CDN's cache, you reduce the number of GET requests sent to Spaces and therefore reduce the user’s overall latency when interacting with your application.

How Do I Implement This?

Several third party CDNs have documentation specifically for Spaces, like Fastly and KeyCDN, and most other CDNs will work with Spaces with minimal configuration.

Please note that in order to use Cloudflare as a CDN, you need to use a tier that supports host header rewrites (which the free tier currently does not).

Avoid Small Files and Use Multi-Part Uploads for Large Files

When Should I Do This?

You should consider the size of your files and the way you upload them to Spaces if you are:

Handling files smaller than 1MB.

Uploading files larger than 500MB.

How Does This Improve Performance?

Spaces is designed for storing and serving moderate to large files. Files 20MB - 200MB in size will give the best performance for both writes and reads. Additionally, combining small files into one larger file will greatly reduce the overall number of requests to your Space compared to handling many small files individually.

How Do I Implement This?

We recommend combining files less than 1MB together into a single, larger file. How you do this will be specific to your particular files and use case, but one example is concatenating daily log files into a monthly file.

Choose the Right Data Center for Your Resources

When Should I Do This?

Choosing the right data center location for your Spaces depends on where the connections to your Spaces come from.

If the connections to your Spaces are from Droplets, you'll see the best performance when you:

If the connections to your Spaces are from end users on the Internet, you'll see the best performance when you use a CDN, regardless of which region your Spaces are in.

How Does This Improve Performance?

Droplets and Spaces in the same data center will have the least amount of latency, but if your application requires connectivity across multiple regions, choose data center locations connected by DigitalOcean's regional backbones. This will provide predictable and stable latency with no packet loss because these connections use our dedicated links instead of the public Internet.

How Do I Implement This?

You can choose the regions for your resources at creation time. For existing infrastructure, you can migrate your Droplets to a new region.

Traffic between Spaces in NYC3 and Droplets in NYC1, NYC2, and NYC3 go over our Northeast regional backbone; traffic between Spaces in AMS3 and Droplets in AMS2, LON1, and FRA1 go over our European regional backbone.

Handle 50x Errors Properly

When Should I Do This?

This recommendation applies any time you upload files to Spaces. You should always make sure your upload application or library correctly handles 50x errors.

How Does This Improve Performance?

With correct error handling and retry logic, your dataset will upload faster and require less human intervention. Additionally, without this functionality, any 50x errors during uploads will create gaps in your data.

How Do I Implement This?

Spaces has a very high degree of compatibility with S3, so one option is to use a well-established S3-compatible client or library for your uploads, like s3cmd.

If you're writing your own code, make sure to implement retry logic with exponential backoff to handle 503 Slow Down responses.

Optimize Your Request Rate

When Should I Do This?

You should consider ways to optimize the number of requests you send to Spaces if:

You send more than 200 requests per second.

Your requests are being rate limited.

You frequently encounter 503 Slow Down responses.

As mentioned in our release notes, on 15 Feb 2018 we resolved an issue where some Spaces customers were being rate limited despite being below the threshold.

How Does This Improve Performance?

To ensure all customers receive a fair share of Spaces’ available throughput in any region, we apply rate limiting at the Space level. Making sure your uploads stay within the threshold will prevent unexpected throttling. As we continue to improve Spaces, we will also re-evaluate this threshold.

How Do I Implement This?

There are several ways to begin optimizing the number of requests you send depending on your use case.

If you plan to push more than 200 requests/second to Spaces (regularly or as part of a one-time upload), open a support ticket so we can help you prepare for the workload and avoid any temporary limits on your request rate.

Use Local or Block Storage Instead

When Should I Do This?

Not all use cases are appropriate for Spaces. Like all object stores, Spaces are best used for static, unstructured data.

You should use local or block storage if:

You're storing dynamic or structured data, like low-latency key/value stores and other databases.

You have a high request rate for your data, like streaming video.

How Does This Improve Performance?

Using local storage or block storage will give you better performance in certain cases because the underlying hardware devices provide low-latency I/O. You can visit Object Storage vs. Block Storage Services to learn more.