The Future of Video Editing Rests on Storage Bandwidth

I, like many others, am looking forward to the release of Apple’s latest powerhouse: the iMac Pro. Promised to breathe fire and leave skid marks across every digital highway on the planet, what’s a geek not to like?

Except… well, as we move into the world of 4K, HDR, and 360° VR video, a fast computer is important but no longer the critical factor. Our storage is. And, as we increasingly move into a collaborative environment, our ability to share projects and media with team members has a direct effect on whether we can meet our budgets and deadlines.

Here’s the problem: Apple focuses on creating computers that are powerful and flexible, combined with its legendary ease of use. All of which I fully approve. But, to achieve that speed Apple is increasingly using internal SSD drives as their main storage.

SSD drives have many advantages. When compared to traditional spinning media, they are:

Extremely fast

Extremely rugged

Extremely small

Extremely quiet

But, they don’t approach the capacity of traditional drives.

Today, most computers sport 250 GB – 1 TB internal SSD drives. The new iMac Pro can be expanded to 4 TB, though pricing hasn’t yet been announced. It is blindingly fast, but its storage isn’t big enough. Which means that we need to connect it to external storage – and that’s where the problem lies.

HERE’S THE CHALLENGE

For editing HD media, any computer released in the last six years will be fine, though older systems will have problems playing or compressing HEVC (H.265) media.

However, when compared to HD:

4K files are 4X bigger

360° VR files are 4X bigger

HDR files are AT LEAST 4x bigger

In other words, a storage system that works perfectly fine for HD will be overwhelmed in both capacity and bandwidth when moving into these new formats. Stepping up to 4K HDR files could generate media storage requirements that are sixteen TIMES what you are using now, with bandwidth requiring hundreds of megabytes per second.

How are we going to handle this?

Yes, we can compress this larger media by working with proxies but this is something you need to think about before you commit to that exciting new HDR project.

For example, when I create new training, I always work in Apple ProRes 4444 because it yields the highest image quality for screen captures. My recent Final Cut Pro X 10.3 training – which had 220 movies in it – generated almost 6 TB of data during production; FAR beyond what any currently affordable SSD can handle.

Bandwidth. The speed that your storage device can transfer data between itself and your computer. Generally measured in mb/s (megabits per second), this number is more useful when converted to MB/s (MegaBytes Per Second) by dividing mbps by 8. Thus, 1000 mbps equals 125 MB/sec.

Thunderbolt 2: About 1,250 MB/second – the rest of the bandwidth is reserved for monitors – provided you have enough devices in your RAID to “fill the pipe.”

Thunderbolt 3: About 2,200 MB/second – the rest of the bandwidth is reserved for monitors – provided you have enough devices in your RAID to “fill the pipe.”

RAID 0: Multiply the bandwidth from the devices list above times the number of drives in it. So, a four-drive RAID 1 using spinning media would be (4) * 125 = 500 MB/second.

RAID 5: Multiply the bandwidth from the devices list above times the number of drives in it minus 1. So, a four-drive RAID 5 using spinning media would be (4 – 1) * 125 = 375 MB/second.

RAID 6: Multiply the bandwidth from the devices list above times the number of drives in it minus 2. So, a four-drive RAID 6 using spinning media would be (4 – 2) * 125 = 250 MB/second.

As every manufacturer will tell you, there is lots of variability in these numbers, but these are good to use for estimates and planning.

NOTE: To get maximum benefit from any protocol, say Thunderbolt 3, both the computer and storage device must support the same protocol. Recently, I was testing the speed of Thunderbolt 2 devices, only to discover that my computer only supported Thunderbolt 1, which meant that the devices were running slower than they were designed. Or, another example, to get the maximum speed from WiFi, both your computer and WiFi receiver need to support the latest protocols.

I WORK ALONE

If you are a solo editor, then your goal should be to purchase the fastest RAID using a connection protocol that is supported by your computer:

Thunderbolt 2 for older systems

Thunderbolt 3 for newer systems

The more devices (drives) a RAID contains, the faster it will go and the more it will store. While there are other protocols – iSCSI, FireWire, eSATA – Thunderbolt will have faster performance, lower costs and better market support.

When purchasing storage my advice is to buy as much storage capacity as you can afford. There is no such thing as “too much storage.”

I WORK IN A GROUP

When you need to share media between members of a local workgroup, things get more complex. And, when you need to share files with members outside the local workgroup, you’ll need professional help in figuring out the best way to set this up.

When I’m creating new training, for example, three of us work on it – all accessing shared storage. Since Thunderbolt is not a “shareable” protocol, we need to use either Ethernet or FibreChannel. Since wiring my office with fiber is way too expensive, I’m forced to use Ethernet.

When sharing files over a network, HD is relatively simple because the bandwidth needs are so small (comparatively). But problems arise as I move into larger format files.

More personal examples:

WiFi speeds in most recent laptops, max out around 46 MB/second.

Low-end switches, like those from NetGear that cost less than $100, limit throughput to about 50 MB/second.

Higher-end 1 Gb Ethernet switches, like those from Cisco, limit throughput to about 125 MB/second.

Last night, I needed to transfer 3.2 TB of data from local storage to a network server via WiFi. The process took 21 hours. Backing up that data via 1 Gb Ethernet, which I need to do every night, took seven hours.

1 Gb Ethernet is not fast enough for 4K HDR media. However, rewiring my office to support 10 Gb Ethernet – which WOULD be fast enough – requires:

Converters on the back of every Mac from Thunderbolt to 10 Gb Ethernet

New cabling in the walls to support 10 Gb Ethernet speeds

New 10 Gb, high-capacity switch

A storage system that supports 10 Gb connectivity

NOTE: The new iMac Pro is announced to support 10 Gb Ethernet natively. It is the only Mac to do so.

As you can imagine, all this gear ain’t cheap. And wiring for fiber, though potentially faster, is even more expensive, as are the switches.

What I’m currently doing is tailoring the video formats I create and edit to match the speed of my storage. Which is an interesting thought: My computers are more than fast enough, but my storage is not – especially when I’m editing in a workgroup.

SUMMARY

Increasingly, projects that used to fit comfortably in a terabyte, are now expanding into dozens, even hundreds of terabytes.

The speed of our computers is more than fast enough – especially the newer ones with high-performance GPUs – to handle almost all these new media formats.

But storage bandwidth – and how we connect our gear – is not keeping up. If you plan to expand into all this new technology be sure to budget for a major expense in storage capacity, storage bandwidth and faster network infrastructure.

Otherwise, you’ll be all ready to edit – and waiting for your media to arrive.

25 Responses to The Future of Video Editing Rests on Storage Bandwidth

Great article. What are your thoughts on using MS’ Surface Studio desktop as a content creation device? Do you see stylus / dial / touch UI paradigms replacing KB / Mouse for professional content creation?

What is the most efficient way to import 4K RAW files (Cinemadng) that were recorded with an Atamos Shogun Inferno External Recorder and SONY FS5, and create proxies to facilitate editing a project in Adobe Premiere Pro?

I followed the workflow outlined on Adobe’s website and spoke with their Tech Support staff, who said I should import 15 clips at a time (I have hundreds of clips), but the process is still taking an inordinately long time.

I was initially trying to import the files from a WD 4TB 3.0 wireless drive, to which I transferred the files from the my SSD cards. I purchased a Samsung SSD 2TB 3.1 2nd Gen. drive with the hope to speed up the importing/creating proxy process. I need to get my footage edited in the next couple of weeks, so I was wondering if you had any suggestions on how to make the process as efficient as possible. Thanks for you help!

Buying the faster drive was smart – WiFi will be too slow for files this big. A Thunderbolt version of the Samsung will be an even better option in the future.

That being said, I don’t know the answer. I forwarded your question to the product team at Adobe and I’ll let you know what I find out if they get back to me.

You might also reach out to Sony Support to see if they have any ideas. AT a guess, it may be easiest to transcode these into a friendlier format, but that’s just a thought. I’ll let you know what else I learn.

Not a member? See The Different Membership Tiers.
Become a member of our Video Training Library & Get unlimited access to our highly-acclaimed training, Thousands of movies available 24-hours a day, 7-days a week.