Transcode your clips for faster *Export* times

It might seem obvious that you'd want to transcode your footage to a more "editing friendly" format so that editing is faster - Cineform, Norman AVC, DNxHD, etc. - but it also has other benefits.

I don't know why I didn't think of this before, but I took an .MP4 file and just exported it and noted the time it took. I then transcoded it to .MP4 again, to the same quality/bitrate, but adjusted so it would run faster in the editor.

The first clip was a little jerky when playing at full resolution, but the second one was as smooth as butter when I dragged the playhead forwards and backwards (which is more work to decode than forwards).

Doing nothing else to it at all, I then exported it with the same settings as the first clip. It took less than half the time to export. I loaded both clips back in and zoomed in to detailed areas to examine them and the quality of both clips was identical. Being able to load the clips in faster meant Hitfilm was able to write them out again faster too. Yay!

So, if you're going to be exporting multiple revisions of something: it makes sense to help Hitfilm get those files in faster, to get them out faster.

Don't just struggle along with jerky files when editing, as everything else (presumably even RAM Preview and Proxy creation, although I didn't time those) and making 'test renders' to see the result at a decent speed will all be slower. Double bad news.

But...double good news if you transcode first.

Comments

Interesting. I often say that for hobbyists like me, there is no NEED for a high end cinema camera ie Canon 300, black Magic anything Red etc. since I make fun little videos that usually only my close circle of friends will see via Youtube.

Therefore even if there is some difference most of us would not notice in a million years.

But it would be fun to see if there really is no difference.

Since you probably still have the clips you can easily prove that they are still the same.

Would you place one clip on track 1 and the other clip on track 2 and set the blending mode on track 1 to Difference? If the video is completely black there is no difference.

Chances are there would be some differences. Cineform and mp4 are both lossy codecs. A decode/re-encode should produce artifacts. Hopefully minimal.

Terry, stacking two different versions of an "identical" file is a good and typical way to measure how much a file has been mangled. Great for photo/video forensics engineers. Blend modes can do some neat things, but are often overlooked in favor of newer techniques. Putting a daytime shot over itself in Multiply used to be step one of digital day-for-night.

I became aware of using the difference blend mode from a Lynda.com Davinci Resolve 12 tutorial series. The Author was discussing steps to take if you are the colorist and receive the project from the editor. The purpose was to insure the EDL conformed to reference video.

My suspicion is that since MP4 is not lossless, it will deteriorate a little every time it's rendered out. I do not think my eyes will detect this, but I thought it would be an interesting exercise.

@BobDiMarzio I don't know if that's part of the Hitfilm day-for-night filter, but footage over itself in Multiply increases midtone and shadow contrast leaving highlights mostly alone. After that, one would use curves to bring down highlights, then add another layer with a blue gradient overlaid in Color Blend Mode at (say) 25-50% opacity.

That's the late-90's/early 2000's method before dedicated plug-ins were created.

I forget who I steal the sentiment from, but many plug-ins/filters are specialized tools that make it easier to do things that could be done with basic curves, color correction wheels, blurs, glows, masked planes, add noise/fractal noise, and blend modes--using a lot of manual steps and multiple layers.

A "fun" exercise might be to approach a specialized filter and analyze what basic corrections it contains...

So cinestyle stacks a blue/orange duotone with a (preset) curves adjustment with added noise and a masked plane for letterboxing. All basic stuff, but in a single easy interface!

Heat/smoke/energy/fluid distortion? Three of those require an add-on in Express--or, create a Composite Shot with the corresponding Fractal Noise and use that as the Source Layer for a Displacement effect. And as a Set Matte for a grade layer of blur for Diffusion!

@Triem23&nbsp; Wow! Another useful thing I did not know about. Thanks for that info. I've always just tweaked the built in but sometimes it wasn't really what I needed and knowing this will come in handy.

@BobDiMario Ah yes, I've used the Difference method to confirm Hitfilm was drifting off when loading in a Handbraked 23.976 fps file and calling it 24fps compared to an original 23.976 fps file.

I just tried the original .MP4 file and the speed trancoded .MP4 file and Difference blended them and....not a pixel difference and a completely black screen.

I then made it full screen on a second monitor (as the Viewer is obviously smaller than full resolution, so would blur the result) and...still all black.

So I slid the top layer over by 1 pixel and yes, there was the 1 pixel offset halo, so it was working.

This was of the two different source files, I didn't compare the Exported files because I only have one of them now, and I'd expect Hittilm to do: same in = same out for each.

So, while .MP4 is a lossy format: at the same bitrate/quality settings (10Mb/s in this case) it's lossy in the same way for the same file via two different encoders, so nothing is necessarily lost by transcoding.

Either that or the changes in R,G,B per pixel between the two versions are so small they only register as almost completely black when Differenced, so...close enough.

Well, I could. But I'm not going to because the whole point is: I'm going to use the Speed trancoded version anyway, so don't care if it's slightly changed from the original. Just like anyone else who transcodes.

If it's changed so little, then the output will also change by a similarly small amount - compared to the original - and I still wouldn't care because: speed and I'm so over the slow original anyway.

For pixel peeping transcode differences to the original I've used the Difference blend mode and add a massive exposure increase to actually see the diffs. Using a gamma of 2.0 typically makes the diffs more visible than a +5 exposure. If you can see the diffs with just the difference blend mode alone then the transcode is really losing quality. With difference, pure black means no diff.

Doing an invert at 50% opacity you can get a gray comparison if that is desired. Middle gray is no diff. To see diffs with this method you need to boost contrast with something like curves.

A screenshot of my test project comparing various transcodes with different codecs.

Obviously don't know what settings you use for your transcode, but if I'm losing even that much (TBH, I played around with Curves and couldn't see anything on mine; but if I have to try too hard: it's insignificant) I'd be happy.

I'm so much more likely to transcode than before, now I know it's a double bonus to do so, as everyone (including me) is always complaining about final render speed.

That is not a lot of difference. It is really tiny given how dim the diffs are. There will be diffs, but the question is how visible (bright) are they. Even withresult bitrate higher than source, and both same codec (ie AVC), there can and likely will be diffs. Macroblock differences. Even if all data is "perfect" but the macroblocks are computed/arranged different, then you can get diffs.

Here is the original.

The following are diffs, with a 5 stop exposure push. You can read SSIM as a percentage. 1.0 = perfect.

Considering the amount of detail in the leaves that it would be 'tempting' for the codec to remove/blur, - and it is in the treetops that there is the most difference - they're all pretty impressively minimally different. On a simpler image there would presumably be less scope to lose detail in the first place. Which is what my test image is - just a talking head - hence possibly why I see no difference.

I don't have any experience with more "normal" footage. Normal being people standing around talking to each other. I have GoPro footage around high frequency backgrounds. A lot of first person view as well. I can envision normal footage could fare better in a transcode.

First person movement with high frequency background is really hard on transcoding. You don't get much of the "good" kinds of compression that you get/find with relatively static scenes. I think it is the "zoom" factor of the first person movement making it hard to reuse parts of one frame in the next. Static or panning like movement is easy to find reuse.

I came up with the crf 20 for fast decode AVC because it gets an SSIM of around 97.5% with my GoPro stuff. Seemed good enough. Honestly one can push it more but that is a personal thing.

I might give Cineform another go (I had other issues with noise from the 'wavelets' in the past), but I'm getting good speed/quality results with .MP4. I'm loathe to change for (possibly) minimal benefits, while the file size is small enough to keep my HDD happy (ingest) and not take up a lot of room too.

Nothing wrong with AVC. People like to blast it as not an 'edit' codec. It does have higher decode overhead than anything. Even with compute heavy features turned off. Like CABAC. CAVLC still compresses better than the bit encoders other 'fast' codecs use. Always a cost for something.

Wavelets like macroblocks have their own compromises. Wavelet artifacts will be different than macroblocks and some may not like one verses the other. People with a Bionic eye like yours are probably a tougher audience to please.

I once tried an All-I AVC fast decode but ran into bugs in Hitfilm. Got support to reproduce, with a Canon DSLR file, but it was a low priority given the low incidence of obvious/immediate issues. I would not trust it even without obvious effects unless I knew it was fixed.

Anyway people use AVC to keep disk consumption under control. So just let AVC, be AVC, but use a GOP that is super short. Like the 8 in my NormanAVC settings. It works damn near as well as All-I AVC scrubbing and it eases Hitfilm's GOP ignorant way of doing things. Heck, even shorter than 8 could have performance benefits in places in Hitfilm. At the same x264 CRF the shorter GOP should raise the result bitrate which of course raises decode overhead. Balancing act.