Optical Flow Stitching – The Future Of 360 Video Post

Optical flow stitching is on the brink of becoming commonplace within 360 video professionals. It is the process of analysing pixel movement across a number of frames and comparing that movement in different images.

Traditional image stitching compares pixels in a single frame image for relative matching accuracy and then applies this to a whole video. Essentially one frame dictates the stitch for everything (unless edited into sections of stitch with a warp transition between). This is a granular methodology matching pixels like a jigsaw across a line.

Optical flow is the process of analysing the objects within the flow of a series of frames and noting how they match and move across a seam. A human can instantly tell if a vertical line such as a building or door frame is broken, but a standard stitch will stitch for a parallax distance, often infinity, so won’t take into account incorrect geometry.

The optical flow software will match the initial frame of the stitch, and then analyze the next series of frames for movement or better match and then apply a skewed avaerge of the frame process to the first frame. It is a complicated algorithm driven by extensive mathematics.

Facebook released a tweak flow algorithm open sourced in 2016, and we are now seeing tweaked versions of it applied by various camera manufacturing companies to their products. This algorithm flows top and bottom into side cameras, and then flows pairs of cameras for objects at different distances.

Optical flow is a processor intensive process. On new FB algorithm based software called Wondertstich software, an optimised windows machine with suitable processing power we see around 1.4 frames per second render to 4K, in comparison to real-time (or 7 frames per second on a mac from standard image stitch software).

As the algorithms become more advanced and the processing requirement lessens, we will see optical flow become more prevalent as it enables much more accurate stereo 3D 360 rendering, therefore making it a format achievable by a larger group of 360 video productions.