Tuesday, December 20, 2016

A new technique for Super Resolution Photography...maybe?

As readers of my posts know, I try and get as much as I can out of the (Canon) camera technology; helped, of course, by Magic Lantern.

So far, accounting for my bias towards landscapes, cityscapes and architecture photography, through various scripts (see the right hand side), we can automatically:

Take exposure brackets, ie according to the dynamic range of the scene. The camera works out the number of brackets to take;

Take focus brackets, from a focus point to the macro end of a lens, or the infinity end, or to the hyperfocal distance (HFD) or from one focus point to another. The camera handles the diffraction and the bracket to bracket overlap;

Take both exposure and focus brackets at the same time;

Take a simulated long exposure brackets, in Full Resolution Silent Picture mode, ie no shutter actions, or with the ‘normal’ mechanical shutter;

Take any number of brackets, ie to use when you are trying to remove people from the scene;

Move the lens to the HFD at the push of a button.

In this post, we are going to take the next step in automatic bracketing, namely: automatic bracketing for super resolution photography.

At the heart of super resolution photography (SRP) is trying to use the equipment you have, like a Canon 5D3 in my case, to emulate a, say, Canon EOS 5DS R. In this post I will show how to enhance the resolution of your camera without spending a penny on a new camera. In my case, using a 5D3, to a camera equivalent to 90 megapixels (MP)

SRP is not new. For example, Hasselblad had a 200 MP camera system, but that 200 MP images was made by a 50 MP sensor, using a special sensor-shift mechanism inside the camera. Typically it would take six separate images, each with slightly different sensor positions with only a pixel of difference between each shot. The camera would then automatically re-align those images and combine them together to produce a photo with four times the amount of resolution.

Last year the Olympus OM-D E-M5 II was announced as the first consumer level camera to feature this, sensor shift, technology. Similar to the Hasselblad , the OM-D E-M5 II takes 8 consecutive photographs with its 16 MP sensor. After shooting these 8 photos, once again each with a different sensor position, it then combines the data from all 8 images into a 40 MP plus image.

But what if you don’t have a camera that can shift the sensor? Are there other ways to achieve a similar result?

Bluntly: yes!

First, you can hand hold and shoot the 8, say, images and post process those images into a SR image in Photoshop. The technique requires hand holding, as taking the images on a tripod would result in image to image perfect alignment; and we need to the images to be different. Having said this, the multi stacking no sensor movement approach has some advantage in reducing noise, ie 8 images (albeit perfectly aligned) will give you a three stop reduction in noise, eg an ISO 800 shot will look like an ISO 100 one.

But such perfect alignment will not help us achieve the SR ‘uplift’.

The problem with hand holding is that you are restricted in your shutter speed, eg to 1/FL, or for an average photographer, say, faster than 1/40s.

To overcome the handholding limitations, I offer the following as an alternative technique, well at least for the Magic Lantern Canon shooter. I’m not aware of anyone else coming up with this idea: but I’m prepared to eat humble pie :-)

The key to my technique is to exploit a known ‘weakness’ in still photography lenses, ie the technique may not work with cine lens (although I can’t test this as I don’t have one).

The feature we are going to exploit is a normal lens’s focus breathing or focal length shortening, feature; which is the name given to the change of focal length (and hence angle of view and magnification) when the focus distance of a lens is changed. It is easy to see the impact of focus breathing by simply looking at an image on the Live View screen at say the closest focus point, and then move to infinity. The image changes on the sensor: which is exactly what we want!

Knowing all lenses exhibit focus breathing, led me to speculate if I could exploit this as form of sensor shifting, ie take multiple images at different focus points.

The immediate thought may be: no way, as the images will be differently out of focus between images. But this is where a pragmatic limitation comes in: use wide angle lenses and ‘limit’ things to the HFD.

As a reminder, if one focuses at the HFD, then every thing between the 1/2 HFD to infinity will be in focus, according the out-of-focus criterion you have chosen, eg the circle of confusion and, if you are being critical, the diffraction blur added in quadrature. Luckily for Magic Lantern shooter, all this Depth of Field maths is built in to ML and accessible via Lua scripting.

The other advantage of focusing at the HFD is that moving to infinity means that only 1/2 of the distance to the HFD is ‘lost’. Thus, say, if the HFD was at 2m, ie the DoF went from 1m to infinity, then moving the focus to infinity would result in an acceptable DoF from 2 to infinity, ie all we lost 1m in the near field.

Move the lens towards infinity and take another image, and repeat until you are at infinity and have taken the required number of images, each with a slightly different image because of the focus breathing OR simply jiggle the lens via the IS;

Ingest the images into your PC and stack them in Photoshop;

Increase the image size by 200%;

Auto align the images;

Turn the image stack into a smart object;

Run the stack mode as medium or mean (and wait a long time!);

Collapse the stack an finish post processing.

The only real problem with the above is step 4, ie repeatedly touching the lens; as manual control of focus position is not good/repeatable as the lens gets towards infinity. The IS jiggle trick works, ie moving the images relative to each other 'randomly' by a pixel, but you have to keep manually intervening.

But help is on hand in the form of my latest Auto Super Resolution Stacking Script (see below).My Lua script runs under ML and will automatically control the lens (which must have AF). The script will find the HFD, and, in the updated version below, jiggle the lens.

The script only requires three inputs:

The user selected number of super resolution images to take, ie 4 to 20;

A user selected delay, 0 to 5 seconds;

What sensor shift strategy to use;

And whether to add a bookmark image at the start and end, ie a dark frame to differentiate the SR stack.

So what are the results?

Well you will have to wait to see as, so far in my experiments all I have done is to get the auto stacking script (see below) running and take a few SR image stacks in our back garden, using my 24mm F/4L at 24mm. The stack comprised of 9 images, covering the HFD (about 1.8m) to the ‘infinity’ end of the lens. Each images was the usual 30MB sized CR2 file, ie 5760 x 3840 pixels. The resultant SR image was 516MB and had 11645 x 7756 pixels. In future posts I will explore some test images.

So why would you bother?

Apart from the curiosity factor, which drives me, there are occasions when you need all that pixel real-estate, eg a super large print on a bill board. Another reason photographer’s enjoy having a lot of megapixels to play with is because it gives us a lot more room to crop and manipulate our images in post production.

Of course, the downside is the shear workload in simulating a super large ‘mega pixel’ sensor, ie a 90MP equivalent on my 5D3. Thus the technique is one to use with care!

For now, without evidence, I offer this ‘new’ approach to super resolution photography, ie shooting on a tripod at any shutter speed, with any (Canon EOS) camera, and look forward to hearing from others regarding their thoughts on this post, especially the Canon Magic Lantern users.Auto Super Resolution Bracketing Script--[[ This script simulates a super resolution stack for post processing...by simulating sensor shifting by changing focus. Because of this, the script is 'limited' to taking...sharp images between the HFD and infinity.This script assumes you are on a tripod, ie not handholding as 'normal' in super resolution stacking (without a special sensor)This version does NOT work with FRSP...at the moment :-)Should be in LVCanon review should be set to OFFUsual caveat: script was written for my workflow and enjoyment, and 'only' tested on a 5D3