PANORAMA TOOLS - USING THE OPTIMIZER

WHAT'S THE OPTIMIZER FOR?

First, it should be understood that running the optimizer is not necessarily an essential part of the stitching
process. PTStitcher does the actual stitching, and relies upon a script file for getting its instructions on what
to do. The script tells it which images to stitch, how they are positioned, and also provides details of the
camera lens(es) and their distortion characteristics. If all these parameters are already accurately known,
(through the use of calibrated equipment), then the optimizer is not actually needed. In most cases, however,
much of this information will be unknown so we use PTOptimizer to work out the missing details as
accurately as possible.

HOW DOES IT ALL WORK THEN?

1. Input images are mapped onto a virtual spherical surface, the idea being to position the images so as to
recreate the original 360x180 view seen from the camera position, or a part of that view. In addition to the
image warping needed to achieve this geometric transformation, the images may also need to be further
distorted to correct for barrel/pincushion distortion.

2. The position of each image on the sphere is identified by yaw, pitch and roll values (parameters y,p and r),
and the lens distortion correction is controlled by the lens parameters a,b and c. The lens horizontal field of
view (HFOV) is a key parameter that enables the size of the remapped image on the spherical surface to be
computed.

3. The images are fitted together rather like a jigsaw puzzle. Fix one image and the optimizer will fit all the
others around that image to form a composite image.

4. The optimizer works out the optimum position of each image, i.e. the best "fit", within the constraints
imposed by the various parameter settings. The control points on matching features in the image overlap
areas tell it how images are to be positioned with respect to other images. If the optimizer is given complete
freedom to move all images into any position, it will fit the images together very nicely, but the composite
image might end up anywhere on the spherical surface.

5. To have some control over where the composite image ends up, you can restrict the position of one or
more images in some way. For example, you might set the y,p and r values of the central image of a
panorama to 0 degrees. This will ensure that the composite image will be approximately centered in the
straight ahead position (where y=p=0 degrees).

6. When a good fit of the images has been established, the composite image needs to be coaxed into a
position such that the y,p and r values of the images match those of the camera taking the photographs.
When this ideal position has been achieved, the horizon in the image will then be aligned with the pitch=0
line (the equator). Vertical features will be aligned with lines representing equal yaw (lines of longitude).
Marking vertical edges and the horizon with t1 and t2 points, respectively, is one way of helping the optimizer
to achieve this blissful state of alignment.

7. The stitcher uses the image parameters worked out by the optimizer (or manually specified) to generate
the output image by projecting the spherical image in various ways e.g. onto a flat surface (rectilinear) or a
cylinder. A portion of the sphere can be output rather than the entire surface, in which case the boundaries of
the selection to be output is symmetrically positioned about the yaw=0, pitch=0 point, i.e. the straight ahead
view. Thus, this window on the view can be made wider or taller but cannot be shifted up or down or
sideways. Of course, the composite image can be shifted on the spherical surface to reveal a different view
in the output window. e.g. if you add 10 degrees to the yaw value of every image, the whole panorama will be
shifted right by 10 degrees, without affecting the alignment of the images with respect to each other. (Think
of a strip of orange peel - the composite image - being slid over the surface of an orange. The strip remains
the same shape wherever it is positioned).

SO HOW DO I RUN THE OPTIMIZER?

You run the optimizer to achieve two distinct objectives: one is to fit the images together as accurately as
possible by aligning the control points; the other is to position the composite image on the virtual spherical
surface such that you have the desired view in the output area, usually also aiming to have a flat horizon and
upright verticals. In any one run, you should be clear about what you want to achieve and set parameters
accordingly. You will generally need to run the optimizer more than once to get the best results, but you can
address either or both objectives together in each run.

Fitting the images together

The optimizer takes the current layout of images as defined in the project file as its starting point and does
its best to improve the layout by making changes to the various parameters. It does this in a cyclical fashion,
hopefully improving the layout a bit more each time round. Note that you can cancel the optimizer at any time
and either accept or reject the current state of the parameters. You control which parameters it is allowed to
vary by checking the parameter boxes on the optimizer form provided in PTGui and PTAssembler etc. The
output from the optimizer is an updated project file (in memory) containing the new set of parameter values.
This then becomes the starting point for the next optimizer run. Everything the optimizer does is aimed at
getting the control points into the best alignment possible. How successful it has been is reported at the end
of each run, when it will give the average control point alignment error and the maximum error (in pixels). You
need to keep an eye on these figures to see how well you are doing. Do remember, though, that these
control point distances need to be assessed in relation to the currently specified size of the output image,
even though you haven't got around to producing an output image yet. Double the output image size and you
will double the average control point distance.

So, let's assume you've just assigned your control points. You might now go on to optimize yaw, pitch and roll
on all the images, which will allow the optimizer complete freedom to move the images around to get them to
align with each other. This is how you would check the boxes (and the 9 images in the project are shown
initially stacked up like a pack of cards, all with y=p=r=0 degrees, and in a 360x180 equirectangular
projection of the whole sphere, like a map of the world):

(Note: For clarity, I have used the equirectangular projection for the output illustrations in order to represent the complete
spherical image space used by the optimizer. In normal use, the projection chosen would be whatever format the ouput was
required to be in, and the width of the view would be adjusted to suit the coverage of the images being processed. PTAssembler
does not have a continuous preview window like PTGui, so individual previews must be generated as and when required.)

After the optimizer has finished, it reports a good
result but the position of the aligned images is not
really much like the result we are aiming for:

While the average control point distance is a useful indicator of how good a fit of the images has been
achieved, this is not the sole measure of the quality of the optimization. Remember that as well as fitting the
images together really well, we also want to get the images into a position on the virtual sphere that
replicates the original scene. So it is more usual to start off by giving the optimizer a helping hand by
assigning rough starting positions to one or more images, which will also speed up the process since the
optimizer has less to do. By specifying the initial position of one image, and then not allowing the optimizer
to move that image, the optimizer has considerably less room for manoeuvre. It can still align all the images
with each other, but the ultimate overall position of the composite image will be determined by the position of
the fixed image. So pick an image that you want to be in the middle of the final panorama and fix that by
setting its yaw, pitch and roll to 0. Then you can check al lthe optimizer's y,p and r boxes except for the
fixed image - the "anchor". The optimizer then produces a result that's somewhat better aligned:

The pitch of the anchored image has a direct effect
on the flatness of the horizon. This animation shows
what happens when the pitch is varied from -5 to
+25 degrees:

If you happen to know the camera was angled
upwards by 15 degrees or so for the anchor image
(as in this series of images), you might as well set
the initial pitch to 15 instead of 0. It all helps. By
doing this, you get this result:

Alternatively, with PTAssembler you might set the
reference point on the horizon (see below).

Generally, rather than optimize everything at once, it is better to optimize only a small number of parameters
in each run, and in any case, not more than there are pairs of control points on each seam. So, in the first
run, optimize just yaw. Then optimize yaw and pitch on the second run. Then optimize yaw, pitch and roll. This
is often quicker than doing everything at once, and the results tend to be better, too. After the images have
been sorted into reasonable positions, you can include lens parameter b in the optimizations to correct for
barrel distortion in the individual images, which will enable the optimizer to get an even better alignment. This
is often all that's required, but some lenses need parameters a and c to be optimized as well in order to get
adequate results. Once you have established a good set of lens parameters, it is worth saving them for use
in future projects.

DO NOT OPTIMIZE THE LENS HFOV PARAMETER unless your project is a 360 degree panorama.
Assuming you have put in a reasonably accurate value derived from the focal length setting in the image exif
data, there is little or nothing to be gained by optimizing this parameter, and the optimizer may well optimize
it down to 0 degrees. But in the case of a 360 panorama, it is ok and indeed desirable to optimize HFOV.

The lens shift parameters d and e are not normally optimized
when working with rectilinear lenses. They are intended to correct for an off-centre
lens and this is more relevant for fish-eye lenses, when it is definitely worthwhile optimizing them. Optimizing
d and e can also be useful for "flat stitching"
applications, for example as described
in Bruno Postle's tutorial.
They are
specified in units of pixels.

Getting the panorama into the final position with correctly aligned horizontals and verticals

Here, we are primarily concerned with relocating the images as a group into a position on the virtual sphere
where vertical features become truly vertical, the horizon is horizontal, and the output area contains the view
that we want. (Correcting converging verticals will not always be appropriate, but when you do want to
correct them, the methods suggested below will be found effective).

Realignment of the panorama can be carried out in several ways. Fix one image in position and all the other
images will be dragged into their new positions around it in the next run of the optimizer. If you can work out
what that position should be, then all well and good. PTAssembler has an option to define a reference point
and reference reference image, which enables the horizon to be aligned reasonably accurately. First set the
reference image on the Optimizer tab (Step 4) and then click on Set Reference Point. Align the red line with
the horizon by following the instructions at the bottom of the panel:

After doing this, you need to optimize with the reference image fixed in position (i.e. y,p and r unchecked),
and the other images free to move. There's no need to optimize the lens parameters if you are just wanting
to reposition the images.

PTGui also provides editing features on its panorama editor window to do a similar job, though maybe less
precisely. It does, however, have a nice button that enables you to reposition the images as a group without
destroying their "fit".

Using t1 and t2 control points to align the panorama

A more accurate method for aligning the panorama is to use the t1 and t2 control point options. The
optimizer will try to bring two features marked by a pair of t1 points into vertical alignment. Similarly, features
marked with t2 points are brought into horizontal alignment. Note that you must only mark features that ought
to be horizontal and vertical as they appear in the output image, which depends on the projection:

Projection

Straight lines preserved in the output

Rectilinear

All

Cylindrical

Verticals
Lines aligned with pitch=0 (e.g. the horizon)

Equirectangular

Verticals
Lines aligned with pitch=0 (e.g. the horizon)

In a cylindrical format, therefore, the only straight line horizontal feature that is preserved in the output image
is the line of the horizon; all other horizontals will become curved, so you mustn't put t2 points on those.

Assign t1 and/or t2 points in addition to the existing control points. Put a pair of t1 points at either end of a
vertical feature by setting the first point in one window and the second point in the other window (and note
that you can select the same image for display in both windows, if necessary). Reliable features to use for
this purpose include the edges of buildings and anything reflected in water. (An object will always be
vertically aligned with its reflection in a still pool). Do this for three or four vertical features, if possible. Here's
an example (from a different series of images) of a t1 point set on the edge of a wall, using the same image
in both windows:

If there is a horizon visible, you can set t2 points on that in a similar way to the t1 points. One pair of t2 points
is not enough to give a flat horizon because it may form an arc or wave that just passes through the two
points. Use three or four pairs strategically positioned to nail it into position. Alternatively, use one pair of t2
points to get the horizon horizontal and a t3 line to get it straight (see t+ below). Here is the assignment of a
single t2 pair on the horizon in images 0 and 4:

There's another pair placed at either end of the horizon. After assigning the t1 and t2 points (both can be
used together), optimize for y,p and r on all images except for y on one image. Keeping the yaw of one
image fixed prevents the panorama shifting sideways. This is how the boxes should be checked:

The optimizer will float the whole composite image into a new position as shown here, where the horizon is
flat and aligns with the line of pitch=0. Any vertical features will also be standing upright. Remember that if
you align the horizon correctly, the verticals will be automatically correct and vice versa. It's easy, really.

Using t3, t4, t5 ... (t+) points for straightening lines

These points are placed in pairs along straight line features. The optimizer will try to bring the set of points
into a straight line. All the points along the first line defined should be type t3 points. t4 will be used for the
next line and so on. The two points of each pair are not placed on identical feaures - just somewhere along
the same edge using the two windows in the same manner as for t1 and t2 points. Very important - you
must only place these points on lines that should remain straight in the selected output projection.
Thus, in a cylindrical projection, only the horizon and verticals are preserved, so don't try to straighten any
other lines. Exceptionally, you can cheat a little, and use t+ points on very short sections of otherwise gently
curved lines in order to persuade overhead electric wires, for example, to join up neatly at the seams. But
only correct very small discontinuities this way. Larger errors should be corrected by editing the output psd
layered file.

For a rectilinear projection, all straight lines are preserved, so t+ points can be freely used. Useful for
calibration purposes to evaluate lens parameters.

Centering the panorama in the output area

Centering the panorama horizontally amounts to adjusting the yaw values of all the images by the same
amount such that the yaw positions of the two images at either end of the panorama are equally displaced
from the yaw=0 point. So if the yaw values of the end images are -60 and +80, subtracting 10 from the yaw
values will equalise them at -70 and +70. You could go through the images one-by-one and do this
subtraction manually. Or, more conveniently, you can just use the "equalize yaw" button on the PTAssembler
Optimizer screen, and it will do all this for you.

Another way is to subtract the10 only from the yaw value of the anchor image and then re-optimize. All the
images will then be dragged into alignment with the anchor and the panorama will become centered. PTGui
has an image centering button too, but you need to be aware that it centers the image vertically as well. So
it's best to use this only early on in the proceedings, otherwise you risk spoiling the vertical positioning
perhaps already established to flatten the horizon.

GENERAL HINTS AND TIPS

Correcting perspective distortion in single images

To do this, you carry out exactly the same procedures described for multiple images. In effect, you stitch just
one image. Generally, use rectilinear input and output formats unless you are inputting a stitched image for
further work, in which case set the lens type appropriate to the projection used (further info below). Set t1 and
t2 points as needed and optimize pitch and roll, with yaw fixed when correcting vertical convergence. It may
be appropriate to optimize yaw, too, when correcting horizontal convergence.

A single image from a panoramic camera
should be entered as lens type cylindrical. The FOV will
generally be known from the camera specifications. The
correct lens type would also need to be specified for an image
taken with a fisheye lens when making perspective
corrections as part of a "de-fishing" process.

Estimating the position of the horizon

If you want to get the perspectives to come out "correctly", it is necessary to align the horizon in the images
with the line of pitch=0. This is easy if you have the horizon visible, but more problematical if you don't.
Here's a couple of suggestions that might help.

1. If you have a wide expanse of level ground/floor in front of the camera, imagine the camera on its tripod or
you holding the camera at some distant point. There might be a doorway, perhaps, to give scale. The
imaginary camera will be on a level with the horizon.

2. Along similar lines, if you took the pictures from the shore by a lake or river, again imagine the camera
positioned on the distant shore and that will be the position of the horizon, because stretches of water like
this are inevitably level.

Picking features for marking with t1 control points

It's important to be sure that any "vertical" features that you mark with t1 points really are vertical. Many lamp
posts and fence posts are not as upright as they might be. Look at similar features nearby and take care not
to pick one that is at odds with all the rest. Objects reflected in water are good sources of perfectly aligned
vertical features, e.g. tops of trees are plentiful. However, remember that tree branches wave about in the
wind, so it's best to mark the tree top and its reflection in the same image rather than in different images
when you have that choice.

Correction of already stitched images

You may have a stitched panorama that would benefit from a bit of tweaking to correct an alignment
problem. It is quite practical to input that image back into Pano Tools as lens type cylindrical or
equirectangular or whatever it happens to be, and carry out any adjustments you like with the optimizer in the
same manner as already described for stitching the images in the first place. Just three points to note:

1. For the lens HFOV, enter the value of the panorama FOV that was originally specified for PTStitcher.

2. If the image was cropped to remove the blank areas that you frequently get at top or bottom, extend the
image canvas to replace these areas before creating the new project. In other words, you ideally need to
re-input the image just as it was originally generated by PTStitcher.