Another type of extension to multi-frame is a recursive one where the
multiple images are not available in batch form but rather are streaming
in serially. One way of formulating this extension is by repeating the classic
2-frame estimate such as the Longuet-Higgens technique
[36] or the algorithm described above [30]. Oliensis
and Thomas [42] and Soatto et al.
[50] sequentially compute such
2-frame estimates and post-process the output with a smoothing Kalman
filter (KF). Here, the measurement vectors and the state vectors are
the same so the KF is linear, completely observable and hence does not
have any linearization problems. In fact, the KF is only acting as a
smoothing filter. It is not really being used in its full capacity as
a state estimator where the measurements are nonlinearly inverted to
obtain state information while keeping track of the state's internal
complex dynamics.

Extended Kalman Filters (EKFs) deal with nonlinearity explicitly and
can be applied to nonlinearly uncover motion and structure instead of
smooth the output of 2-frame techniques. EKF frameworks were utilized
on image sequences by Ayache and Faugeras
[1], Broida and Chellappa [11], Dickmanns and
Graefe [15], Faugeras et al. [20],
Heel [28], Matthies et al. [38] and Young
and Chellappa [62]. A seminal paper by Broida,
Chandrashekhar and Chellappa features a nonlinear EKF for recovering
state information [10] . It does not rely on 2-frame
techniques but rather folds the estimation into the Kalman filtering
equations. The filter is used to nonlinearly invert the measurements
to gather state information. One important deficiency of these
techniques is that camera internal geometry is not always
estimated. This is acceptable for some camera parameters such as skew,
etc. which can be less significant and constant in modern
cameras. However, the focal length (which is related to the zoom)
readily changes in many different video situations. An additional
problem is the perceived unreliability of these techniques due to the
linearization at each time step in EKF calculations. We now outline
our method which estimates focal length and has stronger
stability properties due to parameterization changes.