The validity of feature correspondences plays an important role for feature-correspondence based motion estimation, which leads to the final goal of object tracking. Though different data association methods have been proposed, the problem of feature correspondence is, in general, ill-posed due to either the presences of multiple candidates within search regions or no candidates because of occlusion or other factors. Our research is inspired by how we evaluate the effectiveness of the feature correspondence and how the evaluation will affect motion estimation. The evaluation of template based feature correspondence is achieved by considering the feedback of the latest motion estimation from first visit of Kalman Filtering. Then motion estimation and feature correspondence are re-processed based on evaluation result, which constitutes the second and third visit of Kalman filtering. What makes our work different from others is also that instead of restricting the semantic object tracking in 2D domain, our framework is formulated to recover the 3D depth values of selected features during motion estimation process.