Sounds like gimbal lock to me, kanetan, which is the equivalent of tracking something that starts at the horizon and flys directly overhead to the opposite horizon. As the object passes directly overhead, gimbal lock occurs. The only way to attempt to maintain tracking is by a very rapid 180 degree rotation.

If I may suggest something, kanetan, instead of using the Object/Viewpoint mode (V), just use the Arc Rotate tool (ctrl-r) while you're in Object/Edit mode (A). You can zoom in, rotate, pan, etc in different ways much more easily than when using the Object/Viewpoint tools

(hint: use all three mouse buttons and try rotating/panning outside the circle, inside the circle, and inside the little squares on the circles to limit the axis', move the camera pivot/pan, and change the camera orientation for that view).

Not to mention you can use the arc rotate tool in all the viewports whereas Object/Viewpoint mode only lets you rotate the view in the perspective viewport.