Abstract : Color-based tracking methods have proved to be efficient for their robustness qualities. The drawback of such global representation of an object is the lack of information on its spatial configuration, making difficult the tracking of more complex motions. This issue is overcome by using several kernels weighting pixels locations. In this paper a multiple kernels configuration is proposed and developed in both probabilistic and deterministic frameworks. The advantages of both approaches are combined to design a robust tracker allowing to track location, size and orientation of the object. A visual servoing application in tracking a moving object validates the proposed approach.