Real-time contour tracking in video surveillance suffers the problem of low speed and influence of shadows and noises. In this paper, a novel approach of video tracking is proposed, which employs a faster level-set-based method and a foreground extracting method free of noises. Compared with traditional level-set-based methods, our method avoids the high computational complexity of solving partial differential equations. In addition to the fast tracking method, we analyze the color information in the RGB color space so that the shadows and noises are eliminated in foreground extraction. It is performed without any manual thresholds’ tuning. Furthermore, we make use of the foreground extraction process to get the coarse contour of the object, which is also performed automatically and significantly reduces the time spent in the curve evolution step. With this approach, real-time surveillance system can be implemented. Experiments show that our approach outperforms previous methods for indoor environments.