We propose a fast and automated facial expression recognition system by simplifying both feature extraction and classification. In feature extraction, a skin probability model which adopts look-up table method is devised to swiftly detect the face in the image. Then a multi-layer geometric framework is applied to effectively find the region of facial features. We also use a novel adaptive gradient mask to obtain steady edge information against illumination change and facial feature deformation. In recognition, the deformations of facial features are modeled as stochastic processes by a continuous Markov model which has a simple structure to statistically model time-varying data of facial expression with fast recognition speed and without loss of accuracy. Experimental results show the proposed system can effectively recognize facial expressions from cluttered scenes, illumination change and depth. Comparing to continuous hidden Markov model, the proposed system has great improvement in performance.