A new improved unwrapping algorithm based on the combination of modulation and phase fitting reliability is presented. For the measurement of the sand-like object and the surface with complex reflectivity, the noises such as saturation/cut-off and quasi-speckle in the fringe images are important error sources. The phase unwrapping algorithm only based on fringe modulation is not enough to bypass these error points. In this paper, the characteristics of the phase measurement errors caused by saturation/cut-off are analyzed, and a new phase reliability criterion that is sensitive to both modulation and some intensity noises is defined to reduce the unwrapping errors. With this new reliability criterion and 'flood' unwrapping algorithm, the phase unwrapping is implemented along the more reliable path and the errors are limited in the minimum areas.