In the natural world, physical laws constrain object reflectance. These constraints are learned by the visual system and affect our naturalness perception of images. The goal of this study on natural images is to discover which image statistics are used to determine color naturalness. In a series of experiments, we investigated the role of color in our perception of naturalness by asking subjects to rate the naturalness of the color saturation for different images. We found that observers could easily differentiate unnaturally over or under-saturated images from their original natural counterparts with certain accuracy. Moreover, this ability is not based on the observers' memory of specific pictures nor is it based on high-level knowledge about object color (memory color) since observers could correctly judge natural color saturation for images they had never seen before and for objects with native colors (e.g. oranges) and non-native colors (e.g. cars). Furthermore, natural color saturation judgments differed from global saturation judgments for random-pixel images made from the original natural images indicating that image structure is important for judging natural color because color distributions vary from image to image (e.g. pictures of snow and pictures of fruit). Additionally, both luminance contrast and saturation of an image affect our judgment of color naturalness. If the luminance contrast is reduced without changing chromaticity, the image appears increasingly over-saturated unless color saturation is also reduced. Our findings suggest that critical information lies in the luminance-color relationship. Further investigation shows that as color saturation increases or luminance contrast decreases, color appearance changes from surface color mode to aperture (self-luminous) color mode, and objects unlikely to be self-luminous appear unnatural. We suspect that this mechanism also aids our natural saturation judgment.