People apply previously learned contextual knowledge to facilitate the perception and encoding of scenes. When extrapolation following the employment of contextual information occurs, it sometimes leads to scene memory distortion: people report as if they saw more than they actually had seen. This phenomenon is called the “boundary extension” effect (Intraub & Richardson, 1989). The present study aimed to clarify the effects of contextual information on boundary extension in a more systematic way. Based on the assumption that it is harder to extract contextual information from inverted scenes compared to intact scenes (Kelley, Chun, & Chua, 2003), we presented inverted scenes either during encoding or retrieval to manipulate the level of contextual information and compared the magnitude of boundary extension effect for upright versus inverted scenes. In a series of experiments, we found that scene inversion during encoding, but not during retrieval, significantly reduced boundary extension. We further confirmed this finding in a recognition test for peripheral objects in scenes. Using objects which shared context with, but were absent in the original scene stimuli, we expected that scene inversion would reduce false recognition of those objects. The result showed that false recognition occurs significantly less often for inverted scenes compared to upright scenes. Showing reduced memory distortion for inverted scenes, the current study directly demonstrated that access to contextual information is a critical component of scene extrapolation process.