As we navigate the world we encounter complex visual scenes that we can both categorize and discriminate. Prior studies have reported scene category information in both early visual cortex (EVC) and the scene-selective parahippocampal place area (PPA). However, these studies used only a small number of preselected categories, providing little insight into the discrimination of individual scenes or unbiased test of the categorical structure of the representations. Here we use a multivariate ungrouped approach to establish the differential discrimination and categorical structure of scene representations in EVC and PPA. We presented 96 unique, diverse, and highly detailed scenes in an event-related fMRI paradigm with each scene being a unique condition. The scenes were chosen to equally sample from all the combinations of three broad classes based on apparent depth (near/far), content (manmade/natural), and the gross geometry of the scene (open/closed). We then used multi-voxel pattern analysis to establish how the responses of PPA and EVC. Importantly, neither our stimuli nor analyses had any bias towards any particular organization or categorization of the scene stimuli. We found that the response of both PPA and EVC could be used to discriminate individual scenes from one another. However, the scene representations in these two regions differed in their categorical structure. The response of PPA grouped scenes by their geometry (open/closed) despite differences in their perceptual content, consistent with a role in navigation. In contrast, early visual cortex grouped scenes based on the distance to the nearest foreground object (near/far). In neither region did we find evidence for strong grouping by the scene categories often assumed in the prior literature (e.g. beaches, cityscapes). These results suggest while both regions can discriminate scenes each encodes different aspects of complex scenes, providing insight into the transformation of visual information along the ventral visual pathway.