At any given moment, much of the relevant information about the visual world is in the periphery rather than the fovea. The periphery is particularly useful for providing information about scene structure and spatial layout, as well as informing us about the spatial distribution and features of the objects we are not explicitly attending and fixating. What is the nature of our representation of this information about scene structure and the spatial distribution of objects? In this talk, I'll discuss evidence that representations of the spatial distribution of simple visual features (like orientation, spatial frequency, color), termed spatial ensemble statistics, are specifically related to our ability to quickly and accurately recognize visual scenes. I'll also show that these spatial ensemble statistics are a critical part of the information we maintain in visual working memory – providing information about the entire set of objects, not just a select few, across eye movements, blinks, occlusions and other interruptions of the visual scene.