Visual scene segmentation represents one of the fundamental functions accomplished by the early visual system. Figure-ground assignment, object perception, and attentional allocation all rely upon the ability to rapidly and accurately segment the visual scene into coherent objects and their surrounds. Using synthetic images portraying simple texture-defined figures and an electrophysiological paradigm that allows us to separately monitor cortical responses to figure and background regions, as well as interactions between them, we found distinct neuronal networks responsible for the processing of each. Figure activity was distributed over a network of ventral stream visual areas including the lateral occipital cortex. A separate network, extending from primary visual cortex through the dorsal visual pathway was observed in response to the background region. The activated sites and temporal sequence of these networks was largely invariant with respect to the cues used to define the figure, did not depend on its spatial location or size, and were largely unaffected by attentional instructions. To evaluate border-specific activity, tagged stimuli were presented with figure and background regions composed of different one-dimensional textures such that the figure region remained segmented throughout the duration of the stimulus presentation. Reponses occurring at nonlinear interaction terms resulted in distinct source distributions relative to the figure or background region responses, indicating unique mechanisms that signal border discontinuities. These responses were greatly diminished when the figure region was constructed such that it was continuously segmented, as well as, by the introduction of gaps between the two regions. These non-linear interactions therefore reflect finely tuned pooling activity occurring at the region borders that is highly sensitive to the global segmentation state of the scene. Collectively, these data reflect unique aspects of the object processing hierarchy in which low-level features of the retinal image are abstracted forming the neural basis of visual scene segmentation.