Complex patterns of category selectivity have been demonstrated in several visually-responsive cortical regions. For example, the parahippocampal place area (PPA) responds more strongly to scenes than to other visual stimuli, and it also shows preferences among non-scene stimuli, responding more to buildings than to other objects. We hypothesized that these different aspects of category selectivity might be attributable to early (feedforward) vs. late (recurrent) processing, and thus might impact PPA response at different points in time. To test this, we scanned subjects using a rapid acquisition fMRI protocol (TR = 250 ms) while they viewed four types of images (duration 500 ms; ISI 3 s) in a 2 × 2 design. Images had a large central foreground item that was either a building or a non-building object; independently, these objects were either presented on a scenic background or in isolation with no background. To compute the influence of scenic components, we subtracted responses to stimuli without backgrounds from those with scenic backgrounds. To compute the influences of foreground item category, we subtracted responses to non-building object stimuli from responses to building stimuli. Response differences related to foreground item category (building vs. non-building) were significantly delayed (∼800 ms) compared to response differences related to the presence or absence of scenic backgrounds. We hypothesize that the delayed enhancement of activity for buildings relative to non-building objects may reflect feedback from other cortical regions that identify foreground object category; in this case, enhancing PPA response when the foreground object has navigational relevance. In contrast, scene vs. non-scene discrimination may occur more quickly because it operates on feed-forward calculations intrinsic to the PPA.