Faces are notoriously hard to perceive when turned upside-down. It is often claimed that perceptual decrements reflect a switch from whole-face processing, to a serial analysis of individual features. To test this view, we examined observers' ability to categorise faces briefly presented in their entirety, or viewed through a dynamic aperture that moved incrementally across the facial image. By exposing faces region-by-region, aperture viewing forces observers to use serial feature processing, similar to that supposedly recruited by inverted faces. Consistent with holistic processing accounts, facial identity, gender, and age were categorised more precisely when upright faces were viewed in their entirety, than when viewed through the aperture. Crucially, however, we also observed aperture effects (i.e., greater decision noise) for inverted faces equal to, or greater than, those seen for upright faces (Experiment 1). We go on to show that this pattern is seen irrespective of the direction of aperture transition (Experiment 2) or the nature of the fill used to replace the occluded regions of the to-be-judged image (Experiment 3). A similar pattern is also seen when observers categorise facial expressions (Experiment 4). Finally, we consider the utility of the paradigm for studying individual differences in holistic processing. We present data from fourteen individuals with developmental prosopagnosia that suggest striking variability in the use of whole-face cues in this population (Experiment 5). These results suggest that when interpreting inverted faces, access to the wider face context is far more important than currently believed. Where inversion results in poor local descriptions, possibly due to violation of lighting priors, residual ability to integrate information from contiguous regions may be crucial.