The speed of visual object categorization has been studied with manual Go/No-go paradigms and saccadic reaction time (SRT) paradigms. SRT paradigms require observers to perform choice saccades between parafoveally presented stimuli towards a predefined target category, and provide a more precise description of the lower bound of processing speed. According to a recent study (Visconti di Oleggio Castello & Gobbini, 2015) rapid saccades towards personally familiar (PF) faces can be performed by healthy observers within 180ms. However, this study used a limited number of PF faces, which differed between the few (n=7) observers tested, two of which performed at or near chance level. We tested different cohorts of healthy observers and PS, a case of acquired pure prosopagnosia, to investigate visual categorization across three SRT experiments that controlled for task constraints and breadth of decisional space. Observers performed one gender, and two familiarity categorization tasks. The latter differed in terms of the decisional space, as observers were required to saccade towards one of few, or many possible targets, respectively. Our findings show that healthy and impaired observers' performance varies as a function of decisional space. All observers, including PS, performed binary gender decisions most efficiently; however, the distribution of PS's SRTs differed fundamentally from those of healthy observers. For familiarity decisions, target search for fewer identities was associated with more accurate behavior and faster SRTs. Importantly, like PS, numerous healthy observers' performance was at chance level for familiarity decisions. These observations stress the importance of considering task constraints and procedural aspects in SRT paradigms when attempting to determine processing speed using forced-choice categorizations. Our findings dispute the previous interpretation of SRT modulation attributed to personal familiarity (Visconti di Oleggio Castello & Gobbini, 2015); we argue that rapid SRTs towards PF faces can be entirely accounted for by decisional space constraints.