One of the main challenges in visual psychophysics is to identify the stimulus features on which the visual system bases its computations: they are a pre-requisite for computational models of perception. Here, we use logistic regression for extracting critical stimulus features and predicting the responses of observers in psychophysical experiments. Rather than embedding the stimuli in noise, as is done in classification-image analysis, we infer the important features directly from physically heterogeneous stimuli. Using this approach -which we call ‘decision-image analysis‘- we predict the decisions of observers performing a gender-classification task with human faces as stimuli. Our decision-image models are able to predict human responses not only in terms of overall percent-correct, but predict, for individual observers, the probabilities with which individual faces are (mis-)classified. Comparing the prediction performance of different models can be used to rigorously rule out some seemingly plausible models of human classification performance: We show that a simple prototype classifier, popular in so-called “norm-based” models of face perception, is inadequate for predicting human responses. In contrast, an optimised generalised linear model can predict responses with remarkable accuracy. While this predictor is based on a single linear filter, this filter is not aligned with the first principal component of the stimulus set, in contrast to what has been proposed by proponents of “eigenface-based” models. In addition, we show how decision-images can be used to design optimised, maximally discriminative stimuli, which we use to test the predictions of our models. Finally, the performance of our model is correlated with the reaction times (RTs) of observers on individual stimuli: responses with short RTs are more predictable than others, consistent with the notion that short RTs may reflect earlier, more perceptual decisions modelled well by our decision-images, whereas longer RTs may be indicative of a larger cognitive or top-down component.