Sensory systems are expected to improve their performance as they adapt to new environments. This expectation has been contradicted in studies of visual adaptation. Prolonged exposure to visual stimuli may increase or decrease your sensitivity in a manner that has defied explanation in terms of neural fatigue, lateral interactions, or optimal inference. A solution to this problem was recently proposed from considerations of neuronal parsimony. On this view, visual systems solve the problem of allocation of limited resources. They encode a googol of stimuli using a limited pool of specialized cells. Inspired by the physiological evidence that more cells are allocated to more useful stimuli, we developed a theory of cell allocation according to their utility in basic visual tasks. (For example, cells with small receptive fields are preferred for localizing stimuli, and cell with large receptive fields are preferred for identifying stimuli, i.e., for estimating the frequency content.) We find that the optimal allocation of cells across all potential spatiotemporal stimuli leads to a performance characteristic strikingly similar to the human spatiotemporal contrast sensitivity function (Kelly, 1979). Changes in stimulus statistics are predicted to shift the entire sensitivity function, yielding a diagnostic pattern of gains and losses of sensitivity. We confirmed these predictions by measuring the contrast sensitivity to drifting luminance gratings whose speeds were sampled from distributions with different mean speeds. We also simulated synaptic plasticity in spiking neural networks that afford rapid changes of receptive field size. Notably, unsupervised plasticity in such networks readily leads to global redistribution of receptive fields, shifting the ensuing sensitivity function while requiring no explicit representation of stimulus statistics.