Functional connectivity within a cortical network is essential for computation and behavior. If successful processing results at least partly from connectivity, then anatomical variations in white matter interconnecting a network should correspond to behavioral differences across subjects. However, it is unknown if white matter properties associated with a category-specific network affect category-specific processing. Using fMRI and diffusion-weighted imaging in adults (n=9, ages 18-40), adolescents (n=11, ages 12-16), children (n=13, ages 8-11), and developmental prosopagnosic (face-blind) adults (n=7, ages 25-53), we employ a novel protocol combining measurements of white matter properties, functional selectivity, and behavior in the same subjects. We quantify what we call functionally-defined white matter (FDWM) by extracting fiber tracts that pass near either a face-selective region on the middle fusiform gyrus (mFus-faces) or a place-selective region on the collateral sulcus (CoS-places). We find that in typical adults, diffusion properties in FDWM voxels immediately adjacent to face- and place-selective cortex correlate with accuracy in face or place processing tasks, respectively. Compared to typical adults, this relationship between FDWM local to functional regions and behavior is strikingly atypical in adults with developmental prosopagnosia, suggesting that atypical development of this FDWM-behavior relationship may have perceptual consequences. Lastly, this link between white matter properties and category-processing appears to undergo a protracted development, showing a gradual increase in its strength from childhood, to adolescence, to adulthood. We argue that white matter associated with functional divisions in high-level visual cortex is behaviorally relevant and should be a major component of future research in diffusion-weighted imaging.