Categories create cognitively useful generalizations by leveraging the correlational structure of the world. Although classic cognitive studies have shown that object categories have both intrinsic hierarchical structure (entry-level effects, Rosch et al., 1976), as well as graded typicality structure (Rosch, 1973), relatively little is known about the neural underpinnings of these processes. In this study, we leverage representational similarity analysis to understand how behaviorally relevant category structure emerges in the human visual system. We performed an fMRI experiment in which participants were shown color photographs of 15 subordinate-level categories from each of two basic-level categories (dogs and cars). Typicality for each subordinate within its basic was also assessed behaviorally. We computed the neural correlation distance between all pairs of categories in early visual areas (V1, V2, V3v, hV4) and object-selective cortex (LOC). We found that as we move from low-level visual areas to object-selective regions, neural distances are compressed within object categories, and simultaneously expanded between object categories. This effect arises gradually as we move up the ventral visual stream through V1, V2, V3v, hV4, with a marked increase between hV4 and LOC. Furthermore, within each basic category in LOC, subordinate typicality influences the organization of the neural distance space: highly typical items are brought closer together, while distance between atypical exemplars grows. Again, this effect arises between hV4 and LOC, suggesting that a significant qualitative jump in the differentiation of object categories from one another as independent structures, as well as in their internal organization, occurs in object-selective areas. Our results show that as we move up the ventral visual stream, distances between neural representations of real-world objects warp to facilitate categorical distinctions. Moreover, the nature of this warping may provide evidence for a prototype-based representation that clusters highly typical subordinates together in object-selective cortex.