The fusiform face area (FFA) is a region of the human ventral visual pathway that exhibits a stronger response to faces than to objects. The role of this region within the face perception network is not well understood, and its face selectivity has been debated. Furthermore, it is unclear which specific properties of visual stimuli are systematically reflected in the patterns of activation of this region. There is evidence from various sources that FFA might encode orientation. This includes psychophysics of face selective view-point aftereffects, fMRI adaptation results, and electrophysiological experiments that have revealed neurons that are highly tuned to face orientation in the macaque homologue of FFA. Here we directly explored the encoding of orientation using a combination of functional magnetic resonance imaging (fMRI) and multivoxel pattern analysis (MVPA). We presented subjects with synthetic images of faces and cars that were rotated in depth and presented either above or below fixation. We explored orientation-related information available in fine-grained activity patterns in FFA and early visual cortex. Distributed signals from the FFA allowed above-chance classification of within-category orientation information only for faces. This was also generalized to faces and objects presented in different retinotopic positions. In contrast, classification in early visual cortex resulted in equal, above-chance classification of face and car orientation information, but only when trained and tested on corresponding retinotopic positions. Classification across position was substantially decreased for both categories in early visual cortex. We conclude that category-selective effects of stimulus orientation are reflected in the fine grained patterns of activation in FFA, and that the structure of these patterns is partially translation invariant.