It has been well established that top-down influence plays an important role in processing of objects with which we have high level of expertise. One example is that expectation of visual input of faces or words facilitates the detection of the exemplars from these categories. Recent fMRI studies revealed that responses of some category-specific regions in the ventral occipitotemporal cortex (VOT) can be enhanced by top-down expectations without the presence of real stimuli. However, it is unclear whether such top-down expectations also activate the primary visual cortex differentially. Here we manipulated the participants expectation of object categories (i.e., faces versus words) to address this question. We had two within-subject tasks: the face task and the word task. Each task included a training period and a testing period. For the face task, the training period included increasingly noisy faces and the test period included pure-noise images. For the word task, the training period included increasingly noisy words and the test period included the same pure-noise images as in the face task. Participants were misled that in the testing period 50% of the pure-noise images contained faces or words respectively, and they must detect them. We trained support vector machines (SVM) on the fMRI data scanned during the training periods of the face and word tasks and then tested the learned models with the fMRI data from the testing periods. We found that the discriminant accuracy was 76.57% for V1, 89.64% for V2, and 89.92% for V3~V5. In addition, in the face- and words-preferential areas of VOT (i.e., FFA and VWFA), the accuracy was 61.36%. Our findings suggest that top-down expectations of faces or words can activate the primary visual cortex with even greater differentiation than the face- or word-preferential areas in the VOT.