We are constantly faced with goals that require us to process the same visual stimuli according to different task demands. While much attention has been paid to the perceptual processing of visual stimuli, much less is known about the processing of the task context in which these stimuli occur. Here we used MEG to study the temporal evolution of task representations and their effect on visual category processing. During MEG recordings, participants (N = 17) were shown 8 visually-presented objects embedded in one of four task contexts. Multivariate classification was carried out in a time-resolved manner both for task and category. We found that after presentation of the object stimulus, object information increased rapidly as expected. In contrast, task information increased relatively slowly, peaking around 500 ms post stimulus onset. Using temporal cross-classification, we demonstrate that task context is processed in multiple, distinguishable but partially overlapping stages. To better understand the spatial distribution of task and category information, we used model-based MEG-fMRI fusion (Cichy et al., 2014, Nat. Neurosci.) by combining our MEG data with the results of a previous fMRI study (Harel et al., 2014, PNAS). While early visual cortex exhibited preferential processing of stimulus information and lateral prefrontal cortex preferential processing of task context, we found co-varying task and category information in ventral temporal cortex, indicating that task context may affect visual processing in these brain regions. Together, our results reveal a relatively late involvement of task context in visual processing and highlight the representational dynamics of task context.