In an effort to model perceptual averaging over time, we investigated a critical element in the process: the number of trials that subjects use when generating a temporal average. In Experiment 1, participants were presented with a sequence of vertical lines. After every line's presentation, they were asked to provide an estimate of the average length of all the lines they have seen to that point. One key manipulation was a sudden shift in the mean line length between the first and the second halves of the sequence. If the participants could use all the previous stimuli in their estimates of the moving averages, their estimates would not increase or decrease immediately after the mean shift. However, results suggested the opposite pattern: the participants' estimates of the moving averages followed closely to the shift of mean values, suggesting, contrary to previous reports (e.g., Morgan, Watamaniuk, & McKee, 2000) that they did not make full use of the previous stimuli. We fit the data with an autoregressive model with variable weights assigned to previous stimuli. Best fitting parameters suggested that only the most recent six trials actually contribute to subjects' estimation of the mean. In Experiments 2 and 3, we ran the same task with other visual stimuli, both perceptual (discs varying in diameters) and symbolic (numerical values). The number of "contributors" was similar to that of Experiment 1, suggesting that the result is robust across different stimulus types. These results suggest that temporal averaging is fundamentally limited by the capacity of working memory. Implications for existing models of visual short-term memory are discussed.