Influential models of visual working memory treat each item as an independent unit and assume there are no interactions between items. However, even in displays with simple colored circles there are higher-order ensemble statistics that observers can compute quickly and accurately (e.g., Ariely, 2001). An optimal encoding strategy would take these higher-order regularities into account. We examined how a specific ensemble statistic -the mean size of a set of items- influences visual working memory. Observers were presented with 400 individual displays consisting of three red, three blue, and three green circles of varying size. The task was to remember the size of all of the red and blue circles, but to ignore the green circles (we assume that ignoring the green circles requires the target items to be selected by color, Huang, Treisman, Pashler, 2007; Halberda, Sires, Feigenson, 2006). Each display was briefly presented, then disappeared, and then a single circle reappeared in black at the location that a red or blue circle had occupied. Observers used the mouse to resize this new black circle to the size of the red or blue circle they had previously seen. We find evidence that the remembered size of each individual item is biased toward the mean size of the circles of the same color. In Experiment 2, the irrelevant green circles were removed, making it possible to select the red and blue items as a single group, and no bias towards the mean of the color set was observed. Combined, these results suggest that items in visual working memory are not represented in isolation. Instead, observers use constraints from the higher-order ensemble statistics of the set to reduce uncertainty about the size of individual items and thereby encode the items more efficiently.