We are sounded by ensembles of objects every day. How are outliers in an otherwise homogeneous ensemble represented by our visual system? Are outliers ignored because they are the minority? Or do outliers alter our perception because their presence changes the nature of an otherwise homogenous ensemble? We have previously demonstrated that ensemble representation in human anterior-medial ventral visual cortex is sensitive to changes in the ratio of two types of objects comprising a heterogeneous ensemble. In the present study we investigated how outliers impact object-ensemble representation in this brain region. In an fMRI-adaptation paradigm, we presented a homogenous ensemble containing 25 identical elements followed by another homogenous ensemble containing a majority of identical elements with 0, 2, or 4 outliers. Observers were asked to ignore the outliers and judge whether the two ensembles were mostly same or different. For same judgments, the majority of the elements in the second ensemble were identical to those in the first, except for the outliers which were visually distinct. For different judgments, the majority of the elements in the second ensemble were distinct from those in the first, except for the outliers which were identical to those in the first ensemble. If outliers can be ignored, there should be adaptation in all same judgement trials and release from adaptation in all different judgement trials. Interestingly, in anterior-medial ventral visual cortex, with just 2 or 4 outliers in the same judgement trials, there was a significant release from adaptation compared to when 0 outliers were present. Moreover, with just 4 outliers in the different judgement trials, there was significant adaptation, comparable to the 0 outlier same judgment trials. Together, these results reveal that outliers significantly impact the perception and representation of otherwise homogeneous ensembles in human anterior-medial ventral visual cortex.