Autism is a highly heritable developmental disorder. Endophenotypes are heritable sub-clinical traits associated with the expression of a disorder in the general population. One of the core symptoms, and potential endophenotypes of autism, is difficulty processing eye gaze cues. However, the specific nature of this difficulty is still not well understood. Here, we investigated whether a part of this difficulty is related to impairments in understanding the referential nature of eye gaze; that is following the trajectory of gaze to determine what a person is looking at. We investigated whether individuals who are typically developing (TD), but high in autism traits, have more difficulty tracking eye gaze than do those who are TD with low autism traits. The participants included 120 TD young adults who scored ±1 SD beyond the mean of 2257 young adults screened on the Autism Quotient. We included equal numbers of male and female participants in the high and low trait groups. The eye gaze following task included static images of a person looking at one of many possible objects in a scene. Participants selected one of four labels that identified the target gazed-at object. There was a significant participant sex x autistic trait group interaction (p < .05). Males with high autism traits had lower gaze following accuracy than both males with low autism traits and females with high autism traits (p < .05). For females, there was no difference in performance between the high and low autism trait groups. In addition, scores on gaze following were related to face recognition behavior. These findings indicate that impaired gaze following is a potential endophenotype of autism, but only for males, which is consistent with the notion that autism symptoms and endopheynotypes may manifest differently in females.