Complex social life is thought to be a major driver of complex cognition in primates, but few studies have directly tested the relationship between a given primate species’ social system and their social cognitive skills. We experimentally compared lifespan patterns of a foundational social cognitive skill (following another’s gaze) in tolerant Barbary macaques, Macaca sylvanus, and despotic rhesus macaques, Macaca mulatta. Semi-free-ranging monkeys (N = 80 individuals from each species) followed gaze more in test trials where an actor looked up compared to control trials. However, species differed in ontogenetic trajectories: both exhibited high rates of gaze following as juveniles, but rhesus monkeys exhibited declines in social attention with age, whereas Barbary macaques did not. This pattern indicates that developmental patterns of social attention vary with social tolerance, and that diversity in social behaviour can lead to differences in social cognition across primates.