The ability to estimate elapsed time is fundamental to human behavior, and this ability varies substantially across individuals. Its neural basis remains debated. While some evidence points to a central, modality-independent ‘clock’ underpinning this ability, other empirical data suggest sensory modality-specific ‘clocks’. And whether different brain structures are involved in estimation of shorter and longer temporal intervals remains unclear. Here, we took a new approach to these questions by investigating individual differences in the ability to estimate the duration of stimuli in a large group of healthy observers, and their relationship to brain structure. We examined how well individuals could judge the duration of a stimulus presented for either a short (∼2s) or longer (∼12s) duration in either visual or auditory sensory modalities. We found substantial variation in accuracy across the group of participants; but while variability in participants' performance was highly consistent across modalities, it was much weaker across different estimation durations. We then examined whether these individual differences in behavioral accuracy were reflected in differences in gray matter density using a voxel based morphometry analysis applied to structural MRI images of our participants' brains. Taken together, our data suggest the existence of different modality-independent mechanisms for judging different temporal durations.