Abstract: Dysfunction in inhibitory control of attention was shown in children with Attention Deficit Hyperactivity Disorder (ADHD), Fetal Alcohol Spectrum Disorder (FASD), and elderly with Parkinson's Disease (PD). Previous studies explored the deficits in top-down (goal oriented) and bottom-up (stimulus driven) attention with a series of visual tasks. This study investigates the difference in attentional selection mechanism while patients freely viewed natural scene videos without performing specific tasks, and the difference is utilized to develop classifiers to differentiate patients from controls. These specially designed videos are composed of short (2-4 seconds), unrelated clips to reduce top-down expectation and emphasize the difference in gaze allocation at every scene change. Gaze of six groups of observers (control children, ADHD children, FASD children, control young adults, control elderly, and PD elderly) were tracked while they watched the videos. A computational saliency model computed bottom-up saliency maps for each video frame. Correlation between salience and gaze of each population was computed and served as features for classifiers. Leave-one-out was used to train and test the classifiers. The classifier differentiates ADHD, FASD, and control children with 72% accuracy; another classifier differentiates PD and control elderly with 89% accuracy. A feature selection method was also used to identify the features that differentiate the populations the most. This study demonstrates that attentional selection mechanisms are influenced by PD, ADHD, and FASD, and the behavioral difference is captured by the correlation between salience and gaze. Furthermore, this task-free method shows promise toward future screening tools.