After his playing career ended, he was a scout for the Baltimore Orioles from 1960 to 1967 and again from 1969 to 1972. In 1961, he stepped in as manager for the Leesburg Orioles for the last six games of the season for his only experience as a manager. Both Billy DeMars and Cal Ripken, Sr. had managed the team that season. Scarborough was a member of the Orioles' major league coaching staff in 1968. After leaving the Orioles, he continued his scouting career with the California Angels (1973-1977) and Milwaukee Brewers (1978-1982).