A native of Portland, Oregon, Spencer made his professional boxing debut in May 1960. After building up a 31-5 record, which included wins over contenders Doug Jones, Brian London and Amos Lincoln, he was highly regarded enough to be a part of the eight-man WBA elimination tournament, held after Muhammad Ali had been stripped of the title. Spencer won a 12 round decision over former title-holder Ernie Terrell in August 1967 and was then matched against Jerry Quarry in the semi-finals. Despite entering the bout as a 6/4 favourite, Spencer lost to Quarry by a 12th round TKO.

Spencer lost his next three fights by knockout, to Leotis Martin, Billy Walker and Mac Foster, and was never a serious title contender again. He continued boxing until 1971, though without winning another bout.