In February 1943 he was drafted into the Army and assigned to the Infantry. He spent time in Germany and returned to the Pacific Coast League in 1946. He was sent to Salt Lake City Bees of the Class C Pioneer League for a while but was soon recalled by San Francisco and spent the next ten seasons in the Pacific Coast League with San Francisco, the Portland Beavers in 1949, the Sacramento Solons in 1952 and the Oakland Oaks in 1955, with consistent seasons with high batting averages, home runs and runs batted in. In 1946, he hit the longest home run in the history of Seals Stadium in San Francisco.

Oakland finally sold him to the Cincinnati Redlegs and he got his chance at age 33 to play in the major leagues in 1955. The Detroit Tigers had tried to buy him when he was with Portland but the owner was asking too high a price. His contract was purchased by the Boston Braves in 1948, but he was unable to report to the team because of appendicitis. Already 33 years old when called up by the Redlegs, Joe appeared in 21 games entirely as a pinch hitter and collected only two hits.