James Brown was born in rural Elko, South Carolina (Barnwell County). His parents were Susie Brown and Joseph ("Joe&quot James Gardner (who changed his surname to Brown after Mattie Brown who raised him). Although Brown was to be named after his father Joseph, his first and middle names were mistakenly reversed on his birth certificate. He therefore became James Joseph Brown, as a young child, Brown was called Junior. When he later lived with his aunt and cousin, he was called Little Junior since his cousin's nickname was also Junior. He was of African American with Native American, specifically Apache, descent through his father, and Asian ancestry.

Brown and his family lived in extreme poverty. When Brown was two years old, his parents separated after his mother left his father for another man. After his mother abandoned the family, Brown continued to live with his father and his father's live-in girlfriends until he was six years old. After that time, Brown and his father moved to Augusta, Georgia.

His father sent him to live with an aunt, who ran a house of prostitution. Even though Brown lived with relatives, he spent long stretches of time on his own, hanging out on the streets and hustling to get by. Brown managed to stay in school until he dropped out in the seventh grade.

During his childhood, Brown earned money shining shoes, sweeping out stores, selling and trading in old stamps, washing cars and dishes and singing in talent contests. Brown also performed buck dances for change to entertain troops from Camp Gordon at the start of World War II as their convoys traveled over a canal bridge near his aunt's home. Between earning money from these adventures, Brown taught himself to play a harmonica given to him by his father. He learned to play some guitar from Tampa Red, in addition to learning to play piano and drums from others he met during this time. Brown was inspired to become an entertainer after watching Louis Jordan, a popular jazz and R&B performer during the 1940s, and Jordan's Tympany Five performing "Caldonia" in a short film.

As an adult, Brown legally changed his name to remove the "Jr." designation.[16] In his spare time, Brown spent time practicing his various skills in Augusta-area stalls and committing petty crimes. At the age of sixteen, he was convicted of armed robbery and sent to a juvenile detention center upstate in Toccoa in 1949.

While Brown was in reform school, he became acquainted with Bobby Byrd, who first saw Brown perform in prison. Byrd watched and admired Brown's ability to sing and perform. Byrd's family helped Brown secure an early release after serving three years of his sentence. The authorities agreed to release Brown on the condition that he would get a job and not return to Augusta or Richmond County. After stints as a boxer and baseball pitcher in semi-professional baseball (a career move ended by a leg injury), Brown turned his energy toward music.