The recording sessions saw some tension between the band members, as each had a different vision for the album.[citation needed] Originally conceived as a concept album, the theme of "Southern Accents" became somewhat murky with the inclusion of three songs co-written by Stewart, and three other songs originally planned for the album left off. Songs cut from the track list include "Trailer," "Big Boss Man," "Crackin' Up," "The Image of Me" and "The Apartment Song." These can be found on Playback, a box set released 10 years later that included familiar songs with outtakes, b-sides and other rarities.

While mixing the opening track "Rebels" Petty became frustrated and punched a wall, severely breaking his left hand. Subsequent surgery on his hand left him with several pins, wires and screws to hold his hand together.

The album cover features an 1865 painting by Winslow Homer titled The Veteran in a New Field.