The Union Army recruited and forced nearly 2,000 Blacks (free and slave) into Fort Negley's labor battalions. "Known men of treason," including Belle Meade plantation's William G. Harding, were arrested and lost money, slaves, and supplies to support the project.

Free black families faced mistreatment and struggled to secure livelihoods. Black men were sometimes forced into labor by the Union Army, for example, in the building of Fort Negley in Nashville. The Union army sometimes used the wives and children of black soldiers as laborers. Although employed in a variety of tasks around camp, these families were seen as burdensome to the army.