After studying at the University of Virginia, Randolph "read the law" with an established firm and was admitted to the bar in 1840. He practiced law in Charlottesville, Virginia, and he and Mary lived at his plantation of Edgehill. They moved to the capital of Richmond in 1849. He became active in the community as well as having his law practice. He founded the Richmond Mechanics' Institute and was an officer in the Virginia Historical Society.[2]

Randolph was commissioned a major in the Confederate Army, and later served as a colonel of the artillery in the Battle of Big Bethel. He was promoted to brigadier general on February 12, 1862. Mary Randolph was active in the Richmond Ladies Association, which organized welfare and relief for the war effort.[2]

Randolph was appointed by Jefferson Davis as Secretary of War on March 18, 1862, and he took office on March 24, 1862. He helped reform the department, improving procurement and writing a conscription law similar to one he had created for Virginia. He was most well known for his strengthening the Confederacy's western and southern defenses, but came into conflict with Jefferson Davis over this. With weakening health due to tuberculosis (TB), he resigned on November 17, 1862.[2]

In 1864, Randolph took his family to exile in Europe, staying in England and France. They returned to Virginia in 1866. He died of tuberculosis in March 1867 at his Edgehill plantation.[2] He is buried at Monticello in the Jefferson family graveyard.