Ashe was born in Beaufort, North Carolina. His father, John Baptista Ashe, and brother, John Ashe, both served as Speaker of the North Carolina Colonial Assembly, or House of Burgesses. Ashe became an orphan at the age of nine. He married Mary Porter in 1748; they had three children, including John Baptista Ashe, who would serve in the Continental Congress. After Mary died, Ashe remarried, this time to the former Elizabeth Merrik.

Ashe studied law and was named Assistant Attorney for the Crown in the Wilmington district of the colony.

Ashe's grandson, William Ashe, was a Confederate soldier in the American Civil War, and a son of John B. and the former Eliza Hay. He was killed at Shiloh in Tennessee in 1862, a battle in which William's brother, Samuel Swann Ashe, also fought.[1]