Spencer moved north to Wilmington, Delaware, which had a large free black population. He contributed to the development of the free Africa-American community in this city. [2] There he founded the Union Church of Africans in 1813. (This followed the 1793 establishment in Philadelphia of the African Methodist Episcopal Church by Richard Allen, which was the first independent black church. It had ties to the Methodist Episcopal Church until 1816, when several congregations formed it as a denomination, electing Allen as bishop.

In 1814, Spencer called for the first annual gathering of the Union Church, an event now known as the Big August Quarterly. This has drawn members of this denomination and their descendants together in an annual religious and cultural festival, which continues to be held in the early 21st century.[3]

Thomas Garrett, a Quaker in Wilmington who was an abolitionist and active in the Underground Railroad as a “conductor” of refugee slaves, helped Spencer buy land to build the Mother Church on French Street in Wilmington. Over the course of his lifetime, Spencer began 31 churches, nearly all of them with schools. He became known as the “father of the independent black church movement.”[3]