As the battle over removing Confederate monuments rages across the country in the wake of the white supremacist rally in Charlottesville, Va., there is increasing talk of a compromise: to leave them up, but also build new statues honoring a more diverse set of heroes.

“I can’t help but think that rather than pulling down monuments, as some want to do, rather than tearing down monuments that have graced our cities all across this country for years, we ought to be building more monuments,” Vice President Mike Pence said this week on Fox News. “We ought to be celebrating the men and women who have helped our nation move toward a more perfect union and tell the whole story of America.”

And from the other party, Sen. Tim Kaine, D-Va., said the debate over memorializing the Confederacy isn’t just about “subtraction” of offending statues — though he does want them removed from the U.S. Capitol. “Let’s also talk about whose stories haven’t been told,” Kaine said. He recommended Virginia switch out its Robert E. Lee statue in the Capitol for one of Pocahontas.

Telling the whole story of America, however, is harder than it sounds. Community leaders in Southern cities have faced stiff resistance when they attempted to memorialize civil rights heroes or build monuments or museums explaining slavery.

The issue is particularly salient in Richmond, Va., once the capital of the Confederacy and now a modern, majority-black metropolis that attracts millions of tourists each year. One of the city’s tourist attractions is Monument Avenue, in the historic and elegant Fan District. Five towering statues of Confederate generals and leaders line the street, including a 60-foot-tall Gen. Robert E. Lee and one of Jefferson Davis, president of the Confederacy.

In 1996, the city decided to erect a monument to the African-American tennis great Arthur Ashe, a native of Richmond, alongside the Confederates. The statue attracted controversy, with some residents arguing it was inappropriate to place him next to the heroes of the Confederacy. In the end, Ashe was placed at the far western end of Monument Avenue — putting three blocks’ distance between the famous black athlete and the nearest Confederate commander.

The statue of Confederate Gen. Robert E. Lee that stands in the middle of a traffic circle on Monument Avenue in Richmond, Va. (Photo: Steve Helber/AP)

“That was really the first time that the two histories collided in the same place,” said Ana Edwards, a longtime community activist who is pushing the city to create a nine-acre park memorializing the city’s former slave market in Shockoe Bottom.

In 2003, the city erected a statue at the site of a former ironworks of President Abraham Lincoln and his son Tad, who visited Richmond two days after it fell to the Union Army. “As a Southerner, I’m offended,” Sons of the Confederacy member Bragdon Bowling told the New York Times then. “You wouldn’t put a statue of Winston Churchill in downtown Berlin, would you? What’s next, a statue of Sherman in Atlanta?”

Bowling recently organized a September rally to defend the city’s Lee statue, but canceled it after Charlottesville, saying he didn’t want to be engaged in “social or racial issues.” He did not respond to a request for comment from Yahoo News.

The city’s mayor, Levar Stoney, who is black, has said he personally objects to the Confederate statues but does not believe they should come down. He formed a commission to decide whether and what kind of new memorials should be added to Monument Avenue and how to put the Confederate statues into context.

Stoney has suggested including a statue commemorating black Union soldiers who marched into the city in 1865 and raised the American flag, as well as another statue of Oliver Hill, a civil rights lawyer whose work helped end state-mandated segregation in public schools. Recently, the city unveiled a statue of Maggie Walker, the first black woman to charter a bank, and will create a monument celebrating the Emancipation Proclamation on Brown’s Island next year.

Stoney has also said he thinks the community should focus on improving public education and easing poverty — not just public symbols.

The commission’s contentious first public meeting, held earlier this month before the Charlottesville rally, attracted more than 500 residents — at least one was led out by a police officer for causing a disturbance, according to the local paper. Some shouted that the monuments were tokens of white supremacy and must be destroyed; others objected to the idea of adding any non-Confederate statues to the monument as “politically correct” nonsense.

“Every tourist who comes to Richmond wants to see Monument Avenue; they don’t want to see a politically correct Monument Avenue,” Henry Kidd, a former officer of the Sons of Confederate Veterans group, said at the meeting.

Meanwhile, other residents have given up on the idea that any amount of context could explain why the city is celebrating five men who betrayed the Union and fought to keep slavery. The Charlottesville rally, which attracted neo-Nazis and white nationalists, one of whom is charged with killing a counter-protester, have reignited calls for the statues to come down altogether. That rally was nominally called to show support for a statue of Robert E. Lee that has since been shrouded. After Charlottesville, Stoney directed the commission to consider the removal of the statues altogether for the first time.

“These statues do remain beacons for people with white supremacist, white nationalist and racist agendas,” Edwards said. “It’s an undercurrent of the country that’s still not resolved and is still dangerous for people.”

“You can’t put up a statue of Anne Frank next to one of Adolf Hitler and say now you’ve solved the problem, now you’re telling the full story,” said Phil Wilayto, another Richmond resident who is working with Edwards to memorialize the city’s former role in the slave trade. “No, what you’re doing is saying you honor Anne Frank and Adolf Hitler. You’re historically schizoid.”

Walter Dotts, a preservationist who grew up on Monument Avenue, said over the past few years he had arrived at the view that it may not be possible to put the statues in enough context to explain their “insidious” original intent — to prolong white rule long after the Civil War by enforcing oppressive Jim Crow laws against the black population. He originally thought keeping the statues could provide a “teachable moment” about the past.

“I’m wondering whether now it wouldn’t be better just to take them down and put them in some other venue, I’m not sure what that would be,” Dotts said.

That number has shrunk in recent days, as city leaders and activists removed Confederate monuments in cites from Baltimore, Md. to Austin, Texas. But it still dwarfs the number of statues commemorating heroes of black history around the country.

Brooks said many attempts to honor civil rights heroes are just now coming to fruition around the country. Alabama in 2015 finally put up a plaque where Dr. Martin Luther King Jr. gave his famous speech about the “arc of the moral universe,” though the government has not yet erected a statue honoring the man. The Smithsonian opened an African-American history and culture museum just last year in Washington, after decades of planning. The state of Mississippi is working on a civil rights museum, and the city of Anniston, Miss. plans to memorialize the bus station where Ku Klux Klan members attacked Freedom Riders in 1961. Charleston, S.C., is building a multimillion dollar African-American history museum to examine the city’s role in the international slave trade, and recently overcame opposition to erect a statue to Denmark Vesey, a freed slave who was killed after he was accused of plotting a slave rebellion.

Workers remove the monuments to Robert E. Lee, commander of the pro-slavery Confederate army in the American Civil War, and Thomas “Stonewall” Jackson, a Confederate general, from Wyman Park in Baltimore, Md., Aug. 16, 2017. (Photo: Courtesy of Alec MacGillis/ProPublica via Reuters)

“The fact that we still don’t have a monument to Dr. King is awful,” Brooks said of Alabama. “You know Jefferson Davis is all up around the Capitol.”

Sometimes the hardest part of telling the “whole story” of America is finding funding. In Richmond, Edwards and the Defenders for Freedom, Justice and Equality group is attempting to create a nine-acre park memorializing what was Richmond’s thriving slave trade area — complete with cells where the slaves were kept, auction houses, traders’ offices, and an African burial ground. The area is considered a National Treasure by the National Trust for Historic Preservation. Edwards fought to have a parking lot covering the area removed, and successfully worked to kill a proposal to bury the historic site under a baseball stadium. Now the city is considering the development of just half an acre of the site to memorialize the slave trader Robert Lumpkin’s jail, but Edwards continues to press for the full nine acres.

“We think the iron is hot right now and that this is a good time,” Edwards said. “We are seeing movement.”