It was the summer of 1863, and Abraham Lincoln needed troops. That March, Congress had passed the Enrollment Act, requiring all males between the ages of 20 and 45 to register for a military draft. Since that May, Ulysses S. Grant laid costly siege to the city of Vicksburg, Mississippi, a strategic Confederate fort on the Mississippi River; by June, there would be 80,000 Union soldiers surrounding that city. In late April, “Fighting Joe” Hooker crossed the Rappahannock River, trying to catch Robert E. Lee in a pincer movement. The maneuver failed and the Union lost 17,000 men at the ensuing Battle of Chancellorsville, perhaps Lee’s finest victory. Just two months later, Lee suffered his worst defeat, at Gettysburg. Though victorious there, the Union lost 23,000 men.

The draft began in New York City about two weeks after Gettysburg. The draft would do what all drafts do, which is compel men who do not have the natural constitution of a warrior to become one anyway. You could avoid it by paying $300. Otherwise, you would don the Union blue.

The first day of the draft, Saturday the 11th, went well. The second, Monday the 13th, was a disaster. The Irish had not wanted to work alongside blacks on the docks of Manhattan. They had even less interest in fighting what some called “the nigger war,” so that, presumably, emancipated blacks could come north and take their jobs. Their anger first erupted at the draft offices near today’s United Nations headquarters on the East Side of Manhattan. “The men seemed to be excited beyond expression,” reported The New York Times. The mob “danced with fiendish delight” as it set buildings aflame and attacked blacks, killing dozens.

On the second day, the rioters set upon a four-story house at 339 West 29th Street, in what is today the Manhattan neighborhood of Chelsea. Here, on what was then known as Lamartine Place, stood the graceful home of Quaker abolitionists James Sloan Gibbons and Abigail Hopper Gibbons. It was, according to their friend Joseph H. Choate, “a great resort of abolitionists and extreme anti-slavery people from all parts of the land.” The Hopper-Gibbons house was a known stop on the Underground Railroad, a network of routes and safe houses that, in the first half of the 19th century, whisked runaway slaves across the Mason-Dixon Line. Choate reports that he dined there with William Lloyd Garrison. Present at the dinner was “a jet-black negro who was on his way to freedom.”

An Illustration depicts a mob attacking the home of Abigail Hopper Gibbons (at 339 West 29th Street) during the New York City draft riots, New York. Interim Archives/Getty

Two men on horseback led the mob to the Hopper-Gibbons house. “The horsemen stopped one at each side of the courtyard and allowed about a dozen men with pickaxes into the house while the kept the rest of the mob back,” writes the historian Iver Bernstein. “Finally the advance team was joined by the throng without.” Books were set on fire, art destroyed, furniture defenestrated. Troops fought off the rioters, but after the troops left, the rioters returned.

Choate came to the rescue by “the merest accident.” He had gone to see if there was “trouble in the negro quarters,” only to stumble on the chaos in front of the Gibbons house. One resident of the block, he would later report, had already been “killed in remonstrating with the crowd” (that neighbor, Daniel Wilson, actually appears to have survived the severe beating). Choate went inside but found only pillaging rioters. Nobody from the Hopper-Gibbons clan was there; but two doors down, at the residence of Samuel and Rachel Brown, two Gibbons daughters, Julia and Lucy, lay in hiding. They “threw themselves into my arms,” Choate wrote, “almost swooning.”

It was too dangerous to take the girls down to the street, so Choate went up instead, “over a dozen adjoining roofs,” through the house of Esther and Henry Herrman (other historians hold that they descended through the Hebrew Orphans Asylum; in either case, they had sympathetic Jews to thank). Choate had a carriage waiting. He and the Gibbons daughters made it safely to his house on 21st Street. The riots were finally quelled that Thursday, July 16, in part by troops who had fought at Gettysburg.

Today, the Hopper-Gibbons house is covered in funereal construction mesh. It was bought in 2004 by developer Tony Mamounas; the following year, he sought to build a penthouse atop the four-story structure and was granted permission to do so by the Buildings Department. The city first told Mamounas to stop building in 2009; that same year, the Landmarks Preservation Commission created the Lamartine Place Historic District, which includes the remaining huddle of rowhouses on 29th Street. The Hopper-Gibbons house fell under the district’s auspices, making any further alterations difficult. Mamounas appealed, and there followed a flurry of court motions and recriminations. Finally, the appellate division of the state Supreme Court ruled against Mamounas this February. If the ruling stands, the penthouse will have to come down.

What will remain, in that case, is just another nice building for people who can afford nice things. Mamounas is hardly the villain here; to build and build higher is a primal Gotham urge. He could have been more tactful in dealing with the local preservationists who mounted the campaign against him. He could have also been more blunt. New York is not a city, he might have pointed out, that succumbs easily to history. Or to guilt. Yes, the aforementioned Joseph Choate had heard of a black man lynched on the corner of Sixth Avenue and 32nd Street. And now there stands the Manhattan Mall. You call it forgetting, I call it progress.

People walk past a sign indicating the history of Lamartine Place near the former Hopper-Gibbons House at 339 W 29th St. in Manhattan on April 8, 2015. Jared T. Miller for Newsweek

There is, at least, a sign. Courtesy of the Landmarks Preservation Commission, it hangs from a lamppost on 29th Street and explains the history of the block once known as Lamartine Place. It is, as far as I know, the only acknowledgment of the Draft Riots on any kind of historical marker in the city. No historian I spoke to could think of any commemoration more significant. So, if the city itself has to exert no more effort than to stamp out a single sign on a metal sheet the size of a cafeteria tray, then why should Mamounas have to sacrifice his penthouse? Is a small-time real estate developer really to bear the full weight of history?

The battle over the Hopper-Gibbons House is instructive of a broader New York attitude toward slavery and abolition. That era seems almost too complex for us to remember, eluding the easy narratives of triumph and redemption while calling into question New York’s liberal self-image. Kenneth T. Jackson, a Columbia University professor widely regarded as the preeminent historian of New York City, points out that while Southern cities like Charleston, South Carolina, unequivocally supported slavery and New England ones like Boston thoroughly opposed it, New York was probably the most ideologically conflicted urban center in the nation. Jackson surmised that New York’s complicity in the slave trade remains an “unpleasant topic” to this day. It is not the kind of conversation we can conduct with a well-meaning Starbucks barista. But we will have to have it sooner or later. “There is no future,” Jackson warns, “in denying the past.”

Yet even as its hold on us expires, history manages to intrude like the aggrieved ghost of Hamlet’s father. Prithi Kanakamedala is a historian who organized the “In Pursuit of Freedom” exhibition at the Brooklyn Historical Society last year. She teaches at the Bronx Community College, where 90 percent of the students are either black or Hispanic. Kanakamedala says her students always perk up when learning about the Fugitive Slave Act of 1850, which gave great leverage to Southern states in the apprehension and return of runaway slaves. The law, Kanakamedala’s students tell her, “sounds just like stop-and-frisk.”

On a cold afternoon in February, I set out from the Newsweek offices at the dagger-point of lower Manhattan and walked north, up the crooked spine of the island that is Broadway. My goal was to see as much as possible of the city’s obscured legacy of slavery and abolition. There had been a slave market on Wall Street, abolitionist newspapers where Tribeca is, villages for free blacks in Brooklyn. A patch of Greenwich Village was once known as “The Land of the Blacks.” What still remained of all this, in a 21st century metropolis tumescent with glass and steel?

In his recent book Gateway to Freedom, Columbia University historian and Pulitzer Prize–winner Eric Foner makes two compelling arguments. First, that even after slavery was abolished in New York State in 1827, “the South’s peculiar institution remained central to the city’s economic prosperity,” with money and goods freely crossing the Mason-Dixon Line. But despite its pro-South impulses, the city also became “a crucial way station in the metropolitan corridor through which fugitive slaves made their way from the upper South through Philadelphia and on to upstate New York, New England and Canada.” It was a place that facilitated bondage while preaching freedom.

Foner’s book, by turns scholarly and gripping, includes a map of places in Manhattan and Brooklyn related to the abolitionist enterprise. The map shows 18 sites in Manhattan and five in Brooklyn. I chose what I thought were the most relevant places highlighted by Foner while adding a couple of my own.

A man looks at a display of photographs showing graves at the African Burial Ground Memorial Site, March 1, 2006 in the Lower Manhattan area of New York. Stan Honda/AFP/Getty

My first stop was the African Burial Ground, just north of City Hall. It is, ironically enough, across the street from the U.S. Court of International Trade, the name alluding with sad irony to brigs packed with shackled Africans. The first slaves—11 men—were brought to the Dutch colony of Nieuw Amsterdam in 1626; two years later, three female slaves arrived. They were all owned by the West India Company, which gave them the right to earn wages, marry and own some property.

The British assumed control of New York in 1664 and quickly proved more zealous (and cruel) slave masters, so that by the end of the 18th century, there were 10,727 blacks in what is today New York City and Westchester County, with 77.3 percent of them slaves. “New Yorkers later prided themselves on the notion that in contrast to southern slavery, theirs have been a mild and relatively benevolent institution,” Foner writes. “But New York slavery could be no less brutal than in colonies to the south.” Ninety years before Nat Turner tore through Southampton County, Virginia, in a furious fight for freedom, the “Negro Plot” of 1741 supposedly aimed for New York’s fiery destruction. In the panicked “investigation” that followed, some 30 blacks were burned alive or hanged. Four white accomplices were also executed.

Death, for enslaved Africans, only perpetuated the injustices of life. In 1697, the British masters of New York forbade the burial of blacks in the cemetery of Trinity Church. African burials were conducted outside the settlement’s limits, in what would come to be known as “Negros Buriel Ground.” After the burial ground was closed in 1794, it was covered up, and the city grew above it.

An 1865 map of New York City by mapmaker G. Woolworth Colton .

We rarely think of New York as a slaveholding city, but it had more slaves than any other city except Charleston. Foner writes that on “the eve of the War of Independence...some 20,000 slaves lived within 50 miles of Manhattan island, the largest concentration of unfree laborers north of the Mason-Dixon Line.” Brooklyn was even worse than Manhattan: In 1771, one-third of its population was slave.

By the late 20th century, the site had succumbed the most mundane of modern uses: a parking lot. In 1990, the city sold the land to the General Services Administration (GSA), which set about constructing the Ted Weiss Federal Building. During excavations the following year, 419 bodies were found at a depth of 24 feet. As the history of the site came to light, black activists enjoined the GSA to preserve the ancient cemetery. Wondered the Brooklyn firebrand Charles Barron, a former Black Panther, “Do they then own the remains of your parents because they own the land?”

A man inspects a skeleton at the site of the earliest African American cemetery found in Lower Manhattan in New York, in October 1991. J A Giorano/SABA/Corbis

The African Burial Ground is now a national monument a third of an acre in size; the original cemetery, at 6.6 acres, was approximately 22 times that size. The remains of the slaves were re-interred there in 2003; in 2010, a visitors center opened, a jagged ship’s prow of somber black stone rising from the ground. The Weiss building looms above the ancient cemetery like an annoyed sibling, much younger but much bigger. Reviewing the opening, a critic for The New York Times noted that the memorial “makes the past seem like an excision, a resurrection of an alien time and place, a reminder of what lies deep underfoot.” Above all, the memorial has the feel of a grudging concession. I, for one, can’t think of another museum that devotes so much narrative space to the story of its own creation. Then again, how many institutions can claim to have won a battle against the relentless force that is New York real estate?

The Eternal Indignities

Those who wish to preserve the physical evidence of history always fight a losing battle. Against them are aligned money and time, political avarice and public apathy, not to mention the eternal indignities of gravity and rust. Just one of these forces is usually enough to doom a building for which only the sentimentalist and the historian have any affection.

Still, traces remain, even in this city increasingly dominated by hedge funders and foreign billionaires who have so much of the world’s money but so little of its culture. At 36 Lispenard Street, in the Tribeca neighborhood, stands the former house of David Ruggles, an African-American abolitionist described by Foner inGateway to Freedom as the “leader of a network with connections to anti-slavery activists in Baltimore, Philadelphia, New England and upstate New York.” One of the nation’s first black journalists (he published a magazine called the Mirror of Liberty) and founder of the anti-slavery New York Committee of Vigilance, Ruggles “scoured the wharfs, on the lookout for fugitive slaves.” Today, there is a plaque affixed to the side of the building where he lived and, in 1838, welcomed Frederick Douglass, who was then a runaway slave.

Ruggles’s building remains intact. The bottom floor is a La Colombe Torrefaction coffee shop, its fashionable patrons buzzing about in the glorious refulgence of a winter afternoon. The excellent urban exploration site Untapped Cities had a barista confirm that the basement of the building was “original.” Unsure of what that meant, I asked to see it. A manager readily complied. She took me through the café’s kitchen, down into the drafty basement. Three stone arches did seem, quite obviously, to have been part of the original structure. Behind one these was a small grotto filled with electrical equipment. Once, maybe slaves had cowered there. There is no way to know.

The basement of the present-day La Colombe Torrefaction at 36 Lispenard St. in Manhattan's Tribeca Neighborhood, pictured April 8, 2015. Jared T. Miller for Newsweek

Back upstairs, the cool coffee kids were busy with their iPads and Moleskines. A barista told me she had been stunned to learn of the building’s history, a revelation she earnestly deemed “epic.” I fantasized for days thereafter about being her history teacher.

At 2 White Street was the home of black abolitionist Theodore S. Wright, born a free black and educated at the Princeton Theological Seminary. He warned against those who condemned slavery but did nothing to improve the lot of African-Americans. “It is an easy thing to ask about the vileness of slavery at the South,” he said in an 1837 speech, “but to call the dark man a brother...to treat the man of color in all circumstances as a man and brother—that is the test.”

A two-story affair of brick and wood just south of Canal Street’s ceaseless bustle, the house reminds of a half-bucolic gentility long gone from Manhattan. It turned out that 2 White was an old friend, though one whom it took a moment to recognize. It had once been home to the Liquor Store, a bar so unassuming it didn’t even assume a name, keeping the marquee of the bygone spirit-purveyor. It was, until its death in 2006, one those dusty, dusky spots that are quickly disappearing from the scrubbed face of the city, replaced by “speakeasies” where mixologists work with the solemnity of biochemists. When my friends and I drank there in the early aughts, the Liquor Store was an unpretentious place for people, like myself, who had great pretentions about life in New York but didn’t have the money to live them out. So we drank Bud Light and dreamt.

Pedestrians walk by a building at 2 White St. in Manhattan, April 9, 2015. Jared T. Miller for Newsweek

Now, the building that once belonged to Wright and later belonged to Jim Beam belongs to J. Crew. Though the interior remains handsome, the feel of invasion hangs in the air like Justin Bieber cologne. “We’ve taken over the old Liquor Store,” a J. Crew website proclaims, noting that the building is an “1825 townhouse” and that the “original wood bar is still intact.” There is no mention of Wright’s work. Today, a suit at J. Crew might run you $400; in 1850, that was the price of a slave.

I left Tribeca and headed for Greenwich Village, where there was once a settlement called Little Africa, a refuge for blacks that had strategic value for whites who did not yet possess the whole island of Manhattan. “The Dutch chose to settle the families of former slaves on this land in order to protect the town from incursions by Native Americans,” notes historian Andrew S. Dolkart. “The Africans would serve as a buffer and would be the first settlers attacked during a raid.” But the black settlement survived, becoming the largest of its kind by the midpoint of the 19th century.

And now it is gone, replaced by bars that cater to students from NYU and “authentic” Italian restaurants that cater to tourists from Palookaville. Minetta Street, whose charming curve is a wormhole of serenity, was once known as the Negroes’ Causeway. Today, it is a shortcut to Macdougal Street, which Bob Dylan once roamed. This is where he wrote “Blowin’ in the Wind,” in a coffee shop called the Commons, which later became The Fat Black Pussycat and is now a Mexican joint called Panchito’s. This is what that song says: “How many years can some people exist / Before they're allowed to be free?”

Centuries, it turns out.

A statue of Horace Greeley sits in Horace Greeley Square in midtown Manhattan on April 8, 2015. Jared T. Miller for Newsweek

After a visit to the Hopper-Gibbons House on 29th Street, I headed for the heart of Midtown. Greeley Square Park is where Broadway meets Sixth Avenue, near where Macy’s proclaims itself the world’s largest store and where a malevolent chaos has always held sway. Horace Greeley edited the New-York Tribune, which Foner calls “the nation’s most important antislavery newspaper.” Its offices had been among the targets of the Draft Riots mob. A statue of Greeley presides over the din, but he looks less like the master of the square than a Rip Van Winkle figure emerging, bewildered, into modernity. He’d have no trouble recognizing our own conflicts, though. Macy’s, which dominates the area, recently paid $650,000 to settle a claim that it discriminated against black shoppers.

Kidnappers and Slave Catchers

James P. Hurley was giving a walking tour of Bedford-Stuyvesant when someone threw a bottle at a group with another guide. Hurley, who is white, wanted to highlight the rich history and architectural splendor of the neighborhood. In 1964, there had been rioting there and in Harlem over a white police officer’s shooting of a black teenager. The neighborhood, once a bastion of black gentility, was beginning its decades-long descent into gangs, drugs, no jobs and bad schools.

After the thrown bottle, Hurley stopped giving tours of Bedford-Stuyvesant and instead began teaching a seminar about Bedford-Stuyvesant history at the Pratt Institute Department of Community Development. In 1968, he was approached by Joseph H. Haynes, who had grown up in Bed-Stuy and now worked as a subway engineer. Haynes wanted to show Hurley the lost neighborhood of Weeksville.

Hurley had been hunting for Weeksville for some time, after first spotting a reference to this community of landowning blacks in a 19th century history of Brooklyn by Eugene L. Armbruster. The community had been founded in 1838 by former slave James Weeks and other African-Americans who bought the rural parcels of land, then sold them to brethren. Foner notes that the neighborhood, far from Manhattan and downtown Brooklyn, “offered a modicum of safety from kidnappers and slave catchers.” As the historian Judith Wellman tells it in Brooklyn’s Promised Land, the most thorough history of Weeksville to date, the little village would grow to become the second biggest community of free blacks in antebellum America (the largest was in Carthagena, Ohio).

Hurley hadn’t been able to learn much about Weeksville. The neighborhood seemed lost, just another victim to New York’s relentless regeneration. The commonly repeated narrative has Hurley and Haynes spotting Weeksville from an airplane the latter was piloting over the central Brooklyn area known as Crow Hill. Hurley, who is now 86 years old and lives near Cooperstown, New York, says that “just isn’t true,” though he remembers a subsequent flight during which Haynes took photos of Weeksville with one hand while piloting the airplane with the other. Yet it was an astounding discovery all the same. In the shadows of the massive Kingsborough Houses public housing project stood a dilapidated cluster of houses, on a diagonal strip called Hunterfly Road that had somehow evaded the grid imposed on the city in the early 19th century and survived more recent attempts at “urban renewal.”

Here was Weeksville.

A photograph taken in the early 1970s shows community archaeological excavation and education workshops being conducted at the Hunterfly Road Site, in Weeksville, Brooklyn. Weeksville Heritage Center

“What we found may not look like much,” Hurley said in 1969 to The New York Times, which reported that archaeological digs of Weeksville were being conducted by “dozens of Boy Scouts, local merchants, parents and school children.” The motley crew managed to not only save the four houses from demolition but get them on the National Register of Historic Places in 1972. The 1980s and ’90s were as unkind to this strip as they were to the rest of New York. There were fires, vandals. Local students donated coins. Somehow the houses survived.

Much like the African Burial Ground, Weeksville feels out of place, intruding on the endless jigsaw puzzle of housing projects that dominates the area. Many of the residents of the adjacent Kingsborough Houses—which only make the news for drugs, guns and gangs—are only three or four generations removed from the tea plantations of South Carolina, the pecan groves of Georgia. Whether these descendants of slaves deserve recompense for the atrocities inflicted on their ancestors is one of the great unresolved questions of American society. That question seems especially pressing here.

In late 2013, the Weeksville Heritage Center opened. It is a handsome, modern building, a gleaming new thing in a neighborhood where most things are old and broken. But it is not without its troubles: Weeksville has struggled to raise money and make itself a draw to visitors. The center’s sunny new director, Tia Powell Harris, says that while the center attracts tourists from places like Sweden and China, it has struggled to get local African-Americans through its doors. They do not feel, she says, like it is theirs.

A tintype portrait, taken circa 1875, of an unidentified female resident of the historically African American neighborhood of Weeksville in Brooklyn, N.Y. The Granger Collection

This is especially lamentable because Weeksville is a lesson in self-empowerment, a deviation from the usual victimhood narrative of slavery. Kanakamedala, the Bronx Community College professor, says places like Weeksville remind that the story of African Americans in the 19th century is about more than just bondage. Unwilling to subsist on “lofty ideas of freedom” alone, says Kanakamedala, the blacks of Weeksville sought home ownership and voting rights as means of personal advancement that, at the same time, incrementally improved the social standing of the race. Their struggle may have been quotidian, but that does not make it inconsequential.

But while Weeksville yearns for the cultural limelight, at least it isn’t facing oblivion. The same isn’t true on Abolitionist Place in downtown Brooklyn, where a sad scattering of 19th century houses may be one of the city’s most trenchant links to the abolitionist movement. It is right off the Fulton Mall, the last commercial strip in Brownstone Brooklyn (i.e., white Brooklyn) that caters to blacks. But gentrification is coming to this cheerfully scruffy strip. Glass boxes are rising everywhere, and high-end chains like Shake Shack are eagerly following in their wake.

Abolitionist Place is already home to two hotel towers, with a third in the making. The trio is so hideous, it would give Houston nightmares. In the shadows of these immodest abominations are two humble buildings: 233 Duffield Street, a three-story affair of caramel-colored wood slats, and 227 Duffield Street, whose three stories are a confusion of styles and colors. A window spans nearly the entire length of 227’s ground floor, and there are faded posters facing out to the street. This was once a hair salon. Now, it is a museum of the most rudimentary kind (no admission charge, but also no actual admission).

The posters tell an incredible story, one that the two houses’ erstwhile owners fought, without success, to have the city acknowledge: that the Duffield houses are the last evidence of a system of tunnels and caverns that hid runaway slaves and helped whisk them to safety. “The floor is still earthen,” a reporter for The Brooklyn Rail wrote during a 2007 visit to 227 Duffield Street. “The two side walls are lined with cool gray flagstones. The uniformity of the stones’ pattern is interrupted about three yards from the front of the building. There...a worn wooden beam...separates the wall from an arch that is bordered with stone. Beyond the arch, a new pattern of red brick covers the rest of the wall.”

Lewis Greenstein points to buildings on Duffield St. in Brooklyn, New York, which once contained a system of tunnels that hid escaped slaves, on April 9, 2015. Jared T. Miller for Newsweek

Elsewhere, these may just have been architectural curios. But the house at 227 was occupied by the abolitionists Thomas and Harriet Lee Truesdell. The owners of the two buildings believed the tunnels may have served as a secret passage from Duffield Street to the nearby Bridge Street African Wesleyan Methodist Episcopal Church. Yes, this is a fanciful notion. But the past often seems preposterous, and all history, whether conducted by professionals or amateurs, is an act of imagination. The fewer the facts, the more the imagination fills in.

But the New York of Mayor Michael Bloomberg was not a New York to go weak in the knees for historical fantasies. The city hired an archeological firm, AKRF, to investigate; it found, according to a New York Times report, “no conclusive proof” that the Duffield houses had been part of the Underground Railroad. So that was that.

The owner of 227 Duffield, Joy Chatel, passed away in the winter of 2014, and the fate of her building remains unclear. The other building has been sold, and will soon come down. “We were surrounded by all these hotels,” explains former owner Lewis Greenstein. “We had to get out.”

Officials, including New York City Mayor Michael Bloomberg (R red tie), stand behind four coffins containing the remains of free and enslaved African-Americans, October 3, 2003, after they arrived at Pier 11 on Wall Street, a colonial-era disembarkation point for slave ships in New York. Stan Honda/AFP/Getty

Foner told me that as he has gone around the country, lecturing about Gateway to Freedom, audiences are invariably surprised to learn that “New York was very closely tied into the South,” that Southerners would vacation in New York City with their slaves, that before Brooks Brothers became a symbol of patrician propriety, it provided clothing for slaves. New Yorkers, too, have been hesitant to learn of their city’s past. “We are tolerant and we are multicultural,” says Foner, a lifelong New Yorker. “The Statue of Liberty is our image of ourselves.” He notes, also, how strange it is that we have a national museum commemorating the Holocaust, but not one commemorating slavery.

I left shabby Duffield Street and walked east, into graceful Brooklyn Heights. There, at 86 Pierrepont Street, stood a house that once belong to Lewis Tappan, one of the city’s most ardent abolitionists and a central figure in the case of La Amistad, a Spanish slave ship on which in 1839 the captive Africans staged a successful mutiny (in the Steven Spielberg film Amistad, Tappan is played by Stellan Skarsgård).

I saw no sign of Tappan having lived here; if there is a plaque, it must be very well hidden. An online listing for a rental apartment in the building put the rent at $2,375, lauding the unit’s dishwasher and “excellent light.” You could live there very happily without knowing anything of the building’s past. And many surely have. Would their fleeting months or years at 86 Pierrepont—romances, breakups, dinner parties, lazy Sundays—have been somehow enriched by the presence of a plaque?

Pedestrians walk by a statue of Horace Greeley in Horace Greeley Square in midtown Manhattan on April 8, 2015. Jared T. Miller for Newsweek

If remembrance is to be more than just a history-class platitude, then it requires sacrifice, of space both physical and mental. The two cataracts on the footprint of the World Trade Center demand attention by dint of their enormous size alone: Millions of feet of commercial real estate were ceded to the National September 11 Memorial and Museum. But that usurpation, by itself, is no moral victory. The crushing sound of water, the square abyss, the names etched into granite, these all summon that morning and the profound associations—personal, political, whatever—that 9/11 continues to have. The city gave up this physical space so that you, in turn, give up some of your mental space to think about the 3,000 lives lost on that day. The outrage over people taking selfies there is rooted in the recognition that this contract has been flagrantly, mirthfully breached.

And what of the thousands of lives lost by black New Yorkers throughout the centuries of American slavery? The city has given up very little space for them, though they, too, suffered and died at the mercy of forces that move through flesh and bone like a shock wave, invisible and inexorable. Like the victims of 9/11, the slaves of New York were actors in the great mercantile drama that daily animates the city. But they were unwilling actors. And when they died, nobody plastered posters with their faces in subway stations. Nobody read their names at memorials, and nobody etched their names in stone.

They went into oblivion. But they don’t have to stay there.

Correction: An earlier version of this article misstated who had a bottle thrown at them. It was a guide with another group.