In the panic of The Great Skedaddle of 1861—the hasty escape of white residents from Beaufort, South Carolina, six months after the start of the Civil War—anything that could not be buried or carried was abandoned. Enslaved people were left behind along with all the wealth accumulated from nearby cotton plantations: expensive furniture, horses, and clothing. When the Union Army arrived, there was only one white man left in town and he was dead drunk, or so the legend goes.

As history would have it, what drove white Beaufortonians from their homes was ultimately what saved the town they left behind. Unlike burned and battle-scarred Charleston, 50 miles to the north, the Union Army claimed Beaufort without a fight. Setting up operations in the town’s stately mansions rescued them from destruction. To this day, Beaufort has more surviving antebellum architecture than almost anywhere else in the South.

Beaufort’s intimate, walkable neighborhoods are delightful year-round, and some historic mansions like the Verdier House Museum are always open to the public. But one day each year, some of Beaufort’s most spectacular private properties welcome visitors during the Fall Festival of Houses and Gardens.

The Tidewater. | Photo: Shoshi Parks

Historic home on the edge of the marsh. | Photo: Shoshi Parks

Beaufort river. | Photo: Frank Lesesne

Three Sisters

Beaufort may not be well-known outside of the South, but locally, this town has a reputation. Beaufort and Charleston together with Savannah, Georgia, are what Peach Morrison, executive director of the Lowcountry Tourism Commission, calls the Three Sisters.

“Charleston is the oldest. She’s very well established and very sure of her charms,” Morrison explains. “Savannah is the youngest. She’s beautiful and loves to party. Beaufort is in the middle. She’s the sister who will put her pearls on, get in the truck, back the boat in the water, and go get shrimp for dinner.”

Download the mobile app to plan on the go.

Share and plan trips with friends while discovering millions of places along your route.

Built on the shore of the bucolic Beaufort River, the town is a striking portrait of the beauty bled from antebellum plantation culture, and it stands in stark contrast to the atrocities committed here against enslaved people. Massive mansions line the streets, their broad front porches perfect for rocking chairs and mint juleps. Ancient oak trees drip with Spanish moss. In the marshlands nestled against the shore, water ebbs and flows with the tides.

The Anchorage 1770. | Photo: Katie Wright

During the Civil War, several of Beaufort’s neoclassical manors were converted into hospitals for soldiers injured in battle. My first night in town, I dine on freshly caught fish on the front porch of one of them, Hospital No. 11. It’s now the Anchorage 1770, a charming boutique inn on the river. “Basically the whole town of Beaufort was triage for the Union army,” says Frank Lesesne, who owns the inn together with his wife Amy. “They were doing surgeries on the marble gravestones and after that they were putting the soldiers back in the mansions to convalesce.”

Altogether, 15 of the town’s mansions were turned into hospitals, their walls marked with the graffiti of recovering soldiers dreaming of home.

After dinner, Amy shows off the inn’s interior. Built in 1770, the Anchorage claims to be the country’s largest standing structure built with tabby, a long-gone building material made out of oyster shells, sand, and lime. After the Reconstruction Era, the mansion lived its next life booze-drenched and joyful. Renovations spearheaded by the new owner, a Navy rear admiral, included hidden doors and cubbies throughout the house to conceal his favorite bourbon drink, the Cherry Bounce, from his teetotaling wife. When the admiral was deployed, the Anchorage became the headquarters of the Ribault Social Club, which is now the name of the modern restaurant and bar.

Beaufort River at sunset. | Photo: Katie Wight

“It was going to be a literary society, it was going to have a great library, with all the newspapers from around the country,” Frank says. Instead, the club grew to be known for its dancing, parties, and roulette. “It didn’t evolve, it devolved,” he laughs. After wine and a Cherry Bounce of my own, I too am devolving and I bid the Lesesnes good night.

Going to church

The next morning, the South Carolina heat slaps me in the face as I emerge into the sun-drenched courtyard of The Beaufort Inn ready to explore.

My first stop is the First African Baptist Church. Like much of the South, early Beaufort was built on the backs of enslaved people. But when the town’s white residents fled in The Great Skedaddle, as many as 10,000 black men, women, and children found themselves in a state of unofficial freedom. With the help of northern abolitionist groups, black schools and churches sprang up. For the first time in their lives, black people were paid to harvest the valuable Sea Island cotton plantations to which they’d formerly been chained.

Inside, the two-room space is modest and unadorned. The original wooden pews face the pulpit in neat rows. In the back room, the congregation has collected and hung photos and newspaper clippings from their storied past, including nods to one of the church’s earliest members, Robert Smalls.

Former morgue at The Castle. | Photo: Shoshi Park

First African Baptist Church. | Photo: Shoshi Parks

Smalls was born into slavery around the corner at 511 Prince Street, whose manicured gardens are open to the public today, and sent to work at the Charleston Harbor as a teenager. When the Civil War broke out in 1861, Smalls escaped in the middle of the night on the Confederate ship on which he’d been assigned to work. Commandeered by Smalls, the ship headed straight to the Union Navy with a white bed sheet raised in surrender. A hero in the North, Smalls returned to Beaufort after the war and purchased the home he’d grown up in. He went on to be one of the first African American congressmen in the U.S.

Haunted history

Like many historic towns, Beaufort is alleged to be teeming with ghosts. They are why so many porch ceilings are painted light blue, a local Gullah tradition to keep “haints,” malevolent spirits, from entering the home and wreaking havoc. The porch ceiling of The Castle, an imposing goldenrod manor detailed with touches of medieval drama is painted in “haint blue,” too, but it hasn’t done much good. One of the town’s oldest spirits, Gauche, a jester brought to South Carolina by French explorer Jean Ribault in 1562, has resided here for more than 150 years—or so the legend goes.

The Castle was one of the few Beaufort homes repurchased by the original owner at the war’s end thanks to the valuable china buried beneath the wooden floorboards of an outbuilding during The Great Skedaddle. Inside, The Castle, which was designated as Hospital No. 4 during the occupation, is cavernous. I shuffle forward, one of many eager to see the interior of one of Beaufort’s most famous Grand Dames. A dramatic double-wooden staircase descends from the upper floor at the rear entryway. The kitchen is remarkably small for such an enormous home, but most of the cooking was originally done in a separate building. This kitchen and other modern conveniences, such as railings on the staircase, were added later.

The Castle. | Photo: Shoshi Parks

Several houses and several hours later, I pass by The Castle again, headed for The Marshlands, an 1814 manor where an afterparty is just getting started on the lawn overlooking the river. The line to get in still stretches around the block, the row of people appearing tiny in the shadow of the looming home.

I feel like Pat Conroy got it right when he called Beaufort’s mansions “the summer homes of the creatures of a misused tarot deck.” Beaufort is simultaneously enchanting and strange, an anachronistic town plucked from the past and thrown into a modern era. “We live by the tides,” says Morrison. “It’s a very, very, very special place.”