When James Vann was rising to become the wealthiest businessman in the Cherokee Nation as well as a chief, he decided to build a two-story brick house which would reflect his status. For its construction, Vann brought in professional architects for its design. In addition to providing an education to local Cherokees, the Moravians contributed to the building.

In July 1803, a man named Vogt, perhaps James Vann’s brother in-law, Charles Vogt, and Dr. Henry Chandlee Forman, arrived to begin construction. Work began in late 1803 and the house was completed early in 1804. Both the exterior walls (which are around eighteen inches thick) and the interior walls (which are around eight inches (203 mm) thick) are solid brick. These bricks came from the red clay located on the Spring Place Plantation (Vann House) property. Handwrought nails and hinges came from Vann's own blacksmith shop. Only the interior walls of the third floor are plaster on wood.

The house is a combination of the late Federal style architecture and early Georgian style. Both Georgian and Federal styled homes have two full stories with a half third story. The house has this type of design: the ceilings of both the first and second floor stand at twelve feet, while the ceiling of the third floor stands at only six feet.

The first and second floors have the standard three rooms. On both levels there is a room to the east, a room to the west, and a hallway dividing the two. On the first level, the room to the east is the Vann dining room, while the room to the west is the drawing room, more commonly referred to as a family or living room. On the second floor, the room to the east is the master bedroom and the room to the west is the guest bedroom. Only the third floor, which operated as storage space during James’s life and then as children's rooms during Joseph’s life, strays from this common design.

The third floor is divided into two rooms. The room that the stairway leads into on the third floor is believed to have served as the boys' room. This room is two-thirds the width of the home and has two closets cut into its walls. The second room of the third floor is that of the girls. This room is only one-third the width of the home; however, this room could be shut off from the boy’s room, giving the girls more privacy.

The Vann House also features a basement, which includes two separate rooms. One of which was utilized as a wine cellar. The other is assumed to have been a chamber for misbehaving slaves, to whom James Vann was known to be exceptionally cruel.

The interior of the home is decorated with beautiful colors. The four colors present in the home are red, blue, green, and yellow. White is used throughout the home but only as a filler color. There are two possible reasons for these four colors in the home. The first possibility is that these four colors represent different elements of nature. Red represents the Georgia red clay, blue represents the sky, green represents the trees and grass, and yellow represents the wheat and corn of the harvest. The second possibility is that these four colors are part of Federal style colors.

The red, blue, and yellow seen in the Vann House were often used in other homes of the late seventeen hundreds and the early eighteen hundreds. The only difference between how these colors were used in this home versus how they are used in other homes of the time is the way in which they are distributed. Most homes of the Federal period would concentrate colors in one room, giving a house a red room, blue room, etc. However, in the Vann House the colors have been mixed in almost every room giving the rooms a multi-color appearance, as well as the mantels, doorjambs, and wainscotings, all of which are original to the house. The doors, known as Christian doors, are of special interest. Their construction features a cross and an open Bible.

In addition to the blacksmith shop, the 800-acre (3.2 km2) property around the Vann House included 42 slave cabins, 6 barns, 5 smokehouses, a trading post, more than 1,000 peach trees, 147 apple trees, and a still.

After constructing The Vann House, James lived at the house for 5 years before he was killed at Buffington’s Tavern in 1809. After his death, his favorite child, Rich Joe Vann, which was neither his youngest or eldest child, inherited the house.

After Rich Joe's father died, he made improvement and changes to the new house. After Rich Joe took control of the home he commissioned and paid for decorating the house with hand carvings that adorned the house along with the original colors between 1809 and 1818. Rich Joe hired a father and son construction crew for the work. In 1818, John McCartney and his son James arrived at The Vann House and began their work. While there, the McCartneys added all of the current woodwork in the house including ionic columns. The McCartneys also built the house’s most unusual piece of architecture, a floating staircase in the hallway of the third floor. It received the nickname “floating,” also called hanging, because the second landing of the staircase sits over the first floor hall with no visible supports. This lack of visible structure gives a viewer the illusion that the landing is hanging or floating in midair.

The Vann stairway is called a cantilevered staircase, one of oldest examples of cantilevered construction in Georgia. That was one side of the main entrance, which originally faced the Federal Road and works like a set of scales. To get a set of scales to balance themselves an equal weight must be applied to each side. The staircase of the Vann House works in a similar manner. Though half of the staircase is suspended over the first floor hallway, roughly six inches of the opposite side of the stairway is in a solid brick wall. The brick wall is far denser than the second landing; this means there will never be enough weight on the landing to “tip the scale.”

In 1819, President James Monroe and his three men were on a trip from Augusta to Nashville, they were going to spend the night in the Spartan Moravian mission at Spring Place but President Monroe went to a near location - The Vann House - about a mile away. Rich Joe was 20 years old when he met President Monroe. He found the "Vann House" more comfortable than the mission so he asked Rich Joe permission to spend the night in his house. President Monroe admired the Vann House.[citation needed]

After the Georgia Gold Rush Rich Joe hired a white man, a Mr. Howel,[2] to run Vann House. Although the man never actually worked for Vann, the Cherokee had unknowingly violated a new Georgia law forbidding whites from working for Cherokees without a permit. Leading up to the Cherokee Trail of Tears, Rich Joe, and his family were caught in the midst of the struggle between two opposing claims for the house. Colonel William Bishop and the infamous Georgia Guard tried to take over the house on the grounds of hiring a white man without a permit.. A man, Spencer Riley, who claimed to have won the house in the Land Lottery of 1832, known as the Sixth Georgia Land Lottery, made a claim for the house at the same time. Rich Joe was then evicted by Colonel Bishop.

Colonel Bishop used the house as his local headquarters and permitted his brother, Absalom Bishop, to live there. Riley at the point took action on his claim and settled in the house. In order to get rid of Riley, Bishop took a smoldering log and threw it on the cantilevered steps to smoke him out, causing some damage to the house. This had its intended effect, and Bishop's brother returned to the house. Although Vann and his family lost their home and property, he later sued for the loss and was awarded $19,605 by the government as compensation (It is worth noting that the house alone was valued at $10,000, so the compensation was far from the actual property value.[2]). In November of that year Colonel Bishop imprisoned John Howard Payne for 13 days on the grounds of the house. Payne, noted as composer of "Home, Sweet Home", had been charged with sedition for supporting the claims of the Cherokee over the state of Georgia.

Rich Joe and his family were finally forced out of the house in March, 1835 and moved to Webbers Falls, Oklahoma by following the Trail of Tears. They never returned to Georgia or their house. Over the years, the Vann House has had seventeen different owners. In 1952, J. E. Bradford, a physician who had purchased it in 1920, sold the house to the Georgia Historical Commission and The State of Georgia. At the time of its purchase by the commission, the house was in such a state of disrepair that the roof had come off and the elements were taking their toll.

One of the owners had added an additional room after Rich Joe left Georgia. A restoration project began in 1958, which took six years to complete and included demolishing the additional room that was not present in the original house and repainting the house according to its original color scheme of blue, red, green, and yellow. Today it is administered by Georgia's Parks, Recreation and Historic Sites division of the Department of Natural Resources.

The State of Georgia, Cherokees, and The State of Oklahoma as well as other supporters donated to build a newly designed museum called the "Robert E. Chambers Interpretive Center" in 1999, next to Vann House. It was opened on July 27, 2002, to honor the Cherokee people and their history. The new center also highlights the lives of Chiefs James and Joseph Vann while featuring the history of the Cherokee Nation over the past 200 years, including the infamous Trail of Tears. Robert E. Chambers was named for his supporting the Cherokee, as he was Chatsworth native businessman.