The 1833 mansion, now called the Forbes House Museum, was designed by Boston architect Isaiah Rogers. The Greek Revival design incorporated architectural elements reminiscent of classical temples; accordingly, the Museum's façade boasts two prominent Ionic columns. The layout of the house is symmetrical in design, also typical of the Greek Revival style, with rooms leading from a central front hallway. In 1871, the Captain's son, James Murray Forbes (1845–1937), came to live at Forbes House with his new wife, Alice Frances Bowditch Forbes (1848–1929), and made considerable renovations to the home.

An addition designed by Peabody & Sterns was built on the west side of the structure, and the south porch and carriage house were expanded. The third floor was raised to make room for servants' living quarters, and the plumbing was upgraded from the original 1833 system (Yes, Forbes House was one of the first private homes whose design incorporated indoor plumbing, thanks to Rogers). On the second floor, closets were built in the bedrooms for James and Alice Frances' three children, Mary, Dorothy, and Allan. By this time, the family used Forbes House as their primary home, while also maintaining a residence on Commonwealth Avenue in Boston.

The mansion was originally built for Margaret Perkins Forbes (1773–1856) in 1833 and was used at first in spring and fall only (the family wintered in Boston and summered in Maine). By that time, Margaret was a widow, having lost her husband, Ralph Bennet Forbes (b. 1773), at age 40 in 1824. In 1829, another tragedy occurred: Margaret lost her eldest surviving son, Thomas Tunno Forbes (b. 1802), to a typhoon in Canton. The funds from Thomas' estate (he was the first of his brothers to enter the China Trade), along with monies from his two surviving brothers, Robert Bennet and John Murray, were combined to pay for the construction of the house. Margaret Perkins shared the residence with her four daughters, Emma Perkins, Margaret Perkins, Mary Abbot, and Cornelia Frances. With the exception of Mary Abbot, none were married, and all remained at the mansion for their lifetimes.

The Captain, born in Jamaica Plain, Massachusetts, in 1804, was introduced into the China Trade at age 12 by his uncle Thomas Handasyd Perkins (1764–1854), who, along with his brothers James (1761–1822) and Samuel (1767–1847), established the Boston-based Perkins & Company in 1803, later becoming the Russell & Company.

Ben, as the Captain was affectionately known, began his career as a cabin boy in 1817 on the Canton Packet, and was made a captain by age 20. He made several trips to Canton during his lifetime. In addition to being a China Trade merchant, the Captain was a ship designer, model ship builder, writer, and philanthropist of international stature. In 1834, he married Rose Green Smith (1802–1885). They had four children, Robert Bennet, Jr., Edith, James Murray, and Rose.

Mary Bowditch Forbes (1878–1962), granddaughter of the Captain, was the last family member to live permanently at the mansion. She collected President Lincoln and Civil War memorabilia, now part of the Museum's permanent collection. In 1923, Ms. Bowditch Forbes had a replica of Lincoln's birthplace built on the grounds of the estate. In 1962, she bequeathed the property to her nephew, Dr. H. A. Crosby Forbes, curator of Asian export art, who converted the estate into a museum two years later.

Dr. H. A. Crosby Forbes (b. 1925) founded the Museum of the American China Trade and opened its doors in 1964. Forbes attended Harvard University for his undergraduate and graduate studies, and taught for a time at the Massachusetts Institute of Technology. Most importantly, he was inspired by the Forbes family history—with a total of eleven family members engaged in the American China Trade between 1789–1891—to pursue a long and successful career as a curator of Asian export art. He is the author of the book, Chinese Export Silver, which was published by the Museum of the American China Trade in 1975. Dr. Forbes also spent over two decades as curator of the Peabody Essex Museum, in Salem, Massachusetts.

In 1984 the Museum of the American China Trade became the Captain Robert Bennet Forbes House, and is now called the Forbes House Museum.