Murdoch Glourie (Robert Donat), living in 18th century Scotland is more interested in the local lasses than in fighting the British on the fields of battle, much to his father's dismay. When he is accidentally killed, hiding from his clan's hated rivals, the McLaggen family, he meets his father in Purgatory, who condemns him to remain on Earth as a ghost, haunting the family castle until he obtains an apology from a member of the McLaggen clan. There his ghost remains over the centuries and the castle acquires a reputation for being haunted.

In the 20th century, the castle is occupied by Murdoch's descendant, Donald Glourie (also Robert Donat), who has fallen into heavy debt and is unable to find a buyer for the castle because of its ghost. A young American woman, Peggy Martin (Jean Parker), arrives and instantly falls in love with the ancient castle and, that evening, she brings her wealthy parents (Eugene Pallette and Everley Gregg) to dine with Donald. The dinner and bagpipe entertainment is provided by all of Donald's creditors, who are eager to help insure the sale of the property. Peggy stays in the castle that night to get a better feel for it and, despite Donald's efforts to prevent it, she is visited by Murdoch's ghost, but she only believes it is Donald.

The following day, Donald accepts Mr. Martin's offer to buy the castle, but then is shocked to learn that he plans to move the entire structure to America , "stone by stone, panel by panel." Martin offers Donald the opportunity to come to America and supervise the reconstruction. He accepts, because he and Peggy are falling in love with each other.

On the ship, Murdoch's ghost finds itself in the cargo section among the dissembled castle pieces, and angrily reveals himself to the ship's passengers. Donald returns the money to Mr. Martin when his wife refuses to have anything to do with a castle that includes a ghost. However, a business rival of Martin's offers Donald $100,000 for it, seeing a big marketing and promotion opportunity. Martin immediately counters with a $125,000 bid, to use the castle for his own business promotional purposes.

The news of the arriving ghost is a major media event, and the Martins are greeted with a huge ticker tape parade in New York. In their American castle, they host a highly publicized society gala, promising that the ghost will make a midnight appearance. However, Murdoch having witnessed a gunfight between gangsters at the dock, has gone invisible out of fear of American life.

Mr. Martin's jealous rival, Ed Bigelow (Ralph Bunker) mocks him when the ghost fails to appear. Seeing Martin's faux tartan attire, he announces that he is a true Scotsman, the last remaining kin of the McLaggen clan. Murdoch's ghost makes a dramatic appearance and extracts an apology from the frightened Bigelow. His Earthly mission completed, he is allowed by his father to ascend to Paradise, and Donald begins his new life in America with Peggy.