David Lewis (Peter Gallagher) and his daughter, Rachel (Claire Danes), live year round on Nantucket Island where, two years ago, their wife and mother, Gillian (Michelle Pfeiffer) died in a boating accident on her birthday. David continues to grieve over his loss and spends much of his time, walking on the beach engaged in conversation with Gillian's "ghost."

They are visited for the weekend by Gillian's sister, Esther Wheeler (Kathy Baker) and her husband Paul (Bruce Altman). They also bring along an attractive single woman, whose first name is Kevin (Wendy Crewson), hoping to help David stop obsessing about the past. However, he makes it clear that he is not ready for new relationships.

The Labor Day weekend activities include karaoke singing and a sand castle competition on the beach. Joey (Freddie Prinze Jr.), a local boy with ear and nose rings, takes Rachel to an island keg party, and she returns home, tipsy from her first experience with alcohol, to David's consternation.

His sister-in-law, Esther, is troubled, both by the stagnant state of her own marriage, and by David's inability to respond to Rachel's needs as a young teenager, because of his continued devotion to his dead wife. She tells Paul that she has made a court date for the following Monday to ask for custody of Rachel, at their home on the mainland. Her resolve is hardened when Rachel tells her about David's beach strolls and conversations with Gillian, which he doesn't deny.

Near the end of the weekend, Rachel tearfully announces her decision to go and live with her aunt and uncle. David is devastated, but begins to understand that it is a situation of his own making. As the departing group arrives at the ferry boat and David and Rachel prepare to say goodbye, he announces that he will move back to the mainland to his old job teaching college. He returns to the beach house to close it up for the season, and to have one final conversation with Gillian. He tells her that he has to let her go, and she reluctantly agrees. Before leaving the island, he puts flowers on Gillian's grave site.