Pell is married to the former Nuala O'Donnell, a descendant of the Hartford family and, as such, one of the heirs to the Great Atlantic and Pacific Tea Company fortune. Pell himself is one of the heirs to what started out as the Lorillard tobacco fortune, although the family has been out of the Lorillard firm for generations. Their primary residence is the house known as "Pelican Ledge" overlooking Newport, Rhode Island's Bailey's Beach.

In 1993, during the bitter confirmation battle over Roberta Achtenberg, a lesbian, to the Department of Housing and Urban Development (HUD), Pell stated that his daughter was a lesbian, and that he hoped that it would not be a barrier to federal employment for her. Many agree that Pell, arguably one of the most popular U.S. Senators of the later years of the 20th century, helped secure Achtenberg's nomination, making her the first openly gay person to be confirmed by the United States Senate. Pell now suffers from advanced Parkinson's Disease.