Born Elinor Isabel Judefind in Baltimore, Maryland,[1] she was daughter of William Lee Judefind, a chemist, and his wife, the former Ruth Elinor Schafer. Her paternal grandfather was a Methodist minister.

She married Agnew on May 27, 1942 in Baltimore; he had graduated from Army Officer Candidate School two days earlier. They had four children: Pamela Lee Agnew (Mrs. Robert E. DeHaven), James Rand Agnew, Susan Scott Agnew (Mrs. Colin Neilson Macindoe), and Elinor Kimberly Agnew.[2]