(CNN) -- J.D. Salinger, the famously reclusive author whose 1951 novel, "The Catcher in the Rye," became a touchstone for generations of readers, has died. He was 91.

The author died Wednesday of natural causes at his home in New Hampshire, according to a family statement that his literary agent, Phyllis Westberg, provided Thursday.

"Despite having broken his hip in May, his health had been excellent until a rather sudden decline after the new year," the statement said. "He was not in any pain before or at the time of his death."

Salinger has long been known for his reclusiveness, and "in keeping with his life long, uncompromising desire to protect and defend his privacy there will be no service," the statement said.

"The family asks that people's respect for him, his work, and his privacy be extended to them, individually and collectively, during this time."

Though he wrote more than 30 short stories and a handful of novellas -- many published in The New Yorker and collected in works such as "Nine Stories" and "Seymour: An Introduction" -- Salinger's fame rests on "Catcher," his only novel.

The book is narrated by a teenage boy, Holden Caulfield, who is expelled from a private school, Pencey Prep, in Pennsylvania, and spends the next three days wandering around New York. Caulfield is mistrustful of authority, railing against corrupt adults and "phonies," and plans to decamp for the west.

"An unusually brilliant first novel," The New York Times called it, one of many raves.

Though intended for adult readers, the book touched a nerve among teenagers attracted by Caulfield's nonconformist attitudes and has remained a best-seller since its release. It has gone through dozens of printings in its simple mass-market paperback edition, clad in a simple maroon cover adorned with the title and author's name on the front and back.

It's also been a lightning rod for controversy. Schools have banned it -- troubled by Caulfield's language and attitude as well as his adventures with a prostitute -- and some readers have been obsessed by it, most infamously Mark David Chapman, who murdered John Lennon in 1980.

In 2009, Salinger -- always protective of his work -- sued to stop the publication, sale and advertisement of "60 Years Later: Coming Through the Rye," a novel written by an author calling himself J.D. California.

Jerome David Salinger was born January 1, 1919, in New York City, the son of Sol -- a wealthy meat importer -- and Miriam Salinger. He attended Valley Forge Military Academy in Pennsylvania and spent time at three colleges. He published his first stories in the early 1940s.

In 1942, he joined the U.S. Army. He fought in the D-Day invasion at Normandy as well as the Battle of the Bulge, but suffered a nervous breakdown and checked himself into an Army hospital in Germany in 1945. In December of that year, "I'm Crazy," the first story featuring Caulfield, was published in Collier's.

In 1947, his short story "A Perfect Day for Bananafish," featuring the troubled Seymour Glass, was published in The New Yorker. The Glass family, a group of bright siblings who agonized over their lives, would become the subject of many subsequent stories, all of which were published in The New Yorker and later collected in "Nine Stories," "Franny and Zooey" and "Raise High the Roof Beam, Carpenters and Seymour: An Introduction."

His last published story, "Hapworth 16, 1924," appeared in 1965.

In his later life, Salinger became as well-known for his desire for privacy as his work, and when that privacy was shattered, he would sometimes lash out with lawsuits. A 1987 biography by Ian Hamilton was blocked by a U.S. appellate court; it eventually appeared, in heavily revised form, as "In Search of J.D. Salinger." Salinger also sued a writer, Stephen Kunes, for impersonating him.

He had a nine-month affair with the 18-year-old Joyce Maynard in the early '70s; she later wrote about it in a memoir and auctioned off his letters. (The buyer, Peter Norton, said he would return the letters to Salinger.)

Salinger's daughter, Margaret, wrote a memoir, "Dream Catcher," which was published in 2000. Neither Maynard's nor Margaret Salinger's books were positive, describing the author as controlling and having a host of unusual habits, including drinking his own urine.

Throughout it all, Salinger remained determinedly reserved. He gave interviews sparingly, the last one in 1981 to The (Baton Rouge, Louisiana) Advocate. Occasionally he would release terse statements about his work.

"Salinger had remarked that he was in this world but not of it. His body is gone but the family hopes that he is still with those he loves, whether they are religious or historical figures, personal friends or fictional characters," the statement said.

"He will be missed by the few he was close to every bit as much as by the readers who loved reading him."

Salinger was married three times, most recently to Colleen O'Neil, who survives him. He had two children, Matthew and Margaret, with his second wife, Claire Douglas, whom he divorced in 1967.