While in Italy, Ker saw a first edition of Boccaccio's Decameron, often called the Valdarfers edition. This was a fabled book, which many said did not exist. He paid 100 guineas for it and showed it to his friends in London to huge acclaim. For the next 40 years he collected ancient and curious books, particularly editions of Shakespeare's works and other works which merely mentioned Shakespeare. At his death in 1804 there were 10,000 items. Most were books, but there were also pamphlets and broadside ballad sheets. His library was auctioned in 1812, leading to the formation of the Roxburghe Club. His collection of ballads were later published as the Roxburghe Ballads.

In Jonathan Strange and Mr. Norrell, a 2004 novel by British writer Susanna Clarke, the Duke's affection for the Queen's sister and their subsequent separation are mentioned, and made a premise for an important aspect of the story. One of the title characters, Gilbert Norrell, had a longstanding wish to examine the books of the Duke's library, believing there to be magical texts within. The Duke, being a bibliophile and rich, saw no reason to allow Mr. Norrell the opportunity, and so upon the Duke's death, the new Duke puts the library up for sale in order to pay off court debts. The Duke's library contained several extremely rare and valuable tomes, which Mr. Norrell purchases at auction, causing an increase in friction between himself and Jonathan Strange.