The Prohibition era in America, which lasted for well over a decade and—inconceivable as it might be today—effectively banned the sale and production of booze in the United States, ended with the ratification of the 21st Amendment on Dec. 5, 1933.

Photo colorization by Sanna Dullaway for TIME / Original Image by Kermit Roosevelt / Library of Congress

Col. Theodore Roosevelt with a dead elephant at Meru, Kenya.

The Siena Research Institute (SRI) at Siena College often releases complex expert rankings of all U.S. Presidents. In 2010, the SRI Survey Ranked Roosevelt the #2 President of All Time. Roosevelt took top marks in the survey's categories for 'Imagination,' 'Willingness to Take Risks', 'Court Appointments' and 'Intelligence.'

Armchair historians — and actual ones — have always enjoyed ranking American presidents, recasting Mount Rushmore, debating who was greatest of them all. There are the perennial favorites —Washington, Lincoln, FDR — and the eternally scorned: James Buchanan, Millard Fillmore, Franklin Pierce. Then there are those who voters abused but history redeemed: Herbert Hoover was hung in effigy, his motorcades pelted with rotten fruit, as he left office in the midst of the depression in 1933; yet twenty years later, after his epic humanitarian missions leading post-war disaster relief, he ranked among the most admired men in in America. Harry Truman had a 22% approval rating his last year in office, yet is now exalted for his common sense and steady hand during the dangerous birth of the atomic age.

Among those least inclined to judge and rank the presidents are the presidents themselves. They know the deep and often damaging toll the job takes on all those who hold it, successful or not: the toll on health, on family, on any kind of normalcy. A few weeks after his reelection in 2004, I asked George W. Bush whether he thought more or less highly of his predecessors, now that he’d been in the job a while.

“Of my predecessors? Very interesting,” he replied, and then, without hesitation, “More highly of them all.”

Why? Because “I’ve got a much better appreciation of what they’ve been through.”

Bill Clinton says much the same. “There’s just a general sympathy,” Clinton told Michael Duffy and me, when we interviewed him for our book The Presidents Club, and as if to prove it, he launched into an astonishingly detailed exploration of Franklin Pierce’s presidency, the impact of his son’s tragic death, the challenge of dealing with his party’s inherently untenable coalition: “I’ve thought he had a really admirable life and if he’d had a chance to serve at a different time, he might’ve been different. And you know, I’ve always been fascinated by all this,” Clinton told us, and pointed to the long wall of presidential biographies and memoirs and collections of letters in his office. “I read a really interesting biography of Rutherford Hayes when I was president.”

One president who was especially reluctant to judge his brethren was John F. Kennedy. Once, when asked by Arthur Schlesinger Sr. to participate in a poll of historians that would rank past presidents, Kennedy’s reaction was ferocious: he threw the poll across the room.

“No one,” Kennedy told historian David Herbert Donald early in 1962, “has a right to grade a President — not even poor James Buchanan — who has not sat in his chair, examined the mail and information that came across his desk, and learned why he made decisions.”

On Presidents Day—and as we re-imagine, through Sanna Dullaway’s eye-opening colorizations, some of the figures who have occupied that august, lonely office — it is worth remembering what a democracy asks of the men and women who lead it. That does not mean we suspend judgment: voting is the ultimate ranking of presidential performance. But on a day we celebrate the presidency, we are celebrating service and sacrifice as much as success.