Supreme Court Associate Justice Anthony Kennedy testifies before a House Committee on Appropriations Subcommittee on Financial Services hearing to review the FY 2016 budget request of the Supreme Court of the United States, on Capitol Hill in Washington, Monday, March 23, 2015. (AP Photo/Manuel Balce Ceneta)

For supporters of Hillary Clinton, it was the worst moment since the night they sat under a glass ceiling at the Javits Center in New York, waiting for the victory speech that never came.

Supreme Court Justice Anthony Kennedy, so often the “swing vote” between liberal and conservative viewpoints, announced his retirement.

This was the event that Clinton supporters had dreaded and battled to avoid, writing checks and warning each other that if a Republican won the White House, his appointees to the Supreme Court would roll back everybody’s rights.

It’s not a new charge. Back in 2000, somebody asked Barbra Streisand to name three reasons to vote for Al Gore. “The Supreme Court, the Supreme Court, the Supreme Court,” she answered.

You might be surprised to hear that America’s founders thought of the Supreme Court as the weakest division of the government. “The least dangerous branch,” Alexander Hamilton called it.

You might also be surprised to hear that the framers of the Constitution specifically excluded the judiciary from policy-making. During the Constitutional Convention, there were two attempts to create something called a “Council of Revision,” a panel of judges that would share the power to veto legislation. The Council of Revision was voted down twice, and the veto power was given to the president alone.

For over 150 years there was nothing particularly controversial about the Senate confirmation process for Supreme Court justices. It wasn’t a political job, and vacancies didn’t rile up political factions.

So what happened? How did we get to this place, where a Supreme Court justice can’t retire without half the country coming unglued about it?

To answer that question, you’d have to go back to 1866 and watch the post-Civil War Congress debating the Civil Rights Act to protect the fundamental rights of the freed slaves. At the last minute, lawmakers didn’t have the votes to pass the bill because of this sentence in the first section: “there shall be no discrimination in civil rights or immunities among the inhabitants of any State or Territory of the United States on account of race, color, or previous condition of slavery.”

So Congress cut that sentence out of the bill before the final vote. Simultaneously, a congressional subcommittee was cutting this sentence out of a parallel constitutional amendment, which became the Fourteenth: “All laws, state or national, shall operate impartially and equally on all persons without regard to color or race.”

It’s an unpleasant but undeniable fact of history that the 39th Congress in 1866 didn’t ban racial discrimination. They intentionally preserved segregation. Even the Senate gallery was segregated at the time. And of course those lawmakers weren’t thinking about women’s rights at all.

The intention of that Congress to preserve segregation was perpetuated all the way up until 1954, when the U.S. Supreme Court decided Brown v. Board of Education and ruled racial segregation unconstitutional. The court’s method of reasoning was unusual then, but it isn’t unusual any more. “Because it’s time, that’s why,” has been the theme of more than one landmark ruling since the Brown decision.

But unless the new and improved rights are written into constitutional amendments, what the court giveth, the court can taketh away. That’s what’s made judicial appointments so politicized. Federal judges are effectively making policy, something the Constitution excluded them from doing.

The Constitution has never been amended to ban discrimination on the basis of race or gender.