The Cato Institute’s Roger Pilon bizarrely complains that my Bloomberg View column this week is “selective” in its review of the Supreme Court’s history — as though it were possible for it to be anything else. He then presents his own version of history. I don’t buy that version, but even if it were right it would not undermine my thesis as Pilon thinks it does.

My argument was that the Court’s role in expanding freedom and equality in the U.S. has been vastly exaggerated. As part of that argument I pointed out that the Court rarely struck down laws for the first seven decades of its existence. Pilon counters that legislators were respectful of their enumerated powers during that period and so the Court did not have much to do. His implicit claim, that its record of passivity resulted entirely from its circumstances and not at all from a more modest self-conception, seems to me highly dubious. But even if he’s right, this portion of U.S. history backs up my basic claim: The expansion of freedom and equality during this period was not primarily (or even secondarily or tertiarily) the work of federal courts.

Pilon badly misunderstands the point of my reference to Marburyv. Madison. I didn’t dispute that the case served the cause of freedom; I just pointed out that the principle for which it stood was a kind of judicial self-restraint. It’s an arguable point; I’m in the minority in thinking about it this way. But I didn’t say what Pilon thinks I did.

He then returns to his point about selectivity, dinging me for not mentioning Lovingv.Virginia, Griswoldv.Connecticut, or Lawrencev.Texas. It’s true I didn’t mention all the civil-rights cases of the ’50s and ’60s, using Brown as a stand-in instead. My point about Brown applies to all of them: These decisions might not have been necessary if the Court had not blocked Congress from taking steps to desegregate the South in the 1880s. Pilon adds, irrelevantly, that “both the political and the nonpolitical branches have played their parts in protecti[ng] freedom. That’s part of the genius of our system.” Speaking of selectivity, wouldn’t this have been a good moment for Pilon to mention that he thinks the Civil Rights Act of 1964 is unconstitutional?

And whether or not Griswold and Lawrence were rightly decided, it wasn’t the Supreme Court that made this country largely free of prosecutions for buying contraception or engaging in sodomy, since that condition of freedom preceded the relevant cases.

In my column I suggested that our history does not give us reason to refrain from considering proposals to reduce judicial power. Pilon concludes by claiming that I am somehow contradicting a passing remark by James Madison about how the courts would help enforce the Bill of Rights — as though this remark established that Madison were the libertarian activist that Pilon is. No sale.