SAN DIEGO (AP)  U.S. Supreme Court Justice Anthony Kennedy on Wednesday likened controversy over the court”s decision to allow gay marriage to public reaction over the 1989 ruling that said burning an American flag was protected free speech.

Kennedy, who was the deciding vote in both cases, described how the reaction decades ago was critical at first but changed over time.

His remarks at the 9th Circuit Judicial Conference were his first public comments since he wrote the decision last month that put an end to same-sex marriage bans in 14 states. Kennedy drew the comparison in response to a moderator”s question about how justices weather reaction to closely watched rulings.

“Eighty senators went to the floor of the Senate to denounce the court,” he said of the 1989 ruling. “President Bush took the week off and visited flag factories, but I noticed that after two or three months people began thinking about the issues.”

Kennedy went on to say that a lawyer from Northern California approached him at a restaurant after the flag burning decision to tell him how his father, a prisoner of war in Nazi Germany, came around to the decision.

The former prisoner of war  who secretly sewed red, white and blue cloth together in captivity  stormed into his son”s office and said he should be ashamed to be an attorney, Kennedy said. The lawyer, unsure how to respond, gave his father Kennedy”s concurring opinion.

“He thought about it, came back three days later and said, ”You can be proud of being an attorney,”” Kennedy said, relaying the story.

Kennedy didn”t elaborate on the same-sex marriage ruling or on other decisions in the last term, but he said the justices must decide cases in a fair and neutral way.

“We have to reflect on what these issues mean, and when we have a controversial case  and a very difficult case like (same-sex marriage)  we draw down on a capital of trust, a deposit of trust,” Kennedy said. “We spend that capital of trust, and we have to rebuild that capital. We have to put new deposits, new substance into this reservoir of trust.”

Dissenters in the same-sex marriage ruling accused their colleagues of usurping power that belongs to states and voters. “This court is not a Legislature,” Chief Justice John Roberts wrote in dissent.

The 78-year-old appointee of President Ronald Reagan has long been a crucial swing vote on the court, where four justices are reliable liberals and four others reliably conservative.

He usually votes with the conservatives, but he has crossed the partisan divided on certain issues.

Kennedy also joined a 6-3 majority to uphold the nationwide tax subsidies underpinning President Barack Obama”s health care overhaul.