Former secretary of state Colin Powell joked Wednesday that he remains in the Republican Party so he can continue to “annoy” ultraconservatives.

“I think the party has shifted much further right than where the country is and it should be obvious to party leaders that they cannot keep saying the things that they are saying and doing the things that they were doing and hope to be successful in national level elections in the future, not just 2016,” the retired four-star Army general said at the Washington Ideas Forum in downtown D.C., the Daily Mail reported.

“So I want to continue to be a Republican because it annoys them,” he said, smiling.

Aspen Institute head Walter Isaacson mentioned Mr. Powell’s outspoken support for President Obama, as well as his support for the Iran deal and views on immigration reform that are more commonly held by Democrats.

“You didn’t mention the fact that I worked for about five presidents in a row who were Republicans,” said Mr. Powell, who most recently served as President George W. Bush’s secretary of state.

“Yes, I’m still a Republican,” he added, CNN reported. “In Virginia, you don’t have to declare a party … but I’m still a Republican because I believe in a strong defense, because I believe in the entrepreneurial spirit that is so typical of the Republican Party in the past. But I’m having difficulty with the party now.”

Turning to immigration, he said,” I think most Republicans understand that we need immigration, we are an immigrant nation, it is in our best interest to do it … but there are pockets, as I said previously, there are pockets of intolerance within the Republican party [and] the Republican party had better figure out how to defeat that.”