Walker was one of Trump's first big celebrity endorsers -- praising his former "Celebrity Apprentice" boss as a brilliant man who will truly make America great again.

But after he went public, Walker says he started getting axed from speaking gigs -- most recently getting canceled from an event in California because organizers were afraid Walker would attract anti-Trump protesters.

Walker wouldn't name the groups who 86'd him -- but says it's happened at least 3 times in various states.

In fact, he spoke about the situation this week on the "TMZ Sports" TV show on FS1.

Despite the backlash, Walker says he will NOT stop supporting Trump -- insisting he's the best man for the job.