(CBS News/AP) — A San Francisco-based personal finance firm is among a number of advertisers who have said they’re leaving conservative host Tucker Carlson’s show following his remarks that immigrants make the U.S. “poorer and dirtier.”

It’s the latest example of sponsors leaving a Fox News Channel show after controversy, but experts believe the flap is likely to blow over. So far, the biggest advertisers are sticking with him and his prime-time show, “Tucker Carlson Tonight.”

Carlson said last Thursday there’s pressure from “our leaders” to accept immigrants “even if it makes our own country poorer and dirtier and more divided.” He added Monday that in the Southwest, “thanks to illegal immigration, huge swaths of the region are covered with garbage and waste that degrade the soil and kill wildlife.”

The comments caused a furor on social media. Several advertisers, including San Francisco-based personal finance website NerdWallet and Pacific Life insurance, have pulled advertising from the show. (NerdWallet is a content partner of The Associated Press.) SmileDirectClub said it’s working with its ad buyers to stop running ads during any political opinion shows.

“As a company, we strongly disagree with Mr. Carlson’s statements,” Pacific Life said in a tweet Thursday. “Our customer base and our workforce reflect the diversity of our great nation, something we take great pride in.”

Last night, Tucker Carlson opened his show by saying immigration makes America “poorer, and dirtier, and more divided."

Earlier this year, Laura Ingraham lost some advertisers after she made negative comments about Florida school shooting survivor David Hogg. And last year, Bill O’Reilly saw advertisers abandon him following reports of sexual-misconduct complaints against him. He left the network shortly afterward.

Fox News Channel said in a statement that “left wing advocacy groups” were using social media to “stifle free speech.” The network said it “continues to stand by and work with our advertisers through these unfortunate and unnecessary distractions.” Fox added that all advertisers have switched their ads to other shows, so no revenue was lost.

Carlson said he would not back down to criticism.

“We’re not intimidated,” he wrote. “We plan to try to say what’s true until the last day. And the truth is, unregulated mass immigration has badly hurt this country’s natural landscape.”

Most of Carlson’s biggest advertisers are sticking with the show or staying mum.

MyPillow, which makes pillows and mattress toppers, has no plans to leave. It’s the show’s biggest advertiser in terms of dollars spent, according to Kantar Media.

“I make all of my advertising decisions based on what is best for MyPillow, my customers and my employees,” MyPillow inventor and CEO Mike Lindell said in a statement Tuesday.

The No. 5 top advertiser, AstraZeneca, said it would “continue to assess our advertising purchases regarding the heightened attention surrounding this matter,” but did not announce any action.

The other top three advertisers, Rosland Capital, a precious metals asset firm, and weight-loss companies Jenny Craig and Nutrisystem, didn’t immediately respond to requests for comment.

Allen Adamson, co-founder of brand consultancy Metaforce, said Carlson’s comments, “while damaging in short term, will be supplanted by some other news event. No matter how polarizing your comments are, if you wait long enough someone else will say something more polarizing and take limelight away.”

And advertisers often come back once the controversy dies down. In a statement, NerdWallet said it’s pulling ads “at this time and will be reevaluating any ongoing advertising on this program.”

Jeff Greenfield, co-founder of marketing measurement firm C3metrics, said these types of controversies are “usually short term” and amount to little more than a slap on the wrist. He said shows “don’t feel it unless you permanently pull spending, and most people are not going to do that.”