In the prior weeks, Mr. Magerman, a registered Democrat who calls himself a centrist, had complained to colleagues about Mr. Mercer’s role as a prominent booster of Donald Trump’s presidential campaign.

Now word of Mr. Magerman’s criticism had reached Mr. Mercer, co-chief executive of Renaissance Technologies LLC, one of the world’s most successful hedge funds.

“Those weren’t my exact words,” Mr. Magerman said he told Mr. Mercer, stammering and then explaining his concerns about Mr. Trump’s policy positions, rhetoric and cabinet choices. “If what you’re doing is harming the country then you have to stop.”

Mr. Mercer declined to comment through a spokesman. In a statement, Renaissance’s chairman and founder, Jim Simons, who has been a prominent financial backer of Democrats, said, “I have worked closely with Bob Mercer since he joined our firm almost 25 years ago. While our politics differ dramatically, I have always thought him to be of impeccable character.”

A presidential campaign that divided much of the country also has created tensions within companies. Some senior employees, accustomed to settling grievances behind closed doors, are rebelling in unusually public ways, the polarization playing out for the world to see. Days after the election, a partner at Peter Thiel’s venture-capital firm, Founders Fund, wrote a blog post expressing fears of a Trump presidency, which Mr. Thiel had worked to promote. A spokesman for the firm declined to comment.

After the November election, Grub Hub Inc., chief executive Matt Maloney seemingly suggested in an email to staff that employees who supported Mr. Trump should resign, citing the “hateful politics” of the new president. He later said the message had been misconstrued.

Historically, some leaders of Renaissance, which is based on Long Island, N.Y., have leaned Democratic, including Mr. Simons, who donated to Hillary Clinton’s 2016 presidential campaign.

Some Renaissance executives chafed at the unwanted publicity brought to the firm by Mr. Mercer’s activities during the presidential race, according to people close to the matter. In addition to providing crucial financial help when Mr. Trump’s candidacy was lagging, Mr. Mercer and his daughter Rebekah advised the campaign, suggesting the installation of two Mercer family confidantes, Steve Bannon and Kellyanne Conway, atop the campaign. Those two now hold senior White House positions.

Until now, however, nobody within the tight-lipped hedge fund has gone public with a grievance.

“His views show contempt for the social safety net that he doesn’t need, but many Americans do,” said Mr. Magerman, 48 years old, during an interview with The Wall Street Journal at the Dairy Café, a kosher restaurant he owns in Bala Cynwyd, Pa. “Now he’s using the money I helped him make to implement his worldview” by supporting Mr. Trump and encouraging that “government be shrunk down to the size of a pinhead.”

Mr. Magerman, a 20-year Renaissance veteran who helped design the fund’s trading systems, says he is speaking only for himself, and that there is no sign of a broad insurrection at the firm.

Mr. Magerman makes millions of dollars a year, drives a Tesla and says he gives more than $10 million in charity annually. A research scientist, he is one of 100 partners at the firm, but he isn’t one of Renaissance’s most senior executives.

“I’d like to think I’m speaking out in a way that won’t risk my job, but it’s very possible they could fire me,” he said. “My wife isn’t comfortable with me jeopardizing my job, but she realizes it’s my prerogative and agrees with my sentiments.”

He has concluded that every new piece of code he developed for Renaissance helped Mr. Mercer make more money and gave him greater ability to influence the country.

To try to counteract his boss’s activities, Mr. Magerman says he has been in touch with local Democratic leaders and plans to make major contributions to the party. He says he called Planned Parenthood to offer his assistance and contacted Jared Kushner, Mr. Trump’s son-in-law and White House adviser, to voice his concerns about Ms. Conway and Mr. Bannon. He says he failed to reach Mr. Kushner.

Mr. Magerman says he first spoke with Mr. Mercer in January, when Mr. Magerman, who donates to local schools, called Mr. Mercer to ask for the opportunity to reach out to Rebekah Mercer to offer the administration help on education policy.

During the call, they talked politics, disagreeing about some of the administration’s early steps. After airing his concerns with others at the company, Mr. Magerman received the second call from Mr. Mercer two weeks ago.

The conversation grew strained. After telling Mr. Mercer to stop harming the country, he said Mr. Mercer responded that his goal had been to defeat Mrs. Clinton and that he wouldn’t remain very involved in politics.

“How can you say you’re not involved?” Mr. Magerman said, citing an outside group Rebekah Mercer was involved in that was aimed at boosting Mr. Trump’s agenda.

Mr. Magerman has one idea that would reduce the power of people like Mr. Mercer. He said he was thinking about reaching out to Democratic Sen. Elizabeth Warren (D., Mass.) to craft proposals to reduce speculative trading, which presumably would curtail Renaissance’s profits.

In conversation at his cafe, Mr. Magerman said he hoped his public statements wouldn’t cost him his job. But if he does get fired, he said, he would have more time to devote to politics and other causes.

“This is my life’s work—I ran a group that wrote the trading system they still use,” he said. “But I feel relieved I’m now doing something, and if they fire me, maybe it’s for the best.”

On Thursday morning, after an online version of this story appeared, Mr. Magerman received a new phone call from Renaissance. A representative told Mr. Magerman that he was being suspended without pay and no longer could have contact with the company.