In February 2014, a group of conservative political donors gathered at New York’s Pierre Hotel to strategize about the coming presidential contest.

Robert Mercer, a computer programmer and hedge-fund manager who distrusted the political establishment and loathed Bill and Hillary Clinton, issued a warning: Data he had seen indicated mainstream Republicans such as Jeb Bush and Marco Rubio would have difficulty winning the White House in 2016, according to one of the attendees. He said only a true outsider with a sense of voters’ frustrations could win.

Nearly three years later, Donald Trump is headed to the White House, helped by the 70-year-old Mr. Mercer and his daughter Rebekah, 43. They are poised to become major power brokers in Mr. Trump’s Washington.

Back when Mr. Trump’s candidacy was on the rocks, Mr. Mercer, co-chief executive of hedge fund Renaissance Technologies LLC, provided financial support. Then, in a surprise shake-up in August, two of the Mercer family’s confidantes, Steve Bannon and Kellyanne Conway, were installed atop the Trump campaign following a recommendation from Ms. Mercer.

“The Mercers are incredible people who truly love this country and go all out to protect America and everything it stands for,” Mr. Trump said in a written statement to The Wall Street Journal.

Mr. and Ms. Mercer won’t have any formal roles in the Trump administration. Ms. Mercer, who has been working from Mr. Bannon’s office in Trump Tower, is advising on the selection of nominees to Mr. Trump’s cabinet. Mr. Bannon and Ms. Conway are headed for influential White House roles. Ms. Mercer likely will help lead an outside group designed to support Mr. Trump’s agenda, Ms. Conway has said.

Mr. Mercer and his wife, Diana, were the third-largest donors to Republicans in the 2016 election, after Sheldon and Miriam Adelson and Paul Singer, according to the Center for Responsive Politics. The GOP is now counting on Mr. Mercer to help the party gain more seats in the House and Senate in 2018. Ms. Conway says if the GOP captures more than 60 Senate seats, “Bob Mercer will be one of the people to thank.”

Mr. Mercer and his daughter both declined to comment for this article. This account of Mr. Mercer’s career and his family’s involvement in the Trump campaign is based on interviews with current and former employees of Renaissance and others in the hedge-fund industry, along with people who have dealt with both of the Mercers in the political arena.

Every new administration, especially one built around a political outsider, brings a new crop of people to Washington, where they assume various positions of power, often informal.

It isn’t clear what specific policies or positions, if any, the Mercers are seeking for their support of Mr. Trump. Peter Schweizer, who co-founded a research organization funded by the Mercers, notes: “Bob Mercer does not want to be ambassador to France.”

Mr. Mercer, for his part, is an unlikely kingmaker. A taciturn man, he often sits through meetings without uttering a word. He once told a colleague he preferred the company of cats to humans.

Growing up in New Mexico, Mr. Mercer became fascinated by computers at a young age, he said in a speech several years ago. During the summer after his senior year in high school in 1964, he attended a National Youth Science Camp and learned to program a donated IBM computer.

“Computers were not much of interest to my fellow campers, so I got to use the [computer] all by myself most of the time,” Mr. Mercer said in the speech.

While working toward a physics and mathematics degree from the University of New Mexico at Albuquerque, Mr. Mercer got a job nearby at the Kirtland Air Force Base’s weapons lab. He rewrote the lab’s computer program, making it 100 times faster, but his bosses didn’t do much with his work, he said in the speech.

The experience “left me ever since with a jaundiced view of government-financed research,” he said.

After receiving a Ph.D. in computer science, he joined International Business Machines Corp. and forged a partnership with scientist Peter Brown. Their group helped develop an innovative approach to computerized speech recognition. Mr. Mercer eventually grew frustrated with what he saw as IBM’s inability to capitalize on their advances, two ex-colleagues say. IBM declined to comment.

In 1993, Renaissance, which relies on computers and complex algorithms, hired Mr. Brown. Mr. Mercer decided to join, too, though he initially expressed concern to a colleague about how much trading contributed to society.

When Renaissance researchers encountered a challenge no one could figure out, they would bring it to Mr. Mercer, who would go into seclusion until it was solved, says one person who worked with him. Colleagues dubbed him the “Big Gun.”

Messrs. Mercer and Brown eventually revolutionized Renaissance’s approach to trading stocks and commodities, partly by eliminating the human element from trading decisions and by fine-tuning the firm’s trading models. Last year, Renaissance’s assets climbed above $36 billion, from $27 billion in 2015, even as many rivals stumbled. The two men became co-CEOs in 2010.

Early on, Mr. Mercer’s conservatism was on display in his support for the gold standard and his animosity toward the Clintons. “He thought [Bill] Clinton was a crook,” says former Renaissance employee Nick Patterson.

He often challenged mainstream views. When University of California, Berkeley, physicist Richard Muller visited Renaissance in 2011 to lecture on his doubts about global warming, Mr. Mercer buttonholed him afterward to share his own skepticism.

Mr. Mercer was “distrustful of the science,” Mr. Muller recalls. “It’s a scientist’s duty to be skeptical, and the original measurements had unaccounted systematic problems that could have been important.”

In the 1990s and early 2000s, Ms. Mercer, a Stanford University graduate, worked in Renaissance’s trading business. She left to home-school her children and, with her sisters, open a Manhattan cookie and brownie shop that turned into an online company.

After meeting Andrew Breitbart at a conservative conference, Mr. Mercer and his daughter in 2011 became interested in investing in his right-wing news organization, Breitbart News Network.

Mr. Breitbart introduced the Mercers to his friend Mr. Bannon, a former Goldman Sachs Group Inc. investment banker. Mr. Bannon drew up a business plan and term sheet under which the Mercer family bought nearly 50% of Breitbart News for $10 million, according to public filings and people familiar with the matter.

In March 2012, Mr. Breitbart collapsed on a Los Angeles sidewalk and died of heart failure at age 43. Mr. Bannon and the Mercers convened an emergency meeting in New York to determine Breitbart’s future, according to people familiar with the matter.

Mr. Bannon became the site’s executive chairman, eventually gaining influence over the editorial side of the publication. Breitbart became popular with the “alt-right,” a loose conglomeration of groups, some of which embrace tenets of white supremacy and view immigration and multiculturalism as threats. In November, Mr. Bannon told a Wall Street Journal opinion writer he is an economic nationalist, not a white supremacist. He declined to comment for this article.

Disappointed by President Barack Obama’s 2012 re-election, Mr. Mercer intensified his political activity. In 2013, he and Ms. Mercer, together with Mr. Bannon, started the watchdog group Reclaim New York to track what they viewed as excessive public spending.

That year, Patrick Caddell, a former pollster for Jimmy Carter who has been critical of top Democrats, shared data with Mr. Mercer showing voters were becoming alienated from both political parties and mainstream candidates. After asking Mr. Caddell to do another round of polls and collecting his own data, Mr. Mercer concluded that a major shift was under way, Mr. Caddell says.

Toby Neugebauer, a conservative energy investor who met with Mr. Mercer in 2014, says Mr. Mercer predicted that immigration and trade would be central themes of the 2016 campaign.

“He and Rebekah were looking for an outsider to shake things up in Washington,” Mr. Neugebauer says.

Sometimes, father and daughter would walk through Republican fundraising events locked arm-in-arm. The sociable Ms. Mercer did most of the talking, while her father stood silently beside her.

Breitbart’s online traffic was soaring, validating the Mercers’ faith in Mr. Bannon. When Mr. Mercer hosted Mr. Bannon on Mr. Mercer’s 203-foot yacht, the Sea Owl, Mr. Bannon wore shorts, cursed freely and held forth like a close relation, according to people who visited the yacht.

Mr. Bannon advised the Mercers on which political and media ventures to invest in, escorting potential beneficiaries to Ms. Mercer’s triplex apartment at Trump Place on Manhattan’s Upper West Side, according to people familiar with the situation.

As the 2016 campaign got under way, the Mercers were focused on causing political upheaval rather than backing a specific candidate, friends say. “The Mercers’ view is not really tied to a particular person,” says Mr. Schweizer, who with Mr. Bannon co-founded the Government Accountability Institute, a research group funded by the Mercers.

When Mr. Cruz dropped out of the race last May, the Mercers pivoted to Mr. Trump, the party’s effective nominee. “Bob and Bekah got behind Trump because he was clearly an antiestablishment candidate,” says Mr. Caddell, the pollster. “It’s a philosophical thing. They think the establishment has failed and is self-serving.”

In July, the Mercers gave $2 million to a super PAC supporting Mr. Trump. At a fundraiser in August, Ms. Mercer suggested to Mr. Trump he shake up his campaign, recommending that two of her confidants take control, according to two people familiar with the conversation.

At Renaissance, many employees initially shrugged off Mr. Mercer’s political activities, assuming Mr. Trump would lose, and few were aware of the extent of Mr. Mercer’s involvement with Breitbart, according to people familiar with the firm. Later in the campaign, however, some Renaissance executives chafed at the unwanted publicity that Mr. Mercer’s political activities brought the firm, those people say.

Renaissance’s chairman and founder, James Simons, is a major donor to Democratic causes, including Hillary Clinton’s campaign. Mr. Simons declined to comment.

After Mr. Trump’s upset victory, one of his first decisions was to appoint Mr. Bannon as his chief White House strategist. He later named Ms. Conway as his White House counselor.

Ms. Mercer was appointed to the 16-member executive committee of Mr. Trump’s transition team, which is responsible for recommending administration officials. She has weighed in on personnel decisions including successfully lobbying against Mitt Romney as secretary of state and for Sen. Jeff Sessions as attorney general, according to people familiar with the matter.

Once Mr. Trump takes office, Ms. Mercer is likely to help lead an outside group, funded by her father, aimed at bolstering Mr. Trump’s agenda, Ms. Conway says. Republican operatives expect the organization will build its messages based in part on information collected by Cambridge Analytica, a data firm backed by the Mercers that worked on Mr. Trump’s campaign.

On Dec. 3, Mr. Mercer hosted his annual holiday party at his Long Island estate, Owl’s Nest, a bash at which most guests wear costumes. The party’s theme this year was “heroes and villains.” Mr. Bannon and Ms. Conway brought Mr. Trump as a surprise guest.

As the revelers settled in, Mr. Trump rose to speak, according to two attendees, joking that he had just had his longest conversation ever with Mr. Mercer—two words. He lauded the contributions of Mr. and Ms. Mercer to his campaign, then joined the Mercers, Mr. Bannon and Ms. Conway at the head table.