Hunkered down for his first television interview since he left the White House, an unkempt Steve Bannon succinctly outlined the populist-nationalist mission of Breitbart News. “Our purpose is to support Donald Trump [and] to make sure his enemies know that there’s no free shot on goal,” he told 60 Minutes host Charlie Rose last week. Those enemies include a familiar list of Breitbart targets: establishment lawmakers like Paul Ryan, Mitch McConnell, and the rest of the G.O.P. swamp; undocumented immigrants protected under DACA; the liberal media; White House “globalists” such as Gary Cohn, and so forth.

But the biggest danger to the president and to Breitbart may be their fellow travelers on what Bannon once called “the alt-right,” as became especially clear after Charlottesville. And Bannon was itching to distance himself from the white supremacists, white nationalists, and neo-Nazis that have rallied under Trump and supported his agenda. “They’re getting off a free ride off Donald Trump. They’re getting a free ride,” he exploded, his eyes red, calling them a “small,” “vicious group” that “add[s] no value.” As he condemned them, though, he took a characteristic swipe at the media for continuing to blur the lines between racial extremists and his movement. “I don’t need to be—I don’t need to be lectured—by a bunch of—by a bunch of limousine liberals, O.K., from the Upper East Side of New York and from the Hamptons, O.K., about any of this.”

Prior to Trump’s surprising election, Bannon’s Breitbart pursued, essentially, a no-enemies-on-the-right policy, with a disparate group of believers in its big tent. For years, as Breitbart cultivated a scurrilous following of anti-Islamists, anti-immigrants, and Internet trolls with questionable Photoshop skills and even more questionable taste, Bannon defended his collection of deplorables as people who were simply united by their hatred of the establishment, whatever it was at any given moment. In August 2016, Bannon called Breitbart “the platform of the alt-right,” yoking his site to an ugly strain of American politics at the expense of his own allies. “I’ve talked to people who work with him, and they said, ‘They don’t know why he said that,’” said Morton Klein, echoing several other Bannon associates I’ve spoken to over the past several months. He rolled with it, however, and tended to dismiss complaints about some of the constituencies as political correctness. (Bannon did not return a request for comment.)

And then in August, Charlottesville happened. The nation watched in horror as a group of white supremacists, toting guns and chanting neo-Nazi slogans, chose to defend a statue against liberal protesters by ramming a car into the crowd, injuring 19 and killing one. Whether he deserved it or not, Bannon got swept into the narrative when he was ousted from the White House shortly afterward, just days after Trump claimed that both the supremacists and the antifa were to blame for the violence.

Charlottesville brought to a head something that had been bubbling for a while. Back in April, I reported that the members of Bannon’s ideological world were coming to realize that associating with the “alt-right”—a term also used by neo-Nazi Richard Spencer to describe his own movement—was poisonous to their cause. Though Bannon had counseled the president to stick to his guns over Charlottesville, people close to him say that he’d known for a while that an association with open racialists was a major threat to the rise of populist-nationalism. In fact, said right-wing raconteur-turned-reporter Mike Cernovich, virtually everyone in the upper echelons of the organized Trump base were in agreement that the neo-Nazis, the white supremacists, the anti-Semites, and the rest of them, all had to go.

But when I asked what Bannon and his movement would do to get them out, Cernovich sighed. “Yeah, that’s the issue,” he said. “I don’t know if anybody has the answer about that.”

For Bannon and his allies, purging extremist elements from the far-right movement is hard, because no matter how vocally they repudiate neo-Nazis and their brethren, those groups will hug back harder—especially if there’s a national platform for them to clamber on. “David Duke shows up for every media opportunity,” Bannon complained to Rose, referring to the former K.K.K. grand wizard. One person who met with Bannon recently described him as furious and preoccupied with the Nazi problem. “What the fuck do we do about the Nazis?” he asked at one point during their meeting, according to this person.

Complicating the issue are the rank-and-file Web-based Breitbart constituency that Bannon calls the “Pepes,” after the cartoon frog often used in memes promoting far-right ideology. It has not always been clear where the group’s mischief-making ends and anti-Semitism begins: the “Pepes” themselves gleefully Photoshop their foes into gas chambers, not because they actually believe in white supremacism, some say, but because they simply wanted to see the liberal Internet lose their minds. The Pepes’ outrageous stunts are responsibly in part for fueling the rise of Breitbart, and to an extent, Trumpism, but their refusal to stop tweaking progressives with their use of swastikas has now become a liability.

“Bannon sees the Pepes as kind of like trolls, and not like the Nazis like Richard Spencer and David Duke,” explained the person who spoke to Bannon. “Everybody’s kind of struggling with it. Like, ‘Oh yeah, there’s these people, they’re kind of trollish, they make a lot of jokes or whatever, maybe we wish they were a little bit nicer.’ But that’s different, categorically, than neo-Nazis at rallies throwing Nazi salutes and throwing up Nazi flags.”

Near the top of this heap of problems, however, is Bannon himself. After immortalizing Breitbart as a “platform for the alt-right” during the 2016 Republican National Convention—a slip of the tongue several of his colleagues wish he hadn’t made, in retrospect—Bannon inadvertently linked his site to Spencer’s white nationalist movement. The budding fascist had dubbed his own movement the “alt-right” back in 2010, using it to describe a movement dedicated to marginalizing Jews and minorities from white nations, establishing authoritarian governments, and rolling back equal rights for men and women. And Bannon’s apparent reticence to repudiate hate groups, like William F. Buckley did when he rejected the John Birch Society during the 1960s, only allowed them to grow.

”There are steps he could take to change the editorial side, things he could do to demonstrate, not just in words but in action . . . that he means what he says. That would be the start,” said Jonathan Greenblatt, C.E.O. of the Anti-Defamation League. Bannon, for instance, could stop referring to “globalists”—a term fraught with historical anti-Semitic overtones—or write a major op-ed on Breitbart declaring open war against the white supremacists. But to Greenblatt and the rest of Bannon’s critics, that reluctance seems calculated. “He’s very smart. He knows what he’s doing,” he told me, citing several examples of Breitbart trafficking in winking anti-Semitism: putting Gary Cohn’s name between globe emojis; talking up international banking conspiracies; and using “America First,” a common phrase once uttered by anti-Semite Charles Lindbergh, as a slogan.

Bannon’s positions on race issues remain rather murky, other than that he loves using them as a wedge issue to box in Democrats. “I want them to talk about racism every day,” he recently told The American Prospect. “If the left is focused on race and identity, and we go with economic nationalism, we can crush the Democrats.” But fundamentally, virtually everyone I spoke to agreed that Bannon does not hate Jews in the way that a Nazi, or any other group that calls for white supremacy, would hate Jews. (Bannon’s ex-wife claimed in a sworn statement in 2007 that he had made anti-Semitic remarks in the past, an allegation that Bannon has vehemently denied.) His defenders will point out that several of his senior employees are Orthodox Jews, and that Bannon had gone out of his way to open a Breitbart bureau in Jerusalem. “He’s a philosemite, he’s a supporter of Jewish people in Israel,” said Klein, the president of the pro-Israel Zionist Organization of America, where Bannon is scheduled to speak during a gala in November. (At the moment, Klein suggested that he may be introducing Sheldon Adelson, one of the biggest pro-Israel donors in the country.) Klein, whose parents survived the Holocaust and who criticized Trump for his response to Charlottesville, told me if he had detected a whiff of anti-Semitism from Bannon, he would have nothing to do with him. “If he’s a fine man, and promotes the agenda that we at Z.O.A. believe in, I will have to just tell the truth,” he added.

The biggest white nationalist problem Trump’s movement has, however, may be Trump himself. The majority of Republicans in Washington were baffled when Trump struggled to condemn the white nationalists and neo-Nazis in Charlottesville, not once but twice. “Trump needs to say, “Look, if you’re a Nazi, don’t come to my events, don’t vote for me, period, end of discussion,’” said Cernovich. “Nobody can understand why he hasn’t just said it that way, because I have, and pretty much everybody else has said it,” including Bannon himself on 60 Minutes.

At his own events, Cernovich told me, he had a system in place to deal with any parasitic white supremacists who tried to show up: anyone who threw up a Nazi salute would be booted immediately. “I told mutual acquaintances that my official stance was ‘I’m going to kick you out,’ but my unofficial stance was ‘I’ll kick your ass,’” he said. But there is only so much one man—even one who hawks a self-improvement class called “Gorilla Mindset”—can do. “We don’t like them,” said Cernovich. “We said we don’t like them. We disavowed them, but they keep showing up and saying Trump’s name. What are we going to do about that?”