Ben Affleck addressed the elephant in the room on The Late Show Thursday.

After promoting his new movie, Justice League, Stephen Colbert from the commercial break and confessed he was a big fan of Affleck's 1997 Academy Award-winning drama Good Will Hunting, which the actor wrote and starred in with his best friend, Matt Damon. "Now, the person who made that film for you, the person who started your career, was HarveyWeinstein," he noted. Appearing nervous, Affleck asked, "This is a comedy show, correct?"

"This is a comedy show, but we also talk about the subject of the moment, what the national conversation's about," the host reminded Affleck. "After the revelations—the disturbing and just truly horrifying revelations about what Harvey Weinstein did—did you feel, because you had such a close association with him at the beginning of your career, that you had to do more than simply distance yourself from Harvey Weinstein? Because everybody ran from the exits."

"Yes, and understandably so," the 45-year-old actor replied. "You know, for me, it was awful to see the extent of these terrible crimes. It was hideous. I haven't worked for Harvey in more than 15 years, but nonetheless, I felt this attachment. I did movies like Good Will Hunting, Shakespeare in Love and Chasing Amy—early movies that I really loved doing, when I still was totally brand new. And so, it sort of tainted that a little bit to realize while we were having these experiences and making these movies, there were people who were suffering and dealing with awful experiences. I didn't really know what to do with that, you know? It's hard to know."

Watch

After realizing the scope and severity of Weinstein's actions, Affleck decided to donate his residual money from the Miramax movies he made to RAINN (Rape, Abuse & Incest National Network) and Film Independent. Affleck did so "just because I didn't want to sort of cash a check form the guy, and I thought, 'Maybe I can feel OK about it if it's going to a good cause.'"

Affleck issued a statement via Facebook Oct. 10, saying the allegations against Weinstein made him "angry" and "sick." Hours later, the actor himself was accused of groping Hilarie Burton during a 2003 taping of MTV's TRL. After a clip was brought to Burton's attention on Twitter, the actress replied, "I didn't forget...I was a kid." In an uncensored cold open, shown years later in a retrospective, Burton looked back at the footage with Affleck and said, "He wraps his arm around me and comes over and tweaks my left boob. I'm just like, 'What are you doing?'" A young Burton said, "Some girls like good tweakage here and there. I'd rather have a high-five."

"Girls. I'm so impressed with you brave ones," Burton tweeted, linking to the edited clip. "I had to laugh back then so I wouldn't cry. Sending love." Affleck responded via Twitter, writing, "I acted inappropriately toward Ms. Burton and I sincerely apologize." Eight days later, makeup artist Annamarie Tendler accused him of inappropriately grabbing her during a party in 2014.

Scott Kowalchyk/CBS

So, Colbert wanted to give Affleck the opportunity to explain himself Thursday. "You yourself have been accused of a few things—sexual impropriety—and you've apologized for some of that," Colbert said. "Do feel there is more that you—or all men, especially in Hollywood—have to do to make sure that this isn't a passing thing, where women are listened to?" Affleck didn't disagree that there's more to be done, particularly as more women come forward. "What I was accused of by a woman was of touching her breast while giving her a hug. I don't remember it, but I absolutely apologized for it," he said. "I certainly don't think she's lying or making it up."

"It's just the kind of thing we have to, as men, I think, as we become more aware of this, be really, really mindful of our behavior and hold ourselves accountable and say, 'If I was ever part of the problem, I want to change. I want to be part of the solution.' And to not shy away from these awkward or strange encounters we might have had where we're sort of navigating and not knowing," he continued. Referring to a clip that went viral Wednesday, Affleck said, "I just did an interview where somebody asked me a question—it was a serious question, and I kind of felt uncomfortable and didn't know what to say and laughed awkwardly. It's just a tricky thing to try to handle. I think the most important thing to do is to support the voices coming forward, believe them, and create a business where more women are empowered and in place so less of this happens, and so there is a way of reporting this stuff so that people can feel safe doing it."

Affleck then acknowledged his male privilege, telling the late-night host, "I thought I had a sense of the scope of the problem and I thought I understood it, and the truth is I really didn't. I didn't understand what it's like to be groped, to be harassed, to be interrupted, talked over, paid less, you know, pushed around, belittled—all the things that women deal with, that for me as a man, I have the privilege of not having to deal with. Part of this, for me, has been listening to people I really care about and love as they tell me stories of stuff that has happened to them—this is men and women—and recognizing it's a real thing. I'm not spokesman. I'm not a superhero. I can't change it by myself. I can just be accountable for myself and my actions."

"It certainly feels like a really important moment. It feels like a time when survivors are finding their voice, and people—and I include include myself in this—are really discovering the terrible extent of this problem here in our country. I certainly don't think it's just limited to Hollywood. But I expect if you see it this prevalent in this industry, it must extend to finance, tech and all those other places, and it's really disturbing," the actor told journalist Savannah Guthrie. "It's also inspiring to see people come forward and assert themselves. That's very moving."

As for Weinstein and the dozens of allegations that have been made against the producer, Affleck said, "I knew he was sleazy and kind of a bully, but unfortunately that wasn't that uncommon. I was brand new to Hollywood. I was 24 years old. I had never made a movie. I didn't know much of anything, really. It really is a shame, because although I stopped working with Harvey a long time ago, those movies hold a special place in my heart. To now look back on it and think, 'Gosh, some other people were going through something really ugly, difficult and terrible and disturbing and terrifying while that was happening,' is kind of...I don't know."

"The only thing I can think to do is give my residuals from my Harvey movies to a couple of organizations that I think are making a difference, and try to reconcile that," he said. "It's a way for me to feel better about that early stage of my career, when I made those movies with him."

Last month, Rose McGowan accused Weinstein of rape on Twitter. The actress later suggested she had told Affleck something about the producer's behavior. Asked to set the record straight, Affleck replied, "I don't really want to get into other people's individual stories, because I feel like those are their stories and they're entitled to tell as much or as little of those as they want. I believe Rose. I support her. I really like and admire her tenacity, and I wish her the best."

During his press tour for Justice League, Affleck vowed to do better and look "inward" in light of the allegations made against Hollywood's most powerful men. "That's all you can do, certainly as a man in my position and recognizing how fortunate I am, recognizing the privilege that I have. I'm also really trying to open my eyes and hold myself accountable, because that's all we can really do," he added. "It's a lot harder than pointing a finger at everybody else, really asking, 'What have I done that's crossed the line? How can I do better? How can I be more respectful, more inclusive, more mindful of this?' That's the kind of person that I want to be."

As more people come forward with stories of sexual harassment, Affleck has felt compelled to examine his own behavior in the past. "There's plenty of jokes I wouldn't have made, or things I wouldn't have done," he said. "It's a time when—and it's happened almost overnight—there's a heightened level of awareness. One of the good things about that is the self-consciousness of, 'How am I fitting into the world here? How are my actions affecting other people? How is this interaction going to leave this person feeling?' It's easy to sort of gloss over that stuff and get self-involved. The hard thing is really, as I said, to be mindful and live in the present moment."