“I think that above all what we’ve always said is that it remains vital that anyone that feels victimized should and does have the right to speak out and come forward,” she said. “I obviously support my family, and not everything that’s been reported has been accurate, so I think we’re waiting to get all the information. But of course now is the time for listening, and that’s what we are all trying to do.”

Five women accused Franco, who is nominated at Sunday’s ceremony for outstanding performance by a male actor in a leading role for his movie The Disaster Artist, of sexually exploitative behavior earlier this month.

Last week, Franco skipped the Critics’ Choice Awards, where he won the award for best actor in a comedy. Sources have said the actor has been shaken by the allegations: “He’s in a really bad place,” a source close to the situation told PEOPLE. “His close friends are trying to be there for him.”

“There are people that need to be heard,” the actor told Meyers. “I have my own side of this story, but I believe in these people that have been underrepresented getting their stories out enough that I will hold back things that I could say, just because I believe in it that much,” he said. “So if I have to take a knock because I’m not going to try and actively refute things, then I will, because I believe in it that much.”

In an article published in the Los Angeles Times on Jan. 11, five women accused Franco of abusing his power as an acting teacher and mentor in a sexually exploitative manner. Two students claimed the actor would often become angry on set when they would refuse to film topless, while another former acting student at the film school Franco founded said he once removed safety guards while filming an oral sex scene on the set of the 2015 film The Long Home.

Additionally, actress Violet Paley recounted her previous social media claims that Franco exposed himself and tried to pressure her into oral sex. Though she said they had a consensual relationship, Paley said, “that time wasn’t consensual.” She also alleged that he told her friend to meet him in a hotel when the friend was 17.

“Look, in my life I pride myself on taking responsibility for things that I have done,” Franco told Colbert. “I have to do that to maintain my well being. The things that I heard that were on Twitter are not accurate. But I completely support people coming out and being able to have a voice because they didn’t have a voice for so long. So I don’t want to shut them down in any way.”

The 24th annual Screen Actors Guild Awards are broadcast live from the Shrine Auditorium in Los Angeles on both TNT and TBS starting at 8 p.m. ET.