Kevin Spacey’s alleged sexual misconduct halted production of The Usual Suspects, according to Gabriel Byrne, his costar in the 1995 thriller.

“I did not know honestly then the extent of his violence,” the Irish actor, 67, recently told The Sunday Times. “I mean, he was kind of a joke in that people would say, ‘That’s Kevin,’ but nobody really understood the depth of his predations. It was only years later that we began to understand that [filming] was closed down for a particular reason and that was because of inappropriate sexual behavior by Spacey.”

Spacey, 58, would go on to win the Best Supporting Actor Oscar for his role in the classic, which was directed by Bryan Singer.

The actor was dropped from his show House of Cards in November after Anthony Rapp accused Spacey of assaulting him when he was 14-years-old. Spacey responded by saying “I honestly do not remember the encounter” and apologizing to Rapp for “what would have been deeply inappropriate drunken behavior.”

In a CNN article published in early November, eight anonymous House of Cards employees accused Spacey of creating a “toxic” work environment and displaying “predatory” behavior, allegedly touching staffers without consent and making lewd comments.

Byrne also compared Spacey to disgraced film mogul Harvey Weinstein, saying that they shared “that element of absolute abuse of power.”

He added of Weinstein, “I did three movies with Harvey Weinstein, and I knew he was a sleaze-bag. I knew he was a vile bully and I saw his bullying up close. I saw him be absolutely appalling, not just to women but to men as well. He had very little respect for any kind of human being. He wanted his stars around him.”

A spokesperson for Weinstein previously told PEOPLE in a statement, “Any allegations of non-consensual sex are unequivocally denied by Mr. Weinstein. Mr. Weinstein has further confirmed that there were never any acts of retaliation against any women for refusing his advances.”

Byrne admitted he had heard “vague rumors” about the producer’s bad behavior. “I have to stress that they were vague — of doors being locked and women being compromised,” he said.

“I heard that once or twice from two very well known actresses, but the problem when you hear something like that is, do you pass that on? Because if it’s not true it’s awful, and if it is true it’s not your job to say, ‘Well, I wasn’t there, the door was locked, I don’t know what happened, I just heard the story.’ But I did not know, and many people didn’t know, the extent of the violence that he perpetrated on women.”

Since the allegations against him broke in October, Weinstein has been fired from his powerhouse studio, the Weinstein Company, and his wife, Marchesa designer Georgina Chapman, has announced she’s leaving him. He is also under investigation by police in New York, Los Angeles, Beverly Hills, and the U.K. for sexual assault.