“The minute you release this black man’s face and this white girl’s face, before either of us commented on it, I was already a ‘rapist’ … and a ‘monkey,’ ” he said. “And she’s a ‘whore’ and she’s a ‘slut.’ ”

Olympios has not spoken publicly about the incident since her initial June 14 statement, in which she called herself a “victim” and admitted she had “little memory of that night.” Her attorney Martin Singer previously told PEOPLE that their own investigation into the matter will continue.

Jackson, meanwhile, has detailed his recollection of the encounter to E! News. In the portion of the interview that aired on Monday, he said “that night was probably the wildest night of my entire life.”

Reflecting on the first time that he met Olympios on set, Jackson said the two were “kind of were just complimenting each other on being villains” and joking about how they were “going to dominate Paradise.”

“We started having a little bit more fun, started having a few drinks,” he said. “It got a little bit hot and heavy. She jumped in my arms, we started making out at the bar.”

According to Jackson, Olympios did not appear to be drunk and the whole group was “talking and mingling” before she suggested that the pair go to the pool. (A spokesperson for Olympios had no comment, and a rep for Warner Bros. did not immediately respond to PEOPLE’s request for comment.)

“When you’re a man, mostly African-American man, no matter where you’re at, you always look for things that can help you out,” he said. “And at that moment, I made sure that the cameras followed us. It just seemed too perfect in a sense for me. At that point, that’s when my spidey senses got up.”

Jackson went on to explain that he and Olympios were in the pool “kissing,” “rubbing” and “touching.” Eventually, “things got wild because it was more of like, her being the aggressor, which was sexy because you have a very attractive girl who’s telling you what she wants.”

Jackson said he proceeded to sit on the edge of the pool, and Olympios got up and put “her lady parts like, right on my face.”

“We were already naked at that point,” he said. “We had been naked, like, the second we got into the pool.”

Jackson told E! News that the following day, nothing appeared to be different between him and Olympios — when he saw her, they hugged and spoke before the group all got breakfast together.

“We were all talking about how wild the previous day was,” he said. “She wasn’t mad. We were hanging out. I offered her a shot, but she said that the production had cut her off for drinking for the day.”

The cast continued filming for the next two days, but Jackson says that on Tuesday night (June 6), producers pulled him aside and told him there was an issue and advised him to “bow out.”

“It was stressful — for me, mostly for my mother,” he said. “It’s hard to see your mom cry every single day. It was very difficult.”

Warner Bros. said in a statement last week that the company “does not intend to release the videotape of the incident,” but insists that they “plan to implement certain changes to the show’s policies and procedures to enhance and further ensure the safety and security of all participants.”

A source close to production also told PEOPLE that precautions are being taken to ensure a similar incident won’t occur. According to the source, “in broad strokes, changes were made to ensure everyone’s safety” — particularly in regards to alcohol consumption and consent.

Bachelor in Paradise will return later this summer on ABC, and Jackson’s interview continues Tuesday at 7 p.m. and 11 p.m. ET on E!