In the most recent EW SiriusXM Town Hall interview taped as part of The Jess Cagle Interview, the Watch What Happens Live host sat down with the EW and PEOPLE Director to talk about his new book, Superficial: More Adventures from the Andy Cohen Diaries. During their chat, Cohen shared a cringeworthy story about a conversation he had with Swift at a Met Gala after party, which he thinks made the Grammy winner believe he was a “shady ass bitch.”

Watch Cohen tell the story in vivid detail above, and watch the full episode of The Jess Cagle Interview, available now on the new People/Entertainment Weekly Network (PEN). Go to PEOPLE.com/PEN, or download the PEN app on Apple TV, Roku Players, Amazon Fire TV, Xumo, Chromecast, iOS and Android devices.

Mike Coppola/Getty; Taylor Hill/Getty

As Cohen tells it in the video above, he was sitting by the bar waiting for a drink when he overheard the “Bad Blood” singer looking for a seat with a friend. Even though Swift wasn’t talking to him, Cohen decided to interject anyway.

“I said, ‘Your friend Katy is over there,’ ” recalls Cohen, adding that he has a history of saying exactly the wrong thing to people. “She goes, ‘Katy who?’ I go, ‘Katy Perry.’ ”

Apparently, Swift was far from amused, which makes sense since there might be some bad blood between her and Perry, and made it clear that she didn’t want him to talk about the supposed feud on his show.

Although he wasn’t trying to be “shady,” Cohen realizes that’s how he came off in that little interaction. “Maybe I am a shady ass bitch that I did put it in the book,” he says. “I think the story is about what a douche I am in that instance.”