“It makes me a little paranoid because no matter what, I don’t think any person, or any male, goes into a relationship thinking that they’re going to leave or wouldn’t be there,” he tells PEOPLE at GQ‘s Men of the Year party at the Chateau Marmont in West Hollywood. “I think the circumstances a lot of times dictates things, right?”

But the music mogul says he is not his father, a subject he also addresses on “New Day” from his new album, Watch the Throne.