Should I cut him out of my life? He's clinging onto the past?

He said he was going to meet his brother. He accidently left his phone with me after dropping off a gift. 15 minutes later he called from his ex's phone to pick it up from me. (I saw the name appear on the unknown #). So he was dropping off a gift to her too. I know this because I spoke to her briefly (This was 10:30 night)

He told me they dated for 3 years, lived together. She wasn't interested anymore and starting seeing someone. He said he'd never get back together, and they are friends. She regularly hangs out with his gay brother. She regularly calls him. She took him for a birthday dinner recently. Also, had this feeling that he was recreating memories from his ex. I saw an old picture with them at a pastry place. The same one he took me to. Another old picture of her with the same book he gave me. Also, there was a time he was drunk and allowed his coworker to flirt and put hands on him in-front of me. He's not my boyfriend but still. I know I have insecurities.

We're seeing each other 3 months now. I told him I'm not ready for a relationship, he said he's willing to wait. We haven't kissed yet and are not intimate. I'm shy and awkward. Waited for him to make a move, but I think he was unsure.(my ex was a player and manipulator, my ex before that was abusive and controlling).

His last message: "If that's what the universe is telling you then by all means follow your heart, but considering you're making a preconceived notion about my ex is rather bothersome, I did see my brother that night she was also someone I met after, I understand you're not familiar with ex's being friends, but I was willing to sacrifice things if only things were communicated, I'm sorry you feel that way and I hope the best for you in the future, and if it all means anything you were someone I was really looking forward to go all in with... but I suppose this is for the better take care"