Guys, I am so anxious and confused is he done with me?

I have been talking and going on regular dates with a guy for two months. We see each other 1-2 times a week and talk pretty much everyday but with hour gaps in between messages at times due to both being busy. He has a busy work schedule and a routine after work. When we first started seeing each other he told me he was looking for a girlfriend. Sunday he text me about my weekend and then Monday we saw each other just watched a movie at his house which we normally do on a week night due to work. Everything was normal, very affectionate, loving and told me how much he missed me lately. We started falling asleep on his bed when I said it was getting late and I needed to go and he said nooo just stay a little longer so I did then left.I hadn't heard from him on Tuesday so Wednesday k messaged at 4pm saying hi how was your day? I haven't had a reply. He's never ignored a message between us before so I don't know if he's forgotten or doesn't want to talk anymore. I had once asked him if he wasn't interested anymore would he actually tell me and let me know and he said of course.. But I know guys tend to avoid conversations like that.the reason this is making me so anxious is because when I was 19 a guy 6 years older then me that I hardly knew got on top and had sex with me, I was so shock and intimidated that it took me a good 10 minutes to realise what was happening and that I didn't want this.. He was a stranger. He never spoke to me again after that night. So that's why this situation is making me anxious even though this guy is great and we both said we aren't interested in dating other people but why wouldn't be reply? Confused!!!