Is it too soon to have feelings?

So I"ve been seeing this guy, he is 27 and I am 24 both single parents. We've been talking for about 3-4 months text and calls. We have only gone out a few times since we both work Mon-Fri I work morning until 5pm and he sleeps at 6pm to go into work 11pm to 7am so we can only see each other on weekends and since we both have kids it doesn't always work out. ANYWAY, We went on 3 dates and he is so respectful and such a gentleman. On our 4th date we hung out at his place for a lunch and a movie, we are both adults and decided to be more intimate, then he cooked lunch for us. Anyway I feel like I'm starting to feel a little more attracted to him. I look forward to his calls and texts. And I feel the need to see him more often. I won't be able to this weekend cause I have plans with the fam.