Do guys really not like when girls wear makeup?

When I first went out with this guy I did not wear makeup. However, the second date I wore it. I asked him on the phone if I looked better with or without. He told me he thought I was prettier without anything on my face. Should I not wear any when I am with him? Because it feels good to know that I can be myself, when I am with him.