A colleague of mine was looking me up and down?

A colleague of mine at work asked me to have dinner with him. In the middle of conversation, I noticed, he would give me the once over, although I was sitting infront of him and he could see my outfit without doing so. Was he trying to be rude?