AIBU to still feel awkward after what happened years ago

I used to be a stay at home carer for an elderly lady who I absolutely adore. However I decided to move on to bigger things but still kept in touch with her and her family because they're all really wonderful and kind and we all became good friends.

One of her sons lives with her; and her other son lives in his own place and he is really lovely but he is a LOT older than me (in his 60s now) and I was only 18 when this happened. He used to drop me off and pick me up, he bought me stuff and wouldn't let me pay for it because we both went shopping as that was part of my job. He is nice for doing that. He does say a few "sexual" things but I took it as a joke because that's what we just did. We had a laugh and I didn't think much more of it.

A few years ago, we was having one last Christmas do before I moved onto my new job and towards the end of the night I decided to go home and when I went to give him a quick friendly hug, he immediately went and kissed me. I pulled back , shocked and said ok I'm going now, bye! He proceeded to follow me, seems like he wanted to ask "have I done anything wrong?", I just quickly got into the taxi and felt really ill and sick.

We kind of spoken on/off but I stopped visiting because every time I did, he would turn up and I just felt really awkward. I've now been informed that his mother is not well and I just feel so guilty for neglecting the family because of how he made me feel.

Am I overreacting? I didn't tell anyone anything because I do not want to cause any problems. My husband would absolutely kill him if he found out! (even though we was not together at the time). I feel like I've cheated when I haven't, if that makes sense? I just don't know should I just bite the bullet and see them?