...but I honestly can't say I remember the recoil or the noise of the shot...

I second what Dr. Strangelove said. I was involved in a shooting and all I remember, besides my fixation on the target, was seeing the brass flying out of the weapon. I have no recollection of noise, recoil or anything else. It is quite traumatic.

You will damage your hearing a little bit each time you shoot without hearing protection; the damage is cumulative. I don't think needing hearing protection will be a factor in a self defense situation.