I too have dealt with this problem. I have found that the devil likes to use any remaining guilt or shame from our past against us any chance he can. You need not give details of past sins to anyone. Anyone seeking details may just be looking for gossip. Instead, I just simply say that I am a living example of how Christ and His Church can turn a person's life around. What a better way to use this "problem" than to turn it around and use it to witness for Christ?