I was asked this questions earlier. I thought it's a fair question in contact lens wearers. ​And here are my thoughts on this topic:

You will get closer to having an infection or corneal ulcer. It could from viral, bacterial, acanthomeba, or fungal infection or a mix of them. The worst form is fungal infection. If the infection/ulcer is big and not responding to treatment, You might need a corneal transplant. It’s also painful by the way.

Its not worth it.

If you don't have time to remove them, wear glasses, or do LASIK, or orthokeratology lenses at night.

If you want to continue to wear contacts, and not sure if you are doing it right, just ask your local Optometrist. You can book an appointment if you need to go in and ask more questions. You may have to pay a nominal fee per visit, but call it contact lens coaching if you will. ​Invest in your health. The best things in life are free (your own body/cornea), the second best things are very expensive and not be nearly as good. (The corneal transplant).