Yeah, that's a real pain. One of the disadvantages of using GMail for storage. I learned long ago to keep my GMail account free of all the old messages that I no longer need by removing them from the server. Of course deleting 20,000 messages from your account off the server can take some time. And you might not want to do that.....

Do you really need an IMAP account? The only advantage to using one is if you have other devices that you need to sync mail to such as an iPhone.