Remember, those are actually Indexed Database files, not Folders as we usually see them on our hard drives.
To read, add to or delete from any email folder, the entire folder must load into available RAM.

I had one case where a lady could not send any email. because her Sent Items folder had over 10,000 emails stored in it and it would NOT fit into her available ram. I just deleted the folder and on the next run of O.E., it just made a new Sent Items folder. No harm, No foul!
She now empties that folder and the Deleted Items folder, as a part of her Weekly Maintenance.

That's the only reason that I could think of, why just one program on your computer would run slow.
If I had your computer here, I'd do a very thorough cleanup on the entire system, cleaning out every file that's not needed, tweaking the registry for improved performance, disabling all unneeded Services and UN-Installing all unneeded programs, followed by a Defrag.
I'd also do a very thorough Anti-Virus and Anti-Spyware scan.*

* I've seen computers that ran OK, allbeit very slow, that had over a hundred pieces of spyware on them. Most AV programs do not find or remove Adware and Spyware. I recently worked on a seven year old Dell desktop, that had over 500 pieces of spyware on it. It sure ran better after a good cleaning, with Spybot Search & Destroy and Malware Bytes.

You didn't mention what OS you're using, but I'd have to believe it's Vista or something later. Eh?

I use Windows Mail all the way up through Windows 8 and I prefer it over Windows Live Mail or any alternative email program.
Although, to use it in W7 or W8, some tweaking is required. (but that's a whole different story, and may even incur the wrath of the licensing Police)
I call Windows Mail, "O.E. on steroids"!

While this isn't about this topic I also have Windows Mail running on Windows 7 and how to you handle the spam folder ?
I know you can d/l definition files but can you install them without any problems on Windows 7 ??

Right now I just ignore doing anything because I am not sure what may happen if I do try to install them..

Duh? I'm not real sure just what you're asking. My only Spam folder is in GMail, not on my computer.
So once a week, or so, I go up to GMail and check out my spam folder and empty it as well as my Deleted Items folder.
The SPAM never makes it to my computer, so I don't have to worry about it. *

* When I'm getting email on my PC that I don't ever want to see, I go up to GMail and create a 'Filter' for that address and then I never have to worry about it again. GMail filters it out for me, and it never gets down to my PC.
Even my Cable Co. uses GMail for their email provider, which makes it real handy for me, to create SPAM filters for that too.