In my experience, downloading whole mailboxes does tend to be slow, regardless of the mail service you are using. FastMail tends to download faster than most other services. The question is the number of individual messages and the total size being downloaded.

Terry nudges me to make another point worth bearing in mind. Sometimes, a slow download can be caused by slow writes to the harddrive if it is not in a good condition. You can verify whether that might be the case by copying a few small files to the harddrive, and assessing whether the copy seems unusually sluggish.