Every speaker is different, but with a factory replacement type speaker I am guessing the surround is not a very stiff rubber. It should break in within a few hours of playing time. It might get a little "better" over the course of the next 10-100 hours (who knows) but it should be safe to play loud very soon.
I would worry more about keeping to signal hitting them free of distortion, as that is more likely to blow the speaker than you somehow destroying the surround on a speaker with 2-4mm x-max.