I'll chime in and say that I had experienced this issue before (and that I never found a fix).

Once the 5-man raid was created I could not invite any other low-level characters into the raid. I could remove everyone and reform the 5-man raid with an entirely different set of characters that were standing right next to me, but I was never able to get more than 5 low-level characters into a raid group at one time.