Problems with online

When i play online with my friends, the servers are strange. With one friend we can all join however when one of my other friends sets up a server only a few of us can join and others are excluded. We just get kicked from the server or are rejected before we can even join. I was wondereing whether anyone knew why this was and if there was anything we could do about it so that we could all play on the other server.