you need to move to his server or he needs to move to your server whenever possible. Now, we understand that some servers are FULL most of the times but even though, you can still try to join in. You could also choose to move together to a server that it is not so populated. Remember that server transfers are only allowed once per week.

The guesting feature will arrive in due time and will allow you to invite friends from other servers.