Has the party member already joined your team? KSE only allows you to swap between the npcs who have already joined your party, not to add new party members. Also, you can't have more than 2 npcs in your party.