Getting booted off when playing with buddy shortly after starting game

I'm trying to play multiplayer with a buddy, and we've both been experiencing disconnects. If I host the game, he gets booted out after a few minutes, and can't easily get back in without restarting the entire game, but my game goes on just fine. If we reverse and he hosts the game, I get disconnected after a few minutes, though he is fine. We're both on desktop PCs hardwired, I'm behind a NAT (and have tried opening port 3000 directly to my desktop), he is not behind a NAT, but also cleared port 3000 completely open through his firewall. We were having issues using the direct connection dialog, putting in the straight IPs and port 3000, but using the Join Game from Steam did work to get us in the lobbies (doing the Join Game from the multiplayer and direct IP would just time out and bump us back to the main menu). Both of these installs are completely fresh, I built a new PC and hadn't installed CastleStory on it yet until a few days ago, and he just bought the game yesterday and installed it fresh.