Trying to mount an NFS export, it fails with delay error. Surely, it's a connectivity problem (since not pingable)? I don't get why I can ssh to server to client (and vice versa) though. :S Tested with firewall off on both machines.

Many routers are setup to drop external icmp request to hide themself. If set, ping will never get answer, but other request and service will work as expected, a dedicated tiny-firewall option if you prefer (with 0 relation to the router or host firewall setup).

As you hide your IP i suppose your server and client are on two networks, facing router setup so.
If they are on the same network, this shouldn't apply, so yes you have a connection problem.

Your nfs trouble could just be another trouble, with no relation to your ping problem.

newer nfsutils doesn't allow anymore the lazzy mount of nfsv3 for nfsv4 server.
previously an nfsv4 server offering nfsv3 & 4 service could be mount with a buggy client requesting an nfsv3 mount

damnit i'm not sure how to explain that
- if your client try to mount an nfsv4 mount, the mount must comply with strict nfsv4 (this mean having a nfsroot present, and subdirectories bind to it)
- if your client try to mount an nfsv3 mount from an nfsv4 server, the client must specifically query that, passing nfsvers=3,ver=3

anyway, assuming your ports are forward and open as they should, trying to mount with -vvv will gave a better clue of your problem.