For me it was krb5-user/config being installed. I noticed this by examining /var/log/auth.log and seeing pam_krb5 attempts before pam_unix. Using apt-get remove to uninstall those packages fixed it. Don't remove those packages if you are on a computer requiring kerberos (pam_krb5) obviously. My sudo hang went from a consistent 30s to 0s.

This is hinted at in Halsafar's answer, I have Kerberos enabled on my work VPN but it's useless when I'm off it, so I changed the auth module order in /etc/pam.d/common-auth to use pam_unix before pam_krb5:

On Solaris 10 sudo was hanging for about 30 seconds. With help from truss I was finally able to determine that it was hanging on the quota command which was hanging on an NFS mount. Unmounting the NFS share eliminated the hang. Have not determined what is wrong with the share yet.