I set up a Postfix server on my Ubuntu VPS and it works for sending email from the mail utility in the command line. I set up SPF records, DKIM, pretty much everything I could read about. However, I'd like to be able to send and receive email without SSHing into my box each time.

When I try to set up my server (borogov.es) with my mail client, and I use "mail.borogov.es" as my outgoing mail server, I get "Server not responding." I feel like this is an access issue of some sort, but I'm not sure.

You should not be attempting to send mail from your email client on port 25; rather using 587 (the standard submission port). Port 25 is meant for transfer between mail servers, and many home ISPs block it to reduce spam from compromised home computers.