Some mail not coming through

I've been running the Postfix/Courier/Amavis Sarge setup for years, and added the Stop Spam Before Server setup as well.
In the last few months I have been getting calls that some people are not getting some e-mails.
There is never anything in the mail.log that says any mail even arrived at the server from that sender, or there is a line where mail from that sender to someone else in the domain went through just fine.
I talked to one person sending, and his mail is sometimes bounced with the error "connection to xx.xxxx.xxx(my server) timed out". There is no log entry that a connection was made or dropped from his domain or IP address in the log. I've grep'd for his entire class C and no record of any connection attempt at my end.

Of course it ONLY happens to the two most obsessive-compulsive people on the entire system.

Could it be that my server was 'too busy' at the time and didn't even write a log entry? My server is on a fiber connection so it's unlikely it lost connectivity.