Debugging Authentication

The most important thing to do is to set auth_debug=yes, and preferrably also auth_debug_passwords=yes. After that you'll see in the logs exactly what dovecot-auth is doing, and that should help you to fix the problem.

PLAIN SASL mechanism

With IMAP and POP3 it's easy to log in manually using the IMAP's LOGIN command or POP3's USER and PASS commands (see TestInstallation and TestPop3Installation for details), but with SMTP AUTH you'll need to use PLAIN authentication mechanism, which requires you to build a base64-encoded string in the correct format. The PLAIN authentication is also used internally by both IMAP and POP3 to authenticate to dovecot-auth, so you see it in the debug logs.

The PLAIN mechanism's authentication format is: <authorization ID> NUL <authentication ID> NUL <password>. Authorization ID is the username who you want to log in as, and authentication ID is the username whose password you're giving. If you're not planning on doing a master user login, you can either set both of these fields to the same username, or leave the authorization ID empty.

Encoding with mmencode

printf(1) and mmencode(1) should be available on most Unix or GNU/Linux systems. (If not, check with your distribution. GNU coreutils includes printf(1), and metamail includes mmencode(1). In Debian, mmencode is called mimencode(1).)