To support TLS 1.3, hMailServer needs to be updated to use latest OpenSSL-version.

The latest OpenSSL-version does not come with SSL 3-support. It's possible to compile OpenSSL with SSL3 support, but haven't gotten this to work.

To solve this I'm leaning towards simply removing the SSL 3-support from hMailServer. The options will simply be removed from the UI and the API methods removed. SSL3 has been considered broken for decades, so supporting it may only be confusing to end-users.

The cipher list you enter in the user interface is given as-is to OpenSSL using an OpenSSL API (https://www.openssl.org/docs/man1.0.2/m ... _list.html). OpenSSL parses it and decide what ciphers match. I'm honestly not 100% sure about the syntax for the cipher string itself. The one in hMailServer comes from Mozillas recommendations if I recall correctly.

The cipher list is separate from the TLS versions. By default, OpenSSL enables support for TLS1.0 - 1.3. hMailServer explicitly disables any TLS-version which the user has de-selected in the UI

I just installed the latest build and enabled TLS 1.3. I then verified that I was able to connect to the server using the OpenSSL client and that TLS 1.3 was selected. The cipher was TLS_AES_256_GCM_SHA384. If you go to https://wiki.mozilla.org/Security/Server_Side_TLS and read about their recommendations they say "For services with clients that support TLS 1.3 and don't need backward compatibility, the Modern configuration provides an extremely high level of security." and then mention this cipher.

(I realize that this is a vague reply to your question but that's what I know about this so far)

Have you tried to use the new build already? I'm running it on my server now and was thinking I'll run it for a few days before I put it up on the download section. But I've verified that the communication with external services (such as gmail) is now done using TLS 1.3.

If you're using "STARTTLS (Optional)", I'm not sure tweaking this has so much value since doing a downgrade-MITM-attack would be trivial in those cases anyway. If you have configured hMailServer to enforce TLS then it makes more sense.

I get so many unencrypted connections on port 25, don't see how I could force StartTLS on port 25.
However, I also don't allow AUTH on port 25 (using the ini setting), so there are no usernames / passwords accessible

Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

In TLS 1.3, SNI is required and I hadn't implemented that in hMailServer. If hMailServer connects to pop.gmail.com:995 without including SNI, then pop.gmail.com will complete the TLS-handshake but return an incorrect SSL-certificate. This incorrect SSL-certificate has a Subject/Issuer/etc all containing something like "client did not send SNI; fix your client", so it's basically Google's way of finding bugs in clients, which appears to have worked this time.

Are you trying to use the master branch or 5.7-branch with the new OpenSSL/Boost? I have not merged the fixes to the master branch/5.7-branch yet - I will do that later this week. You can check the 5.6.8 branch to see what changes I had to do. The short version is that a few of the OpenSSL/Boost API:s hMailServer was using has been removed (like 2-3 of them) so I had to make a few small changes to no longer use them.