The handshake failed due to an unexpected packet format

Most likely your server requires explicit SSL, sometimes also known as TLS.

It is called explicit SSL mode, because after the connection is established, client explicitly issues a command to the server that initiates SSL/TLS negotiation.

This is in contrast to implicit SSL mode, where SSL negotiation is initiated just after successful connection. In implicit mode server and client knows to use SSL, because client uses default protocol port, that is commonly used for secured traffic.

First try to connect to your server without SSL:

// C#
client.Connect("mail.example.com");

' VB.NET
client.Connect("mail.example.com")

Then, before logging-in, start explicit SSL negotiation. The command name differs for different protocols:

Explicit SSL (aka TLS)

The code is exactly the same no matter which protocol (IMAP, POP3 or SMTP) you use.

// C#
client.Connect("mail.example.com");
client.StartTLS();

' VB.NET
client.Connect("mail.example.com")
client.StartTLS()

StartTLS method negotiates security protocol with the server and secures the channel using SSL or TLS. Now, your connection is secured.