there are no browsers, just connections between custom written programs

The first authentication scheme that came to my mind was... client signs their messages with the private key, server confirms messages via the public key. Looks good.

But why not just using simple login/password authentication in that case? I know it's not a brilliant idea sending secret info over wires but is it really bad if everything is done via secure HTTPS anyway? What are disadvantages of that method?

Authentication based on asymmetric keys is already supported by SSL under the name "client certificate". Indeed, a signature is involved. If you want to use a client asymmetric key, you are encouraged to use this SSL feature which is already implemented by SSL libraries.

Using an asymmetric key for authentication is useful when the identity of the client is asserted by a third party: an entity, distinct from the server, makes the initial identification (aka "certification" with certificates), and the server relies on this identification. This is a separation of roles which may or may not be useful in your context. Another way of viewing it is the following: when the client connects to the server, in a login+password scenario, then the client sends its password to the server. Therefore the server learns the password. A consequence is that the client must not use the same password if it is connecting to several servers which do not trust each other. With a client certificate, the client can use the same certificate with every server.

If you do not need the extra functionalities of certificates, then a login+password is fine. Actually, if the client works in an automatic fashion without a user entry, then the "password" is stored somewhere in the client, not typed by a human. Therefore, that password can be a bunch of random bytes, which will be much more robust against exhaustive search than a human-remembered password. Under these conditions, you may want to use SSL/TLS-PSK which will build on this "pre-shared key" and avoid the need for any certificate, neither client or server.

Similarly, if the password is indeed typed or remembered by a human, consider TLS-SRP which also needs no client or server certificate, and, moreover, tolerates passwords of limited entropy.

"Therefore the server learns the password." Can't the password be hashed by the client, instead of by the server?
–
Antoine PinsardFeb 21 '13 at 14:22

If the password is hashed by the client, then the hash itself is the password: showing it to the server is sufficient to be granted access. The server still learns a "password-equivalent" data element.
–
Thomas PorninFeb 21 '13 at 15:23

If I'm interpreting your question correctly, what you're asking boils down to essentially "Why not just use passwords instead of keys for SSH".

The only thing that's typically insecure about passwords is their stupid human users. Typically password strength will be a lot lower (a LOT lower) than any key based system. This opens you up to more attack vectors such as brute force, which of course can be managed via correct password policies etc.

Short version - Nothing inheriently wrong, it's just a less secure and you have to decide if that's acceptable to you or not.