Can you tell me how you do it ?
Without keys I do this :
- open Dolphin
- click on the "address-bar".
- enter "fish://user@server".
- if not saved in the wallet, I get a messagebox where I can enter the
password.
- access files

How do you specify the key to use ?

I configured the ssh server to accept passwordless connections. This
requires setting up a private/public key pair and then adding the public
key to /.ssh/authorized_keys on the "server".