Instead of using set -o vi you add 'set editing-mode vi' to you $HOME/.inputrc file you have vi-keybinding-loveless in all applications that support readline. You can also set this to system wide by added this to /etc/inputrc.