I'm a heavy vim user, but I still prefer the emacs-style key bindings for bash. If I want to do vim-style editing on the command line, I sometimes use Ctrl-X Ctrl-E to edit the line in my default editor (which is vim).
`.inputrc` is your friend. I have `v` (in normal mode) configured to open the line in an editor, because muscle memory sometimes makes me try to enter visual mode when editing a command.
I do the same. This might be overboard to some, but I went a step further and installed a vim plugin that brought many of the emacs-style key bindings into vim.
YMMV, of course.