I do all my web development ssh-ing to a Linux box. On my server the Mac ‘delete’ button worked fine in shell, but was deleting forward in Vim.
Here’s a fix!
Run a command in your terminal window:
stty -a | grep erase
and check if it says
erase = ^?;
If it is set to something else (e.g. ^H) then put the following line in both .bashrc and in either .bash_profile or /etc/profile:
if tty --quiet ; then
stty erase '^?'
fi
Good luck!
Taken from http://www.ibb.net/~anne/keyboard.html