I am new to vi, actually I have started learning vi from today and I have got stuck at the behavior of the backspace key. Actually when I fired up vi on my Ubuntu 12.04 for the first time my backspace key was working normally but after that it has started behaving strangely. Whenever I press the backspace in the insert mode it just moves one place to the left instead of erasing the character. How can I get back the default backspace functionality? Please note that I don't want to install vim or set nocompatibilty.

That's the traditional vi behaviour. If you've got the option to set nocompatibility, then you're already using vim.
–
Stéphane ChazelasMay 16 '13 at 19:27

I am not aware that vi is installed on Ubuntu. vi in Linux world in my experience is just a symbolic link to vim. If you want to learn real vi either install nvi or hairloom project vi.
–
Predrag PunosevacMay 18 '13 at 2:11

vim can be made to behave more like vi. This can be done by giving the command vi instead of vim from the commandline, where vi is only a sym-link to vim, in which case vim is opened in vi mode.
You can check this with :set compatible?.

As mentioned by the previous answer, the effect of the backspace becomes only visible after leaving in insert mode when set compatible is enabled.

Note:
In vi-compatible mode, you cannot backspace over text which was previously entered (before entered insert mode) or eol's or indentation in insert mode.
see :help 'bs'

'backspace' 'bs' string (default "")
global
{not in Vi}
Influences the working of <BS>, <Del>, CTRL-W and CTRL-U in Insert
mode. This is a list of items, separated by commas. Each item allows
a way to backspace over something:
value effect
indent allow backspacing over autoindent
eol allow backspacing over line breaks (join lines)
start allow backspacing over the start of insert; CTRL-W and CTRL-U
stop once at the start of insert.
When the value is empty, Vi compatible backspacing is used.
For backwards compatibility with version 5.4 and earlier:
value effect
0 same as ":set backspace=" (Vi compatible)
1 same as ":set backspace=indent,eol"
2 same as ":set backspace=indent,eol,start"

Try out the different settings to understand their meaning: Enter characters/line breaks and indentation in insert mode, exit and reenter insert mode and then try backspacing.

Users who are not familiar with vi behaviour and don't insist on using plain vi (not recommened anyway) should :set backspace=indent,eol,start. Afaik on Debian there is usually a system-wide config file installed with this setting.