This VM I am in is a bit old and somewhow apt-get is not pulling the latest packages... so I need it to get it manually.
As many other open source software built with make: Get a release, from http://ftp.jaist.ac.jp/pub/GNU/emacs/ for example, then compile and install.

$ wget http://ftp.jaist.ac.jp/pub/GNU/emacs/emacs-24.3.tar.gz
$ tar -xzvf emacs-24.3.tar.gz
$ cd emacs-24.3
$ less INSTALL; # Just take a glance of the install notes
$ ./configure; # Check the output is not something weird
$ make; # No errors should appear here
$ sudo make install; # Gets moved to somewhere in the PATH