To remove a template, close VS and THEN make sure the template is removed from both locations to remove the template permanently.
A better way to install a template is to use the Export Template Wizard to make the exported template an installable extension, this can be later uninstalled in the standard way.