[Joomla] robots.txt file update as of version 3.4

If you've recently updated to Joomla 3.4 you might have noticed this message:

A change to the default robots.txt files was made in Joomla! 3.3 to allow Google to access templates and media files by default to improve SEO. This change is not applied automatically on upgrades and users are recommended to review the changes in the robots.txt.dist file and implement these change in their own robots.txt file.

So why was there a change? But first:

Why is there a robots.txt file with every Joomla distribution?

The robots.txt file indicates which parts of your site you don't want to be accessed by a search robot (also known as crawlers). The file is part of the Robots exclusion standard and most search engines follow it. The file, for example, lets the crawler know that it is allowed to visit folder xyz, but it isn't allowed to view the resources in folder abc. The protocol is purely advisory - it won't stop bots that don't adhere to that standard from indexing those resources.