This changes the "discourage search engines" option to output a `noindex, nofollow` robots meta tag. `Disallow: /` is removed from the `robots.txt` to allow search engines to discover they are requested not to index the site.
Disallowing search engines from accessing a site in the `robots.txt` file can result in search engines listing a site with a fragment (a listing without content).
Props donmhico, jonoaldersonwp.
Fixes #43590.
git-svn-id: https://develop.svn.wordpress.org/trunk@45928 602fd350-edb4-49c9-b593-d223f7449a82