Crawler for Brandwatch.com.,a social media monitoring company. The bot will crawl blogs, forums, news sites, and all kinds of social media content. The content is indexed, much like a search engine, allowing our users to find the pages that mention the words they are interested in.

Below are some methods to control the access that this bot (title) has to your site and or pages. The methods below may not work if the bot does not pay attention to the limits you have established.
Select a method: Robots.txt, Meta Tags, htaccess,

Using Robots.txt:

A Robots.txt file is placed in the root of your website. Good bots will first look for and review the robots.txt file before either continuing on to other pages on your site or leaving if they are not allowed. For more information about robots.txt files, visit robotstxt.org.

Do not allow any bot (user-agent) to access any part of your siteUser-agent: *
Disallow: /

Allow any bot (user-agent) to access any part of your siteUser-agent: *
Disallow:

Do not allow magpie-crawler to access any part of your siteUser-agent: magpie-crawler
Disallow: /

Allow magpie-crawler to access any part of your siteUser-agent: magpie-crawler
Disallow:

Allow magpie-crawler to access your site, but magpie-crawler is not allowed to access the "admin" folderUser-agent: magpie-crawler
Disallow: /admin

Allow magpie-crawler to access your site, but magpie-crawler is not allowed to access the "admin" folder and the "photos" folderUser-agent: magpie-crawler
Disallow: /admin
Disallow: /photos

Using Meta Tags:

You can use meta tags in your pages to help control the access bots have to your site. If you use a template for all your pages, you can add the meta tags in between the
<head> and
</head> and it will work on all the pages using that template. If you want to control specific pages, you can add the meta tags on individual pages in between the
<head> and
</head> instead.

Do not allow magpie-crawler to access your page(s)<meta name="magpie-crawler" content="noindex">

Allow magpie-crawler to access your page(s) and follow the links to more pages<meta name="magpie-crawler" content="index, follow">

Using HTACCESS:

You may have the ability to add and or modify an htaccess file on your server. The htaccess file can be used to control the bots at the server level. Add the following to the .htaccess file on your server to block specific bots from visiting your site. Be sure to replace the Enter User Agent with the user-agents for the bots you would like to block simliar to the user-agent (magpie-crawler) listed in the example below.