Web crawling

At times, undisclosed third parties attempt to automatically crawl our web pages. Web crawling may be done for several purposes, including to gather intelligence (such as search results) with the purpose of delivering information to an application.

Automated Web crawling and robotic activities from third parties are generally disallowed across our properties. Interested parties should utilize the Bing Search API to access multiple supported source types. Customers can sign up to use the Bing API on Windows Azure Marketplace. Bing offers two options:

Non-compliance with Microsoft policies and guidelines will result in immediate blocks and further actions from Microsoft.

Under all circumstances, malicious activities are strictly disallowed. Malicious activities include, but are not limited to: those intended to game, defraud or deceive Microsoft or any of its subsidiaries, partners, customers, affiliates or end users (for example by manipulating click measurement activity or click-based advertising payments); attempts not to be identified; and non-compliance with applicable regulations, industry best practices, corporate policies, etc.

Individual exceptions may be available under limited circumstances, entirely within the discretion of Microsoft and revocable at any time without prior notice. Please contact TQException@microsoft.com to submit an exception request.

If any such exceptions are granted, the following guidelines must be heeded:

Identification. Web crawling activities will need to be disclosed to Microsoft and identified through appending a value to the User Agent (“UA”). This value must be communicated to Microsoft.

Sponsored listings. Clicking on any ads is strictly disallowed.

Traffic thresholds. Traffic limits may be communicated by Microsoft.

Data usage.

Information gathered as a part of the crawling activities must not be used for fraudulent or other malicious purposes, as described above, or in any way that may cause harm to Microsoft or any of its subsidiaries, partners, customers, affiliates or end users.

The collection of individual end-user personal information, including but not limited to internet protocol (“IP”) address, click activity and other browsing behaviors, browser and search history, etc., is strictly prohibited.

Microsoft is a member of the Internet Advertising Bureau (“IAB”), and we are committed to a high standard of industry engagement and best practices. This includes honoring the IAB’s Spiders & Bots Best Practices.