The Ubiquity and Danger of Web Scraping

Industry Leaders

The web scraping industry leaders are Screen-Scraper, Mozenda, Diffbot, and Scrapinghub. Additionally, a number of websites like Freelancer.com, Upwork and Guru.com host ads providing freelance and company web scraping services, as well as ads seeking web scraping services. The latest addition to the web scraping economy is Spinner Bot, a web scraping software that allows users to push requests across multiple proxies.

Web scraping is a software method used to extract information from websites. It often includes transforming unstructured website data into a database for analysis, or repurposing stolen content for the scraper's own online operations. Not only does web scraping pose a critical challenge to company branding, it can also threaten sales and conversions, lower SEO rankings or undermine the integrity of content that took considerable time and resources to produce.

Through analysis of top web scraping platforms and services, Distil Networks' 2016 Economics of Web Scraping Reportuncovers the ubiquity and danger of this practice. The following findings outline how the democratization of web scraping lets perpetrators effortlessly steal sensitive information on the web.