Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

http://scrapy.org/

## What is scrapy_model ?

It is just a helper to create scrapers using the Scrapy Selectors allowing you to select elements by CSS or by XPATH and structuring your scraper via Models (just like an ORM model) and plugable to an ORM model via ``populate`` method.

Import the BaseFetcherModel, CSSField or XPathField (you can use both)

```pythonfrom scrapy_model import BaseFetcherModel, CSSField```

Go to a webpage you want to scrap and use chrome dev tools or firebug to figure out the css paths then considering you want to get the following fragment from some page.

def post_parse(self): # executed after all parsers # you can load any data on to self._data # access self._data and self._fields for current data # self.selector contains original page # self.fetch() returns original html self._data.url = self.url

class DummyModel(object): """ For tests only, it can be a model in your database ORM """