Having fun web crawling with phantomJs

CodeProjectA couple of weeks ago, a colleague of mine showed me this cool tool called phantomJs.This is a headless browser, that can receive javascript to do almost anything you would want from a regular browser, just without rendering anything to the screen.This could be really useful for tasks lik

A couple of weeks ago, a colleague of mine showed me this cool tool called phantomJs.This is a headless browser, that can receive javascript to do almost anything you would want from a regular browser, just without rendering anything to the screen.

This could be really useful for tasks like running ui tests on a project you created, or crawling a set of web pages looking for something.

...So, this is exactly what i did!

There's a great site I know of that has a ton of great ebooks ready to download, but the problem is that they show you only 2 results on each page, and the search never finds anything!

Realizing that this site has a very simple url structure (e.g.: website/page/#), I just created a quick javascript file, telling phantomjs to go through the first 50 pages and search for a list of keywords that interest me. If i find something interesting, it saves the name of the book along with the page link into a text file so i can download them all later.