If you are a Web developer, these new features of our crawler can help you to mine programming “gems” by examining, isolating, and collecting script lines that have been embedded in the source code of documents. And by crawling or accessing URLs of external files or link tags, programmers can view and dissect hidden scripts.

To learn more about these new features or previously added features, visit

Back in 03-25-11 we released the minerazzi web crawler and link checker tool. As a beta, it is not perfect and needs improvements. Actually, this is the online version of the crawler used by the minerazzi search architecture (beta). Hopefully, it will evolved into a diagnostic tool.

We mentioned that the online version will undergo several changes, all intended to provide an online “web crawler for the masses”. The idea is to put users in control of the crawling process since current crawlers lack of human intuition with regard to the next URL to crawl from a to-do list.

Since at this time we haven’t launch an official blog, this post goes…

We are excited to announce several updates to the minerazzi crawler. This is the online version of the indexing crawler used by the minerazzi search engine (beta).

The long-term goal is to turn this version into a multifunctional mining platform and a crawler for the IT masses; i.e., a crawler to be used by IR researchers, data miners, webmasters, developers, etc. That is, a crawler that even Web designers and the average public can use.