As I have been dealing more and more with web page related security I have found the need for a tool that can crawl a web page and search for potential malicious data.
Many of you might already know urlquery.net and the services they provide. It is a great site which loads your web page and tries to find malicious content which might not be visible to the end user.
It does this by loading the web page in firefox, and passing the traffic through Snort, and a couple of other analysis engines. It will also create a graph over the HTTP requests as well as a and overview over the HTTP headers.

Having a real Firefox, with real plugins loading your site proves to be very useful as more and more malware is getting quite sneaky, and will only serve malware to a very limited amount of visitors to avoid detection.

All these features is gold for anyone working with web page security (eg. a hosting provider).

The only problem for me is that urlquery.net isnt open source, which means that every test I do will become public, which I might not be ready for.

I looked into Thug, but lacking a web interface/report viewing interface, I did not find it perfect enough for my use.
I had therefore to create something on my own.
This has resulted in FjoSpidie, a now Open Source Spider/Honey Client.

This spider runs Firefox through Selenium and records the traffic with tcpdump which is passed through Snort to search for known, malicious content.