The SitePoint Forums have moved.

You can now find them here.
This forum is now closed to new posts, but you can browse existing content.
You can find out more information about the move and how to open a new account (if necessary) here.
If you get stuck you can get support by emailing forums@sitepoint.com

If this is your first visit, be sure to
check out the FAQ by clicking the
link above. You may have to register
before you can post: click the register link above to proceed. To start viewing messages,
select the forum that you want to visit from the selection below.

ASP site crawler. How to make one?

Hi,
I am intereted in understanding how does a crawler works. By crawler I mean something like the some search engines use to fetch automatically sites on the net. What is the method an ASP script like that can look up the content of a site?

Its not that hard really. I've made the equivalent of a crawler in PHP.

One key element though is regular expressions, so if you are unfamiliar with them become familiar. (on a side note I don't know if ASP includes regular expressions out of the box or if they are something you'd need at add in)