Hmmm, I dunno, I never looked up much of anything about making web crawlers. I know it can be done it php, as I am pretty sure I have seen it mentioned somewhere before but other than that, I don't think I can help much.

Off the top of my head it sounds like you need to go to a page and read the HTML in the page - assuming the page has HTML. I do know that you can read HTML with php, in particular you could potentially find all links (that are not javascript links) using PHP DOM. Beyond that, this tutorial might help some.