Small Web Agents using VB - Part II

Introduction

In the previous article, we saw a simple VB application which will pull out the HTML page of a particular URL. In this article, we will build a small web crawler which will crawl through all the links in the given URL.

Setting up the Visual Basic Environment with required Components and Libraries:

Open Visual Basic and create a new project (user Standard EXE).

Select Project -> References from the main menu and add the following Microsoft Libraries:

Microsoft HTML Object Library

Add Microsoft Windows Common Controls to the toolbox as follows. Select Project -> Components from the main menu. The Components window will open. With the controls tab selected, scroll down and click the check box preceding the components:

Microsoft Windows Common Control 6.x

Set up the UI for the Crawler

Add a label, two button controls, a listbox, and a treeview control as below:

Add the code for the Crawler:

On click of the start button, populate the list box with all the links under the given URL: