Stan's Blog

Tips and tricks on how to use Maxprog products

Web Dumper Incremental downloads

With Web Dumper you can download entire Websites off of the Internet, and save them on your hard drive for later offline browsing. The downloaded web sites are saved on your hard drive with their directory structure intact by default. Indeed, Web Dumper automatically downloads HTML documents along with their embedded pictures, sounds, movies and so on while it screens them to look for any enclosed links to other documents.

But what if you want to download sequences of documents, pictures or directories? Web Dumper supports that too. It is known as 'Incremental Download'.

Indeed, you can use Web Dumper to fetch contents from the web, not only by using an HTML index file but also directly by name! Let's see...

You have probably seen directories or files named by number or a word followed by a number like '/01/pict1.jp', '/01/pict2.jp', '/01/pict3.jp',..., '/02/pict1.jp'... etc... right? This type of structure follows a very simple pattern. With the Incremental download feature you can fetch those contents, it is just a matter of setting the starting URL properly.

Incremental download URL

The starting URL here has been set to

http://www.maxprog.com/test[1-2]/picture[001-002].jpg

See the text surrounded by brackets? They are the sequencing setting parameters.

Indeed, test[1-2] will be replaced by 'test1' and 'test2' and picture[001-002].jpg with 'picture001.jpg' and 'picture002.jpg'.

As a result Web Dumper will try to download the following files:

http://www.maxprog.com/test1/picture001.jpg

http://www.maxprog.com/test1/picture002.jpg

http://www.maxprog.com/test2/picture001.jpg

http://www.maxprog.com/test2/picture002.jpg

By setting the starting URL to http://www.maxprog.com/picture[12-15].jpg you would get instead:

http://www.maxprog.com/picture12.jpg

http://www.maxprog.com/picture13.jpg

http://www.maxprog.com/picture14.jpg

http://www.maxprog.com/picture15.jpg

The sequence tag format is simple:

- An opening bracket '['

- The starting number, with or without heading zeros

- An hyphen '-'

- The ending number, with or without heading zeros

- A closing bracket ']'

You can insert as many sequence tags inside the URL!

How does it work?

By clicking on the 'Dump' button, Web Dumper will process the sequence tags inside the starting URL. The resulting URLs are added to the download queue. Web Dumper automatically starts downloading the files. If the file is not found, an error is shown next to the bad URL. It doesn't affect the other URLs. Once finished, just click on the 'Downloads' button. You will be taken to the folder where the pictures have been downloaded.

It is recommended to use the right starting and ending numbers. A web server may deny access temporally to a client (you) after a given amount of errors.