Why not become a lifetime supporting member of the site with a one-time donation of any amount? Your donation entitles you to a ton of additional benefits, including access to exclusive discounts and downloads, the ability to enter monthly free software drawings, and a single non-expiring license key for all of our programs.

You must sign up here before you can post and access some areas of the site. Registration is totally free and confidential.

first, I am talking about webpages that have 1,2,3,next etc links (like google results)I need a 'bot' that will click 'next' in a webpage, go to the next webpage, save it and then click next, go to the next, save it, etc

I think you can use httrack for that. Just pass it the page's address and configure it to download all the pages linked by that one at a depth of '1'.If the page has other links not related to the search, they will be downloaded too, but I suppose you could delete those manually or something. I think httrack can ignore domains, so if those other pages are all in the same domain (the domain of the original page), you could just ignore that one and you'd get only the interesting pages

the problem is that I need to do this inside the web browser, because the website needs authentication, which is not easy to achieve in webpage offline downloaders (it is not webpages in http://user:pass@domain.com format, but it requires web form authentication)

the problem is that I need to do this inside the web browser, because the website needs authentication, which is not easy to achieve in webpage offline downloaders (it is not webpages in http://user:pass@domain.com format, but it requires web form authentication)

Sounds like a job for either GreaseMonkey, AutoIt and AutoHK but unless you're willing to provide some details I don't think anyone will be able to help:

eg.

GreaseMonkey - you need to provide access to the site so as to be able to create a userscript to do the actions you want.AutoIt/AutoHK - you might get away with providing a screenshot of the site so as to give reference to mouse movement/actions and/or key input.

I think these are the most likely automated options barring a dedicated program.

If the website is using a form for verification then it most likely sets a cookie and you could use a website downloader that can use the cookie.

Try FireFox with DownThemAll! - it can supposedly download all links on a page.

« Last Edit: August 28, 2008, 08:02:07 PM by 4wd »

Logged

I do not need to control my anger ... people just need to stop pissing me off!

what I want to do is to save the google results webpage (the one with the numbers at the bottom), then click to go to the next google results webpage, save, go to next, save, etc (in other words I need to save all the webpages of the above mentioned urls)

all the above must be done within the web browser, because the website needs me to first authenticate via a web form

I'd think you would have to use Foxmarks to sync your bookmarks.Then go to your Foxmarks web site where all your links are and work with them from there.Of course you need Firefox also which I guess you have.

Would the addon, 'Download Them All', work?

Or you can use a download manager and the addon 'Copy all Links'.

Copy and paste them into the manager, which ever one is built into Firefox, and there are a few. Which one to choose would depend on it's options that you need.

If the sequential pages have some sort of numbering rule in their URL (most do, I think), then you could copy the starting URL, duplicate it as many times as required in an editor, change the numbering as required for each URL (with 400+ items, I would probably do this step in Excel or something similar), and ask Scrapbook to down them all in a folder.

If you use the first method I mentioned (giving Scrapbook a list of URLs to save), it saves the web pages in the background, meaning it doesn't load the pages into Firefox. There's a small pop up showing the progress:

It saves one page at a time, with a small delay (a couple of seconds) in between, so it won't overwhelm the server. You may safely ignore the progress dialog (which would take some time if you give it a long list) and continue to use Firefox.

When it's done, the progress dialog goes away and another small message box pup up from the lower-right corner telling you "capture completed".