REPLACE all of the links instead of the first one....
----- Original Message -----
From: "Steve VanSlyck" <s.vanslyck at spamcop.net>
To: "Central OH Linux User Group - 432xx" <colug-432 at colug.net>
Date: Fri, 17 Dec 2010 17:48:36 -0500
Subject: Re: [colug-432] Data mining
> Thanks Rich. Here's domething odd.
>> When I open the origninal HTML file which worked, and release all of the
> links except the first one which had a session ID that displayed in
> Firefox (unless I've gone mad again), THAT file works!
>> Creating a new one does not, but using the original does.
>> Does this reveal anything useful?
>> ----- Original Message -----
> From: Richard Hornsby <richardjhornsby at gmail.com>
> To: Central OH Linux User Group - 432xx <colug-432 at colug.net>
> Date: Fri, 17 Dec 2010 15:04:27 -0600
> Subject: Re: [colug-432] Data mining
>> >
> > This type of web scraping was a major part of my job for about 5 years,
> but I'm missing something also -- they're doing something weird and I
> haven't quite nailed down what yet. Going directly to the search results
> page using GET variables works fine, no need to post the form.
> >
> >
>http://franklincountyoh.metacama.com:80/do/searchByParcelId?taxDistrict=010&parcelNbr=008436
> >
> > On that page is what appears to be a standard HTML link to the taxinfo
> page -- but the target TAXINFO script is looking for something - the
> "stateful" things I can think of are: a sessionid of some sort in the
> URL, a cookie, or a referrer. I've tried all three and none are working.
> The other thing I thought was that it was a wget UA problem, but that
> wasn't it either.
> >
> > wget
>>--referer="http://franklincountyoh.metacama.com:80/do/searchByParcelId?taxDistrict=010&parcelNbr=008436"
> -U "Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-HK) AppleWebKit/533.18.1
> (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5" --load-cookies cookies
>>"http://franklincountyoh.metacama.com/do/selectDisplay?select=TAXINFO&curpid=01000843600"
> >
> >
> > at the moment I'm stumped and have to be missing something I should
know
> about. I have to run to an event practice in a bit, but I'll keep
> messing with it. There is a way -- we just haven't found it yet.
> >
> >
> >
> > On Dec 17, 2010, at 13:25 , Steve VanSlyck wrote:
> >
> > > If you go to
> > >
> > > http://franklincountyoh.metacama.com/do/searchByParcelId> > >
> > > then enter, for example, parcel no. 010-008436
> > >
> > > you are taken to
> > >
> > >
>>http://franklincountyoh.metacama.com/do/searchByParcelId;jsessionid=2C5A034D6CE2B1FF816F095759EF5347
> > >
> > > (which contains what I assume to be a session ID).
> > >
> > > If you then click on the link labeled "Tax/Payment Info" you are
taken
> to
> > >
> > >
>>http://franklincountyoh.metacama.com/do/selectDisplay?select=TAXINFO&curpid=01000843600
> > > You can easily print that page to a PDF file if you like.
> > >
> > > So all that works fine.
> > >
> > > Yesterday, I was able to grab a large number of TAXINFO pages by
> > > (a) creating an HTML page on my desktop with a link to each of the
> parcels
> > > I was interested in, and
> > > (b) created a PDF file using Acrobat to grab the page and one level
> down).
> > >
> > > It didn't work at first, so I then
> > >
> > > (a) opened the page, grabbed the session ID link,
> > > (b) left the page open,
> > > (c) pasted the link into my local HTML page,
> > > (d) and then created the PDF file from Acrobat as before.
> > >
> > > It worked like a dream. Well, last night anyway.
> > >
> > > Today, however, it will not work at all. I've tried using the same
> session
> > > ID, using a fresh session ID, opening Acrobat first, opening Firefox
> (or
> > > IE first) and so on.
> > >
> > > I don't know enough of what's going on under the hood, however, to
> recreate
> > > my earlier success. I don't even know if the need for a session ID
was
> > > the problem or what I did (whatever it was) that made it go.
> > >
> > > Are there any Internet experts here that can help? I really don't
want
> to
> > > download 200 pages manually.
> > >
> > >
> > > _______________________________________________
> > > colug-432 mailing list
> > > colug-432 at colug.net> > > http://lists.colug.net/mailman/listinfo/colug-432> >
> >
> > _______________________________________________
> > colug-432 mailing list
> > colug-432 at colug.net> > http://lists.colug.net/mailman/listinfo/colug-432> >
> _______________________________________________
> colug-432 mailing list
>colug-432 at colug.net>http://lists.colug.net/mailman/listinfo/colug-432>