On Wed, Oct 27, 2010 at 10:31 AM, Troy Wical <troy@wical.com> wrote:
> > Warning: Unknown header line: 'ath-Name:
> http://type2.com/ezmlm-archives/index.cgi?list=type2&cmd=monthbydate&month=201009'
> from program spider.pl
> > err: External program failed to return required headers Path-Name:
> Regarding the "Unknown header line" error, I'm having a heck of a time
> finding anything related to that. Full debugging has been activated, and
> I've gone through and tried to look at the previous URL's that may be
> throwing it off, but no luck yet. Maybe a break in the garage, playing
> mechanic, will help me come back refreshed.
>
I have not looked at that code in, well, years. Swish *should be working
with bytes, so my guess is that the spider is telling swish that the content
is one byte longer than it really is.
http://dev.swish-e.org/browser/swish-e/trunk/prog-bin/spider.pl.in#L1409
# Re-encode the data for outside of Perl
1407 eval {
1408 # Need to only require Encode here?
1409 $$content = Encode::encode( $server->{charset}, $$content )
1410 if $server->{charset};
1411 };
1412 if ( $@ ) {
1413 print STDERR "Warning: document '", $response->request->uri, "'
could not be encoded to charset '$server->{charset}'\n";
1414 delete $server->{charset};
1415 }
$content should now be a reference to a string of bytes.
1416
1417 $server->{counts}{'Total Bytes'} += length $$content;
1418 $server->{counts}{'Total Docs'}++;
1419
1420
1421 # ugly and maybe expensive, but perhaps more portable than "use
bytes"
1422 my $bytecount = length pack 'C0a*', $$content;
1423
This is a wild guess, but what if you replace that with:
my $bytecount = length $$content;
It's probably the same but that's how I would get the length from a string
of bytes.
The other thing, if you really want to battle this, is to output the spider
to a file and then use an editor and try and figure out the length
difference -- or maybe just add an extra space character before the
Path-Name line where it's failing and then feed that to swish.
--
Bill Moseley
moseley@hank.org
_______________________________________________
Users mailing list
Users@lists.swish-e.org
http://lists.swish-e.org/listinfo/users