There are starting to be some good Ruby libraries for screen-scraping,
too.

There's a simple but good version of WWW::Mechanize (you can find it
via the 'gems' Ruby library if you have that installed). And REXML is
a fantastic XML parsing library, with XPath built in so you don't have
to do so much procedure stuff as you do with some of the Perl modules.

This won't help much with the Javascript mess, though. (And yes, I've
found similar awful cruft in dealing with scraping financial services
sites. I think it must be output of some middle-ware app that folks
use to make web sites. I had to deal with one recently that had *no*
way of navigating via regular HTML; only Javascript links! Truly
annoying.)