A real problem with screen scraping has developed at findU over the
last couple days. The program Weather Display added a feature that
allows one to display the weather in real time of any station on
APRS. Unfortunately, the author implemented this by hitting findU's
rawwx.cgi every second, for each user of this feature. He has agreed
to remove it from future versions and parse the data directly, but
the code is in the wild.
Yesterday I wrote an automated routine to identify screen scrapers
and block their IP automatically, but that ran afoul of an authorized
user with dynamic IP. I will periodically be running detecting code
manually and locking out IP addresses which are screen scraping
without any warning. The prohibition on scraping is not new, but
because my time has been extremely limited due to hurricane recovery
and configuring the new server, I have not been policing as
carefully as I should have. The situation reached a critical point in
the last few days when the added load of end-of-year maintenance
pushed the server over the brink, resulting in loos of data being fed
to the National Weather Service. With the screen scrapers shut down I
will have enough capacity. If you use a script manually to grab lat/
lon or some other data it will not trigger the code searching the
log, you will be noticed though if you hit the server on an automated
basis to grab data rather than parsing it yourself.
If you are screen-scraping, and really believe this the only way you
can get the data, explain your situation to me.
Steve K4HG