If this is your first visit, be sure to
check out the FAQ by clicking the
link above. You may have to register
before you can post: click the register link above to proceed. To start viewing messages,
select the forum that you want to visit from the selection below.

Enjoy an ad free experience by logging in. Not a member yet? Register.

Imitating a browser on a server

We're trying to access a URL with a script (currently PHP). Literally every browser is able to access this URL without trouble, but they are trying to protect themselves from spiders and bots, and when we try to read the URL from our script we get "Bad User Agent". The people who own the URL don't seem to be able to figure out how to get us listed as "good user agents"- they are using code provided by an outside vendor and are not the most technically acute people around.

It's critical that we work around this problem. One approach we're batting around is to find a way to mimic a browser and just read the contents of the page we need to read through a browser window. Can anybody suggest where we might find some example code that would show us how to do this?

I'd recommend using PHP's cURL Library (A lot of great documentation and user-examples available @ php.net). You will need to "spoof" your user agent to look like Internet Explorer or Firefox (or any supported browser)