Scrapinghub Support Center

How can we help you today?

Specific site using http 404 to circumvent ban detection

x

xingzhouliu

started a topic
about 1 year ago

I've recently seen some important sites returning http 404's instead of other codes to ban ip's. This behavior appears only when I use crawlera, not other proxies or ips using the same headers, randomized intervals, etc., and circumvents crawlera's ban detection.

Anyone else run into this, and are there any possible fixes down the road?

Best Answer

n

nestor
said
about 1 year ago

I've added a ban rule to handle this cases of 404s so that Crawlera will retry the request with a different IP if it receives this response.