Don’t look that it is a nice Amazon spider, it might be stealing your content!

After reading Neil Patel’s post about some content marketing tips, I’ve adjusted my .htaccess file in order to block web scrapers from Amazon Web Services (a platform that can be used for scraping websites).

Update: I use now Sucuri Website Firewall which blocks malicious traffic very effectively. If you use this product, you will not need to do such tricks like this one in my article below.

Here are the instructions for Appache to include in your website’s .htaccess file:# Blocking AWS scrapers (Amazon Web Services IP's) deny from 103.246.148.0/23 deny from 103.246.150.0/23 deny from 103.4.8.0/21 deny from 103.8.172.0/22 deny from 107.20.0.0/14 deny from 107.23.255.0/26 deny from 122.248.192.0/18 deny from 172.96.97.0/24 deny from 174.129.0.0/16 deny from 175.41.128.0/18 deny from 175.41.192.0/18 deny from 176.32.104.0/21 deny from 176.32.112.0/21 deny from 176.32.120.0/22 deny from 176.32.125.0/25 deny from 176.32.64.0/19 deny from 176.32.96.0/21 deny from 176.34.0.0/19 deny from 176.34.128.0/17 deny from 176.34.159.192/26 deny from 176.34.32.0/19 deny from 176.34.64.0/18 deny from 177.71.128.0/17 deny from 177.71.207.128/26 deny from 177.72.240.0/21 deny from 178.236.0.0/20 deny from 184.169.128.0/17 deny from 184.72.0.0/18 deny from 184.72.128.0/17 deny from 184.72.64.0/18 deny from 184.73.0.0/16 deny from 185.143.16.0/24 deny from 185.48.120.0/22 deny from 203.83.220.0/22 deny from 204.236.128.0/18 deny from 204.236.192.0/18 deny from 204.246.160.0/22 deny from 204.246.164.0/22 deny from 204.246.168.0/22 deny from 204.246.174.0/23 deny from 204.246.176.0/20 deny from 205.251.192.0/19 deny from 205.251.192.0/21 deny from 205.251.224.0/22 deny from 205.251.228.0/22 deny from 205.251.232.0/22 deny from 205.251.236.0/22 deny from 205.251.240.0/22 deny from 205.251.244.0/23 deny from 205.251.247.0/24 deny from 205.251.248.0/24 deny from 205.251.249.0/24 deny from 205.251.250.0/23 deny from 205.251.252.0/23 deny from 205.251.254.0/24 deny from 205.251.255.0/24 deny from 207.171.160.0/20 deny from 207.171.176.0/20 deny from 216.137.32.0/19 deny from 216.182.224.0/20 deny from 23.20.0.0/14 deny from 27.0.0.0/22 deny from 43.250.192.0/24 deny from 43.250.193.0/24 deny from 46.137.0.0/17 deny from 46.137.128.0/18 deny from 46.137.192.0/19 deny from 46.137.224.0/19 deny from 46.51.128.0/18 deny from 46.51.192.0/20 deny from 46.51.216.0/21 deny from 46.51.224.0/19 deny from 50.112.0.0/16 deny from 50.16.0.0/15 deny from 50.18.0.0/16 deny from 50.19.0.0/16 deny from 52.0.0.0/15 deny from 52.10.0.0/15 deny from 52.12.0.0/15 deny from 52.15.0.0/16 deny from 52.16.0.0/15 deny from 52.18.0.0/15 deny from 52.192.0.0/15 deny from 52.196.0.0/14 deny from 52.2.0.0/15 deny from 52.20.0.0/14 deny from 52.200.0.0/13 deny from 52.208.0.0/13 deny from 52.216.0.0/15 deny from 52.218.0.0/18 deny from 52.218.128.0/18 deny from 52.218.192.0/18 deny from 52.218.64.0/18 deny from 52.219.0.0/20 deny from 52.219.16.0/22 deny from 52.219.20.0/22 deny from 52.219.24.0/21 deny from 52.219.32.0/21 deny from 52.219.40.0/22 deny from 52.220.0.0/15 deny from 52.222.0.0/17 deny from 52.222.128.0/17 deny from 52.24.0.0/14 deny from 52.28.0.0/16 deny from 52.29.0.0/16 deny from 52.30.0.0/15 deny from 52.32.0.0/14 deny from 52.36.0.0/14 deny from 52.4.0.0/14 deny from 52.40.0.0/14 deny from 52.44.0.0/15 deny from 52.46.0.0/18 deny from 52.48.0.0/14 deny from 52.52.0.0/15 deny from 52.54.0.0/15 deny from 52.57.0.0/16 deny from 52.58.0.0/15 deny from 52.62.0.0/15 deny from 52.64.0.0/17 deny from 52.64.128.0/17 deny from 52.65.0.0/16 deny from 52.66.0.0/16 deny from 52.67.0.0/16 deny from 52.68.0.0/15 deny from 52.70.0.0/15 deny from 52.72.0.0/15 deny from 52.74.0.0/16 deny from 52.76.0.0/17 deny from 52.76.128.0/17 deny from 52.77.0.0/16 deny from 52.78.0.0/16 deny from 52.79.0.0/16 deny from 52.8.0.0/16 deny from 52.80.0.0/16 deny from 52.84.0.0/15 deny from 52.86.0.0/15 deny from 52.88.0.0/15 deny from 52.9.0.0/16 deny from 52.90.0.0/15 deny from 52.92.0.0/20 deny from 52.92.16.0/20 deny from 52.92.248.0/22 deny from 52.92.252.0/22 deny from 52.92.32.0/22 deny from 52.92.39.0/24 deny from 52.92.40.0/21 deny from 52.92.48.0/22 deny from 52.92.52.0/22 deny from 52.92.56.0/22 deny from 52.92.60.0/22 deny from 52.92.64.0/22 deny from 52.92.68.0/22 deny from 52.92.72.0/22 deny from 52.92.76.0/22 deny from 52.92.80.0/22 deny from 52.92.92.0/22 deny from 52.93.0.0/24 deny from 52.93.1.0/24 deny from 52.93.12.0/22 deny from 52.93.16.0/24 deny from 52.93.2.0/24 deny from 52.93.3.0/24 deny from 52.93.4.0/24 deny from 52.93.8.0/22 deny from 52.94.0.0/22 deny from 52.94.10.0/24 deny from 52.94.11.0/24 deny from 52.94.12.0/24 deny from 52.94.13.0/24 deny from 52.94.192.0/22 deny from 52.94.196.0/24 deny from 52.94.197.0/24 deny from 52.94.198.0/28 deny from 52.94.198.16/28 deny from 52.94.198.32/28 deny from 52.94.198.48/28 deny from 52.94.198.64/28 deny from 52.94.198.80/28 deny from 52.94.204.0/23 deny from 52.94.206.0/23 deny from 52.94.208.0/21 deny from 52.94.216.0/21 deny from 52.94.224.0/20 deny from 52.94.252.0/23 deny from 52.94.254.0/23 deny from 52.94.4.0/24 deny from 52.94.5.0/24 deny from 52.94.6.0/24 deny from 52.94.7.0/24 deny from 52.94.8.0/24 deny from 52.94.9.0/24 deny from 52.95.0.0/20 deny from 52.95.100.0/22 deny from 52.95.104.0/22 deny from 52.95.128.0/22 deny from 52.95.132.0/22 deny from 52.95.16.0/21 deny from 52.95.192.0/20 deny from 52.95.212.0/22 deny from 52.95.24.0/22 deny from 52.95.240.0/24 deny from 52.95.241.0/24 deny from 52.95.242.0/24 deny from 52.95.243.0/24 deny from 52.95.244.0/24 deny from 52.95.245.0/24 deny from 52.95.246.0/24 deny from 52.95.247.0/24 deny from 52.95.248.0/24 deny from 52.95.249.0/24 deny from 52.95.251.0/24 deny from 52.95.252.0/24 deny from 52.95.255.0/28 deny from 52.95.255.112/28 deny from 52.95.255.128/28 deny from 52.95.255.144/28 deny from 52.95.255.16/28 deny from 52.95.255.32/28 deny from 52.95.255.48/28 deny from 52.95.255.64/28 deny from 52.95.255.80/28 deny from 52.95.255.96/28 deny from 52.95.28.0/24 deny from 52.95.30.0/23 deny from 52.95.34.0/24 deny from 52.95.35.0/24 deny from 52.95.36.0/22 deny from 52.95.40.0/24 deny from 52.95.48.0/22 deny from 52.95.52.0/22 deny from 52.95.56.0/22 deny from 52.95.60.0/24 deny from 52.95.61.0/24 deny from 52.95.62.0/24 deny from 52.95.63.0/24 deny from 52.95.64.0/20 deny from 52.95.80.0/20 deny from 52.95.96.0/22 deny from 54.144.0.0/14 deny from 54.148.0.0/15 deny from 54.150.0.0/16 deny from 54.151.0.0/17 deny from 54.151.128.0/17 deny from 54.152.0.0/16 deny from 54.153.0.0/17 deny from 54.153.128.0/17 deny from 54.154.0.0/16 deny from 54.155.0.0/16 deny from 54.156.0.0/14 deny from 54.160.0.0/13 deny from 54.168.0.0/16 deny from 54.169.0.0/16 deny from 54.170.0.0/15 deny from 54.172.0.0/15 deny from 54.174.0.0/15 deny from 54.176.0.0/15 deny from 54.178.0.0/16 deny from 54.179.0.0/16 deny from 54.182.0.0/16 deny from 54.183.0.0/16 deny from 54.183.255.128/26 deny from 54.184.0.0/13 deny from 54.192.0.0/16 deny from 54.193.0.0/16 deny from 54.194.0.0/15 deny from 54.196.0.0/15 deny from 54.198.0.0/16 deny from 54.199.0.0/16 deny from 54.200.0.0/15 deny from 54.202.0.0/15 deny from 54.204.0.0/15 deny from 54.206.0.0/16 deny from 54.207.0.0/16 deny from 54.208.0.0/15 deny from 54.210.0.0/15 deny from 54.212.0.0/15 deny from 54.214.0.0/16 deny from 54.215.0.0/16 deny from 54.216.0.0/15 deny from 54.218.0.0/16 deny from 54.219.0.0/16 deny from 54.220.0.0/16 deny from 54.221.0.0/16 deny from 54.222.0.0/19 deny from 54.222.128.0/17 deny from 54.222.58.0/28 deny from 54.223.0.0/16 deny from 54.224.0.0/15 deny from 54.226.0.0/15 deny from 54.228.0.0/16 deny from 54.228.16.0/26 deny from 54.229.0.0/16 deny from 54.230.0.0/16 deny from 54.231.0.0/17 deny from 54.231.128.0/19 deny from 54.231.160.0/19 deny from 54.231.192.0/20 deny from 54.231.224.0/21 deny from 54.231.232.0/21 deny from 54.231.240.0/22 deny from 54.231.244.0/22 deny from 54.231.248.0/22 deny from 54.231.252.0/24 deny from 54.231.253.0/24 deny from 54.231.254.0/24 deny from 54.232.0.0/16 deny from 54.232.40.64/26 deny from 54.233.0.0/18 deny from 54.233.128.0/17 deny from 54.233.64.0/18 deny from 54.234.0.0/15 deny from 54.236.0.0/15 deny from 54.238.0.0/16 deny from 54.239.100.0/23 deny from 54.239.104.0/23 deny from 54.239.108.0/22 deny from 54.239.114.0/24 deny from 54.239.116.0/22 deny from 54.239.120.0/21 deny from 54.239.128.0/18 deny from 54.239.16.0/20 deny from 54.239.192.0/19 deny from 54.239.2.0/23 deny from 54.239.32.0/21 deny from 54.239.4.0/22 deny from 54.239.48.0/22 deny from 54.239.52.0/23 deny from 54.239.54.0/23 deny from 54.239.56.0/21 deny from 54.239.64.0/21 deny from 54.239.8.0/21 deny from 54.239.96.0/24 deny from 54.239.98.0/24 deny from 54.239.99.0/24 deny from 54.240.128.0/18 deny from 54.240.192.0/22 deny from 54.240.196.0/24 deny from 54.240.197.0/24 deny from 54.240.198.0/24 deny from 54.240.199.0/24 deny from 54.240.200.0/24 deny from 54.240.202.0/24 deny from 54.240.203.0/24 deny from 54.240.204.0/22 deny from 54.240.208.0/22 deny from 54.240.212.0/22 deny from 54.240.216.0/22 deny from 54.240.220.0/22 deny from 54.240.225.0/24 deny from 54.240.226.0/24 deny from 54.240.227.0/24 deny from 54.240.228.0/23 deny from 54.240.230.0/23 deny from 54.240.232.0/22 deny from 54.240.236.0/22 deny from 54.240.240.0/24 deny from 54.240.244.0/22 deny from 54.240.248.0/21 deny from 54.241.0.0/16 deny from 54.241.32.64/26 deny from 54.242.0.0/15 deny from 54.243.31.192/26 deny from 54.244.0.0/16 deny from 54.244.52.192/26 deny from 54.245.0.0/16 deny from 54.245.168.0/26 deny from 54.246.0.0/16 deny from 54.247.0.0/16 deny from 54.248.0.0/15 deny from 54.248.220.0/26 deny from 54.250.0.0/16 deny from 54.250.253.192/26 deny from 54.251.0.0/16 deny from 54.251.31.128/26 deny from 54.252.0.0/16 deny from 54.252.254.192/26 deny from 54.252.79.128/26 deny from 54.253.0.0/16 deny from 54.254.0.0/16 deny from 54.255.0.0/16 deny from 54.255.254.192/26 deny from 54.64.0.0/15 deny from 54.66.0.0/16 deny from 54.67.0.0/16 deny from 54.68.0.0/14 deny from 54.72.0.0/15 deny from 54.74.0.0/15 deny from 54.76.0.0/15 deny from 54.78.0.0/16 deny from 54.79.0.0/16 deny from 54.80.0.0/13 deny from 54.88.0.0/14 deny from 54.92.0.0/17 deny from 54.92.128.0/17 deny from 54.93.0.0/16 deny from 54.94.0.0/16 deny from 54.95.0.0/16 deny from 67.202.0.0/18 deny from 72.21.192.0/19 deny from 72.44.32.0/19 deny from 75.101.128.0/17 deny from 79.125.0.0/17 deny from 87.238.80.0/21 deny from 96.127.0.0/17 # End of Blocking AWS scrapers (Amazon Web Services IP's)

The source list of AWS IPs is taken from here (thanks Sid from appsecho.com for helping me update the list).

Also, AWS may be a source of hacking attempts. So it’s an additional reason to block these scrapers.

However, please keep in mind that there may be some NOT harmful services working from some of these IP ranges (e.g. CloudFront CDN). So, if you use such services, you will need to edit the IP-deny list appropriately.

If you are not sure what you do, or if you want more protection from a malicious traffic, then look at external website firewall services (I’m using Sucuri WAF (CloudProxy), which is comparatively affordable).

I hope you enjoyed the article! You can read my free researches on resources and tools for bloggers and small business owners on this website. By the way, if we haven't met before - my name is Michael Bely.

Comments

Hey Michael, thanks for sharing these IPs that we should block in order to host a more powerful and working webpage focusing upon the basic business. I also read Neil Patel’s blog but couldn’t see that specific post in past about what you were talking in this update.

> So basically it’s people using AWS services to scrape your content and use it, not actually Amazon, right? Absolutely.

> Have you found it made a big difference (In terms of saving bandwidth etc)? As regards saving bandwidth, there are just 3 weeks have passed after I implemented AWS blocking (not much time to judge), but anyway here are the results:

– I’ve compared the period of 22 days from AWS blocking implementation till now (i.e. from the 24th of Jul till the 14th of Aug) with the period of 22 days before the implementation (from the 2nd of Jul till the 23th of Jul). – I’ve got 83% more visitors (with the same pageviews per visitor stats) [according to Google Analytics] – I’ve got just 30% increase of bandwidth. [according to AWStats] Thus, roughly, I’ve saved 40% of bandwidth ((1.83-1.3)/1.3 = 40%).

However, I believe, that websites with a different popularity can get a different bandwidth savings.