GTrends SE 1.6 has a new feature that, when used in conjunction with the TOR proxy, makes it much easier to get around Google bans. This tutorial will explain how to enable that feature.

First, set TOR’s authentication method to “Password”. If you have installed TOR as part of the Vidalia bundle you can find this option in the Advanced tab of the Settings window. Choose any password you like and make sure you memorize it. Note : don’t tick the box that says Randomly Generate (if you do the program won’t tell you what password it generated).

You need to restart TOR for the changes to take effect. In Vidalia you can do this by first clicking “Stop Tor”, followed by “Start Tor”.

Now all that is left is configuring GTrends SE to enable automatic identity switching. Head to Settings/Proxies and tick “Automatically switch to new identity when needed”. Enter the TOR password in the appropriate box. Also make sure that “Use a proxy” is checked and TOR’s proxy address (usually 127.0.0.1:8118) is in the proxy list. Finally, uncheck “Automatically pause if getting Forbidden results” in Settings/Speed.

Click OK. You’re set 🙂

Notes

I recommend removing all other proxies (except TOR) from the proxy list when enabling identity switching.

GTrends SE expects the TOR control port to be at 127.0.0.1:9051. At this time the address is hardcoded and can’t be changed. Automatic identity switching will not work if TOR is not running on localhost and/or the control port number is different.

If an identity doesn’t work , GTrends SE will wait 15 seconds before trying again and switching to a new one. This delay was added to account for slower network connections where it takes a while to get a new identity.

You can probably disable “Play sound if blocked by Google” now (if you wish).

The script is fairly basic, but it might be useful for those interested in a web-based, free alternative to GTrends SE. The PHP script uses a slightly different algorithm (the results might be unexpected…) and requires a Google account. You will need some basic webserver administration/PHP skills to set it up.

In the beginning, GTrends SE was intended for those who participated with TDC 2007. So I didn’t describe the algorithm here, as TDC’ers would already be familiar with it. However, now I’ve gotten some inquiries from new users as to how, exactly, is the number in the “Searches” column calculated. So here’s the algorithm, in short.

The search volume is calculated by comparing it to the reference keyword (Settings -> Criteria -> Reference keyword). The program sends a query with both keywords to Google Trends and calculates the searches based on the ratio of the reference/unknown keyword bar graphs. The keyword phrase that needs to be checked is wrapped in “quotes”.

There’s one curious implication this algorithm has – for keywords that have very small search volume, it will always report the exact same (small) number of searches. The exact figure produced depends on the reference keyword and how many searches it has.

You get those similar results because Google Trends shows 1-pixel graph bars for those keywords (keyword_bar_size = 1). Trends will even show those 1-pixel bars for keywords that don’t get any searches at all. Obviously, plugging this value (1) into the above equation, you will always get the same result.

So if you notice lots of keywords with the same small number in the “Searches”, blame Google Trends, okay? 😉

Google has banned the default reference keyword used by GTrends SE from Google Trends. Damn.

Therefore, I just uploaded a new version, which uses a different keyword.

Note that if you already have GTrends SE installed, downloading the new version will not automatically change your ref. keyword (this is by design). You need to change it manually by clicking “Settings” and entering a keyword and searches/day in the “Reference keyword” panel. I recommend you find your own baseline keyphrase, but, if you really don’t have a keyword to use, ask me (I won’t put the keyphrase in public here so that it doesn’t get abused).

My hosting provider is having some technical issues with email, so I’m afraid I won’t be able to use the newsletter to notify people about this change.

Some GTrends SE users have emailed me asking why they sometimes get “Forbidden” results and how to get rid of them. In a nutshell – you will get a “Forbidden” result when Google decides to ban your IP address (temporarily) because they detect that you are using an automated tool to search Google.com. GTrends SE needs to do Google searches to determine the “Competition” numbers.

In this tutorial I’ll explain how to prevent this block. Google’s proprietary detection algorithms are hard to fool, but the tricks I’ll mention below should be effective for most users.

Use longer Google grace period

The “grace period” is how long the application will wait between consecutive requests. Longer grace period = less likely to be blocked. 12 to 16 seconds work well in most cases. The setting is found under Settings -&gt Speed. Note that you can safely set the Google Trends grace period to a lower value (e.g. 2 seconds) to compensate for any loss in speed.

Enable “Always check searches first, competition second”.

The idea is to decrease the number of searches that GTrends SE needs to do by quickly eliminating worthless keywords. This helps decrease the time needed to process a list of keywords and decreases the chances of getting a Google ban.

You can find this feature under Settings -&gt Criteria. Make sure the “Enable this feature” box is checked, set “Action” to “Don’t check the keywords any further” and set the “Less than X searches per day” value as high as you can afford to. For example, if you set it to “70 searches per day”, keywords that have less than 70 searches/day will not be fully checked – GTrends SE will stop analyzing those keywords as soon as the number of searches is determined.

There are no explicit rules on what numbers you should set in the “Negative threshold” section – choose what makes sense for the niche you’re researching.

Get a good proxy

Using an anonymous proxy is a very effective way to avoid being blocked by Google (though you need to switch the proxies now and then so that the proxies themselves don’t get banned). The real challenge is finding a good proxy – most of the freely available ones are of dubious quality and may have been already banned, even before you try to use them. I’ve already written a few posts on this topic :

While a few users have had success with commercial, subscription-based proxy providers, I’m reluctant to recommend a specific site. A few Google searches will likely turn up a number of proxy services – e.g. Socksify – use at your own risk.