20 February 2010

In the context of something completely unrelated, I was looking for a fairly general pattern in the Google 1TB corpus. In particular, I was looking for verbs that are sort of transitive. I did a quick grep for 5grams of the form "the SOMETHING BLAHed the SOMETHING." Or, more specifically:

grep -i '^the [a-z][a-z]* [a-z][a-z]*ed the [a-z]*'

I then took these, lower cased them, and then merged the counts. Here are the top 25, sorted and with counts:

1 101500 the surveyor observed the use 2 30619 the rivals shattered the farm 3 27999 the link entitled the names 4 22928 the trolls ambushed the dwarfs 5 22843 the dwarfs ambushed the trolls 6 21427 the poet wicked the woman 7 15644 the software helped the learning 8 13481 the commission released the section 9 12273 the mayor declared the motion 10 11046 the player finished the year 11 10809 the chicken crossed the road 12 8968 the court denied the motion 13 8198 the president declared the bill 14 7890 the board approved the following 15 7848 the bill passed the house 16 7373 the fat feed the muscle 17 7362 the report presented the findings 18 7115 the committee considered the report 19 6956 the respondent registered the domain 20 6923 the chairman declared the motion 21 6767 the court rejected the argument 22 6307 the court instructed the jury 23 5962 the complaint satisfied the formal 24 5688 the lord blessed the sabbath 25 5486 the bill passed the senate

What the heck?! First of all, the first one is shocking, but maybe you could convince me. How about numbers 4 and 5? "The trolls ambushed the dwarfs" (and vice versa)? These things are the fourth and fifth most common five grams matching my pattern on the web? "The poet wicked the woman"? What does "wicked" even mean? And yet these all beat out "The bill passed the house" and "The court instructed the jury". But then #23: "The prince compiled the Mishna"??? (#30 is also funny: "the matrix reloaded the matrix" is an amusing segmentation issue.)

If we do a vanilla google search for the counts of some of these, we get:

1 10900 the surveyor observed the use 4 7750 the trolls ambushed the dwarfs 5 7190 the dwarfs ambushed the trolls 6 ZERO! the poet wicked the woman 15 20200000 the bill passed the house 22 3600000 the court instructed the jury

This just flabbergasts me. I'm told that lots of people have expressed worries over the Google 1TB corpus, but have never actually heard anything myself... And never seen anything myself.

Does anyone have an explanation for these effects? How can I expect to get anything done with such ridiculous data!

I've heard of similar discoveries of problems in this data set (pretty informal, conversations at GALE meetings, NIST workshops).

Another piece of evidence that says they may be significantly screwed up is that no one is using them in MT, at least, no one "major" finds them to be indispensable. There's usually a bit of hedging (e.g. things like "we had to do some stupid pruning, so the fact we didn't see significant gains can be attributed to that"). I'm quite sure that if these were really reasonable 5-gram counts, someone in the MT rat race would have found a way to use them.

I think the algorithm google use to collect the corpus is buggy. Only human verified corpus could be perfect but for extremely large corpus it is too difficult to verify. Only solution could be to optimize the algo further.

Very revealing post, thanks! (who knows how many weeks of useless debugging I haven't wasted trying to debug algorithms because of assuming the corpus was right! Believing in the data is a scientist's classic mistake ;-)

I think there was too much expectation regarding Google's corpus. Until now, I have no compelling reasons to consider their research particularly impressive, and I have no proof that they care or are even aware of how to build a corpus, either. Actually, Google is known to be generally dismissive of NLP and to rely on ever-increasing amounts of data to build their applications. That's an excellent foundation, but clearly has some unaddressed problems.

As for the comment about using this corpus for MT, I am not sure that would constitute a promising approach. The most obvious use of corpora in MT involves parallel corpora, and I wouldn't expect a corpus which fails to fulfill minimum quality standards to have been parallelized.

I agree with the commenter who says Google's scraper may be buggy. If you are not familiar with it, I strongly recommend you to look for information on the WaCky corpus, built by NLP professionals with a methodology (for a change ;-) and much more reliably:

it's worth keeping in mind that there is no such thing as a uniform or even representative sample of the web. the sample space of "the web" isn't defined since there are an infinite number of autogenerated pages, pages you can't always get to, and the like. For example, maybe the scraper found a game website and kept clicking the "attack the troll" button and generated 20,000 pages of a dwarf attacking a troll. or whatever.

i feel like that people hope that "the web" is a big textual corpus that's fairly representative of typical ways to use language, but no one has ever done a real investigation to find out under what circumstances this is true.

from what i've heard, web page deduplication is a really hard problem but also a really critical one for this. maybe whoever did the scrape used a crappy deduplication scheme -- that might explain the dwarf/troll thing, for example.

Brendan, Tom: Yes, I think it's clear that the ngram counts != Google counts. That's fine with me. Surely everyone (I hope) knows that there are dedup issues for anything on the web. But even taking that stuff into account, I can't reconcile in my head the differences, especially the 20k that turned into a zero!

I guess what I'm saying is that I would expect some sort of vague correlation between the counts. Obviously I don't expect them to be the same, or even on the same order of magnitude.

20k could turn into zero if they all came from the same set of pages that got filtered out later. (e.g. a spam site with fake content designed to manipulate pagerank).

Aside from those cases, I think that overall the counts will still have correlation to the "actual google counts" though. If you found the actual google counts for the 5-gram results ranked #1-100, #1001-1100, #5001-5100, etc, I think that the higher ranked sets will have higher average actual google counts than the lower ranked sets.

Those 20K+ occurrences of trolls/dwarfs in the n-grams, like the 7K hits on the Web, are basically all from the same sentence (repeated over and over on the Web).

I wouldn't say the Google n-gram counts are bad -- the repetition of that sentence, like the other strange ones, is a legit phenomenon on the Web. However, I completely agree that for some NLP applications, the counts might not be the best ones.

It's often more informative to count each unique sentence on the Web only once. If you do that, even over a billion Web pages, you count "the trolls ambushed the dwarfs" only a handful of times (e.g., I got 5 occurrences of that when searching for "ambushed" as a predicate in TextRunner, which btw might be a better source of counts for your needs here).

The thing with the dwarfs makes some sense at least. It appears to me that this is due to a very common issue in creating corpora from the web: duplicates and near-duplicates. The web is full of copy-paste text. These duplicate snippets are not always easy to spot, there are several algorithms but they tend to fail if the copied text snippet is short. Googling reveals that the dwarf-thing actually comes from a review of a book by Terry Pratchett. Quite naturally, every online book store will have the standard review written by the publisher in the article description.

Google hit counts can change very quickly. As pointed out in other comments, Google may just block a site from being indexed, or the Terry Pratchett book might run out of stock world wide and disappear from the shops.

Maybe there is an easier and less suspicious explanation: every top-ranked 5-gram will almost by definition be an outlier. The chance of observing a random sequence of 5 words is very, very small. Of course language is not random, but also the chance of observing a sequence of 5 words actually used in language is still very small. Thus you could predict in advance that the top-ranked 5-grams will not be "standard" language. This however holds for every corpus.

sorry to bring in a bit of ethnic knowledge, but "the Prince compiled the Mishna" probably refers to a well-known event in the history of Judaism when rabbi Juhuda HaNasi (Juda the Prince) codified basic elements of the jewish law

Notice that the first four of Hal's weird 5-grams show up just in this small snippet. When I checked the cached copy of the page that was pointed to, Google told me that the search term occurred only in the referring pages. Probably someone has created a link farm using these 5-grams to boost the rank of the pages they point to when these terms are used as search queries

Not that I am defending the data but here are some variables to consider...

Could it be spam?Spam can be seemingly random but the same thing may posted by bots everywhere, four years later this post is still getting spammed.

The web changes fast.The data is from 2006 and the google algorithm has vastly improved since then. They have only recently added contextualization. Maybe now they are able to weed out oddities and segment a list of plays now.

Whether the data is correct and/or valid does not change our need to verify its usefulness which is why I greatly appreciated your post.

Really? A natural language processing blog and you can't see that you may have a language snippet cut out of context? "the surveyor observed the use" -> The auditing team assigned to 'the surveyor observed the use' of high quality tools and noted it in their reports.

To the anonymous led here from hacker news, the post is talking about relative frequency, not whether that snippet exists or not (he showed it does right in the post, just at a much lower frequency than others that are actually more representative).

These articles are fantastic; the information you show us is interesting for everybody and is really good written. It’s just great!! Do you want to know something more? Read it...: Great Flowers delivery service through flower2world.

I recently came across your blog and have been reading along. I think I will leave my first comment. I don’t know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often.