Surprising mathematical law tested on Project Gutenberg texts

February 22, 2016

A picture of Zipf's law at CRM. Credit: UAB

Zipf's law in its simplest form, as formulated in the thirties by American linguist George Kingsley Zipf, states surprisingly that the most frequently occurring word in a text appears twice as often as the next most frequent word, three times more than the third most frequent one, four times more than the fourth most frequent one, and so on.

The law can be applied to many other fields, not only literature, and it has been tested more or less rigorously on large quantities of data, but until now had not been tested with maximum mathematical rigour and on a database large enough to ensure statistical validity.

Researchers at the Centre de Recerca Matemàtica (CRM), part of the Government of Catalonia's CERCA network, who are attached to the UAB Department of Mathematics, have conducted the first sufficiently rigorous study, in mathematical and statistical terms, to test the validity of Zipf's law. This study falls within the framework of the Research in Collaborative Mathematics project run by Obra Social "la Caixa". To achieve this, they analysed the whole collection of English-language texts in the Project Gutenberg, a freely accessible database with over 30,000 works in this language. There is no precedent for this: in the field of linguistics the law had never been put to the test on sets of more than a dozen texts.

According to the analysis, if the rarest words are left out - those that appear only once or twice throughout a book - 55% of the texts fit perfectly into Zipf's law, in its most general formulation. If all the words are taken into account, even the rarest ones, the figure is 40%.

"It is very surprising that the frequency of occurrence of these words should be determined by a single-parameter formula. The famous Gaussian bell curve, for example, needs two parameters, position and width, to adjust to the real data", explains Álvaro Corral, a CRM researcher attached to the UAB Department of Mathematics and coordinator of the research. "If we ignored words that appear three, four or five times in a whole work, the percentage of books that follow Zipf's law could be even higher".

In mathematical terms, the law states that if all the words are ranked by frequency of use, the second most frequently occurring one appears half as often as the most frequent one; the third, 1/3 as often and, in general, the word occupying the position n appears 1/n times as often as the most frequent one.

In fact, the most general formulation of the law includes an exponent a, so that the relationship is 1/na. Though this complicates the formula a little, the frequency fits very closely for values of "a" very near to 1 (i.e. as if no exponent had been added). There are other formulations of the law that are mathematically more complex, but all have a single free parameter.

The researchers studied the validity of the three most frequently used formulations of Zipf's law in all the English-language texts (31,075 books) in the Project Gutenberg database, and they observed that one of these formulations fits, with statistically significant results (p>0.05), the frequency of occurrence of all the words in over 40% of the books in the collection, texts that contain between 100 and over a million words.

"Zipf's law has generated much debate, but always basing its validity on certain specific examples", points out Álvaro Corral. "It seems obvious that in today's age of Big Data and high-performance computers, we need to focus on large-scale analysis of the law, and these results are a big step in that direction".

"Although literature is regarded as one of the greatest expressions of creative freedom, not even major authors like Shakespeare or Dickens escape the tyranny of Zipf's law", concludes Dr Corral.

Related Stories

The idea that we can learn about possible extraterrestrial (ETI) communication systems by studying non-human communications on Earth is similar to the astrobiological idea that one might learn more about exobiology by studying ...

(PhysOrg.com) -- Linguists have thought for many years the length of words is related to the frequency of use, with short words used more often than long ones. Now researchers in the US have shown the length is more closely ...

It's a good thing some words have many meanings—ambiguous words actually make communication easier and may be an inevitable consequence of a language's evolution, according to a new SFI working paper by External Professor ...

Physicists have identified a mechanism that may help explain Zipf's law – a unique pattern of behavior found in disparate systems, including complex biological ones. The journal Physical Review Letters is publishing their ...

(Medical Xpress) -- Do you ever wonder about the stuff that makes up words? Why is a word a word, what goes into forming it, what's its history or why is it long or short? Scientists at the Massachusetts Institute of Technology ...

(Phys.org)—A team of researchers with the University of Edinburgh has found evidence that the "Thames Beater" was a weapon that could be used to kill another person—perhaps at times, with a single blow to the head. In ...

The best decisions are made on the basis of the average of various estimates, as confirmed by the research of Dennie van Dolder and Martijn van den Assem, scientists at VU Amsterdam. Using data from Holland Casino promotional ...

One of the world's 7,000 languages vanishes every other week, and half - including scores of indigenous North American languages—might not survive the 21st century, experts say. To preserve as much linguistic diversity ...

Fossilised ticks discovered trapped and preserved in amber show that these parasites sucked the blood of feathered dinosaurs almost 100 million years ago, according to a new article published in Nature Communications today.

Everyone wants to be with their family for Christmas, but spare a thought for a group of orphan fossils that have been separated from their parents since the dawn of animal evolution, over half a billion years ago.

4 comments

In the pre-Web days, I once read a four page essay written just to prove that the English word 'the' was unnecessary. Indeed the whole essay seems normal and clear when I read it, even though 'the' never occurred.I have since been unable to find the essay anywhere.