By default, Lingua::Identify assumes normal mode, but others are available.

In dummy mode, instead of actually calculating anything, Lingua::Identify only does the preparation it has to and then returns a bunch of information, including the list of the active languages, the selected methods, etc. It also returns the text meant to be analised.

Do be warned that, with langof_file, the dummy mode still reads the files, it simply doesn't calculate language.

langof_file works just like langof, with the exception that it reveives filenames instead of text. It reads these texts (if existing and readable, of course) and parses its content.

Currently, langof_file assumes the files are regular text. This may change in the future and the files might be scanned to check their filetype and then parsed to extract only their textual content (which should be pretty useful so that you can perform language identification, say, in HTML files, or PDFs).

To identify the language a file is written in:

$language = langof_file($path);

To get the most probable language and also the percentage of its probability, do:

($language, $probability) = langof_file($path);

If you want a hash where each active language is mapped into its percentage, use this:

%languages = langof_file($path);

If you pass more than one file to langof_file, they will all be read and their content merged and then parsed for language identification.

After getting the results into an array, its first element is the most probable language. That doesn't mean it is very probable or not.

You can find more about the likeliness of the results to be accurate by computing its confidence level.

use Lingua::Identify qw/:language_identification/;
my @results = langof($text);
my $confidence_level = confidence(@results);
# $confidence_level now holds a value between 0.5 and 1; the higher that
# value, the more accurate the results seem to be

The formula used is pretty simple: p1 / (p1 + p2) , where p1 is the probability of the most likely language and p2 is the probability of the language which came in second. A couple of examples to illustrate this:

English 50% Portuguese 10% ...

confidence level: 50 / (50 + 10) = 0.83

Another example:

Spanish 30% Portuguese 10% ...

confidence level: 30 / (25 + 30) = 0.55

French 10% German 5% ...

confidence level: 10 / (10 + 5) = 0.67

As you can see, the first example is probably the most accurate one. Are there any doubts? The English language has five times the probability of the second language.

The second example is a bit more tricky. 55% confidence. The confidence level is always above 50%, for obvious reasons. 55% doesn't make anyone confident in the results, and one shouldn't be, with results such as these.

Notice the third example. The confidence level goes up to 67%, but the probability of French is of mere 10%. So what? It's twice as much as the second language. The low probability may well be caused by a great number of languages in play.

In order to identify the language a given text is written in, we repeat a given process for each active language (see section LANGUAGES MANIPULATION); in that process, we look for common patterns of that language. Those patterns can be prefixes, suffixes, common words, ngrams or even sequences of words.

After repeating the process for each language, the total score for each of them is then used to compute the probability (in percentage) for each language to be the one of that text.

The "Small Word Technique" searches the text for the most common words of each active language. These words are usually articles, pronouns, etc, which happen to be (usually) the shortest words of the language; hence, the method name.

This is usually a good method for big texts, especially if you happen to have few languages active.

Please do not contribute with modules you made yourself. It's easier to contribute with unprocessed text, because that allows for new versions of Lingua::Identify not having to drop languages down in case I can't contact you by that time.

Use make-lingua-identify-language to create a new module for your own personal use, if you must, but try to contribute with unprocessed text rather than those modules.

Check the language a given text file is written in, supposing you happen to know it's either Portuguese or English:

use Lingua::Identify qw/langof set_active_languages/;
set_active_languages(qw/pt en/);
my $text = join "\n", <>;
# identify the language by letting the module decide on the best way
# to do so
my $language = langof($text);