Introducing the Natural Language Toolkit (NLTK)

Natural language processing (NLP) is the automatic or semi-automatic processing of human language. NLP is closely related to linguistics and has links to research in cognitive science, psychology, physiology, and mathematics. In the computer science domain in particular, NLP is related to compiler techniques, formal language theory, human-computer interaction, machine learning, and theorem proving. This Quora question shows different advantages of NLP.

In this tutorial I'm going to walk you through an interesting Python platform for NLP called the Natural Language Toolkit (NLTK). Before we see how to work with this platform, let me first tell you what NLTK is.

What Is NLTK?

The Natural Language Toolkit (NLTK) is a platform used for building programs for text analysis. The platform was originally released by Steven Bird and Edward Loper in conjunction with a computational linguistics course at the University of Pennsylvania in 2001. There is an accompanying book for the platform called Natural Language Processing with Python.

Installing NLTK

Let's now install NLTK to start experimenting with natural language processing. It will be fun!

Installing NLTK is very simple. I'm using Windows 10, so in my Command Prompt (MS-DOS) I type the following command:

pip install nltk

If you are using Ubuntu or macOS, you run the command from the Terminal. More information about installing NLTK on different platforms can be found in the documentation.

If you are wondering what pip is, it is a package management system used to install and manage software packages written in Python. If you are using Python 2 >=2.7.9 or Python 3 >=3.4, you already have pip installed! To check your Python version, simply type the following in your command prompt:

python --version

Let's go ahead and check if we have installed NLTK successfully. To do that, open up Python's IDLE and type the two lines shown in the figure below:

If you get the version of your NLTK returned, then congratulations, you have NLTK installed successfully!

So what we have done in the above step is that we installed NLTK from the Python Package index (pip) locally into our virtual environment.

Notice that you might have a different version of NLTK depending on when you have installed the platform, but that shouldn't cause a problem.

Working With NLTK

The first thing we need to do to work with NLTK is to download what's called the NLTK corpora. I'm going to download the whole corpora. I know it is very large (10.9 GB), but we are going to do it only once. If you know which corpora you need, you don't need to download the whole corpora.

In your Python's IDLE, type the following:

import nltk
nltk.download()

In this case, you will get a GUI from which you can specify the destination and what to download, as shown in the figure below:

I'm going to download everything at this point. Click the Download button at the bottom left of the window, and wait for a while until everything gets downloaded to your destination directory.

Before moving forward, you might be wondering what a corpus (singular of corpora) is. A corpus can be defined as follows:

Corpus, plural corpora; A collection of linguistic data, either compiled as written texts or as a transcription of recorded speech. The main purpose of a corpus is to verify a hypothesis about language - for example, to determine how the usage of a particular sound, word, or syntactic construction varies. Corpus linguistics deals with the principles and practice of using corpora in language study. A computer corpus is a large body of machine-readable texts.

Stop Words

Sometimes we need to filter out useless data to make the data more understandable by the computer. In natural language processing (NLP), such useless data (words) are called stop words. So, these words to us have no meaning, and we would like to remove them.

NLTK provides us with some stop words to start with. To see those words, use the following script:

Notice that concordance() returns every occurrence of the word language, in addition to some context. Before that, as shown in the script above, we tokenize the read file and then convert it into an nltk.Text object.

I just want to note that the first time I ran the program, I got the following error, which seems to be related to the encoding the console uses:

The Gutenberg Corpus

Project Gutenberg (PG) is a volunteer effort to digitize and archive cultural works, to "encourage the creation and distribution of eBooks". It was founded in 1971 by Michael S. Hart and is the oldest digital library. Most of the items in its collection are the full texts of public domain books. The project tries to make these as free as possible, in long-lasting, open formats that can be used on almost any computer. As of 3 October 2015, Project Gutenberg reached 50,000 items in its collection.

NLTK contains a small selection of texts from Project Gutenberg. To see the included files from Project Gutenberg, we do the following:

Conclusion

As we have seen in this tutorial, the NLTK platform provides us with a powerful tool for working with natural language processing (NLP). I have only scratched the surface in this tutorial. If you would like to go deeper into using NLTK for different NLP tasks, you can refer to NLTK's accompanying book: Natural Language Processing with Python.