This is the recommended installation procedure, as it provides ready access to the examples and allows you to modify anything you might need. This is especially useful if you if you want to submit another task to the repository.

All needed data will be downloaded to ~/ParlAI/data, and any non-data files (such as the MemNN code) if requested will be downloaded to ~/ParlAI/downloads. If you need to clear out the space used by these files, you can safely delete these directories and any files needed will be downloaded again.

Worlds, agents and teachers

The main concepts (classes) in ParlAI:

world - defines the environment (can be very simple, just two agents talking to each other).

agent – an agent in the world, e.g. the learner. (There can be multiple learners.)

teacher – a type of agent that talks to the learner, implements one of the
listed before.

After defining a world and the agents in it, a main loop can be run for training, testing or displaying, which calls the function world.parley(). The skeleton of an example main is given in the left panel, and the actual code for parley() on the right.

Actions and Observations

All agents (including teachers) speak to each other with a single format -- the observation/action object (a python dict).
This is used to pass text, labels and rewards between agents.
It’s the same object type when talking (acting) or listening (observing), but a different view (with different values in the fields).
The fields are as follows:

Each of these fields are technically optional, depending on your dataset, though the 'text' field will most likely be used in nearly all exchanges.

For a fixed supervised learning dataset like bAbI, a typical exchange from the training set might be as follows (the test set would not include labels):

repeat_label: basic class for merely repeating all data sent to it (e.g. for piping to a file, debugging)

local_human: takes input from the keyboard as the act() function of the agent, so a human can act in the environment

Examples

This directory contains a few particular examples of basic loops.

base_train.py: very simple example shows the outline of a training/validation loop using the default Agent parent class

display_data.py: uses agent.repeat_label to display data from a particular task provided on the command-line

display_model.py: shows the predictions of a provided model on a particular task provided on the command-line

eval_model.py: uses the named agent to compute evaluation metrics data for a particular task provided on the command-line

build_dict.py: build a dictionary from a particular task provided on the command-line using core.dict.DictionaryAgent

drqa: shows how to train the attentive LSTM DrQA model of Chen et al. on SQuAD.

Tasks

Our first release included the following datasets (shown in the left panel), and accessing one of them is as simple as specifying the name of the task as a command line option, as shown in the dataset display utility (right panel):

Over 20 tasks were supported in the first release, including popular datasets such as
SQuAD, bAbI tasks, MCTest, WikiQA, WebQuestions, SimpleQuestions, WikiMovies, QACNN, QADailyMail, CBT, BookTest, bAbI Dialog tasks,
Ubuntu, OpenSubtitles, Cornell Movie, VQA-COCO2014.
Since then, several datasets have been added such as VQAv2, VisDial, MNIST_QA, Personalized Dialog, InsuranceQA, MS MARCO, TriviaQA, and CLEVR. See here for the current complete task list.

Choosing a task in ParlAI is as easy as specifying it on the command line, as shown in the above image (right). If the dataset has not been used before, ParlAI will automatically download it. As all datasets are treated in the same way in ParlAI (with a single dialog API), a dialog agent can in principle switch training and testing between any of them. Even better, one can specify many tasks at once (multi-tasking) by simply providing a comma-separated list, e.g. the command line “-t babi,squad”, to use those two datasets, or even all the QA datasets at once (-t #qa) or indeed every task in ParlAI at once (-t #all). The aim is to make it easy to build and evaluate very rich dialog models.

Each task folder contains:

build.py file for setting up data for the task (downloading data, etc, only done the first time requested, and not downloaded if the task is not used).

agents.py file which contains default or special teacher classes used by core.create_task to instantiate these classes from command-line arguments (if desired).

worlds.py file can optionally be added for tasks that need to define new/complex environments.

To add your own task:

(optional) implement build.py to download any needed data

implement agents.py, with at least a DefaultTeacher (extending Teacher or one of its children)

if your data consists of fixed logs, you can use extend DialogTeacher and thus core.data.TextData, in which case you just need to write your own setup_data function which provides an iterable over the data according to the format described in core.data

if your data uses other fields, write your own act() method which provides observations from your task each time it's called

MTurk

An important part of ParlAI is seamless integration with Mechanical Turk for data collection, training and evaluation.

Human Turkers are also viewed as agents in ParlAI and hence person-person, person-bot, or multiple people and bots in group chat can all converse within the standard framework, switching out the roles as desired with no code changes to the agents. This is because Turkers also receive and send via a (pretty printed) version of the same interface, using the fields of the observation/action dict.

We currently provide three examples: collecting data, human evaluation of a bot, and round-robin chat between local humans and remote Turkers.

The mturk library contains the following directories:

core: this directory contains the core code for setting up AWS backend that supports the MTurk chat interface, code for HIT creation and approval, and the wrapper class MTurkAgent which encapsulates the MTurk interface into a standard Agent class.

tasks: this directory contains three sample MTurk tasks.

qa_data_collection: get questions and answers from turkers, given a random paragraph from SQuAD.

multi_agent_dialog: round-robin chat between two local human agents and two Turkers.

To run an MTurk task:

Go into the directory for the task you want to run.

Run python run.py -nh <num_hits> -na <num_assignments> -r <reward> [--sandbox]/[--live], with <num_hits>, <num_assignments> and <reward> set appropriately. Use --sandbox to run the task in MTurk sandbox mode before pushing it live.

To add your own MTurk task:

create a new folder within the mturk/tasks directory for your new task

implement task_config.py, with at least the following fields in the task_config dictionary:

hit_title: a short and descriptive title about the kind of task the HIT contains. On the Amazon Mechanical Turk web site, the HIT title appears in search results, and everywhere the HIT is mentioned.

hit_description: a description includes detailed information about the kind of task the HIT contains. On the Amazon Mechanical Turk web site, the HIT description appears in the expanded view of search results, and in the HIT and assignment screens.

hit_keywords: one or more words or phrases that describe the HIT, separated by commas. On MTurk website, these words are used in searches to find HITs.

task_description: a detailed task description that will be shown on the HIT task preview page and on the left side of the chat page. Supports HTML formatting.

implement run.py, with code for setting up and running the world where MTurkAgent lives in.

(Optional) implement worlds.py, with a world class that extends from World.

Please see the MTurk tutorial to learn more about the MTurk examples and how to create and run your own task.

Support

If you have any questions, bug reports or feature requests, please don't hesitate to post on our Github Issues page.