NLP-progress

Repository to track the progress in Natural Language Processing (NLP), including the datasets and the current state-of-the-art for the most common NLP tasks.

Dialogue

Dialogue is notoriously hard to evaluate. Past approaches have used human evaluation.

Dialogue act classification

Dialogue act classification is the task of classifying an utterance with respect to the fuction it serves in a dialogue, i.e. the act the speaker is performing. Dialogue acts are a type of speech acts (for Speech Act Theory, see Austin (1975) and Searle (1969)).

Switchboard corpus

The Switchboard-1 corpus is a telephone speech corpus, consisting of about 2,400 two-sided telephone conversation among 543 speakers with about 70 provided conversation topics. The dataset includes the audio files and the transcription files, as well as information about the speakers and the calls.

The Switchboard Dialogue Act Corpus (SwDA) [download] extends the Switchboard-1 corpus with tags from the SWBD-DAMSL tagset, which is an augmentation to the Discourse Annotation and Markup System of Labeling (DAMSL) tagset. The 220 tags were reduced to 42 tags by clustering in order to improve the language model on the Switchboard corpus. A subset of the Switchboard-1 corpus consisting of 1155 conversations was used. The resulting tags include dialogue acts like statement-non-opinion, acknowledge, statement-opinion, agree/accept, etc.
Annotated example:Speaker: A, Dialogue Act: Yes-No-Question, Utterance: So do you go to college right now?

ICSI Meeting Recorder Dialog Act (MRDA) corpus

The MRDA corpus [download] consists of about 75 hours of speech from 75 naturally-occurring meetings among 53 speakers. The tagset used for labeling is a modified version of the SWBD-DAMSL tagset. It is annotated with three types of information: marking of the dialogue act segment boundaries, marking of the dialogue acts and marking of correspondences between dialogue acts.
Annotated example:Time: 2804-2810, Speaker: c6, Dialogue Act: s^bd, Transcript: i mean these are just discriminative.
Multiple dialogue acts are separated by “^”.

Dialogue state tracking

Dialogue state tacking consists of determining at each turn of a dialogue the
full representation of what the user wants at that point in the dialogue,
which contains a goal constraint, a set of requested slots, and the user’s dialogue act.

Second dialogue state tracking challenge

For goal-oriented dialogue, the dataset of the second Dialogue Systems Technology Challenges
(DSTC2) is a common evaluation dataset. The DSTC2 focuses on the restaurant search domain. Models are
evaluated based on accuracy on both individual and joint slot tracking.

Wizard-of-Oz

The WoZ 2.0 dataset is a newer dialogue state tracking dataset whose evaluation is detached from the noisy output of speech recognition systems. Similar to DSTC2, it covers the restaurant search domain and has identical evaluation.

Retrieval-based Chatbot

The main task retrieval-based chatbot is response selection, that aims to find correct responses from a pre-defined index.

Ubuntu Corpus

The Ubuntu Corpus contains almost 1 million multi-turn dialogues from the Ubuntu Chat Logs. The task of Ubuntu Corpus is to select the correct response from 10 candidates (others are negatively sampled) by considering previous conversation history. You can find more details at here. The Evaluation metric is recall at position K in N candidates (Recall_N@K).

Generative-based Chatbot

The main task of generative-based chatbot is to generate consistent and engaging response given the context.

Personalized Chit-chat

The task of persinalized chit-chat dialogue generation is first proposed by PersonaChat. The motivation is to enhance the engagingness and consistency of chit-chat bots via endowing explicit personas to agents. Here the persona is defined as several profile natural language sentences like “I weight 300 pounds.”. NIPS 2018 has hold a competition The Conversational Intelligence Challenge 2 (ConvAI2) based on the dataset. The Evaluation metric is F1, Hits@1 and ppl. F1 evaluates on the word-level, and Hits@1 represents the probability of the real next utterance ranking the highest according to the model, while ppl is perplexity for language modeling. The following results are reported on dev set (test set is still hidden), almost of them are borrowed from ConvAI2 Leaderboard.