Paper summaryshagunsodhani# Improving Word Representations via Global Context and Multiple Word Prototypes
## Introduction
* This paper pre-dated papers like Glove and Word2Vec and proposed an architecture that
* combines local and global context while learning word embeddings to capture the word semantics.
* learns multiple embeddings per word to account for homonymy and polysemy.
* [Link to the paper](http://www.aclweb.org/anthology/P12-1092)
## Global Context-Aware Neural Language Model
### Training Objective
* Given a word sequence *s* (local context) and a document *d* in which the sequence occurs (global context), learn word representations while learning to discriminate the last correct word in *s* from other words.
* *g(s, d)* - scoring function giving liklihood of correct sequence.
* *g(s<sup>w</sup>, d)* - scoring function giving liklihood of last word in *s* repalced by a word *w*.
* Objective - *g(s, d)* > *g(s<sup>w</sup>, d)* + 1 for any other word *w*.
### Architecture
* Two scoring components (neural networks) to capture:
* Local Context
* Map word sequence *s* into an ordered list of vectors *x = [x<sub>1</sub>, ..., x<sub>m</sub>]*.
* *x<sub>i</sub>* - embedding corresponding to *i<sup>th</sup>* word in the sequence.
* Compute local score *score<sub>l</sub>* by using a neural network (with one hidden layer) over *x*.
* Preserves word order and syntactic information.
* Global Context
* Map document *d* to an ordered list of word embeddings, *d = (d<sub>1</sub>, ..., d<sub>k</sub>)*.
* Compute *c*, the weighted average of all word vectors in document.
* The paper uses *idf* score for weighting the documents.
* *x = * concatenation of *c* and vector of the last word in *s*.
* Compute global score *score<sub>g</sub>* by using a neural network (with two hidden layers) over *x*.
* Similar to bag-of-words features.
*score = score<sub>l</sub> + score<sub>g</sub>*
* Train the weights of the hidden layers and the word embeddings.
### Multi-Prototype Neural Language Model
* Words can have different meanings in different contexts which are difficult to capture when we train only one vector per word.
* Solution - train multiple vectors per word to capture the different meanings.
* Approach
* Gather all the fixed-sized context windows for all occurrences of a given word.
* Find the context vector by performing weighted averaging of all the words in the context window.
* Cluster the context vectors using spherical k-means.
* Each word occurrence in the corpus is re-labeled to its associated cluster.
* To find similarity between a pair of words *(w, w')*:
* For each possible cluster of *i* and *j* corresponding to the words *w* and *w'*, find distance between cluster centers for *i* and *j* and weight them by the product of probabilities of *w* belonging to *i* and *w'* belonging to *j* given their respective contexts.
* Average the value over the *k<sup>2</sup>* pairs.
## Training
* Dataset
* Wikipedia corpus
* Parameters
* 10-word windows
* 100 hidden units
* No weight regularization
* 10 different word embeddings learnt for words having multiple meanings.
## Evaluation
* Dataset
* WordSim-353
* 353 pairs of nouns
* words represented without context
* contains human similarity judgements on pair of words
* The paper contributed a new dataset
* captures human similarity judgements on pair of words in the context of a sentence
* consists of verbs and adjectives along with nouns
* for details on how the dataset is constructed, refer the paper
* Performance
* Proposed model achieves higher correlation to human scores than models using only the local or global context.
* Performance can be improved by removing the stop words.
* Using multi-prototype approach (multiple vectors for the same word) benefits the model on the tasks where the context is also given.
## Comments
* This work predated the more general word embedding models like [Word2Vec](https://gist.github.com/shagunsodhani/176a283e2c158a75a0a6) and [Glove](https://gist.github.com/shagunsodhani/efea5a42d17e0fcf18374df8e3e4b3e8). While this model performs good at intrinsic evaluation tasks like word similarity, it is outperformed by the more general and recent models on downstream tasks like NER.

# Improving Word Representations via Global Context and Multiple Word Prototypes
## Introduction
* This paper pre-dated papers like Glove and Word2Vec and proposed an architecture that
* combines local and global context while learning word embeddings to capture the word semantics.
* learns multiple embeddings per word to account for homonymy and polysemy.
* [Link to the paper](http://www.aclweb.org/anthology/P12-1092)
## Global Context-Aware Neural Language Model
### Training Objective
* Given a word sequence *s* (local context) and a document *d* in which the sequence occurs (global context), learn word representations while learning to discriminate the last correct word in *s* from other words.
* *g(s, d)* - scoring function giving liklihood of correct sequence.
* *g(s<sup>w</sup>, d)* - scoring function giving liklihood of last word in *s* repalced by a word *w*.
* Objective - *g(s, d)* > *g(s<sup>w</sup>, d)* + 1 for any other word *w*.
### Architecture
* Two scoring components (neural networks) to capture:
* Local Context
* Map word sequence *s* into an ordered list of vectors *x = [x<sub>1</sub>, ..., x<sub>m</sub>]*.
* *x<sub>i</sub>* - embedding corresponding to *i<sup>th</sup>* word in the sequence.
* Compute local score *score<sub>l</sub>* by using a neural network (with one hidden layer) over *x*.
* Preserves word order and syntactic information.
* Global Context
* Map document *d* to an ordered list of word embeddings, *d = (d<sub>1</sub>, ..., d<sub>k</sub>)*.
* Compute *c*, the weighted average of all word vectors in document.
* The paper uses *idf* score for weighting the documents.
* *x = * concatenation of *c* and vector of the last word in *s*.
* Compute global score *score<sub>g</sub>* by using a neural network (with two hidden layers) over *x*.
* Similar to bag-of-words features.
*score = score<sub>l</sub> + score<sub>g</sub>*
* Train the weights of the hidden layers and the word embeddings.
### Multi-Prototype Neural Language Model
* Words can have different meanings in different contexts which are difficult to capture when we train only one vector per word.
* Solution - train multiple vectors per word to capture the different meanings.
* Approach
* Gather all the fixed-sized context windows for all occurrences of a given word.
* Find the context vector by performing weighted averaging of all the words in the context window.
* Cluster the context vectors using spherical k-means.
* Each word occurrence in the corpus is re-labeled to its associated cluster.
* To find similarity between a pair of words *(w, w')*:
* For each possible cluster of *i* and *j* corresponding to the words *w* and *w'*, find distance between cluster centers for *i* and *j* and weight them by the product of probabilities of *w* belonging to *i* and *w'* belonging to *j* given their respective contexts.
* Average the value over the *k<sup>2</sup>* pairs.
## Training
* Dataset
* Wikipedia corpus
* Parameters
* 10-word windows
* 100 hidden units
* No weight regularization
* 10 different word embeddings learnt for words having multiple meanings.
## Evaluation
* Dataset
* WordSim-353
* 353 pairs of nouns
* words represented without context
* contains human similarity judgements on pair of words
* The paper contributed a new dataset
* captures human similarity judgements on pair of words in the context of a sentence
* consists of verbs and adjectives along with nouns
* for details on how the dataset is constructed, refer the paper
* Performance
* Proposed model achieves higher correlation to human scores than models using only the local or global context.
* Performance can be improved by removing the stop words.
* Using multi-prototype approach (multiple vectors for the same word) benefits the model on the tasks where the context is also given.
## Comments
* This work predated the more general word embedding models like [Word2Vec](https://gist.github.com/shagunsodhani/176a283e2c158a75a0a6) and [Glove](https://gist.github.com/shagunsodhani/efea5a42d17e0fcf18374df8e3e4b3e8). While this model performs good at intrinsic evaluation tasks like word similarity, it is outperformed by the more general and recent models on downstream tasks like NER.