Topics

semantic relations

In Predicting the relevance of distributional semantic similarity with contextual information

We first set up a human annotation of semantic links with or without contextual information to show the importance of the textual context in evaluating the relevance of semantic similarity, and to assess the prevalence of actual semantic relations between word tokens.

Page 1, “Abstract”

They are not suitable for the evaluation of the whole range of semantic relatedness that is exhibited by distributional similarities, which exceeds the limits of classical lexical relations, even though researchers have tried to collect equivalent resources manually, to be used as a gold standard (Weeds, 2003; Bordag, 2008; Anguiano et al., 2011).

Page 1, “Introduction”

One advantage of distributional similarities is to exhibit a lot of different semantic relations , not necessarily standard lexical relations.

We hypothetize that evaluating and filtering semantic relations in texts where lexical items occur would help tasks that naturally make use of semantic similarity relations, but assessing this goes beyond the present work.

Page 2, “Introduction”

We present the experiments we set up to automatically filter semantic relations in context, with various groups of features that take into account information from the corpus used to build the thesaurus and contextual information related to occurrences of semantic neighbours 3).

Page 2, “Introduction”

In other words, is there a semantic relation between them, either classical (synonymy, hypernymy, co-hyponymy, meronymy, co-meronymy) or not (the relation can be paraphrased but does not belong to the previous cases) ?”

Page 3, “Evaluation of lexical similarity in context”

We differ from all these evaluation procedures as we do not focus on an essential view of the relatedness of two lexical items, but evaluate the link in a context Where the relevance of the link is in question, an “existential” view of semantic relatedness .

Page 8, “Related work”

This helps cover non classical semantic relations which are hard to evaluate with classical resources.

F-score

In Predicting the relevance of distributional semantic similarity with contextual information

In case one wants to optimize the F-score (the harmonic mean of precision and recall) when extracting relevant pairs, we can see that the optimal point is at .24 for a threshold of .22 on Lin’s score.

Page 4, “Evaluation of lexical similarity in context”

Other popular methods (maximum entropy, SVM) have shown slightly inferior combined F-score , even though precision and recall might yield more important variations.

Page 6, “Experiments: predicting relevance in context”

As a baseline, we can also consider a simple threshold on the lexical similarity score, in our case Lin’s measure, which we have shown to yield the best F-score of 24% when set at 0.22.

Page 6, “Experiments: predicting relevance in context”

If we take the best simple classifier (random forests), the precision and recall are 68.1% and 24.2% for an F-score of 35.7%, and this is significantly beaten by the Naive Bayes method as precision and recall are more even ( F-score of 41.5%).

Page 7, “Experiments: predicting relevance in context”

Also note that predicting every link as relevant would result in a 2.6% precision, and thus a 5% F-score .

Page 7, “Experiments: predicting relevance in context”

overall best F-score of 46.3% is reached with Random Forests and the cost-aware learning method.

Page 7, “Experiments: predicting relevance in context”

Table 3 sums up the scores for the different configurations, with precision, recall, F-score and the confidence interval on the F-score .

precision and recall

In Predicting the relevance of distributional semantic similarity with contextual information

Figure 3 shows the influence of the threshold value to select relevant pairs, when considering precision and recall of the pairs that are kept when choosing the threshold, evaluated against the human annotation of relevance in context.

Page 4, “Evaluation of lexical similarity in context”

In case one wants to optimize the F-score (the harmonic mean of precision and recall ) when extracting relevant pairs, we can see that the optimal point is at .24 for a threshold of .22 on Lin’s score.

Page 4, “Evaluation of lexical similarity in context”

Figure 3: Precision and recall on relevant links with respect to a threshold on the similarity measure (Lin’s score)

Page 4, “Experiments: predicting relevance in context”

We have seen that the relevant/not relevant classification is very imbalanced, biased towards the “not relevant” category (about 11%/89%), so we applied methods dedicated to counterbalance this, and will focus on the precision and recall of the predicted relevant links.

Page 6, “Experiments: predicting relevance in context”

Other popular methods (maximum entropy, SVM) have shown slightly inferior combined F-score, even though precision and recall might yield more important variations.

Page 6, “Experiments: predicting relevance in context”

We are interested in the precision and recall for the “relevant” class.

Page 7, “Experiments: predicting relevance in context”

If we take the best simple classifier (random forests), the precision and recall are 68.1% and 24.2% for an F-score of 35.7%, and this is significantly beaten by the Naive Bayes method as precision and recall are more even (F-score of 41.5%).

similarity measure

Appears in 4 sentences as: similarity measure (4)

In Predicting the relevance of distributional semantic similarity with contextual information

A distributional thesaurus is a lexical network that lists semantic neighbours, computed from a corpus and a similarity measure between lexical items, which generally captures the similarity of contexts in which the items occur.

Page 1, “Introduction”

Figure 3: Precision and recall on relevant links with respect to a threshold on the similarity measure (Lin’s score)

Page 4, “Experiments: predicting relevance in context”

A straightforward parameter to include to predict the relevance of a link is of course the similarity measure itself, here Lin’s information measure.

Page 5, “Experiments: predicting relevance in context”

This is already a big improvement on the use of the similarity measure alone (24%).

contextual information

Appears in 3 sentences as: contextual information (3)

In Predicting the relevance of distributional semantic similarity with contextual information

We first set up a human annotation of semantic links with or without contextual information to show the importance of the textual context in evaluating the relevance of semantic similarity, and to assess the prevalence of actual semantic relations between word tokens.

Page 1, “Abstract”

We present the experiments we set up to automatically filter semantic relations in context, with various groups of features that take into account information from the corpus used to build the thesaurus and contextual information related to occurrences of semantic neighbours 3).

Page 2, “Introduction”

To verify that this methodology is useful, we did a preliminary annotation to contrast judgment on lexical pairs with or without this contextual information .

distributional similarities

In Predicting the relevance of distributional semantic similarity with contextual information

They are not suitable for the evaluation of the whole range of semantic relatedness that is exhibited by distributional similarities , which exceeds the limits of classical lexical relations, even though researchers have tried to collect equivalent resources manually, to be used as a gold standard (Weeds, 2003; Bordag, 2008; Anguiano et al., 2011).

Page 1, “Introduction”

One advantage of distributional similarities is to exhibit a lot of different semantic relations, not necessarily standard lexical relations.

Page 1, “Introduction”

For each pair neighboura/neighbourb, we computed a set of features from Wikipedia (the corpus used to derive the distributional similarity ): We first computed the frequencies of each item in the corpus, f reqa and f reqb, from which we derive

semantic similarity

Appears in 3 sentences as: semantic similarity (3)

In Predicting the relevance of distributional semantic similarity with contextual information

We first set up a human annotation of semantic links with or without contextual information to show the importance of the textual context in evaluating the relevance of semantic similarity , and to assess the prevalence of actual semantic relations between word tokens.

Page 1, “Abstract”

We hypothetize that evaluating and filtering semantic relations in texts where lexical items occur would help tasks that naturally make use of semantic similarity relations, but assessing this goes beyond the present work.

Page 2, “Introduction”

We proposed a method to reliably evaluate distributional semantic similarity in a broad sense by considering the validation of lexical pairs in contexts where they both appear.