Syndicate

ADEL: ADaptable Entity Linking

Submitted by Raphael Troncy on 06/28/2018 - 13:36

Tracking #: 1935-3148

Authors:

Julien Plu

Giuseppe Rizzo

Raphael Troncy

Responsible editor:

Guest Editors LD4IE 2017

Submission type:

Full Paper

Abstract:

Four main challenges can cause numerous difficulties when developing an entity linking system: i) the kind of textual documents to annotate (such as social media posts, video subtitles or news articles); ii) the number of types used to categorise an entity (such as PERSON, LOCATION, ORGANIZATION, DATE or ROLE); iii) the knowledge base used to disambiguate the extracted mentions (such as DBpedia, Wikidata or Musicbrainz); iv) the language used in the documents. Among these four challenges, being agnostic to the knowledge base and in particular to its coverage, whether it is encyclopedic like DBpedia or domain-specific like Musicbrainz, is arguably one of the most challenging one. In this work, we propose to tackle those four challenges. In order to be knowledge base agnostic, we propose a method that enables to index the data independently of the schema and vocabulary being used. More precisely, we design our index such that each entity has at least two information: a label and a popularity score such as a prior probability or a PageRank score. This results in a framework named ADEL, an entity recognition and linking hybrid system using linguistic, information retrieval, and semantics-based methods. ADEL is a modular framework that is independent to the kind of text to be processed and to the knowledge base used as referent for disambiguating entities. We thoroughly evaluate the framework on six benchmark datasets: OKE2015, OKE2016, NEEL2014, NEEL2015, NEEL2016 and AIDA. Our evaluation shows that ADEL outperforms state-of-the-art systems in terms of extraction and entity typing. It also shows that our indexing approach allows to generate an accurate set of candidates from any knowledge base that makes use of linked data, respecting the required information for each entity, in a minimum of time and with a minimal size.

The authors present ADEL, an entity recognition and linking hybrid system using linguistic, information retrieval, and semantics-based methods. ADEL is
a modular framework that is independent to the kind of text to be processed and to the knowledge base used as referent for disambiguating entities. The authors evaluate ADEL on six benchmark datasets: OKE2015, OKE2016, NEEL2014,NEEL2015, NEEL2016 and AIDA. ADEL is found to outperform state-of-the-art systems on both extraction and entity typing, although performance on linking is less competitive (though not by much).

Overall, the paper was easy to read and follow, and the strongest point was the experimental section. The authors used many benchmarks, and evaluated against various systems, which lends confidence in the system given the plethora of entity recognition, typing and linking systems out there. The authors also present an interesting approach for modularly combining many of the key steps involved in a full entity recognition and linking pipeline.

Primarily, ADEL jointly addresses four challenges commonly encountered in building such pipelines:

(1) First, the authors can deal with texts of different natures, including social media and more formal texts like newswire. Typically, different systems are applicable for different kinds of text, so a unified architecture for handling both is an important contribution.
(2) Second, the authors can handle non-English data, which extends the scope and utility of the work to more use cases.
(3) Third, the authors can recognize and link multiple entity types. This is now standard in many systems of this kind.
(4) Fourth and perhaps most importantly, the entity linking module in ADEL is somewhat agnostic to the KB used for linking. The authors are correct that many linking systems do make assumptions about the KB they are linking to and that domain/KB transfer can be difficult. Therefore, I consider this to be an important contribution.

Other strong points are, as noted above, the extent of the evaluations, and also that the authors note real-world projects like NexGenTV and ASRAEL for which such a modularly configurable system may be useful.

My only suggestion to the authors is to proofread the paper a couple of times to make the writing more crisp. Otherwise, I believe the article should be accepted.

Review #2

By Anastasia Dimou submitted on 30/Jul/2018

Suggestion: Accept

Review Comment:

The paper is significantly reworked compared to the previous, all my comments are resolved and the proposed work is original and has significant results. However it still lacks a bit in terms of typos and grammar/syntax mistakes. I would suggest the paper to be accepted but I would also strongly recommend to do a thorough proof reading before submitting the final version.

I have a couple small remarks bellow per section but they are mostly related to small clarifications, so nothing that would prevent the paper from being accepted:

abstract
Let me explain how it reads: we identify 4 challenges, the 3rd one (being agnostic of the knowledge base) is the most challenging one, but we will still tackle all 4. --> perhaps that description could be a bit improved.

introduction
It would be beneficial for the paper if there was an association between the contributions and challenges, namely which contribution(s) address(es) which challenge(s).

related work
"Table 1 details the extraction or recognition techniques adopted" but in the table (not in the caption) everything comes under entity extraction and recognition is only mentioned as subcategory of extraction. Is this valid and on purpose?

"We can see that the systems have difficulties to propose a way to tackle these challenges, as they address at most two challenges and sometimes none" --> Hmm, if there are systems that do not address any of the challenges? what do they do then?! I mean, are you sure that you identified all possible challenges? and there are no others that these tools address? Could you please clarify this?

The related work with all tables is really good! The discussion is mainly per table. Are there any cross-table comments?

I would suggest to make the tables caption more detailed? Eg add to the caption a small explanation of what each column represents and the same, ie a small explanation, for the frequently encountered cell values, eg syntactic features.

approach

The extractors that are mentioned in the extractors module are not in figure 2 where other extractors are mentioned? (or at least not with the same name). More, while it is mentioned that they are 6, the detailed text mentions 7.

detail: Listing 1 goes beyond the text frame, same for the link that follows it

It is mentioned that "The module is composed of two steps: i) indexing and ii) search optimization." but earlier in the general description of the architecture it is mentioned that "The second part, (Index), contains the module Indexing.". ie only one module. I think those two need to be aligned.

evaluation

The bolding is not consistent in the evaluation's tables values. While it is mentioned that the best system is in bold, this is not always the case. Unless the best system is judged based on F1 but if so, that should be clarified.

evaluation/conclusions

I would suggest to provide a back reference, explaining/summarising how each challenge was resolved.

I mention a couple of minors, but a few repeat and there are more throughout the text:
"each entity has at least two information" --> this "two information" needs to be rephrased I think
The age of the modern artificial intelligence as started in the middle of the 1940s. --> (has) started?
"Alan Turing as stated the earliest artificial intelligence problem" --> stated?
"depending of the task" --> depending on
"they may exist" --> there may/might exist
"The first step consists in extracting all entities that will be indexed using a SPARQL query." --> it needs to be rephrased
"The best configuration for doing entity recognition is the same than for the extraction." --> same as

Along with these four claims, the authors suggest that they have a highly modular system. However, the evaluation section, in my opinion, fails to strongly demonstrate or backup these claims.

Let's start with the modular system claim. The authors point out examples like AIDA, TagME, Babelfy as static non-adaptable system, non-modular systems. For instance, for AIDA one would need to change the code to include a new KB. I see that as engineering change and not an algorithmic one. So I don’t necessarily buy the argument that it's not knowledge base independent. Similarly for Babelfy, authors claim a linear formula can't be added. Why would they want to do that?

From an engineering/system architecture and design point of view, yes ADEL is modular. I want to compliment the authors for the effort and the design. But I see that as a system contribution and not as a research contribution in the context of this journal.

If I apply the author's own definition about the need for code-changes as some-what static systems, even ADEL is not completely modular. The Overlap Resolution module uses a manual mapping between different types (LOC, Place etc.) across different NER systems. If one would add a new NER/new type, I see a "code-change" required in ADEL. The index construction also seems a somewhat manual process, with the end user/developer required to construct the SPARQL queries to generate a new index for a new KB (including the PageRank computation).

Let's consider the textual independence claim. The authors did evaluate the system across several different benchmarks which include different genre data - from news like articles to tweets. However, I attribute the textual independence to system/architecture as opposed to algorithmic. The system architecture allows one to put together a different pipeline for different genre of data. E.g. you choose appropriate POS/NER for tweets and choose another one for news data. The system is designed well for bring your own recognition system + linking system. I don’t see much of novel research contribution here. The evaluation section clearly shows that ADEL uses Stanford CoreNLP models all the way.

On the subject of language independence, authors do admit that they haven't achieved it and point to the fact that NERs/POS models for different languages can be added to the system (which again is a nice architecture contribution …). I recommend the authors to look into work in cross-lingual, multi-lingual embeddings and their applications in building truly language agnostic models (including NERs).

Knowledge base and Entity type independence is more of system design. In theory, most of the related work stated in this paper can be engineered to work with different KBs and entity types.

The evaluation section should focus on backing up the claims. For instance, there is too much emphasis on "recognition". I see those results more of a reflection of the performance of the CoreNLP models (and their different combinations) as opposed to providing evidence for the contributions. ADEL being an Entity Linking system, as authors themselves admit, is yet to achieve satisfactory results in the linking process (Tables 9, 10, 11, 14 to cite a few examples). The ADEL linker formula needs more work, and authors have pointed to promising interesting next steps.

I reviewed the author comments for the previous reviews and I would agree with Reviewer 3. I would consider this paper as a good systems paper and not a full research paper. I feel the novel contributions - model combination, overlap resolution module and indexing are not strong enough to warrant a full research paper. It’s also unclear to me regarding what's new in this paper and what's previously published.

I definitely recommend this paper to be resubmitted as a systems paper.

(a small gripe - in section 6.2, please reference the appropriate table numbers in the subsections, will make it easier to read).