GERBIL
Ricardo Usbeck, Michael Röder, Axel-Cyrille Ngonga Ngomo et al. · International World Wide Web Conference (WWW) · 2015
We present GERBIL, an evaluation framework for semantic entity annotation. The rationale behind our framework is to provide developers, end users and researchers with easy-to-use interfaces that allow for the agile, fine-grained and uniform evaluation of annotation tools on multiple datasets. By these means, we aim to ensure that both tool developers and end users can derive meaningful insights pertaining to the extension, integration and use of annotation applications. In particular, GERBIL provides comparable results to tool developers so as to allow them to easily discover the strengths and weaknesses of their implementations with respect to the state of the art. With the permanent experiment URIs provided by our framework, we ensure the reproducibility and archiving of evaluation results. Moreover, the framework generates data in machine-processable format, allowing for the efficient querying and post-processing of evaluation results. Finally, the tool diagnostics provided by GERBIL allows deriving insights pertaining to the areas in which tools should be further refined, thus allowing developers to create an informed agenda for extensions and end users to detect the right tools for their purposes. GERBIL aims to become a focal point for the state of the art, driving the research agenda of the community by presenting comparable objective evaluation results.
Cited by 190 works. doi:10.1145/2736277.2741626
Authors
- Ricardo Usbeck
- Michael Röder
- Axel-Cyrille Ngonga Ngomo
- Andreas Both
- Martin Brümmer
- Diego Ceccarelli
- Marco Cornolti
- Didier Cherix
- Bernd Eickmann
- Paolo Ferragina
- Roberto Navigli A
- Francesco Piccinno
- Giuseppe Rizzo
- Harald Sack
- René Speck
- Raphaël Troncy A
- Jörg Waitelonis
References
- WordNet: An electronic lexical database . Ed. by Christiane Fellbaum. Cambridge, MA: MIT Press, 1998. Pp. xxii, 423.
- WordNet
- Introduction to the CoNLL-2003 shared task
- Large-Scale Named Entity Disambiguation Based on Wikipedia Data
- Learning to link with wikipedia
- Reproducible Research in Computational Science
- The Automatic Content Extraction (ACE) Program – Tasks, Data, and Evaluation
- DBpedia spotlight
- BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network
- Robust Disambiguation of Named Entities in Text