Agreement measures

    Annotation, evaluation and agreement measures in corpus linguistics
    2019, Vol. XXIV-1, pp. 111-129

    The establishment and evaluation of annotated resources as well as automatic annotation systems have growing interest in linguistics and NLP. However, evaluation methods are not always well mastered and used, particularly in linguistics, where the sequential nature of textual data requires specific methods. In this article, we first analyze the notions of corpus and annotation in their multiple forms, then focus on the results of the evaluation questions. In particular, we emphasize the difference in nature between the evaluation of an annotation system and the evaluation of manually multi-annotated resources. We provide an overview of existing evaluation tools and their possible use for each of these two cases.