In this paper we focus on syntactic annotation consistency within Universal Dependencies (UD) treebanks for Russian: UD_Russian-SynTagRus, UD_Russian-GSD, UD_Russian-Taiga, and UD_Russian-PUD. We describe the four treebanks, their distinctive features and devel- opment. In order to test and improve consistency within the treebanks, we reconsidered the experiments by Martínez Alonso and Zeman; our parsing experiments were conducted using a state-of-the-art parser that took part in the CoNLL 2017 Shared Task. We analyze error classes in functional and content relations and discuss a method to separate the errors induced by annotation inconsistency and those caused by syntactic complexity and other factors.
annotation consistency, Universal Dependencies, Russian treebanks, dependency parsing
Alzetta, C., Dell’Orletta, F., Montemagni, S., and Venturi, G. (2018). Dangerous relations in dependency treebanks. In Proceedings of the 16th International Workshop on Treebanks and Linguistic Theories (TLT16), pages 201–210, Praha, Czechia.
Boguslavsky, I., Iomdin, L., Timoshenko, S., and Frolova, T. (2009). Development of the russian tagged corpus with lexical and functional annotation. In Metalanguage and Encoding Scheme Design for Digital Lexicography. MONDILEX Third Open Workshop. Proceedings. Bratislava, Slovakia, pages 83–90.
Boyd, A., Dickinson, M., and Meurers, W. D. (2008). On detecting errors in dependency treebanks. Research on Language and Computation, 6(2):113–137.
Brants, T. (1997). The negra export format for annotated corpora. University of Saarbrücken, Germany.
Brants, T. and Skut, W. (1998). Automation of treebank annotation. In Proceedings of the Joint Conferences on New Methods in Language Processing and Computational Natural Language Learning, pages 49–57. Association for Computational Linguistics.
de Marneffe, M.-C., Grioni, M., Kanerva, J., and Ginter, F. (2017). Assessing the annotation consistency of the universal dependencies corpora. In Proceedings of the Fourth International Conference on Dependency Linguistics (Depling 2017), pages 108–115.
De Smedt, K., Rosén, V., and Meurer, P. (2015). Studying consistency in ud treebanks with iness-search. In Proceedings of the Fourteenth Workshop on Treebanks and Linguistic Theories (TLT14), pages 258–267.
Dickinson, M. and Meurers, W. D. (2003). Detecting inconsistencies in treebanks. In Proceedings of TLT, volume 3, pages 45–56.
Dozat, T., Qi, P., and Manning, C. D. (2017). Stanford’s graph-based neural dependency parser at the conll 2017 shared task. Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 20–30.
Dyachenko, P., Iomdin, L., Lazursky, A., Mityushin, L., Podlesskaya, O., Sizov, V., Frolova, T., and Tsinman, L. (2015). Sovremennoe sostoyanie gluboko annotirovannogo korpusa tekstov russkogo yazyka (syntagrus). Trudy Instituta Russkogo Yazyka im. V. V. Vinogradova, (6):272–300.
Iomdin, L. and Sizov, V. (2009). Structure editor: a powerful environment for tagged corpora. Research Infrastructure for Digital Lexicography, page 1.
Kaljurand, K. (2004). Checking treebank consistency to find annotation errors. Technical report at ResearchGate, https://www.researchgate.net/publication/265628472_Checking_treebank_consistency_to_find_annotation_errors.
Kulick, S., Bies, A., Mott, J., Maamouri, M., Santorini, B., and Kroch, A. (2013). Using derivation trees for informative treebank inter-annotator agreement evaluation. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 550–555.
Martínez Alonso, H. and Zeman, D. (2016). Universal dependencies for the ancora treebanks. Procesamiento del Lenguaje Natural, (57):91–98.
Mediankin, N. and Droganova, K. (2016). Building NLP pipeline for russian with a handful of linguistic knowledge. In Chernyak, E., Ilvovsky, D., Skorinkin, D., and Vybornova, A., editors, Proceedings of the Workshop on Computational Linguistics and Language Science, pages 48–56, Aachen, Germany. NRU HSE, CEUR-WS.
Mel’cuk, I. A. (1981). Meaning-text models: a recent trend in soviet linguistics. Annual
Review of Anthropology, 10(1):27–62.
Nivre, J., de Marneffe, M.-C., Ginter, F., Goldberg, Y., Hajic, J., Manning, C., McDonald, R., Petrov, S., Pyysalo, S., Silveira, N., Tsarfaty, R., and Zeman, D. (2016). Universal dependencies v1: A multilingual treebank collection. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016), pages 1659–1666, Portorož, Slovenia. European Language Resources Association.
Straka, M. and Straková, J. (2017). Tokenizing, pos tagging, lemmatizing and parsing ud 2.0 with udpipe. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 88–99, Vancouver, Canada. Association for Computational Linguistics.
Zeman, D., Popel, M., Straka, M., Hajic, J., Nivre, J., Ginter, F., Luotolahti, J., Pyysalo, S., Petrov, S., Potthast, M., Tyers, F., Badmaeva, E., Gökirmak, M., Nedoluzhko, A., Cinková, S., Hajic jr., J., Hlavácová, J., Kettnerová, V., Urešová, Z., Kanerva, J., Ojala, S., Missilä, A., Manning, C., Schuster, S., Reddy, S., Taji, D., Habash, N., Leung, H., de Marneffe, M.-C., Sanguinetti, M., Simi, M., Kanayama, H., de Paiva, V., Droganova, K., Martínez Alonso, H., Çöltekin, Ç., Sulubacak, U., Uszkoreit, H., Macketanz, V., Burchardt, A., Harris, K., Marheinecke, K., Rehm, G., Kayadelen, T., Attia, M., Elkahky, A., Yu, Z., Pitler, E., Lertpradit, S., Mandl, M., Kirchner, J., Fernandez Alcalde, H., Strnadova, J., Banerjee, E., Manurung, R., Stella, A., Shimada, A., Kwak, S., Mendonça, G., Lando, T., Nitisaroj, R., and Li, J. (2017). CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 1–19, Vancouver, Canada. Association for Computational Linguistics.
