Natural language processing (NLP) tools are often developed with the intention to ease human processing, a goal which is hard to measure. Eye movements in reading are known to reflect aspects of the cognitive processing of text (Rayner et al., 2013) and is becoming available in consumer products. We explore how eye movements reflect aspects of reading that are of relevance to NLP system evaluation and development. In this paper we present an analysis of the differences between reading automatic sentence compressions and manually simplified newswire using eye-tracking experiments and readers’ evaluations. We show that both manual simplification and automatic sentence compression provide texts that are easier to process than standard newswire, and that the main source of difficulty in processing machine-compressed text is ungrammaticality. Importantly, we find that grammatical errors introduced by automatic processing and grammatical complexity stemming from human authors affects reading behavior and the subjective assessment of sentence readability differently. Especially the proportion of regressions to previously read text is found to be sensitive to the differences in human- and computer-induced complexity. This finding is relevant for evaluation of automatic summarization, simplification and translation systems designed with the intention of facilitating human reading.
Marcelo Adriano Amancio, UK Sheffield, and Lucia Specia. 2014. An analysis of crowdsourced text simplifications. In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR)@ EACL, pages 123–130.
Jørg Asmussen. 2001. Korpus 2000. Korpuslingvistik (NyS30).
Rebekah George Benjamin. 2012. Reconstructing readability: Recent developments and recommendations in the analysis of text difficulty. Educational Psychology Review, 24(1):63–88.
Carl-Hugo Bjornsson. 1983. Readability of Newspapers in 11 Languages. Reading Research Quarterly, 18(4):480–497.
Philippe Blache, Stephane Rauzy. 2012. Robustness and processing difficulty models. a pilot study for eye-tracking data on the french treebank. In 24th International Conference on Computational Linguistics, page 21.
Bernd Bohnet. 2010. Very high accuracy and fast dependency parsing is not a contradiction. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 89–97. Association for Computational Linguistics.
William Coster and David Kauchak. 2011. Simple English Wikipedia: a new text simplification task. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, volume 2, pages 665–669. Association for Computational Linguistics.
Johan Falkenjack and Arne J¨onsson. 2014. Classifying easy-to-read texts without parsing. In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR)@ EACL, pages 114–122.
Lijun Feng, Martin Jansche, Matt Huenerfauth, and Noémie Elhadad. 2010. A comparison of features for automatic readability assessment. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 276–284. Association for Computational Linguistics.
Rudolph Flesch. 1948. A new readability yardstick. Journal of applied psychology, 32(3):221.
Kenneth Holmqvist, Marcus Nyström, Richard Andersson, Richard Dewhurst, Halszka Jarodzka, and Joost Van de Weijer. 2011. Eye tracking: A comprehensive guide to methods and measures.
Sigrid Klerke and Anders Søgaard. 2012. DSim , a Danish Parallel Corpus for Text Simplification. In Proceedings of Language Resources and Evaluation (LREC 2012), pages 4015–4018.
Kevin Knight and Daniel Marcu. 2000. Statisticsbased summarization-step one: Sentence compression. In AAAI/IAAI, pages 703–710.
Matthias T Kromann. 2003. The Danish Dependency Treebank and the DTAG treebank tool. In Proceedings of the Second Workshop on Treebanks and Linguistic Theories (TLT), page 217.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data.
Ryan McDonald. 2006. Discriminative sentence compression with soft syntactic evidence. In EACL.
Anneli Olsen. 2012. The tobii i-vt fixation filter. Tobii Technology.
Slav Petrov, Dipanjan Das, and Ryan McDonald. 2011. A universal part-of-speech tagset. Arxiv preprint ArXiv:1104.2086.
Keith Rayner, Kathryn H Chace, Timothy J Slattery, and Jane Ashby. 2006. Eye movements as reflections of comprehension processes in reading. Scientific Studies of Reading, 10(3):241–255.
Keith Rayner, Alexander Pollatsek, and D Reisberg. 2013. Basic processes in reading. The Oxford Handbook of Cognitive Psychology, pages 442–461.
Sarah E Schwarm and Mari Ostendorf. 2005. Reading Level Assessment Using Support Vector Machines and Statistical Language Models. In Proceedings of the 43rd Annual Meeting of the ACL, pages 523–530.
Advaith Siddharthan and Napoleon Katsos. 2012. Offline sentence processing measures for testing readability with users. In Proceedings of the First Workshop on Predicting and Improving Text Readability for target reader populations, pages 17–24. Association for Computational Linguistics.
Sara Stymne, J¨org Tiedemann, Christian Hardmeier, and Joakim Nivre. 2013. Statistical machine translation with readability constraints. In Proceedings of the 19th Nordic Conference on Computational Linguistics (NODALIDA’13), pages 375–386.
Sowmya Vajjala and Detmar Meurers. 2014. Exploring measures of readability for spoken language: Analyzing linguistic features of subtitles to identify age-specific tv programs. In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR)@EACL, pages 21–29.
Kristian Woodsend and Mirella Lapata. 2011. WikiSimple: Automatic Simplification ofWikipedia Articles. In Twenty-Fifth AAAI Conference on Artificial Intelligence.
Zhemin Zhu, Delphine Bernhard, and I. Gurevych. 2010. A monolingual tree-based translation model for sentence simplification. In Proceedings of The 23rd International Conference on Computational Linguistics, pages 1353–1361. Association for Computational Linguistics.
