This paper deals with the relation between speech and form of co-occurring iconic hand gestures. It focuses on multimodal expression of eventualities. We investigate to what extent it is possible to automatically classify gestural features from the categorization of verbs in a wordnet. We do so by applying supervised machine learning to an annotated multimodal corpus. The annotations describe form features of gestures. They also contain information about the type of eventuality; verb Aktionsart and Aspect; which were extracted from plWordNet 2.0. Our results confirm the hypothesis that the Eventuality Type and Aktionsart are related to the form of gestures. They also indicate that it is possible to some extent to classify certain form characteristics of gesture from the linguistic categorization of their lexical affiliates. We also identify the gestural form features which are most strongly correlated to the Viewpoint adopted in gesture.
Multimodal eventuality expression; iconic co-speech gesture; wordnet; machine learning
Polish WordNet. Wroclaw University of Technology. http://plwordnet.pwr.wroc.pl/wordnet/.
Tennant; R. and M. Brown The American Sign Language Handshape Dictionary. Washington; DC:
Gallaudet University Press (2010).
Alibali; M. W.; Heath; D. C.; and Meyers; H. J. Effects of visibility between speakers and listeners on gesture production: Some gestures are meant to be seen. Journal of Memory and Language; 44:159–188 (2001).
Becker; R.; Cienki; A.; Bennett; A.; Cudina; C.; Debras; C; Fleischer; Z.; M. Haaheim; T. Mueller; K. Stec; and A. Zarcone. Aktionsarten; speech and gesture. In Gesture and Speech in Interaction ’11; (2011).
Bressem; J. A linguistic perspective on the notation of form features in gestures. Body – Language – Communication. Handbooks of Linguistics and Communication Science. Berlin; New York: Mouton de Gruyter (2013).
Brinton; L. The structure of modern English: A linguistic introduction. John Benjamins Publishing Company (2000).
Comrie; B. Aspect. Cambridge: Cambridge University Press (1976).
Cohen; J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement; 20(1):37–46 (1960).
Duncan; S. Gesture; verb Aspect; and the nature of iconic imagery in natural discourse. Gesture; 2(2):183–206 (2002).
Eisenstein; J.and Davis; R. Gesture features for coreference resolution. In Renals; S.; Bengio; S.; and Fiscus; J.; editors; MLMI 06; pages 154–155 (2006).
Eisenstein; J.and Davis; R. Gesture improve coreference resolution. In Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL; pages 37–40; New York (2006).
Fellbaum; C.; WordNet: An Electronic Lexical Database. MIT Press; Cambridge; MA (1998).
Fujie; S.; Ejiri; Y.; Nakajima; K.; Matsusaka; Y.; and Kobayashi; T. A conversation robot using head gesture recognition as para-linguistic information. In Proceedings of the 13th IEEE International Workshop on Robot and Human Interactive Communication; 159–154 (2004).
Grice; H. Logic and Conversation. Syntax and Semantics; 3:41–58. Academic Press; New York (1976).
Jokinen; K. ; Navarretta; C.; and Paggio; P. Distinguishing the communicative function of gesture. Proceedings of MLMI (2008).
Karpi´nski; M.; Jarmolowicz-Nowikow; E.; Malisz; Z.; Szczyszek; M.; Juszczyk; J. Rejestracja; transkrypcja i tagowanie mowy oraz gestów w narracji dzieci i doroslych. Investigationes Linguisticae; 17 (2008).
Kendon; A. Gesture: Visible Action As Utterance. Cambridge University Press; Cambridge (2004).
Kipp; M. Gesture Generation by Imitation - From Human Behavior to Computer Character Animation. Boca Raton; Florida (2004).
Kita; S. and A. Özyürek. What does cross–linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language; 48(1):16–32 (2003).
Kopp; S.; Bergmann; K.; and Ipke; W. Multimodal communication from multimodal thinking – towards an integrated model of speech and gesture production. Semantic Computing; 2(1):115–136 (2008).
Krauss; R. M.; Chen; Y.; and Gottesman; R. F. Lexical gestures and lexical access. a process model. In McNeill; D.; editor; Language and Gesture; pages 261–283. Cambridge University Press; New York (2000).
Laskowski; L. Kategorie morfologiczne j?ezyka polskiego — charakterystyka funkcjonalna. PWN; Warszawa (1998).
Levin; B. English Verb Classes and Alternations: A Preliminary Investigation. University of Chicago Press; Chicago (1993).
Lis; M. Annotation scheme for multimodal communication: Employing plWordNet 1.5. In Proceedings of the Formal and Computational Approaches to Multimodal Communication Workshop. 24th European Summer School in Logic; Language and Information (ESSLLI’12) (2012).
Lis; M. Influencing gestural representation of eventualities: insights from ontology. In Proceedings of the 14th ACM International Conference on Multimodal Interaction (ICMI’12); 281–288; (2012).
Lis; M. Multimodal representation of entities: A corpus-based investigation of co-speech hand gesture. PhD dissertation; University of Copenhagen (submitted).
Louwerse; M.; Jeuniaux; P.; Hoque; M.; Wu; J.; and Lewis; G. Multimodal communication in computermediated map task scenarios. In Sun; R. and Miyake; N.; editors; Proceedings of the 28th Annual Conference of the Cognitive Science Society; Mahwah; NJ. Erlbaum (2006).
Louwerse; M. M.; Benesh; N.; Hoque; M.; Jeuniaux; P.; Lewis; G.; Wu; J.; and Zirnstein; M. Multimodal communication in face-to-face conversations. In Sun; R. and Miyake; N.; editors; Proceedings of the 29th Annual Conference of the Cognitive Science Society; Mahwah; NJ. Erlbaum (2006).
Maziarz; M. Non-lexical verb synsets in upperhierarchy levels of polish wordnet 2.0. Technical report; Wroclaw University of Technology (2012).
Maziarz; M.; Piasecki; M.; Szpakowicz; S.; Rabiega-Wi´sniewska; J. and B. Hojka. Semantic relations between verbs in polish wordnet 2.0. Cognitive Studies; (11):183–200 (2011).
McNeill; D. Hand and Mind: What Gestures Reveal About Thought. University of Chicago Press; Chicago (1992).
McNeill; D. Gesture and Thought. University of Chicago Press; Chicago (2005).
Melinger; A. and Levelt; W. Gesture and the communicative intention of the speaker. Gesture; 4(2):119–141 (2005).
Mlynarczyk; A. Aspectual pairing in Polish. PhD dissertation; University of Utrecht (2004).
Morency; L.-P.; de Kok; I.; and Gratch; J. A probabilistic multimodal approach for predicting listener backchannels. Autonomous Agents and Multi-Agent Systems; 20:70–84 (2009).
Morency; L.-P.; Sidner; C.; Lee; C.; and Darrell; T. Contextual recognition of head gestures. In Proceedings of the International Conference on Multimodal Interfaces (2005).
Morency; L.-P.; Sidner; C.; Lee; C.; and Darrell; T. Head gestures for perceptual interfaces: The role of context in improving recognition. Artificial Intelligence; 171(8–9):568–585 (2007).
Navarretta; C. Anaphora and gestures in multimodal communication. In Proceedings of the 8th Discourse Anaphora and Anaphor Resolution Colloquium (DAARC 2011); pages 171–181; Faro; Portugal
(2011).
Navarretta; C. and Paggio; P. Classification of feedback expressions in multimodal data. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL’10); pages 318–324; Uppsala; Sweden (2010).
Parrill; F. Viewpoint in speech–gesture integration: Linguistic structure; discourse structure; and event structure. Language and Cognitive Processes; 25(5):650–668 (2010).
Parrill; F.; Bergen; B. and P. Lichtenstein. Grammatical aspect; gesture; and conceptualization: Using co-speech gesture to reveal event representations. In Cognitive Linguistics; 24(1): 135–158 (2013).
Peirce; C. S. Collected Papers of Charles Sanders Peirce (1931-58). Hartshorne; P. Weiss and A. Burks; Cambridge; MA: Harvard University Press (1931).
Poggi; I. Iconicity in different types of gestures. Gesture; 8(1):45–61 (2008).
Ramchard; G. Post-davidsionianism. Theoretical Linguistics; 31(3):359–373 (2005).
