Published: 2016-09-16
ISBN: 978-91-7685-679-6
ISSN: 1650-3686 (print), 1650-3740 (online)
This paper presents an analysis of the temporal alignment be- tween head movements and associated speech segments in the NOMCO corpus of first encounter dialogues [1]. Our results show that head movements tend to start slightly before the onset of the corresponding speech sequence and to end slightly after, but also that there are delays in both directions in the range of -/+ 1s. Various factors that may influence delay duration are investigated. Correlations are found between delay length and the duration of the speech sequences associated with the head movements. Effects due to the different head movement types are also discussed.
[1] P. Paggio, J. Allwood, E. Ahls´en, K. Jokinen, and C. Navarretta, “The NOMCO multimodal nordic resource - goals and characteristics,” in Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC’10). Valletta, Malta: European Language
Resources Association (ELRA), 2010.
[2] D. McNeill, Hand and Mind: What Gestures Reveal About Thought. University of Chicago Press, Chicago, 1992.
[3] D. McNeill, Gesture and thought. University of Chicago Press, 2005.
[4] A. Kendon, Gesture: Visible Action as Utterance. Cambridge University Press, 2004.
[5] S. Kita and A. O¨ zyu¨rek, “What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking,” Journal of Memory and Language, vol. 48, no. 1, pp. 16–32, 2003.
[6] J.-P. De Ruiter, “The production of gesture and speech,” in Language and Gesture. Cambridge University Press, 2000.
[7] D. Bolinger, Intonation and its parts: Melody in spoken English. CA: Stanford: Stanford, 1986.
[8] A. Kendon, “Gesture and speech: two aspects of the process of utterance,” in Nonverbal Communication and Language, M. R. Key, Ed. Mouton, 1980, pp. 207–227.
[9] D. P. Loehr, “Gesture and intonation,” Ph.D. dissertation, Georgetown University, 2004.
[10] D. P. Loehr, “Aspects of rhythm in gesture and speech,” Gesture, vol. 7, no. 2, 2007.
[11] T. Leonard and F. Cummins, “The temporal relation between beat gestures and speech,” Language and Cognitive Processes, vol. 26, no. 10, pp. 1457–1471, 2010.
[12] G. Giorgolo and F. A. Verstraten, “Perception of ‘speechand-gesture’ integration,” in Proceedings of the International Conference on Auditory-Visual Speech Processing 2008, 2008, pp. 31–36.
[13] U. Hadar, T. Steiner, E. C. Grant, and F. C. Rose, “Head movement correlates of juncture and stress at sentence level,” Language and Speech, vol. 26, no. 2, pp. 117–129, 1983.
[14] U. Hadar, T. Steiner, and F. C. Rose, “The timing of shifts of head postures during conversation,” Human Movement Science, vol. 3, no. 3, pp. 237–245, 1984.
[15] M. Kipp, Gesture Generation by Imitation – From Human Behavior to Computer Character Animation. Boca Raton, Florida: Dissertation.com, 2004.
[16] C. Navarretta and P. Paggio, “Verbal and Non-Verbal Feedback in Different Types of Interactions,” in Proceedings of LREC 2012, Istanbul Turkey, May 2012, pp. 2338–2342.
