PARALINGUISTIC FEATURES IN STUDENTS' STORYTELLING WITHIN MULTIMODAL COMMUNICATIONS

Didik Rinan Sumekto(1*), Hanandyo Dardjito(2), Silvester Goridus Sukur(3), Yunita Susi Andriani(4), Rismalla Ma'im Olam(5), Dyah Ayu Sulistyorini(6),

(1) Sarjanawiyata Tamansiswa University, Indonesia
(2) Sarjanawiyata Tamansiswa University, Indonesia
(3) Sarjanawiyata Tamansiswa University, Indonesia
(4) Sarjanawiyata Tamansiswa University, Indonesia
(5) Sarjanawiyata Tamansiswa University, Indonesia
(6) Sarjanawiyata Tamansiswa University, Indonesia
(*) Corresponding Author

Abstract


This study assessed paralinguistic features in Indonesian university students' contextual interaction during storytelling. Data collection was recorded from respondents' two video-based storytelling performances, while a self-rated questionnaire was distributed to 235 respondents out of 481 undergraduate English education students using simple random sampling. Data analysis used a mixed-methods approach to qualify students' paralinguistic features using the eduistic linguistics annotator (ELAN) and to quantify the paralinguistic features using statistical analyses through the significance of .05. The findings revealed that the ELAN analyzed the contextual interaction among freshmen, sophomores, juniors, and seniors. The students’ paralinguistic features corresponded with the lexical and semantic evidence, which approached the function of monosyllabic and bisyllabic words, nonverbal expressions, and interpretations. Bodily gesture quantitatively showed moderate category for 44.7% (t = 2.434; p = .016), articulation showed attributable category for 54.0% (t = 3.789; p = .000), facial expression showed moderate category for 61.7% (t = 2.472; p = .014), and voice loudness showed attributable category for 47.7% (t = 4.121; p = .000). Herein, positive and significant attribution were shown by these paralinguistic features towards students' contextual interactions in storytelling for 34.9% with the multiple regressions (F = 7.990, R² = .349, and p < .000). The paralinguistic features empirically address the multimodal communication modes to improve teaching and learning activities.

Keywords


contextual interaction, multimodal communication, storytelling

Full Text:

PDF

References


Aihara, R., Takiguchi, T., & Yasuo Ariki, Y. (2015). Individuality-preserving voice conversion for articulation disorders using phoneme-categorized. ACM Transactions on Accessible Computing, 6(4), 1-17. https://doi.org/10.1145/2738048

Ali, A. Z. M., Segaran, K., & Hoe, T. W. (2015). Effects of verbal components in 3D talking-head on pronunciation learning among non-native speakers. Educational Technology & Society, 18(2), 313-322.

Asadi, S., Wexler, A. S., Cappa, C. D., Barreda, S., Bouvier, N. M., & Ristenpart, W. D. (2019). Aerosol emission and super emission during human speech increase with voice loudness. Scientific Reports, 9(2348), 1-11. https://doi.org/10.1038/s41598-01938808z

Behera, A., Matthew, P., Keidel, A., Vangorp, P., Fang, H., & Canning, S. (2020). Associating facial expressions and upper-body gestures with learning tasks for enhancing intelligent tutoring systems. International Journal of Artificial Intelligence in Education, 30, 236-270. https://doi.org/10.1007/s40593-020-00195-2

Bursic, S., Boccignone, G., Ferrara, A., Alessandro D’Amelio, A., & Lanzarotti, R. (2020). Improving the accuracy of automatic facial expression recognition in speaking subjects with deep learning. Applied Sciences, 10, 1-15. https://doi.org/10.3390/app10114002

Byun, S., Yoon, S., & Jung, K. (2020). Comparative studies on machine learning for paralinguistic signal compression and classification. The Journal of Supercomputing, 76(1), 8357-8371. https://doi.org/10.1007/s11227-020-03346-3

Cabrera, L., Lorenzi, C., & Bertoncini, J. (2015). Infants discriminate voicing and place of articulation with reduced spectral and temporal modulation cues. Journal of Speech, Language, and Hearing Research, 58(3), 1033-1042. https://doi.org/10.1044/2015_JSLHR-H-14-0121

Cánovas, C. P., Javier Valenzuela, J., Carrión, D. A., Olza, I., & Ramscar, M. (2020). Quantifying the speech-gesture relation with massive multimodal datasets: Informativity in time expressions. PLoS ONE, 15(6), 1-18. https://doi.org/10.1371/journal.pone.0233892

Caridakis, G., Wagner, J., Raouzaiou, A., Lingenfelser, F., Karpouzis, K., & Andre, E. (2013). A cross-cultural, multimodal, affective corpus for gesture expressivity analysis. Journal of Multimodal User Interfaces, 7, 121-134. https://doi.org/10.1007/s12193012-0112-x

Chanes, L., Wormwood, J. B., Betz, N., & Barrett, L. F. (2018). Facial expression predictions as drivers of social perception. Journal of Personality and Social Psychology, 114(3), 380-396. https://doi.org/10.1037/pspa0000108

ELAN. (2021, July 6). ELAN–Previous releases. Nijmegen: Max Planck Institute for Psycholinguistics. https://archive.mpi.nl/tla/elan/previous

Esling, J. H., Moisik, S. R., Benner, A., & Crevier-Buchman, L. (2019). 1 – Voice and voice quality. Cambridge Core: University of Sussex Library https://doi.org/10.1017/9781108696555.001

Gosztolya, G., & Tóth, L. (2018). A feature selection-based speaker clustering method for paralinguistic tasks. Pattern Analysis & Applications, 21(1), 193-204. https://doi.org/10.1007/s10044-017-0612-0

Griethuijsen, R. A. L. F., Eijck, M. W., Haste, H., Brok, P. J., Skinner, N. C., Mansour, N., Gencer, A. S., & BouJaoude, S. (2014). Global patterns in students’ views of science and interest in science. Research in Science Education, 45(4), 581-603. https://doi.org/10.1007/s11165-014-9438-6

Gupta, R., Audhkhasi, K., Lee, S., & Narayanan, S. (2016). Detecting paralinguistic events in audio stream using context in features and probabilistic decisions. Computer Speech and Language, 36, 72-92. https://doi.org/10.1016/j.csl.2015.08.003

Haj, M. E., Daoudi, M., Gallouj, K., Moustafa, A. A., & Nandrino, J-. L. (2018). When your face describes your memories: Facial expressions during retrieval of autobiographical memories. Reviews in the Neurosciences, 29(8), 861-872. https://doi.org/10.1515/revneuro-2018-0001

Ishi, C. T., Mikata, R., & Ishiguro. (2020). Person-directed pointing gestures and interpersonal relationship: Expression of politeness to friendliness by android robots. IEEE Robotics and Automation Letters, 5(4), 6081-6088. https://doi.org/10.1109/LRA.2020.3011354

Kano, T., Takamichi, S., Sakti, S., Neubig, G., Toda, T., & Nakamura, S. (2018). An end-to-end model for cross-lingual transformation of paralinguistic information. Machine Translation, 32(2), 353-368. https://doi.org/10.1007/s10590-018-9217-7

Kaňok, M., & Novotný, M. (2019). Effect of age and gender on articulation of voiced and voiceless stop consonants in Czech. Clinician and Technology, 49(3), 97-101. https://doi.org/10.14311/CTJ.2019.3.05

Kita, S., Alibali, M. W., & Chu, M. (2017). How do gestures influence thinking and speaking? The gesture-for-conceptualization hypothesis. Psychological Review, 124(3), 245-266. http://doi.org/10.1037/rev0000059

Kita, S. (2009). Cross-cultural variation of speech-accompanying gesture: A review. Language and Cognitive Processes, 24(2), 145-167. https://doi.org/10.1080/01690960802586188

Kozminska, K. (2018). Intonation, identity and contact-induced change among Polish-speaking migrants in the U.K. Journal of Sociolinguistics, 23(1), 29-53. https://doi.org/10.1111/josl.12318

Kwon, A. Y., Vallotton, C. D., Kiegelmann, M., & Wilhelm, K. H. (2018). Cultural diversification of communicative gestures through early childhood: A comparison of children in English-, German-, and Chinese-speaking families. Infant Behavior and Development, 50, 328-339. https://doi.org/10.1016/j.infbeh.2017.10.003

Labuschagne, P. A. (2017). Non-verbal and paralinguistic political communication in sport: An analysis of images of Springbok rugby. Communication: South African Journal for Communication Theory and Research, 43(1), 80-102. https://doi.org/10.1080/02500167.2017.1307243

Leach, S., & Weick, M. (2020). When smiles (and frowns) speak words: Does power impact the correspondence between self-reported affect and facial expressions? British Journal of Psychology, 111(1), 683-701. https://doi.org/10.1111/bjop.12433

Leung, Y., Oates, J., & Chan, S. P. (2018). Voice, articulation, and prosody contribute to listener perceptions of speaker gender: A systematic review and meta-analysis. Journal of Speech, Language, and Hearing Research, 61(2), 266-297. https://doi.org/10.1044/2017_JSLHR-S-17-0067

Li, H., Yuan, Y. C., Bazarova, N. N., & Bell, B. S. (2018). Talk and let talk: The effects of language proficiency on speaking up and competence perceptions in multinational teams. Group & Organization Management, 44(5), 953-989. https://doi.org/10.1177/1059601118756734

Li, K., Jin, Y., Akram, M. W., Han, R., & Chen, J. (2019). Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy. The Visual Computer, 36, 391-404. https://doi.org/10.1007/s00371-019-01627-4

Lierde, K. M. V., Letter, M. D., Vermeersch, H., Roche, N., Stillaert, F., Lemmens, G., Peeters, P., Rogiers, X., Blondeel, P., & Corthals, P. (2015). Longitudinal progress of overall intelligibility, voice, resonance, articulation and oromyofunctional behavior during the first 21 months after Belgian facial transplantation. Journal of Communication Disorders, 54, 42-56. https://doi.org/10.1016/j.jcomdis.2014.09.001

Lin, Y-. L. (2020). A helping hand for thinking and speaking: Effects of gesturing and task planning on second language narrative discourse. System, 91, 1-10. https://doi.org/10.1016/j.system.2020.102243

Mortaji, L. (2018). Effects of sustained impromptu speaking and goal setting on public speaking competency development: A case study of EFL college students in Morocco. English Language Teaching, 11(2), 82-98. http://doi.org/10.5539/elt.v11n2p82

Myers, B. R., & Finnegan, E. M. (2015). The effects of articulation on the perceived loudness of the projected voice. Journal of Voice, 29(3), 3909-390.e15. https://doi.org/10.1016/j.jvoice.2014.07.022

Nandamudi, S., & Scherer, R. C. (2019). Airflow vibrato: Dependence on pitch and loudness. Journal of Voice, 33(6), 815-830. https://doi.org/10.1016/j.jvoice.2018.05.007

Nilsen, E. S., Rints, A., Ethier, N., & Moroz, S. (2016). Mother–child communication: The influence of ADHD symptomatology and executive functioning on paralinguistic style. Frontiers Psychology, 7(1203), 1-11. https://doi.org/10.3389/fpsyg.2016.01203

Nugrahaningsih, N., & Porta, M. (2019). Soft biometrics through hand gestures driven by visual stimuli. ICT Express, 5(2), 94-99. https://doi.org/10.1016/j.icte.2018.04.014

Park, Y., & Stepp, C. E. (2018). The effects of stress type, vowel identity, baseline f0, and loudness on the relative fundamental frequency of individuals with healthy voices. Journal of Voice, 33(5), 603-610. https://doi.org/10.1016/j.jvoice.2018.04.004

Pérez-Espinosa, H., Martínez-Miranda, J., Espinosa-Curiel, I., Rodríguez-Jacobo, J., & Avila-George, H. (2017). Using acoustic paralinguistic information to assess the interaction quality in speech-based systems for elderly users. International Journal of Human-Computer Studies, 98, 1-13. https://doi.org/10.1016/j.ijhcs.2016.09.013

Pichel, B. (2016). From facial expressions to bodily gestures: Passions, photography and movement in French 19th-century sciences. History of the Human Sciences, 29(1), 27-48. https://doi.org/10.1177/0952695115618592

Šebesta, P., Trebický, V., Fialová, J., & Havlícek, J. (2019). Roar of a champion: Loudness and voice pitch predict perceived fighting ability but not success in MMA fighters. Frontiers Psychology, 10(859), 1-9. https://doi.org/10.3389/fpsyg.2019.00859

Sinha, S., Jain, A., & Agrawal, S. S. (2017). Empirical analysis of linguistic and paralinguistic information for automatic dialect classification. Artificial Intelligence Review, 51(2), 647-672. https://doi.org/10.1007/s10462-017-9573-3

Slepian, M. L., & Carr, E. W. (2019). Facial expressions of authenticity: Emotion variability increases judgments of trustworthiness and leadership. Cognition, 183, 82-98. https://doi.org/10.1016/j.cognition.2018.10.009

So, W-. C., & Wong, K. -Y. M. (2017). Tracing the development of spatially modulated gestures in the manual modality in nonsigning Chinese-speaking children. Applied Psycholinguistics, 39(3), 527-544. https://doi.org/10.1017/S0142716417000510

Sumekto, D. R., & Setyawati, H. (2020). Revealing lecturer’s paralinguistic attribution: How the visual manner contributes to students’ non-cognitive skills. Indonesian Journal of Applied Linguistics, 9(3), 559-571. https://doi.org/10.17509/ijal.v9i3.23206

Sumekto, D. R., Taufiqulloh, T., Setyawati, H., Hikmah, S., & Ghozali, I. (2021). Understanding the paralinguistic features disclosure depicted in the lecturer’s visual modes of writing class instruction. 3L: Language, Linguistics, Literature, 27(4), 173-192. https://doi.org/10.17576/3L-2021-2704-13

Thacker, J. M., Chambers, C. G., & Graham, S. A. (2018). Five-year-olds’ and adults’ use of paralinguistic cues to overcome referential uncertainty. Frontiers Psychology, 9(143), 1-10. https://doi.org/10.3389/fpsyg.2018.00143

Tooley, K. M., Konopka, A. E., & Watson, D. G. (2018). Assessing priming for prosodic representations: Speaking rate, intonational phrase boundaries, and pitch accenting. Memory & Cognition. 46(4), 625-641. https://doi.org/10.3758/s13421-018-0789-5

Trujillo, J. P., Vaitonyte, J., Simanova, I., & Özyürek, A. (2019). Toward the markerless and automatic analysis of kinematic features: A toolkit for gesture and movement research. Behavior Research Methods, 51, 769-777. https://doi.org/10.3758/s13428-01810868

Uştuk, Ö., & Aydın, S. (2016). The effects of the use of paralinguistic cues on foreign language anxiety among English as a foreign language speaker. Innovation in Language Learning and Teaching, 12(3), 289-302. https://doi.org/10.1080/17501229.2016.1211133

Weisbuch, M., Grunberg, R. L., Slepian, M. L., & Ambady, N. (2016). Perceptions of variability in facial emotion influence beliefs about the stability of psychological characteristics. Emotion, 16(7), 957-964. https://doi.org/10.1037/emo0000123

Wessel-Tolvig, B., & Paggio, P. (2016). Revisiting the thinking-for-speaking hypothesis: Speech and gesture representation of motion in Danish and Italian. Journal of Pragmatics, 99, 39-61. https://doi.org/10.1016/j.pragma.2016.05.004

Wood, A., Martin, J. D., Alibali, M. W., & Niedenthal, P. M. (2018). A sad thumbs up: Incongruent gestures and disrupted sensorimotor activity both slow processing of facial expressions. Cognition and Emotion, 33(6), 1-14. https://doi.org/10.1080/02699931.2018.1545634

Wu, H., Zhang, S., Liu, J., Qiu, J., & Zhang, X. (2018). The gesture disagreement problem in free-hand gesture interaction. International Journal of Human–Computer Interaction, 35(3), 1-13. https://doi.org/10.1080/10447318.2018.1510607

Zarate, J. M., Tian, M., Woods, K. J. P., & Poeppel, D. (2015). Multiple levels of linguistic and paralinguistic features contribute to voice recognition. Scientific Reports, 5(1), 1-9 https://doi.org/10.1038/srep11475




DOI: https://doi.org/10.24071/llt.v27i2.8701

Refbacks

  • There are currently no refbacks.


Copyright (c) 2024 Didik Rinan Sumekto, Hanandyo Dardjito, Silvester Goridus Sukur, Retno Untari

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Indexed and abstracted in:

     

 

 

LLT Journal Sinta 1 Certificate (S1 = Level 1)

We would like to inform you that LLT Journal: A Journal on Language and Language Teaching has been nationally accredited Sinta 1 by the Ministry of Education, Culture, Research and Technology of the Republic of Indonesia based on the decree  No. Surat Keputusan 169/E/KPT/2024. Validity for 5 years: Vol ... No 1, 20... till Vol ... No 2, 20...


Sinta 1 certificate to post here asap. Thank you for your patience and understanding. 

 

  

 

This work is licensed under CC BY-SA.

Creative Commons Attribution-ShareAlike 4.0 International License

 

Free counters!


 LLT Journal: A Journal on Language and Language Teaching, DOI: https://doi.org/10.24071/llt, e-ISSN 2579-9533 and p-ISSN 1410-7201, is published twice a year, namely in April and October by the English Language Education Study Programme of Teacher Training and Education Faculty of Sanata Dharma University, Yogyakarta, Indonesia.