Analyzing Audio Survey Responses: Insights from a Research Project on Political Trust

Empirical Social Science Research Seminar Series
15 May 2024

Camille Landesvatter

University of Mannheim

Hello! Who is your speaker today?

  • 👋 Camille Landesvatter

  • 📍 Research Associate at the MZES, University of Mannheim

  • 🏫 Studied social sciences and sociology at the University of Stuttgart and Mannheim

  • 🎓 Started my PhD in Sociology in 2020 under the supervision of Florian Keusch and Paul Bauer (TrustME project)

  • 🏠 Moved to Berlin in 2021 and visited the WZB for a research stay

  • Finished my PhD last month! 🎉

Today’s Agenda

    1. Audio Data in the Social Sciences and in Surveys: What insights can we gain by leveraging spoken responses in surveys to understand respondents’ views more deeply?
    1. Emotion Analysis: What insights can we gain through automated emotion analysis of audio responses?
    1. Emotions in Political Trust: Can we use audio data and emotion analysis to measure political trust? Do emotions affect political trust judgments?

Audio Data in the Social Sciences and in Surveys

What is Audio Data?

Audio Data in the Social Sciences and in Surveys

  • Audio data = information represented in the form of sound waves captured in a digital format (e.g., WAV)

Figure 1: Waveform of an audio file (amplitude over time).
  • Sources for audio data for the social sciences: Interviews, Social Media (e.g. live streams, Youtube), public speeches and debates, political talk shows and podcasts, press conferences, recorded audio survey answers.

What are characteristics of Audio Data?

Audio Data in the Social Sciences and in Surveys

  • Audio Data contains rich information including paralinguistic elements
    • e.g., pitch, volume, laughter and sighs, tone of voice, pause and silence, emotional cues

Cognitive Processing Modes:

  • “System 1” versus “System 2” language (Lütters et al. 2018)
    • System 1: Intuitive, Automatic, and Fast Thinking → Spoken language
    • System 2: Analytical, Deliberate, and Slow Thinking → Written language

Audio Answers in Surveys

Audio Data in the Social Sciences and in Surveys

  • Spoken answers compared to written answers are longer, more elaborate and detailed (Gavras et al. 2022, Höhne & Gavras 2022, Lütters et al. 2018, Revilla et al. 2020)

Explanations:

  • Gavras et al. 2022: Audio formats facilitate the answer process by enabling open narration, intuitive and spontaneous answers (≠ intentional and conscious text answers).

  • Revilla et al. 2020: Speaking requires less effort than typing and voice formats make survey answering easier and quicker.

  • Unfortunately, spoken answers compared to written answers increase response times and non-response rates (Revilla et al. 2020, Lütters et al. 2018)

Methods for the Analysis of Audio Data

Audio Data in the Social Sciences and in Surveys

  • Increasing number of methods to analyze audio data

  • Intersection of Computer Science, Computational Linguistics and Computational Social Science

  • Examples of Methods:

    • Automatic Speech Recognition (“speech-to-text”)

    • Natural Language Processing

    • Speaker Diarization / Speaker Identification

    • Environmental Sound Analysis

    • Speech Emotion Recognition

Emotion Analysis

Speech Emotion Recognition

Emotion Analysis

  • Why use Speech for Emotion Analysis? → Recognizing emotions from text is difficult because it is stripped of all paralinguistic and acoustic features
  • Speech Emotion Recognition (SER) is part of the field of Automated Emotion Recognition, further intersecting with disciplines such as Neurocomputing and Affective Computing
  • Evolution of Models from traditional Machine Learning models (e.g., logistic regression) to deep learning based methods, e.g. SpeechBrain (Ravanelli et al. 2021)

Emotion Recognition = Sentiment Analysis?

Emotion Analysis

The concept of emotions is not always clearly distinguished from similar phenomena such as mood, affect, and feeling. (Gabriel et al. 2023, 39)

  • Sentiment: the valence of a feeling (e.g., positive versus negative)

  • Emotions: a more complex and multi-dimensional state of feeling further characterized by their intensity as well as their cognitive evaluations (e.g., other-person control for anger)

  • Affect = “an umbrella term that is used to refer to both emotions and moods” (Lee, Dirks, and Campagna 2023, 549)

Research Project:
Emotions in Political Trust

Background

Emotions in Political Trust

  • Conventional notion of political trust where trust judgments are made upon the basis of risk calculations and rational choice-making processes
  • Recently challenged by the idea of an “affect-based” form of political trust (e.g., Theiss-Morse and Barton 2017)
“A decision to trust a government organization may […] not always be conscious and/or rational.” (Grimmelikhuijsen 2012, 57)
  • “Automatic Hot Cognition” (Lodge and Taber 2013)

Research Questions and Motivation

Emotions in Political Trust

What thought processes and associations come to respondents’ minds when prompted to discuss ‘politics’ and their level of trust?
Are individual trust judgments in surveys driven by affective components?

Motivation:

  1. Research on determinants of political trust generates knowledge on how trust evolves and changes over time.

  2. The influence of emotions on political trust is crucial in order to understand how the media such as television uses emotions as an instrument for influencing people, for example by showing politicians’s emotional appearances in the media (Gabriel et al. 2023)

Questionnaire Design

Emotions in Political Trust

Political Trust Question “How often can you trust the federal government in Washington to do what is right?” closed-ended, 4 categories (Always; Most of the time; About half of the time; Never; Don’t Know
Probing Question  “The previous question was: ‘How often can you trust the federal government in Washington to do what is right?’.
Your answer was: ‘About half of the time’.
In your own words, please explain why you selected this answer.” 		open-ended, audio request, SVoice tool (Höhne, Gavras and Qureshi 2021)

Data

Emotions in Political Trust

  • Self-administered web survey, September 6 to October 27, 2023

  • U.S. non-probability sample; \(n\)=1,474 with 491 audio open answers

  • quota-based (U.S. Census Bureau 2015) with challenges in obtaining sufficient participants in the oldest age category (58+)

Methods

Emotions in Political Trust

Figure 1: Methods for Sentiment and Emotion Analysis.

Results: Sentiment

Emotions in Political Trust

  • The commonly used trust in government question used in many surveys produces predominantly negative associations and thought processes.

Figure 2: Sentiment Classification with three categories by classifier (BERT vs. GPT).
Note. n=491 open-ended answers.

Results: Sentiment

Emotions in Political Trust

  • Associations and thought processes with a positive sentiment positively influence trust scores, and vice versa → Associations with the question wording and thought processes matter.

Figure 3: Linear model of sentiment and a five-category trust score (bi- and multivariate).
Note. GPT-based classification. Reference category is negative sentimemt.

Results: Emotions

Emotions in Political Trust

  • Only 17% of the probing answers contain emotional language.

Figure 4: Emotion Classification obtained from SpeechBrain.
Note. Analysis of n=491 open-ended answers. Number of observations for each sentiment category: 408 (neutral), 44 (angry), 18 (sad), 21 (happy).

Results: Emotions

Emotions in Political Trust

  • Respondents with a happy emotional condition have increased trust scores.

Figure 5: Linear model of emotion and a five-category trust score (bi- and multivariate).
Note. SpeechBrain-based classification.

Discussion

Emotions in Political Trust

  • We find that individual’s associations, thought processes and feelings influence how much they report to have trust in politics.

  • But we find few respondents to use emotional language in their answers and there is no consistent effect of emotions on reported trust score.

Two follow-up questions:

  • Maybe a survey setting is a too formal setting top elicit emotions?
    • Could depend on circumstances, e.g. are others around?
  • How does SpeechBrain classify? Can we generate a human benchmark?

How could future research on voice and emotions look like?

Outlook on future research agenda

Outlook

  • Overall, audio data plays a crucial role for studying societies, as spoken language is one of humanity’s most important means of communication, expression and information exchange in various fields (e.g., public speeches and debates, political talk shows and podcasts, press conferences).

  • → We need more applied research to gain more experiences with this type of data.

  • Subfields to research: speech-to-text algorithms (Landesvatter, Behnert, Bauer 2023, Meitinger, Sluis, Schonlau 2024), ethical considerations and privacy concerns, survey and experiments design to collect audio data, etc.
  • Multi-disciplinary approaches are useful (e.g., experts from linguistics, sociology, psychology, and computer science).

Speech-to-Text algorithms

Outlook

  • Landesvatter, Camille, Jan Behnert, and Paul C. Bauer. 2023. “Comparing Speech-to-text Algorithms for Transcribing Voice Data from Surveys.” SocArXiv. October 10. doi:10.31235/osf.io/vk6wj.

Thank you for your Attention!

References

Gabriel, Maier, Masch, and Renner. 2023. Political Leaders, the Display of Emotions, and the Public: An Empirical Study on the Coverage and Effects of Emotions in German Politics. Nomos.

Gavras, Höhne, Blom, and Schoen. 2022. “Innovating the collection of open-ended answers: The linguistic and content characteristics of written and oral answers to political attitude questions.” Journal of the Royal Statistical Society. Series A, 185(3):872-890.

Grimmelikhuijsen, Stephan. 2012. “Linking Transparency, Knowledge and Citizen Trust in Government: An Experiment.” International Review of Administrative Sciences 78(1):50–73.

Höhne and Gavras. 2022. “Typing or Speaking? Comparing Text and Voice Answers to Open Questions on Sensitive Topics in Smartphone Surveys.” Available at SSRN: https://ssrn.com/abstract=4239015 or http://dx.doi.org/10.2139/ssrn.4239015.

Lee, Kurt, and Rachel L. Campagna. 2023. “At the Heart of Trust: Understanding the Integral Relationship Between Emotion and Trust.” Group & Organization Management 48(2):546–80.

Lodge, Milton, and Charles S. Taber. 2013. The Rationalizing Voter. Cambridge University Press.

Lütters, Friedrich-Freksa, and Egger. 2018.“Effects of Speech Assistance in Online Questionnaires.” Presented at the General Online Research Conference, Vol. 18.

Ravanelli, Parcollet, Plantinga, et al. 2021. “SpeechBrain: A General-Purpose Speech Toolkit.” arXiv.

Revilla, Couper, Bosch, and Asensio. 2020. “Testing the Use of Voice Input in a Smartphone Web Survey.” Social Science Computer Review 38(2):207–24.

Theiss-Morse and Dona-Gene Barton. 2017. “Emotion, Cognition, and Political Trust.” Pp. 160–75 in Handbook on Political Trust. Edward Elgar Publishing.

Camille Landesvatter — Analyzing Audio Survey Responses