Building an ICCN Multimodal Classifier of Aggressive Political Debate Style: Towards a Computational Understanding of Candidate Performance Over Time

ABSTRACT

Understanding the implications of aggressive political debate style amid corrosive modes of campaign politics requires fine-grained analyses of political performance, attending to multiple communication modalities. Politicians’ facial expressions, emotional tone, and speech content can all independently convey aggression and dominance, and often work in combination for purposes of emphasis. Yet micro-coding individual visual, tonal, and verbal features across more than a handful of debate segments becomes extremely labor intensive, hampering research, especially historical, longitudinal, and cross-cultural work. To address this limitation, we develop a novel multimodal classifier using an Interaction Canonical Correlation Network (ICCN) that incorporates video and audio features with speech coding of candidate debate performance, trained on a 20% sample of 10-second segments from each of the first televised U.S presidential debates between 1980 and 2020. In the analysis, we demonstrate this classifier can accurately detect aggression by political candidates in U.S. debates. We sharpen its performance by distinguishing between debate eras characterized by lower and higher levels of aggression and validate the approach by comparing the performance of unimodal with multimodal classification. This classifier opens new avenues for computational social science research, including explaining candidate behavior within debates at a larger scale and across different eras.

KEYWORDS:

• Computer vision
• Dominance displays
• Interaction Canonical Correlation Network
• multimodal classification
• political performance
• Trump
• presidential debates

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

For Data Coding, Codebooks and Computer Code:

• Data Coding and Codebooks of 1976 through 2020 Presidential Debate Videos

BOX DATA ARCHIVE: https://uwmadison.box.com/s/kuqqypkcgcwr5miaoag5iu855upooqty

• Computer Code for the Interaction Canonical Correlation Network Method

GITHUB REPOSITORY:

https://github.com/zsun227/ICCN

Notes

¹ To verify that Clinton’s lower level of expressiveness compared to Trump was not an artifact of coding bias or a misapplication of coding criteria due to gender differences between the candidates, we compared our earlier coding of the 2012 presidential debates between Barack Obama and Mitt Romney against the expressiveness of Trump in 2016. To make valid comparisons across election years, behavior measures for Trump (coded at 10-second intervals in 2016) were aggregated to 30-second intervals – the unit of time analyzed for our 2012 coding. Comparisons between Trump and both Obama and Romney show the same pattern of pronounced expressiveness by Trump relative to both male candidates (see Bucy et al., 2020, p. 645), with Clinton largely mirroring the norms of recent Republican and Democratic contenders. Thus, the differences observed in 2016 were due to a sharp increase in expressiveness by Trump rather than a muted performance by Clinton due to gender dynamics on stage.

² The LSTM transforms a time series of either raw audio or raw video features gathered over the 10-sec. time interval into a single fixed-length feature vector. A simpler alternative would be to concatenate the raw features into a high dimensional vector for use in the classifiers, as concatenation has a fixed (and finite) memory, while the LSTM is a recursive network that can retain information over a longer period. A comparison between ICCN and the different baseline methods can be found in Sun et al. (2020).

³ The codebook was developed for a more in-depth times-series analysis of candidate behavior in presidential debates (see Bucy et al., 2020). Only selected variables are used here.

Additional information

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Research Foundation of Korea Grant funded by the Korean Government, NRF-2016S1A3A2925033, and by the Vice Chancellor for Research and Graduate Education at the University of Wisconsin–Madison through funds provided to D.V.S. Additional support was provided by funding to E.P.B. from the Marshall and Sharleen Formby Regents Endowed Professorship in the College of Media and Communication at Texas Tech University.

Notes on contributors

Dhavan V. Shah

Dhavan V. Shah is the Jack M. McLeod Professor of Communication Research and Louis A. & Mary E. Maier-Bascom Chair at the University of Wisconsin, where he is Director of the Mass Communication Research Center (MCRC) and Research Director of the Center for Communication and Civic Renewal (CCCR), both in the School of Journalism and Mass Communication. His research centers on (1) the influence of message construction and processing in social evaluations and behaviors, (2) the capacity of mass and interpersonal communication, primarily through online networks, to shape civic engagement, participation, and trust, and (3) the effects of computer-mediated interactions, particularly support expression, on the management of cancer, aging, and addiction.

Zhongkai Sun

Zhongkai Sun is currently working as an Applied Scientist at Amazon Alexa AI, where his research interest is leveraging large language models to advance conversational AI. His research explores aspects such as reinforcement learning with human feedback for controllable text generation, and the integration of multimodal and knowledge graphs with large language models. Before joining Amazon, Zhongkai Sun received his Ph.D. from the University of Wisconsin-Madison with his dissertation on multimodal language analysis.

Erik P. Bucy

Erik P. Bucy is the Marshall and Sharleen Formby Regents Professor of Strategic Communication in the College of Media and Communication at Texas Tech University, where he teaches and conducts research on misinformation, visual and nonverbal communication, and public opinion about the press. He is the co-author of Image Bite Politics: News and the Visual Framing of Elections (with Maria Elizabeth Grabe) and past editor of Politics and the Life Sciences, an interdisciplinary journal published by Cambridge. Bucy is a US-UK Fulbright Scholar and Honorary Fellow of the Mass Communication Research Center at the University of Wisconsin-Madison.

Sang Jung Kim

Sang Jung Kim is an assistant professor at the University of Iowa in the School of Journalism and Mass Communication. She studies the interaction between technology, politics, and social identity, with particular attention to the mediating role of social media platforms and the spread of information to the public. She explores the identities of message creators and message receivers on social media platforms—including racial identity, gender identity, and political identity—and utilizes both experimental methods and computational approaches to understand how consumers and creators of such content introduce and are impacted by biases.

Yibing Sun

Yibing Sun is a Ph.D. student in the School of Journalism and Mass Communication at the University of Wisconsin–Madison. Her research is centered on the production and effects of visual content within the networked communication environment. Specifically, she seeks to unravel the intricate relationship between textual and visual content and explore how memes are disseminated online using computational methods and experiments.

Mengyu Li

Mengyu Li is a Ph.D. student in the School of Journalism and Mass Communication at the University of Wisconsin–Madison. Using computational and experimental methods, she studies the impact of multi-modal, multi-affordance, and multi-platform social media in shaping (mis)perceptions and driving action in the realms of gender politics and public health.

William Sethares

William A. Sethares is a Professor in the Department of Electrical and Computer Engineering at the University of Wisconsin in Madison. He has been a scientific researcher at the Rijksmuseum in Amsterdam and is the Honorary International Chair Professor at the National Taipei University of Technology. His research interests include adaptation and learning in image and signal processing with a special focus on applications of natural language processing to problems in communications, the social sciences, and the arts.