Measures of Argument Strength: A Computational, Large-Scale Analysis of Effective Persuasion in Real-World Debates

ABSTRACT

The present research examined how value-free and value-driven measures of argument strength (MAS) can be computationally extracted using a theory-driven approach at scale in a naturalistic setting by analyzing a total of 7,961 real-world debates and 42,716 judgments in rhetorical quality. In the first study, value-free MAS was significantly related to the rhetorical quality of arguments (i.e. their persuasiveness). The results indicate that the side that provides more information-source citation, less quantitative specificity, more unique words, and more abstract language is more likely to be perceived as convincing in dialectical argumentation, where two people are exchanging opposing arguments. In the second study, the added influence of value-driven MAS is investigated. The results show that the similarity between the moral values represented in arguments and those that are salient to argument receivers predicts the rhetorical quality. The research demonstrates how rhetorical quality can be measured and predicted at scale, and how naturally generated arguments can be used for scientific progress in persuasion research.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Supplemental data

Supplemental data for this article can be accessed online at https://doi.org/10.1080/19312458.2023.2230866

Notes

¹ https://www.reddit.com/r/changemyview

² https://www.debate.org

³ https://www.kialo.com

⁴ web.archive.org has archived some pages of the website. Here is an example of a debate: https://web.archive.org/web/20220319231240/https://www.debate.org/debates/Intelligence-Nature-vs-Nurture/1/

⁵ https://docs.python.org/3/library/re.html

⁶ https://pypi.org/project/word2number/

⁷ https://www.nltk.org/

⁸ https://github.com/cjhutto/vaderSentiment

⁹ https://osf.io/ax5mg/?view_only=c31b414de0544a62be3d2e97aa18e4e3

¹⁰ https://github.com/medianeuroscience/emfdscore

Additional information

Notes on contributors

Sungbin Youk

Sungbin is a Ph.D. candidate in the Department of Communication at the University of California, Santa Barbara, and a researcher in the Media Neuroscience Lab. His research focuses on the accurate assessment and effective resolution of problematic media use. His research integrates (1) computational and neuroscience approach to persuasion, and (2) psychological and behavioral approach to excessive media consumption, such as short-form videos. By bridging these interdisciplinary perspectives, he aims to provide a comprehensive understanding of problematic media use and contribute to the development of evidence-based strategies to promote healthier media habits.

Musa Malik

Musa Malik is a graduate student in the Department of Communication at the University of California, Santa Barbara, and a researcher in the Media Neuroscience Lab. He holds a BS in Neuroscience from New York University, Shanghai and a MA in Communication from the University of California, Santa Barbara.

Yibei Chen

Yibei Chen received her Ph.D. in Communication from the University of California, Santa Barbara, and is an incoming Postdoctoral Associate at the McGovern Institute for Brain Research at Massachusetts Institute of Technology (MIT). Her research focuses on narrative comprehension (i.e., information processing) in the brain and mind.

Frederic R. Hopp

Frederic R. Hopp (Ph.D., UC Santa Barbara) is Assistant Professor for Political Communication at the University of Amsterdam’s School of Communication Research. He is interested in the moral content of human communication and how moralized messages are cognitively processed and motivate behavior. Frederic leverages a method-theory synergy that combines behavioral experiments, neuroimaging, natural language processing, and machine learning. His work has received numerous awards and been published in scientific journals including Nature Human Behavior, Journal of Communication, Behavior Research Methods, and Computational Communication Research.

René Weber

René Weber received his Ph.D. (Dr.rer.nat.) in Psychology from the University ofTechnology in Berlin, Germany, and his M.D. (Dr.rer.medic.) in Psychiatry and CognitiveNeuroscience from the RWTH University in Aachen, Germany. He is a Professor in theDepartment of Communication and the Department of Psychological and Brain Sciencesat the University of California in Santa Barbara, director of UCSB’s Media NeuroscienceLab (https://medianeuroscience.org), and member of UCSB’s Neuroscience Institute(https://www.nri.ucsb.edu). He also holds a Visiting Professorship position at EwhaWomans University in Seoul, South Korea. He was among the first media psychologyscholars who regularly use computational approaches and brain imaging technology toinvestigate various topics related to media industries, from the appeal of mediaentertainment, diversity and inclusion in the media, the impact of media violence, to thepersuasiveness of campaigns. He has published four books and more than 150 journalarticles and book chapters. His research has been supported by grants fromnational scientific foundations in the United States and Germany, as well as throughprivate philanthropies and industry contracts. He is a Fellow of the InternationalCommunication Association.