Large Language Models Can Argue in Convincing Ways About Politics, But Humans Dislike AI Authors: implications for Governance

ABSTRACT

All politics relies on rhetorical appeals, and the ability to make arguments is considered perhaps uniquely human. But as recent times have seen successful large language model (LLM) applications to similar endeavours, we explore whether these approaches can out-compete humans in making appeals for/against various positions in US politics. We curate responses from crowdsourced workers and an LLM and place them in competition with one another. Human (crowd) judges make decisions about the relative strength of their (human v machine) efforts. We have several empirical ‘possibility’ results. First, LLMs can produce novel arguments that convince independent judges at least on a par with human efforts. Yet when informed about an orator’s true identity, judges show a preference for human over LLM arguments. This may suggest voters view such models as potentially dangerous; we think politicians should be aware of related ‘liar’s dividend’ concerns.

KEYWORDS:

• artificial intelligence
• rhetoric
• large language models
• political methodology
• political debate

Disclosure statement

No potential conflict of interest was reported by the author(s).

Supplemental data

Supplemental data for this article can be accessed online at https://doi.org/10.1080/00323187.2024.2335471.

Notes

1. See e.g. Spirling (2023) for discussion as to why open source LLMs may be generally preferable to proprietary efforts.

2. For example, Gallup has historically asked “In your view, should immigration be kept at its present level, increased or decreased?”.

3. For 4 prompts, a large run of 300 arguments was generated. We did this to get a general sense of performance, in terms of how many ‘usable’ arguments we should expect: see Supporting Information (SI) C.

4. SI D gives more information on the curation process.

5. The task was fielded through MTurk with a random treatment assignment. Through random chance, the control group was slightly larger than the treatment. We also had several more people in the treatment group vs. the control fail to complete the task adequately to be included in the analysis.

Additional information

Notes on contributors

Alexis Palmer

Alexis Palmer is a PhD candidate at New York University.

Arthur Spirling

Arthur Spirling is the Class of 1987 Professor of Politics at Princeton University.