Comparing Chatbots and Online Surveys for (Longitudinal) Data Collection: An Investigation of Response Characteristics, Data Quality, and User Evaluation

ABSTRACT

As chatbots are gaining more popularity than ever, they have recently been considered as interesting tools for survey administration in social science research. To explore this idea, we investigated the extent to which there are differences in response characteristics and data quality between a traditional, web-based survey and a conversational, chatbot-based survey (which we integrated in an instant messaging app). In addition, we zoomed into how respondents evaluate both survey modes. Using a longitudinal design, we also explored how response characteristics evolved over a period of two weeks. Overall, we did not find evidence that chatbots might be better survey administration tools than web surveys. On the contrary, the web survey often seemed to generate more favorable response characteristics and data quality. Finally, when it comes to user perceptions, we found that the chatbot survey was evaluated less favorably in terms of perceived enjoyment, usefulness, and security. Based on these results, we draw conclusions about whether chatbots can be considered as valid alternatives for traditional web survey methods.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

¹ Participants were allowed to create a new Skype account to take part in the study, if desired. In this case, they received instructions on how to download Skype on their mobile device and to enable notifications, to ensure they received the invite from the chatbot on a daily basis (should they be in the chatbot condition).

² Light quotas were added to ensure variance in terms of gender and age to partially reflect the online population in The Netherlands. For education, these light quotas were not enforced given the uneven distribution of Skype users across education levels.

³ https://osf.io/ws6ax/

⁴ https://osf.io/p7rvj/

⁵ The chatbot was published with the Bot Channels Registration service from Microsoft Azure.

⁶ Calculated as Levenshtein Distance using Python’s FuzzyWuzzy package.

Additional information

Funding

This work was funded by the Digital Communication Methods Lab (RPA Communication) of the University of Amsterdam

Notes on contributors

Brahim Zarouali

Brahim Zarouali is Assistant Professor in Persuasive Communication & Digital Media at the Institute of Media Studies (IMS) at the KU Leuven. His research interests center around the impact of persuasive communication through digital media and data-driven technologies on people and society. With this focus, he aims to find out how (vulnerable) people can be empowered so that they can interact with these digital innovations in a critical and informed way.

Theo Araujo

Theo Araujo is Scientific Director of the Amsterdam School of Communication Research (ASCoR) and Associate Professor in the Department of Communication Science at University of Amsterdam. His research focuses on the increasing adoption of artificial intelligence and related technologies within our communication environment, including conversational agents and automated decision-making.

Jakob Ohme

Jakob Ohme is Research Group Lead at the Weizenbaum Institute and Fellow at the Digital Communication Methods Lab at the University of Amsterdam. His research interests center around the impact of digital and mobile communication processes on political behavior and news flows, generational differences in media use and political socialization, and the development of new methodological approaches in political communication and journalism.

Claes de Vreese

Claes de Vreese is University Professor of Artificial Intelligence and Society, with a special focus on media and democracy at the University of Amsterdam. He holds the Chair in political communication at the Amsterdam School of Communication Research (ASCoR). His research interests include the role of automation, algorithms, and artificial intelligence in democratic processes.