BotOrNot: A Platform for Conducting Experiments with Undisclosed Chat Agents

Abstract

Chatbots are omnipresent in today's online environments in work and private life. While early chatbots were easy to identify, recently released open-domain chatbots, such as GPT3 and Blenderbot2, increasingly blur the line between human and chatbot interaction. Dedicated research is required to better understand how different design configurations of open-domain chatbots impact their users, whether users are able to distinguish between human and bot chat agents, and how users respond to the undisclosed identity of their counterpart. However, there is a lack of experimental platforms that integrate state-of-the-art chatbots in order to enable such research. We therefore propose BotOrNot, which enables large-scale experimental research with participants in a Turing test setting. Participants are matched with either Blenderbot2/GPT3 or another human participant and tasked to figure out whether the counterpart is a human or bot. We designed the platform in a way that it allows to adapt the settings of the experiment to enable different experimental scenarios and follow an open approach allowing to integrate future bots via an API. Participants can personalize their avatars and chatbots can also be personalized with regards their personality and avatar.