Although large language models (LLMs) have only recently become ubiquitous, psychological distress following user-LLM interaction has already been documented in both the scientific literature and the popular media. Such distress may be related to the tendency of some LLMs to provide sycophantic responses that uncritically reinforce the views of the user. To support users’ healthy emotional regulation, flexible, emotionally aware AI systems are needed.
When Joseph Weizenbaum released the first chatbot ELIZA in the 1960s, it became clear that humans might treat chatbots as human-like. A reported anecdote tells that Weizenbaum’s secretary asked him to leave the room so that she would have some privacy with ELIZA1. Half a century on, one common function of AI-powered chatbots is to be artificial companions in everyday life2.
The very frequent use of AI chatbots—with some people even developing dependence or delusions3,4—can be related to user behavioral patterns (e.g., anthropomorphism of AI models4) or to traits and states (e.g., state of loneliness2), as well as to system design (e.g., AI sycophancy, the tendency for a chatbot to be overly agreeable). In this Comment, we focus on AI sycophancy because of the systemic risks that it can pose to vulnerable users. A user in despair, for instance, might become strongly attached to or even dependent on an AI companion that exhibits high agreeability.
By encouraging users to forget that they are interacting with a machine, moreover, sycophantic models may encourage even greater anthropomorphism. Recent evidence shows that participants prefer sycophantic chatbots to more neutral alternatives5 such that the AI industry may have strong economic incentives to retain this communication style in commercial systems. In other words, AI sycophancy can appear as a design element to maximize user engagement with the AI product6.
Sycophantic AI systems are of particular concern because they may habitually and uncritically reinforce users’ views, failing to challenge even harmful beliefs. For example, when interacting with an individual with paranoid delusions, a sycophantic chatbot will likely reinforce the user’s distorted worldview. The results of these kinds of interaction have received extensive coverage in the popular media under the general label of “AI Psychosis.” We emphasize, however, that “AI Psychosis” is not a clinically recognized disorder—indeed, such experiences may be more accurately described as delusional7—and that much remains to be understood about the nature and causal pathways involved in problematic human-AI interactions4,8.
How to ameliorate the problems arising from AI sycophancy remains an open question involving complex trade-offs and multiple stakeholders. Importantly, moderate levels of AI sycophancy are not necessarily bad or likely to result in psychological distress, analogous to how normal levels of agreeableness can enhance human-to-human communication9. Indeed, the release of LLMs with reduced positivity bias, charm, and tendencies toward user confirmation has been accompanied by anecdotal complaints, with many users requesting a return of older, warmer models such as GPT-4o (in line with the experimental preference data discussed above)10.
Extract — continue reading at the source.
