Growing numbers of people frequently turn to Large Language Models (LLMs) for concerns from various areas of life. Because its content has become ubiquitous, examining user perceptions of LLM-produced text and its distinguishability from human writing is crucial. This study investigated how seven LLMs (ChatGPT-4o, ChatGPT-3.5, Llama 3, Gemma, Gemini, Phi-3, Orca-2) describe themselves when compared to humans.
LLM-generated outputs and human responses were collected using psychological questionnaires (Big Five, self-esteem, narcissism) and tailored prompts to elicit short self-descriptive texts. Six LLM and ten human judges then evaluated personality traits and text origin based on these self-descriptions (140 total, half LLM-generated). Results show that current LLM outputs can convey personality-consistent cues in questionnaires and texts.
The perception of such significantly differed from that of humans. Both LLM and human judges accurately inferred personality from short self-descriptions and correctly differentiated between LLM-generated and human text. Linguistic analysis and participant feedback revealed textual cues distinguishing LLM-produced from human language and offered insights into how LLM versus human judgment and decisionmaking differ.
Large Language Models (LLMs) are neural networks based on the Transformer architecture, trained on vast amounts of textual data, and often fine-tuned using techniques like reinforcement learning through human feedback (RLHF). Chatbots driven by LLM architecture are becoming increasingly prevalent in today’s society, employed in a variety of contexts. Humans could interact with LLMs as customer service bots but also use them as resources in their daily lives.
Cases of LLMs being used as replacements for genuine human connection, e.g., as virtual partners or therapy bots, have also been observed1,2,3. Recent studies highlighted the trend of increasing anthropomorphization and personalization of LLMs4. The former reflects a cultural phenomenon that has first been termed ELIZA effect by Joseph Weizenbaum5.
Anthropomorphism describes a fundamental drive to attribute human traits to non-human entities, a strategy to interpret behavior and establish a sense of relatedness6. More human-like LLMs are viewed more favorably by humans, having been associated with beneficial results, especially in organizational contexts7. Anthropomorphic LLMs were associated with better communication, higher user satisfaction, greater trust, and improved company reputation compared to non-anthropomorphic ones8,9,10.
Extract — continue reading at the source.
