Language-based assessments for experienced well-being: accuracy and external validity across behaviors, traits, and states

While language is central to human communication, psychological science has relied on self-reported rating scales for decades because it was difficult to quantify language effectively. Large language models (LLMs) have changed that in recent years. Here, we used LLMs with machine learning to compare traditional ratings against language-based responses (short essays and descriptive words) of experienced well-being over 12 weeks from 764 individuals from an overburdened population highly exposed to emotional experience, U.S. hospitality workers.

While accuracies were comparable by internal criterion (across 9 affective traits; words r = .53 vs. .51 for ratings), both language formats doubled the ratings’ accuracy (up to r = .18 and .24 for words and essays, respectively) across 24 external criteria: traits, demographics, and behavior. Beyond improved accuracy, language responses offered psychological insights: words provided concise emotional nuances, whereas essays encompassed broad life contexts. Though each language-based format differed in strengths and weaknesses, both demonstrated greater information-richness than traditional rating scales.

Oslo Business School, Oslo Metropolitan University, Oslo, Norway Department of Psychology, School of Behavioral and Brain Sciences, University of Texas at Dallas, Richardson, USA Department of Computer Science, Stony Brook University, Stony Brook, USA Ryan L. Ganesan, Oscar Kjell, Syeda Mahwish & H. Andrew Schwartz Department of Psychology, Lund University, Lund, Sweden Department of Psychological Sciences and Human Development, Cornell University, Ithaca, USA Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA, USA Renaissance School of Medicine, Stony Brook Medicine, Stony Brook, USA College of Connected Computing, Vanderbilt University, Nashville, USA Oscar Kjell has co-founded a start-up using computational language assessments to diagnose mental health problems.

The data are drawn from a larger study titled “Data Science for Unhealthy Drinking (DS4UD).” The study, including all methods, was approved by the University of Pennsylvania Institutional Review Board (IRB), and all methods were carried out in accordance with relevant guidelines and regulations. All participants provided digital informed consent before participating in the study. In the consent information, participants were given details about the study design and their commitment, along with contact information for study personnel.

They were also informed of their right to withdraw from the study at any time, and that their data would only be analyzed at the group level and that no sensitive data would be made public. Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Below is the link to the electronic supplementary material.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.