Social media platforms mediate how billions form opinions and engage with public discourse. As autonomous AI agents increasingly participate in these spaces, understanding their behavioral fidelity becomes critical for platform governance and democratic resilience. Previous work demonstrates that LLM-powered agents can replicate aggregate survey responses, yet few studies test whether agents can predict specific individuals’ reactions to specific content.
This study benchmarks LLM-based agents’ accuracy in predicting human social media reactions (like, dislike, comment, share, no reaction) across 120,000 + unique agent-persona combinations derived from 1,511 Serbian participants and 27 large language models. In Study 1, agents achieved 70.7% overall accuracy, with LLM choice producing a 13%-point performance spread. Study 2 employed binary forced-choice (like/dislike) evaluation with chance-corrected metrics.
Agents achieved Matthews Correlation Coefficient (MCC) of 0.29, indicating genuine predictive signal beyond chance. However, conventional text-based supervised classifiers using TF-IDF representations outperformed LLM agents (MCC of 0.36), indicating that the predictive signal derives from semantic text content rather than from any capacity for individualized behavioral simulation. The genuine but modest predictive validity of zero-shot persona-prompted agents suggests that, while current accuracy is insufficient for precise individual targeting, the capacity to predict reactions at rates above chance warrants attention in discussions of AI-driven influence and social simulation methodology.
The advantage of zero‑shot agents is that they require no task‑specific training, which makes large‑scale deployment easy across diverse contexts, including election campaigns and mass‑scale manipulation. Limitations include single-country sampling. Future research should explore multilingual testing and fine-tuning approaches.
The idea that large language models (LLMs) can act as proxies for human participants in social and behavioral research has moved within a few years from a speculative proposition to a rapidly expanding research program. Park1 placed 25 LLM-powered generative agents in a sandbox environment and found that they organized social events, formed relationships, and coordinated daily routines in ways that human observers judged believable, while foundation models trained on large-scale behavioral data have since achieved human-level prediction of cognition across diverse paradigms2. That early proof of concept has given way to more ambitious projects.
Park3 grounded generative agents in two-hour interviews with 1,052 individuals and reproduced their General Social Survey responses at 85% of the participants’ own two-week test-retest accuracy, and Yang et al.4 scaled the approach to one million agents in the OASIS platform, simulating information spread, group polarization, and herd effects on Reddit-like and X-like platforms. Together these advances suggest that LLM-based social simulation could become a general-purpose tool for testing policy interventions, modelling opinion dynamics, and stress-testing platform design where real-user experiments would be impractical or unethical5. In a complementary line of work, Altera.AL6 introduced Project Sid, demonstrating that 10 to over 1,000 LLM-powered agents placed in a Minecraft environment could autonomously develop specialized professional roles, adhere to and modify collective rules through democratic processes, and engage in cultural and religious transmission across multiple simulated societies.
Extract — continue reading at the source.
