Large-scale surveys are central to investigating economic preferences but are costly and slow. Can large language model (LLM)-simulated responses substitute for human survey data? We conducted a virtual Global Preference Survey, simulating six experimentally validated economic preferences for 67,201 matched human respondents across 74 countries.
Across 20 LLM configurations with diverse prompts and robustness checks, we found systematic but modest individual-level alignment (87.5% of Spearman correlations have 95% confidence intervals (CIs) above zero), with LLMs recovering the direction of associations in 77.8% (95% CI, (71.6%, 83.1%); CI-based consistency ratio) of gender, age and cognitive-ability dimensions, but only 48.1% (95% CI, (45.1%, 51.2%)) of cultural dimensions. Ablation and error decompositions suggest weaker alignment when removing demographic information than when removing country labels, and in many models alignment is weaker in economically undeveloped and less digitally connected countries. LLM-simulated responses may recover some individual- and group-level patterns but are not yet reliable proxies for cross-national representative surveys in macro-level inference or policy evaluation.
Large-scale surveys are an important tool in economics. By directly eliciting latent preferences, they complement revealed-preference approaches and support both causal and descriptive inference at the population scale1,2. However, implementing such surveys is expensive and slow.
Researchers need to construct sampling frames, train enumerators, provide respondent incentives and coordinate complex field operations. Recent advances in large language models (LLMs) have prompted an intriguing alternative: using LLMs to generate responses for surveys or experiments3,4,5,6,7,8,9,10. This approach can lower costs and accelerate iterations without human recruitment.
Despite this promise, concerns remain about using LLMs’ simulated responses to replace human data, given inherent algorithm biases such as identity flattening11; systematic skew towards Western, educated, industrialized, rich and democratic populations12,13,14,15,16; overestimated statistical power7,17; and other potential risks18,19,20,21. Prior work has focused largely on political opinions6,17,22,23,24,25,26 and on social and moral attitudes12,13,14,27,28,29, while paying much less attention to economic preferences, even though these preferences play an important role in linking individual behaviour to market outcomes and social welfare, and constitute one of the most compelling use cases for LLMs in efforts to improve individual and societal well-being. Preference heterogeneity is a building block of modern economic theory: it helps explain individual decisions30,31,32, socio-economic status33,34 and macroeconomic outcomes35,36,37, and it is indispensable for policy design and evaluation38,39,40.
Understanding group-level heterogeneity in preferences is especially important in development economics, which seeks to capture how policies affect different populations rather than only the average effect. If LLMs can be shown to successfully predict human economic preferences, they may be used to generate counterfactual data for policy analysis, potentially alleviating the fundamental problem of causal inference—namely, that the counterfactual is unobserved41. However, the credibility of such applications depends on the models’ ability to reproduce well-documented patterns of heterogeneity.
Extract — continue reading at the source.
