Could artificial intelligence answer surveys on behalf of people in dozens of countries? For researchers and policymakers, the possibility is tempting. A language model can generate thousands of responses within minutes, while interviewing real people across national borders takes months of recruitment, translation and careful fieldwork.
Yet new research suggests that the shortcuts could obscure the differences that make international surveys useful in the first place. An analysis of 67,201 human respondents across 74 countries found that simulated answers from large language models reproduced some demographic patterns but did a considerably worse job representing cultural differences between populations.
The paper, published in Nature Human Behaviour on 12 October 2026, compared real economic preferences with answers generated by several widely used AI models. Across the tested cultural comparisons, simulated answers reproduced the direction of human patterns in only 48.1% of cases. For age, gender and cognitive ability, the corresponding figure was 77.8%.
It is an important warning for the emerging practice of treating AI-generated responses as substitutes for survey participants. A model that appears to understand broad human tendencies may still miss how those tendencies differ between countries, including places whose experiences are less well represented in digital information.
Why researchers want AI-generated survey respondents
Opinion surveys and behavioral studies help governments, businesses and academic researchers understand how people make decisions. Large international surveys can reveal whether preferences differ between younger and older people, across income groups or among populations facing different economic circumstances.
But such evidence is expensive to collect. Researchers must adapt questionnaires to local languages, recruit participants, maintain consistent procedures and correct for sampling differences. This limits the frequency and geographic reach of ambitious studies.
Generative AI offers a seemingly convenient alternative. A researcher could supply a model with a hypothetical respondent’s age, gender and country, then ask the same questions used in a real survey. If the simulated answers resemble actual human responses, the technique might assist early-stage research or help explore a question before collecting new data.
The crucial condition is whether the model reproduces the relevant differences accurately. Agreement about general trends is not enough if the evidence will be used to design policy or compare how people in distinct societies respond to financial incentives.
The new study, led by Fadong Chen and colleagues, tested this idea at unusually large scale. Rather than asking whether AI-generated survey data merely looked plausible, the team assessed how closely the simulated responses matched real answers from a well-established international dataset.
Six preferences that shape economic decisions
The benchmark was the Global Preference Survey, conducted as part of the 2012 Gallup World Poll. That project collected information about more than 80,000 people across 76 countries, representing much of the world’s population and economic activity. The new AI comparison used 67,201 matched respondents from 74 countries for whom the necessary information was available.
Researchers focused on six economic preferences: willingness to take risks, patience, positive reciprocity, negative reciprocity, altruism and trust. These concepts describe different aspects of behavior relevant to financial decisions, cooperation and social exchange.
Risk-taking concerns how people approach uncertainty. Patience captures choices between sooner and later rewards. Positive reciprocity relates to returning kindness, while negative reciprocity concerns responses to unkind treatment. Altruism refers to helping others, and trust concerns expectations about other people.
These preferences matter because average national differences and variations within societies can affect how researchers interpret saving, entrepreneurship, cooperation and institutional behavior. A convincing synthetic survey would need to reproduce more than a broad tendency to answer politely or favorably.
For each human respondent, the researchers assembled an AI persona using the individual’s gender, age, cognitive ability and country or region. The model was asked survey questions in the relevant language, with financial amounts adapted to local currency where applicable. The resulting AI answer could then be compared with that person’s recorded human response.
These were simulated respondents, not new human participants. The language models did not know the private thoughts of the people they were imitating. They generated plausible answers using demographic information and patterns learned during training.
Nearly 28 million AI answers, tested across models
The scale of the experiment was substantial. The authors examined 20 AI configurations, including main model and language conditions as well as follow-up tests in which prompts or response settings were changed. Twelve principal configurations formed the main comparisons, while eight additional conditions tested robustness.
Models came from OpenAI, Google and DeepSeek. The tested configurations included different generations of GPT systems, Google’s Gemini models and DeepSeek-V3. The researchers also compared answers produced in the survey’s local language with English-only prompting.
Across all conditions, the work produced approximately 27.8 million individual AI-generated survey answers and about 1.39 million simulated respondent profiles counted across the configurations. This is not the same as surveying 1.39 million additional people: the same human profiles could be simulated repeatedly under different AI setups.
The main analysis compared AI and human preferences at several levels. The team tested agreement between matched individuals, examined age and gender differences, compared national averages and investigated broader cultural patterns. They also evaluated whether synthetic data could reproduce previously published findings based on the original survey.
Different tests are necessary because a dataset may reproduce a broad group average while getting many individuals wrong. Conversely, a model may approximate familiar demographic relationships without accurately locating a country relative to others on a particular preference.
The encouraging result came with a substantial qualification
At the individual level, many AI predictions were positively associated with real human answers. Across the six preference domains and twelve main configurations, 63 of 72 correlations, or 87.5%, were significantly positive after accounting for multiple comparisons.
That sounds reassuring until the size of the agreement is considered. On average, the AI-generated preferences explained only 1.3% of the variance in individual human preferences. Put differently, the presence of many positive correlations did not mean the models could accurately reconstruct what particular people had said.
The average mean squared error for standardized individual preference scores was 1.827. Since the scores were standardized to have unit variance, an error of that magnitude indicates that the differences between simulated and actual preferences remained large relative to the variation in human responses.
The distinction illustrates a basic statistical principle. With tens of thousands of cases, even a small association may be reliably distinguishable from zero. Statistical significance does not necessarily make an estimate useful for predicting an individual person’s choices.
National averages offered only limited improvement. At the country level, AI-generated preferences explained 4.2% of the variation in human country averages, while their mean squared error was 1.722. Aggregating individuals reduced some noise, but it did not eliminate the underlying mismatch.
Indeed, the study found that different language models were often more similar to each other than they were to the corresponding human survey results. Model agreement, by itself, therefore cannot be treated as evidence that synthetic survey answers faithfully represent a population.
Demographic similarities concealed cultural mismatches
One of the clearest results emerged when the researchers separated demographic patterns from cultural ones. They examined whether the direction of observed human differences was reproduced by AI across factors such as gender, age and cognitive ability.
Across those demographic comparisons, AI predictions captured the direction of human associations in 77.8% of cases, with a reported 95% confidence interval of 71.6% to 83.1%. That suggests models retained meaningful information about some broad patterns associated with familiar demographic characteristics.
But for the assessed cultural dimensions, the equivalent rate was only 48.1%, with a 95% confidence interval of 45.1% to 51.2%. This was close to an even split between matches and mismatches in direction. It did not establish that the AI could reliably recreate cultural differences between countries.
This comparison does not mean that an AI got 48.1% of every survey question correct, nor that all cultural relationships were always reversed. It concerns whether the direction of selected human cultural associations was reproduced under the researchers’ evaluation method.
Nevertheless, the contrast matters because cross-country differences are often the very reason researchers conduct an international study. If synthetic data smooth away important distinctions, an analysis might produce an attractive global average while misrepresenting the people for whom a policy is intended.
Why a country label was not enough
The team then asked whether the problem could be traced to the information used to build the simulated people. Each model received demographic details and a country or region label. One possibility was that supplying more culturally relevant location information would substantially improve the predictions.
To investigate this, the researchers ran prompt experiments that systematically removed some information. In one setup, the model received demographics without a country label. In another, it received the country but not the demographic characteristics. Additional variations removed age, gender or cognitive ability separately.
The results suggested that removing demographic information generally weakened alignment more than removing the country label. In statistical models, adding country controls left the association between predicted and observed preferences largely unchanged, whereas accounting for demographics substantially reduced it.
That finding does not imply that nationality or cultural context is unimportant to human preferences. Instead, it suggests the simple country-label prompt did not capture those influences very effectively in the AI simulations tested.
The authors also found country-structured errors. In many models, discrepancies tended to be smaller in economically more developed and digitally connected countries. The relationships were not universal across every model, and the researchers did not establish that digital access itself caused better performance.
For settings such as South Africa, this is an important consideration rather than a directly measured country-specific conclusion. Researchers should not assume that a system tested mainly on familiar demographic patterns will represent local economic experiences merely because the prompt includes a country name.
Published findings were difficult to reproduce
The authors included a more practical test: could the synthetic survey reproduce conclusions that researchers had previously obtained using human responses? They considered 35 published country-level associations drawn from work using the Global Preference Survey.
Even the strongest-performing language model reproduced just 16 of the 35 associations, or 45.7%, with the same direction and statistical significance when judged using the original studies’ 10% significance criterion.
This result is particularly relevant to policy evaluation. Synthetic data may look coherent in isolation, but an empirical conclusion is useful only if the generated information supports the correct comparisons and relationships.
The researchers identified additional weaknesses. The models systematically overestimated women’s prosocial preferences in some domains relative to the human benchmark. They also tended to generate more concentrated, less diverse responses than the human participants.
Raising the models’ response temperature, a setting that allows more variation in outputs, increased diversity to some extent but did not close the gap fully. Making the answers more varied did not automatically make the simulated populations more representative.
These findings suggest that the main limitation is not only an insufficient number of artificial survey participants. Generating far more answers from a model that has similar systematic biases may simply reproduce those biases at larger scale.
What this means for research and business decisions
Companies increasingly use AI to summarize customer feedback, evaluate marketing ideas and explore how different audiences might react to messages. Synthetic personas can help teams formulate hypotheses and identify questions worth testing with real people.
But this study cautions against using those personas as if they were replacements for properly sampled customer research. Differences in purchasing power, culture, social norms and institutional trust may be flattened or distorted by models that are convincing at a conversational level.
For researchers, there is also a distinction between idea generation and measurement. A simulated respondent might help draft a questionnaire or expose an ambiguous prompt, while still being an unreliable source of evidence about what a real population believes or prefers.
Previous Research Today reporting on AI-generated peer reviews has shown that machine-generated judgments can change when prompted with prior opinions. That finding addresses a different application but reinforces why plausible output should be assessed against independent evidence.
Similarly, research on scientists’ experiences with more capable AI illustrates the difference between a technological capability and the actual outcome it produces for users. Improvements in fluency or speed do not by themselves guarantee better evidence.
In business, work on using social-media information for marketing strategy offers another reminder that collecting data and interpreting people accurately are separate tasks. AI may help process information, while survey design and local validation remain essential.
A defensible workflow would use AI-generated personas to explore possible questions, then test the resulting hypotheses with genuine respondents. That approach preserves the efficiency of AI-assisted preparation without treating statistical imitation as human testimony.
Important limitations of the comparison
The authors were careful to identify where their findings might overstate or understate modern AI’s abilities. The human benchmark came from a survey conducted in 2012, whereas the model-based simulations used much newer systems. Preferences can change over time, and changes in society over more than a decade may contribute to observed differences.
Conversely, some material from the original survey may have appeared in AI training datasets. If a model encountered related information during training, parts of the apparent agreement could reflect learned patterns or memorization rather than an ability to infer real preferences from limited details.
The study used a defined persona-prompting protocol, often with deterministic temperature settings. Richer cultural descriptions, retrieval from local sources or specialized training could produce different results. The study does not establish a permanent ceiling on future technology or test every possible form of synthetic respondent.
It also assessed particular economic preference instruments, not all public opinion or every aspect of consumer behavior. The results should not be interpreted as proof that all generative AI applications fail or that no synthetic data can ever assist research.
Nevertheless, the researchers included multiple model families, local-language and English-only conditions, prompt variations and checks on response diversity. The limitations therefore do not erase the basic observation that this sizeable audit found weak agreement at several levels relevant to international inference.
Before replacing respondents, validate the representation
Large language models have made it easier than ever to produce answers that resemble survey responses. That convenience is valuable for brainstorming and testing research materials, but the central requirement of a representative survey is not how quickly an answer is produced. It is whether the answer reliably describes the people being studied.
Across 74 countries, this analysis found that AI-generated answers captured some broad demographic differences while explaining only a small portion of the variation in actual preferences and reproducing fewer than half of the tested cultural patterns in direction.
For the growing market in synthetic audiences and AI-generated survey panels, the message is clear: synthetic responses should be independently validated against real human data before being used to make consequential decisions. Until then, they are better treated as research aids than as substitutes for the public.
Source Information
Study title: Simulated responses from large language models are weak proxies for human economic preferences across 74 countries
Authors: Fadong Chen, Qijun Wang, Lei Wang and Sudeep Bhatia
Journal: Nature Human Behaviour
Publication date: 12 October 2026
Study design: Secondary-data benchmarking study comparing real international economic-preference survey responses with simulated responses across 20 language-model and prompt configurations; individual, demographic, country-level and cultural alignment analyses, including robustness tests
Sample or scope: 67,201 matched human respondents in 74 countries from the Global Preference Survey, approximately 27.8 million AI-generated answers and 1.39 million simulated persona instances across 20 configurations; six economic preference domains
Key finding: Model responses explained only 1.3% of individual-level human-preference variance and reproduced the direction of 48.1% of tested cultural associations, versus 77.8% of age, gender and cognitive-ability associations
