How accurate are synthetic respondents compared with real people?
What independent, named studies have found, where synthetic answers go wrong, and what nobody knows yet.
Synthetic respondents can match the average answers of real survey samples on some questions, but independent tests find large, hard-to-predict errors. In Pew Research Center's September 2026 test, AI stand-ins for real panelists missed the human results by 12 percentage points on average, and by more for some subgroups. They are not a substitute for asking real people.
What do independent studies find?
Averages can come close; distributions, subgroups and relationships between answers are where synthetic samples fail. Only academic papers and research bodies are listed here, not vendor validation studies.
| Study | What was tested | Main finding |
|---|---|---|
| Argyle et al., 2023 | GPT-3 given thousands of backstories of real US survey participants | Answer distributions approximated those of many demographic subgroups, which the authors call "algorithmic fidelity" |
| Santurkar et al., 2023 | Language model opinions against 60 US demographic groups | Substantial misalignment, as large as the Democrat-Republican gap on climate change, and it persisted when the model was told which group to represent |
| Bisbee et al., 2024 | ChatGPT personas rating 11 groups, against the American National Election Studies | Averages close to the survey; far less variation; 48% of regression coefficients significantly different, with the sign reversed in 32% of those |
| Brand, Israeli and Ngwe, 2025 | GPT-3.5 Turbo answering conjoint studies on household products and computers | Replicated willingness to pay for existing features; needed fine-tuning on earlier human results for new features |
| Park et al., 2024, revised 2026 | Agents of 1,052 real Americans built from two-hour interviews | Reached 83% to 86% of people's own two-week test-retest consistency on General Social Survey items; demographics-only agents reached 74% |
| Pew Research Center, 2026 | AI "digital twins" of panelists answering nearly 300 questions in three 2026 survey waves | Average error of 12.4 percentage points; on 28% of questions the gap exceeded 15 points |
Where do synthetic respondents go wrong?
The same six failure patterns recur across these studies.
- Too little variety. Opinions cluster around the majority view. In Pew's test, nearly half of the questions had at least one answer choice that no AI respondent picked, and the AI put the share of Hispanic adults likely to follow the World Cup at 97% against a real 43%.
- Bigger errors for subgroups. Pew's average error was 16.1 points for Republicans and Republican leaners and 15.1 for Black adults, against 12.4 overall.
- Wrong relationships between answers. Even where averages matched, Bisbee and colleagues found that analyses relating one answer to another often reached different conclusions from the real survey.
- Instability. Small changes in prompt wording shifted results, and the same prompt gave different results over about three months as the model changed (Bisbee et al.).
- Overconfidence. On one knowledge question, Pew's twins gave the right answer 98% of the time where 52% of people did, and across questions real respondents were about four times as likely to say "not sure".
- Anything new. Recent events, new product categories and features without earlier human data are where models have least to go on (Pew; Brand, Israeli and Ngwe).
What is known, and what isn't?
The evidence is consistent on the risks and thin on where synthetic answers can be trusted.
Reasonably well established:
- Synthetic samples can approximate some population averages, especially on well-documented topics.
- Richer data about real individuals improves predictions of those individuals' answers (Park et al.).
- Errors are larger for subgroups, and they are hard to predict in advance; Pew concludes that AI polling is not a replacement for probability-based surveys of real people.
Not known:
- How accurate any given vendor's synthetic panel is. There is no shared benchmark, and validation studies published by vendors are rarely independent or peer reviewed.
- How results hold outside the US, in business-to-business audiences or in small niche segments; most independent tests use US general-population political and social surveys.
- Whether a result will still hold after the next model update.
The Market Research Society's Advanced Insights and Analytics Council (2025) takes the same line: synthetic data should never replace real human interactions, must be validated against real-world findings, and inherits the biases of the data behind it.
How should you use synthetic respondents, if at all?
Use them to prepare research with real people, not to replace it.
- Good uses: pretesting questionnaire wording, generating hypotheses, rehearsing a discussion guide and listing possible reactions to explore with real people.
- Label the output. Say it is simulated wherever it appears, and never present it as what real people think.
- Validate before you rely. Ask a real sample the same questions and compare totals, subgroups and the spread of answers, not only the averages.
- Record versions. Note the model, prompt and persona version, and rerun to see how much results move between runs.
PulseLake's Terms put the limits plainly: synthetic output is not real research, is not a prediction or a statistical finding, and is not representative of any population. PulseLake's synthetic research uses explicit, versioned personas with clear provenance and a path into human validation. Our article on synthetic respondents and their limits covers the uses in more detail.
Frequently asked questions
Does giving the AI more data about each person make it more accurate?
For predicting one person's answers, yes: in Park and colleagues' study, agents built from two-hour interviews reached 83% of people's own test-retest consistency, against 74% for agents given demographics only. For population estimates it is not enough on its own: Pew gave its digital twins 71 extra attitude variables per person and still saw errors of about 12 points.
Can fine-tuning or calibration fix the errors?
Partly, and only where human data already exist. Brand, Israeli and Ngwe found that fine-tuning on earlier human survey results helped a model estimate willingness to pay for new features within the same product category, but may not carry over to unrelated categories.
Does a synthetic sample have a margin of error?
Not in the statistical sense. A margin of error describes random sampling from a real population. Synthetic answers are generated by a model, so the usual formula does not measure how far they are from the truth; only comparison with a real sample does.
Which AI model gives the most accurate synthetic respondents?
Rankings change with each model release. In Pew's comparison on one survey wave, Claude Opus 4.6 erred by 11.4 points, GPT-5.1 by 13.3 and GPT-5 nano by 17.4, and in a further test GPT-5.1 made opinion look more extreme than it is while Opus 4.6 made it look more moderate.
Sources
Sources for the facts on this page, last checked October 9, 2026.
- Pew Research Center, Can AI Stand In for Human Survey-Takers? Not Really (30 September 2026) checked October 9, 2026
- Pew Research Center, Methodology: silicon samples (30 September 2026) checked October 9, 2026
- Bisbee et al., Synthetic Replacements for Human Survey Data? The Perils of Large Language Models, Political Analysis 32(4), 2024 checked October 9, 2026
- Argyle et al., Out of One, Many: Using Language Models to Simulate Human Samples (arXiv 2209.06899; Political Analysis, 2023) checked October 9, 2026
- Santurkar et al., Whose Opinions Do Language Models Reflect? (arXiv 2303.17548, 2023) checked October 9, 2026
- Park et al., LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals (arXiv 2411.10109, 2024, revised 2026) checked October 9, 2026
- Brand, Israeli and Ngwe, Using LLMs for Market Research (MSI Report 25-136, November 2025) checked October 9, 2026
- Market Research Society, AI and Synthetic Data in Market Research (Advanced Insights and Analytics Council, 2025) checked October 9, 2026
Related guides
- Synthetic respondents and their limits
- What is an AI-moderated interview and how does it work?An AI-moderated interview is a research interview in which an AI model asks the questions, follows a guide and probes answers, with a real participant.
- What questions should you ask in a concept test?A concept test asks the same questions about every concept: appeal, relevance, uniqueness, believability, purchase intent, price and why. A ready-to-use set.
Run research end to end. Keep the knowledge working.
One AI-native operating system for market research and insight professionals — from study design and evidence generation to agents, institutional knowledge, delivery and action.