How accurate are synthetic respondents? Validation methods and benchmarks
Assess synthetic respondents against human evidence for a defined audience and decision, without assuming a universal accuracy rate.
Synthetic respondents do not have one accuracy rate that applies across questions, audiences and decisions. Their usefulness must be evaluated against suitable human evidence for a defined task. A model that produces convincing answers or reproduces a familiar result has not necessarily demonstrated that it can predict a new audience's responses.
The practical question is therefore narrower than “Does synthetic research work?” Ask what the simulation is intended to estimate, what evidence it uses and what kind of error would change the decision.
What does accuracy mean in this context?
Accuracy can refer to several different comparisons. A model might reproduce an overall response distribution, rank concepts in the same order as a human study or identify themes that researchers find useful. These are different achievements.
A supplier should name the measure and show how it was calculated. “Similarity” without a definition is difficult to interpret. An average can also hide a poor result for one audience or one question.
Choose the comparison before seeing the generated result. Otherwise, it is too easy to select whichever measure makes the output look strongest.
What does published research show?
Pew Research Center's September 2026 experiment compared simulated respondents with human survey results across three survey waves. Pew concluded that the tested approach was not an adequate replacement for public-opinion polling. That finding concerns the design and use studied; it is not a universal score for every synthetic research product.
NORC's discussion of synthetic respondents provides another primary source for assessing the continuing role of human data. Read the methodology and the intended use when interpreting any research claim.
For a buyer, the lesson is to ask for evidence that matches the planned task. A published result on public opinion, a vendor's concept test and an internal questionnaire rehearsal answer different questions.
Why can a realistic answer still be wrong?
Language models can produce a fluent explanation under an assigned persona. That fluency does not establish that the explanation reflects the experience of a real person in the target population.
The input can also shape the result in ways that are easy to overlook. A persona described as price-conscious may predictably object to an expensive offer. The output may reflect the assigned assumption rather than new evidence about how common that objection is.
Ask what the model knew before answering. If it had access to the result used for evaluation, apparent agreement may reflect answer exposure. If the evaluation uses a well-known historical study, consider whether it measures reproduction of existing material rather than prediction of new evidence.
How should a validation study be designed?
Begin with a written decision and a defined audience. Specify the instrument, stimuli, model, persona inputs and generation process. Decide which outputs will be compared and what level of disagreement would be consequential.
Keep evaluation material separate from the material used to build or tune the simulation. Preserve the generated answers before inspecting the human comparison so the process does not become a series of adjustments toward the desired result.
Use an appropriate human study as the comparison. Human research has its own sampling and measurement limitations, which should be documented rather than treated as nonexistent.
A useful protocol also identifies the decisions outside the evaluation's scope. Evidence for choosing interview topics does not automatically support a sales forecast or a population estimate.
How can distribution error be explained clearly?
Consider a hypothetical question with two options. If 60% of human respondents choose the first option and the simulation produces 70%, the difference is 10 percentage points. That is not the same as a claim that the simulation is “90% accurate.”
For questions with several options, compare each option using a stated measure and explain how results are aggregated. Preserve question-level results so a good average cannot conceal a large error on a consequential item.
If the decision concerns a ranking, report ranking agreement separately from distribution error. Two methods can select the same leading concept while disagreeing substantially about the strength of preference.
Why do subgroup checks matter?
An overall result can look acceptable while the model performs poorly for a smaller group. The relevant groups depend on the decision: purchase responsibility, accessibility needs, geography or experience with the product may matter more than a generic demographic split.
Plan those comparisons before fieldwork where possible. Small human subgroup samples can themselves be uncertain, so interpret disagreements with the study design in mind.
Do not hide a failure by pooling groups until the average improves. Record where the simulation is useful, where evidence is insufficient and where human research should lead.
What should repeated runs establish?
Repeated generation can help reveal sensitivity to wording, instructions and model configuration. Record what changes and whether the conclusion remains stable.
Stability is different from external validity. A model can repeat the same answer consistently and still disagree with people. Use repeated runs to understand the process, not as a substitute for the human comparison.
A model update, changed audience or new product can also require a fresh evaluation. Validation belongs to a defined configuration and use, rather than permanently to a product name.
Can generated responses increase the sample size?
They increase the number of generated records. They do not create additional recruited people. Report human participants and simulation runs separately.
If a method combines observed and modelled data, explain the method, origin of each record and uncertainty assumptions. Do not apply conventional survey precision claims merely because a file contains many generated rows.
This distinction should appear beside the finding. A reader should not need to discover in an appendix that a quoted “respondent” was a model output.
How does PulseLake fit into the workflow?
PulseLake is a research operating system that connects study design and evidence collection with structured analysis, interpretation and reporting. Its research frameworks organise questions and measures into dimensions and sub-dimensions. That structure gives the evidence a shared meaning and makes it usable for analysis beyond an individual response or survey.
Validation must preserve the meaning of each measure. A shared dimension name alone is insufficient: question wording, audience, scale and calculation method must also support the comparison between simulated and observed evidence.
Its evidence provenance and persistent study context can help keep those steps understandable. They do not establish a universal accuracy guarantee. Researchers remain responsible for the method and the interpretation.
Use synthetic exploration to improve the next question. When the decision requires fresh evidence about people, collect that evidence and show how it changes the initial hypothesis.
Frequently asked questions
What accuracy percentage should we expect?
There is no defensible universal percentage. Ask for the measure, evaluation design, audience and error distribution relevant to the intended use.
Can synthetic respondents predict product sales?
A simulated preference does not by itself establish future sales. Any forecasting use needs a suitable model, observed evidence and validation for that specific decision.
Does a larger synthetic sample solve bias?
No. More generated records can repeat the same modelling assumptions or errors. Assess the underlying process and keep generated counts separate from human sample sizes.
What should a vendor provide?
The evaluation method, relevant human comparison, model and input details, limitations and subgroup results. Ask which changes would require the validation to be repeated.
