How accurate are synthetic respondents?
An honest look at the back-tests — where synthetic respondents are strong, where they are weak, and why you should demand a calibration score.
6 min read · Updated October 22, 2026
The honest answer: it depends, and you should measure it
Accuracy is not a single number — it depends on the topic and on how richly the respondents are grounded. That is why the responsible way to talk about synthetic accuracy is to back-test it against real data rather than quote a flattering headline.
A calibration back-test takes a real, labelled poll, simulates every real respondent, and reports how closely the prediction matched reality — mean error, total variation, and the gap per segment.
What the back-tests actually show
On well-grounded questions, the method can get close. Back-testing against the Cooperative Election Study (a public-domain dataset of over 60,000 real US respondents), a richly grounded simulation landed about 2 points off the real 2020 US presidential vote and predicted roughly 86% of individual respondents correctly. Across a set of six public-opinion benchmarks the average error was about 10 points, with several within single digits.
The lesson in those two numbers is the whole story: rich grounding (adding attributes like party and ideology) pulled the average error down sharply, while thin grounding on the same questions was far worse. Accuracy is earned by the data you feed in.
Where synthetic respondents are weak
They are weakest on polarised, identity-driven topics and on the tails of a distribution — minority and extreme views are systematically under-represented. They are also sensitive to prompt framing, and without an explicit skepticism prior they drift toward agreeable yes-saying.
None of this makes them useless. It makes them a directional instrument with known error bars — which is fine, as long as you know the error bars.
What to demand from any tool
If a synthetic-audience tool will not show you its own accuracy, treat its numbers as decoration. The one thing that separates a usable instrument from a confident guess is a calibration score you can check on your own topic. Ask for it, and weight every result by what it says.
What is a good calibration error?
Lower is better — 0 means the prediction matched reality exactly. In public back-tests a well-grounded question landed about 2 points off and the six-benchmark average was about 10 points. Treat anything in the low single digits as a confident read and double-digit errors as directional.
Why do synthetic respondents under-represent extreme views?
Language models regress toward the average and miss the tails, so minority and extreme positions get smoothed out. It is a structural limitation — calibrate on your topic and validate high-stakes calls with real fielding.
What is silicon sampling?
Silicon sampling is the research technique behind synthetic audiences: conditioning a language model on real respondent profiles to simulate survey answers. Here is the idea and its limits.
Synthetic survey vs real survey: when to use which
Synthetic and real surveys solve different problems. A practical guide to when a synthetic survey is the right first read and when you must field a real one.
What is a synthetic audience?
A synthetic audience is a set of AI respondents grounded in real population data, used to simulate a survey in minutes. Here is how it works, what it is good for, and where it breaks.