Synthetic audiences are triage tools. Human taste still finds the message

Synthetic audiences are good at telling you which messages to kill. They are not good at telling you which message will win. The best evidence published this year says the best frontier model reproduces less than half of what real consumers actually react to, so the job of finding the message still belongs to people with taste, gut and creative range.

We work in synthetics every day at Open Target, and we think the category is genuinely useful. That is exactly why it needs to be positioned correctly. Use synthetic panels to generate hypotheses and clear out weak options quickly. Do not use them to forecast how real people will respond, and calibrate every synthetic panel against a small real sample before it informs spend.

What ConsumerSimBench actually measured

In May 2026, researchers Tianyu Wang, Jiajun Li and Jianghao Lin published ConsumerSimBench, a benchmark that asks a blunt question: can large language models predict how a crowd of consumers will react to something?

The setup is more rigorous than most vendor demos. The team took 1,553 real topics from Chinese social media and broke real public reactions into 23,122 atomic, auditable criteria across four families: sentiment flashpoints (what triggers emotion), emotion keywords, positive aspects consumers praise, and negative aspects they criticize. Each model's simulated reaction was scored on coverage, meaning the share of those real reaction points it reproduced, with yes/no judgments rather than a vague quality score.

They tested 13 frontier models. The results:

• Gemini-3.1-Pro led with 47.8% coverage.
• GPT-5.2 reached 35.8%.
• Claude-Opus-4.6 reached 32.8%.
• GPT-4o came last at 29.7%.

So the best simulator on the market missed more than half of what real people said. The misses were not random. The two hardest families were sentiment flashpoints and negative aspects: 49.3% of flashpoint criteria and 41.6% of negative-aspect criteria were covered by no model at all. In other words, the models were worst at the two things a marketer most needs to know: what will make people feel something, and what they will complain about.

The authors' diagnosis is the part every marketer should read. Models "recognize the event topic and produce plausible affect, but fail at attention-anchor selection." They can sound emotional while failing to anticipate which trigger will dominate the conversation. That is a precise description of the gap between a competent message and a winning one.

Two caveats keep this honest. The benchmark uses Chinese-language social media, so the exact numbers may not transfer to every market. And a structured reasoning prompt actually reduced coverage, while a multi-agent generate-and-reflect pipeline improved one model only from 32.9% to 37.6% on a subset. Clever prompting does not close this gap.

The other 2026 evidence points the same way

ConsumerSimBench is not an outlier. The NIM Marketing Intelligence Review (Nuremberg Institute for Market Decisions) tested synthetic respondents against real ones and found they matched real brand choices about 79% of the time, with an average deviation of 1.2 points on a 7-point scale. The problems were directional: synthetic data "consistently overestimated positive attitudes," especially toward well-known brands, and produced significantly less variation than real people. The authors conclude synthetic data is currently best suited to "early-stage concept testing or lower-stakes applications."

Google Research's ConvApparel work on simulated shoppers found the same realism gap from another angle. A trained discriminator "confidently identified nearly all simulated conversations as synthetic." Simulators grounded in real data beat purely prompted ones, but a gap to human behavior remained. The prompted simulator stayed "unnaturally polite and patient" even when facing a bad shopping agent.

Personia, a synthetic research vendor, published a useful roundup of the 2026 validation studies that lands in the same place: synthetic research approximates aggregate, mainstream patterns reasonably well and struggles with individuals, niche objections, positivity bias and controlled experiments. It also notes that no study has yet shown synthetic research improves business decisions. When a vendor says that about its own category, take it seriously.

Why human gut still finds the message

Put the evidence together and a clear pattern emerges. Synthetic audiences regress toward the plausible middle. They are too positive, too uniform and too polite, and they miss the specific trigger that makes a real audience care.

Winning messages live almost entirely outside that middle. They come from noticing the odd detail a customer said on a sales call, the objection nobody says out loud, the joke that only lands with this audience, the tension a category has been ignoring. That is pattern recognition built from exposure to real people, plus the taste to know which strange idea is the right one. It is what good strategists and creatives do, and it is exactly the attention-anchor selection ConsumerSimBench found models are worst at.

This does not make synthetics useless. It makes them a filter. A synthetic panel is very good at telling you a message is confusing, generic or off-category. It is much worse at telling you which of three strong options will break through. Use it where its error profile is acceptable.

A human-first workflow

This is the sequence we recommend for message development:

1. Start with humans and real signal. Mine sales calls, support tickets, reviews and customer interviews. A strategist writes the core tensions and a creative lead drafts a wide set of territories, including some that feel risky.
2. Use synthetics to expand and stress-test. Ask a synthetic panel to generate objections, misreadings and alternative angles for each territory. Treat the output as a hypothesis list, not a verdict.
3. Triage, do not rank. Cut messages the panel consistently finds confusing or generic. Do not promote a message because synthetic personas loved it, because positivity bias is the documented failure mode.
4. Calibrate against a small real sample. Before synthetic results inform spend, run the same stimuli past a small real audience. If synthetic and real reactions disagree on what matters, trust the real sample and adjust how much weight the panel gets.
5. Let humans make the call, then let the market decide. The final shortlist is a judgment call by people with taste. Live tests with real budget settle it.

Questions to ask a synthetic research vendor

Any serious vendor should be able to answer these directly:

• What real data are your personas grounded in, and how recent is it? Grounded simulators beat prompted ones in the published work.
• Have you validated against real respondents in my category and market, and can I see the comparison, including where you were wrong?
• How do you measure and correct for positivity bias and compressed variance?
• What decisions do you recommend your tool not be used for?
• Can I run a side-by-side calibration against my own small real sample before committing?

A vendor that cannot answer the last two questions is selling a forecast it cannot back up.

What to do about it

1. Reclassify synthetic panels in your process as triage. Use them to kill weak messages and surface objections, not to pick winners or forecast response rates.
2. Budget a small real calibration sample for every synthetic study that will influence spend, and compare results before acting.
3. Discount synthetic enthusiasm. Treat high positive scores, especially for an established brand, as a warning sign rather than a green light.
4. Put a named human owner on the final message decision, and require them to explain the insight behind it in plain language.
5. Run the vendor questions above before renewing or signing any synthetic research contract.