AI Panels: Sample Accuracy Is an Illusion

One of the core principles of market research is sampling precision: ensuring that the people you survey accurately represent your target population. This allows researchers to make confident, evidence-based decisions about products, services, or marketing strategies.

Synthetic respondent panels, however, turn this principle on its head. When you “draw a sample” from an AI panel, you are not sampling real people. You are giving the AI instructions — prompts — about the characteristics of the respondents you want it to simulate.

The resulting “sample” is entirely constructed from patterns in a black-box model. There is no real population behind it, no verification of behaviors, and no way to confirm that the generated responses reflect actual human opinions.

Prompt Engineering, Not Research

In essence, what the AI panel does is prompt engineering, not data collection. The quality and “accuracy” of your insights depend entirely on how well the prompt is written, not on the underlying data.

• Accuracy in traditional research – measures how closely the survey responses reflect the real world.
• Accuracy in synthetic panels –  measures how well the AI follows the instructions embedded in the prompt.

This has nothing to do with actual access to demographic or behavioral data. You are not sampling 500 people from a verified panel — you are asking an algorithm to pretend to be 500 people with certain characteristics.

And, of course, the person writing that prompt is usually your vendor, not you. They decide how the AI is instructed to generate responses, how demographics are represented, and how behaviors are “simulated.” In other words, your data is filtered through the vendor’s assumptions, interpretations, and design choices — adding yet another layer of abstraction and uncertainty.

What about the Depth of Responses?

One of the ways AI panels create the appearance of a large sample is through repeated prompting. To simulate multiple respondents, the system runs the model multiple times with slight variations in its inputs or internal randomness. Each run produces a slightly different answer, which vendors present as if it were a separate individual.

This process is often called “sampling” — but it’s very different from what market researchers mean by sampling. In research, sampling involves selecting real people from a defined population to ensure representativeness and statistical validity. In AI, “sampling” is a mathematical term: it refers to the probabilistic process the model uses to select the next word in a sequence.

For example, when a model generates text, it doesn’t deterministically pick the “most likely” word every time. It uses a small amount of randomness to choose among high-probability options. By running this process repeatedly, vendors can produce multiple variations of responses to the same prompt — and call it a panel of respondents.

Each output is unique, but it’s generated by the same underlying model, drawing on the same internal patterns. This is how AI creates the illusion of hundreds of distinct voices — but it is not based on individual people or verified opinions.