PulseLake logoPulseLake

How do you compare messages or options with simulated audiences?

A worked example from a real run that compared four headlines, with the full sample report to explore.

Updated
The short answer

Put the options side by side and ask every simulated persona the same questions. Preference analysis then shows which option each persona chooses, how strongly, and the reasons and trade-offs behind the choice. The reasons are a better guide to the next test than the ranking alone, so use them to pick the finalists for real research.

Choosing between messages, features, bundles or positioning options is common, and it is often settled by opinion. Preference analysis gives the choice a structure: the same options, the same questions and an explanation of why different people lean one way.

When is preference analysis the right method?

Use it when you have two or more explicit alternatives and need to choose or prioritise. Examples are two headlines for a launch, three feature sets for a plan, or a bundle with and without a service. If you have only one option, use concept testing. If you do not yet know the audience, start with audience discovery.

What does a sample preference analysis report look like?

The example below is a real run that compared four candidate hero headlines for the landing page of an invented company, Northline Automation, which designs, installs and supports warehouse and factory automation. Each headline would sit above the same subhead. Four simulated personas, in finance, executive, operations and engineering roles, answered eight questions twice.

Sample report · Preference analysisSimulated · fictional company

Northline Automation: which hero headline?

Choose the landing-page headline that automation buyers prefer and that most clearly conveys an end-to-end, accountable integration partner, and understand why.

Strong signal

Proceed to a controlled pilot with explicit success measures and risk controls.

  • Concept clarity75.6
  • Preference strength75.6
  • Purchase intent70.6
  • Personas4
  • Runs each2
  • Simulations8
  • Options compared4
  • Option B chosen8 of 8 runs

Purchase intent by segment

  • Finance · Director · Mid (500–5k)72.5
  • Operations · Senior · Enterprise (5k+)65
  • Engineering · Manager · Enterprise (5k+)72.5
  • Executive · Manager · Enterprise (5k+)72.5
It is specific about the operating environment, covers the main stages of delivery, and emphasizes accountability. From a finance and risk perspective, having one team responsible is more meaningful than a broad promise about putting robots to work.
Simulated response · What most influenced your preference
I might choose Option C if testing showed that it generated materially better engagement without reducing lead quality. I would also reconsider if Northline could not substantiate the 'one accountable team' claim across equipment, software integration, and ongoing maintenance.
Simulated response · What would make you choose differently

A simulated example for a fictional company, run on synthetic personas from a stated catalogue. It estimates how a defined audience might respond to the material shown; it does not measure real people, and company names are invented.

The overall signal is strong, 73.9 out of 100. All eight runs chose Option B, the headline that says the work is “assessed, installed and supported by one accountable team”, and all eight rated their intent to choose or recommend it “probably”. The reason given was specificity: it names the service, the operating environments and who is accountable. The accepted downside was length and lower memorability.

A unanimous result deserves caution. AI-generated personas tend to be agreeable, so agreement among them is a prompt to verify, not proof (Nielsen Norman Group). The more useful part of the report is the list of conditions that would change the choice.

How do you read the report?

Every persona-research report follows the same order. These six steps walk through the example above.

  1. Start with the decision brief

    The ring is the overall signal: 73.9 out of 100, labelled “Strong signal”. It combines the dimension scores with equal weight, so read it as a direction, not a verdict. Beside it sits the line the evidence supports acting on.

    In the report: Decision brief
  2. Find the dimension pulling the score

    Concept clarity is the strongest dimension at 75.6; Purchase intent is the weakest at 70.6. An average can hide one weak dimension, and that is usually where the next round of work belongs.

    In the report: Performance across dimensions
  3. Read the choice and the reasons

    The report records the pick for every run. In this example all eight runs chose the same option, so the useful part is the reasons given and the conditions under which a persona would choose differently.

    In the report: Direct answers to the approved research questions
  4. Read findings with their prevalence

    Each finding states how many of the 8 simulations it appeared in, next to a quoted simulated answer. A point raised in 8 of 8 runs carries more weight than one raised once.

    In the report: Reported findings, prevalence and simulated voice
  5. Check who was simulated and why

    The panel is 4 personas drawn from a catalogue with stated filters, each answering 2 times. The report lists every profile and why it was selected, so you can judge whether the right people were in the room.

    In the report: Who was simulated—and why
  6. Finish with the limits and the next step

    Every report ends with what the evidence cannot show and the real-world checks to run next. The simulated answers are hypotheses to test with real people, not findings about them.

    In the report: Limitations and appropriate use · Where to go next

Can you explore the whole report yourself?

Yes. The frame below holds the complete sample report, including each persona’s choice and reasons for every run.

Interactive sample reportOpen full screen ↗

What would you do after a result like this?

The report turns the choice into a test plan:

  • use the winning option only if the “one accountable team” promise can be backed by clear contractual ownership
  • support the headline with evidence of how one accountable team reduces handoffs
  • test the winner against the runner-up using qualified enquiries, not clicks alone, as the main outcome
  • monitor whether the longer headline reduces comprehension or engagement

That is the right use of the result: it narrows four options to two and says what to measure.

What can a simulated preference analysis not tell you?

It cannot tell you what real buyers will choose or how much a choice is worth. Simulated participants have matched the direction of published experiments only qualitatively (Horton et al.). Take the top two options into an A/B test or a survey with a proper sample, and let observed behaviour decide.

A short sequence

  1. Name the decision and write each option out in full, with the same length and detail.
  2. Fix the questions and the criteria before you see the options side by side.
  3. Read the strength of the preference, then the reasons, then the conditions that would change it.
  4. Take the top two options into a real test.

Frequently asked questions

How many options should I compare?

Two to four is usually enough. Beyond that, the reasons become hard to separate and the questions get long. The example on this page compared four headlines.

Does the top-ranked option win?

Not on its own. Check how strong the preference is and whether it holds across segments, then read the conditions under which personas would choose differently. A narrow lead that flips between segments is a reason to test again.

How is this different from concept testing?

Concept testing looks at one proposition. Preference analysis sets alternatives against each other and asks why one is chosen, which makes it the right method for headlines, feature sets and bundles.

Sources

Sources for the facts on this page, last checked October 17, 2026.

  1. Horton, Filippas and Manning, Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus? (arXiv 2301.07543) checked October 17, 2026
  2. Nielsen Norman Group, Synthetic Users: If, When, and How to Use AI-Generated “Research” (2024) checked October 17, 2026

Run research end to end. Keep the knowledge working.

One AI-native operating system for market research and insight professionals. Study design, evidence generation, agents, institutional knowledge, delivery and action, all in one place.