The summary dropped the user who disagreed

On October 5 Jakob Nielsen gave the synthetic-user argument its cleanest form. Use AI personas to open up the problem, then put real people in front of the thing before you ship it. He called it syntheticipation and wrote the rule as “Diverge with silicon; validate with carbon.”

He’s right, and he has the receipts for it. When Nielsen Norman Group ran synthetic users against three studies they had already done with real people, the synthetic answers came back agreeable and flat. A real learner said “I didn’t finish all the courses.” The synthetic one said “I completed all the courses I mentioned.” Nielsen’s conclusion from it is “Real users are meaner. Cherish that.”

So the rule guards one door. It decides who answers the questions. It doesn’t say anything about who writes up the answers.

The fight is over the part almost nobody uses

User Interviews surveyed 150 research professionals in May. Eight percent actively use tools that simulate participants. 28% reject the idea outright and 47% call themselves skeptical and want more evidence first.

In the same survey, 80% said they use AI regularly in their work. Maze asked nearly 500 researchers and product people over the new year and found 69% using AI on at least some research projects, up 19 points in a year.

Read those together. The loud argument is about a tool fewer than one in ten researchers has adopted. The quiet default, already in most studies, is a model reading the transcripts and telling you what’s in them.

Trust peaks exactly where it shouldn’t

Lyssna asked 300 practitioners how much they trust AI across the synthesis tasks. Trust falls as the task takes more interpretation: 82.9% for summarizing, down to 25.6% for visualization.

That ranking makes sense if summarizing is compression. It isn’t. Summarizing is selection, and something has to be left out for the summary to be shorter than the thing it summarizes. What matters is what gets cut, and that has now been measured.

What a summary does to an objection

A study published in March benchmarked 18 language models on aggregating opinions and scored how well each summary represented the people in the source. Minority views lost. Bias against them ran 34.4 times larger than random bias, consistently, across every model and every topic they tested. The paper separates people who see themselves as a minority from positions that are semantically in the minority, and finds both under-represented in the generated summaries.

A January paper by Aghaebe and colleagues names the mechanism. Summarizers resolve disagreement while they write. Handed a set of people who don’t agree, the model converges on the dominant view and the dissent comes out the other side as consensus.

The failure here isn’t inaccuracy. The summary is accurate about the majority. The problem is that the one person who said the thing nobody else said is the cheapest line to cut, and that person is usually the reason the study was worth running.

On the Cross-Sell work at Best Egg, applicant interviews before design started turned up this, from someone who had just been declined: “I probably won’t use you again. It made me mad and frustrated.” One person, one sentence, and it set the shape of the work. I put it in the before-state flow next to the dead end and it’s still on the case study. A good themed summary would have filed it under negative sentiment at decline. True, and nothing to design from.

The quote that argues against the recommendation

When a team brings me a synthesis, I ask for the strongest quote that argues against the recommendation. Not the limitations slide. A sentence a real person said that makes the recommendation harder to make.

If nobody can produce one, I don’t assume the finding is wrong. I assume nobody has checked whether the disagreement survived the write-up, and the study goes back until someone has looked. Usually the objection is sitting in a transcript nobody reopened. The check costs an afternoon, because the raw material never left the building.

The same question works in an interview. Not whether a candidate uses the research tools, which everyone does now, but what the inconvenient finding in their last study was and what they did about it.

Where the rule needs extending

Validating with carbon and writing up with silicon gets you a study with real people in it and a deliverable that reads like agreement. The participants were real. The disagreement died in the summary, and the team acted on the summary.

So extend the rule one step past the participant. The objection belongs in the synthesis, in the words the person used, with the recommendation it cuts against named next to it. If it isn’t in there, the study isn’t finished, however many real humans sat for it.