Designers disagree and the judge decides

The measurement problem with generative interfaces is real, and the field has been straight about it. When the screen gets assembled per person, there’s no fixed A to put against B. The design agency Orizon said it plainly in August: “A/B testing gets harder when there’s no fixed ‘A.‘” Their other line is the sharper one. Time on page means nothing if the page is different every time.

So the answer that’s settled in is to grade the output instead. Score each render with a model, calibrate that model against expert human labels, ship when the agreement is good enough. Statsig’s guide to testing AI products recommends judge scoring for exactly this reason. Arize’s says to compare the judge’s decisions with trusted labels and keep human review involved in defining the rubric.

I’d do the same thing. What I’d argue with is what happens to that rubric once somebody writes “expert human labels” and starts tuning toward them.

What the platform holds still

Start with what the experiment can still do, because it’s narrower than people assume. Optimizely’s bucketing documentation is specific: a user with the same ID always sees the same variation, because the platform hashes user IDs consistently. That’s a guarantee about the variant. It was never a guarantee about the screen.

Which means the test runs fine. Put the generator in one arm and a drawn design in the other, and the number that comes back is about the generator. It says the policy beat the page, on average, for that population, in that window. It says nothing about the screen one applicant got at 11 on a Sunday night.

That isn’t the method failing. It’s the method answering the question you asked it. The design question used to ride along for free, because the thing sitting in the arm was a screen somebody drew and somebody else approved. Now the thing in the arm is a generator and the screens arrive after the result.

The labels don’t agree

So the judge carries the design question by itself, which makes calibration the step worth reading slowly.

In April, Yi-Hao Peng, Samarth Das, Jeffrey Bigham and Jason Wu put 20 trained designers in front of generated mobile interfaces, everyone with at least a year of professional UI experience. Each rated the same 600 pairs. 12,000 labels in total. On the plain question of which interface in a pair was better, their average Cohen’s kappa came out at 0.25, on a scale where 1.0 is perfect agreement and zero is chance.

Agreement that low normally means a sloppy task or careless raters. The authors went and checked. Thirteen of the designers wrote 1,378 rationales on the pairs they’d split over, and the disagreement held up as systematic. They were weighting different design dimensions, not guessing. The authors’ conclusion is the part to take away: a single aggregate judge loses that signal.

A second group reached it from the other side. In July, Zheng Wu and four colleagues wrote that a judge of this kind reflects only one implicit viewpoint, and that a homogeneous one erases disagreement between user subgroups. Replacing the single judge with a panel of grounded personas moved correlation with human ratings from 0.716 to 0.922. Their panelists still agreed on which model was better overall. They diverged on the individual dimensions.

What a deployment threshold actually is

Read the calibration step again with that in hand. Appen sells rubric design for judges and describes the work honestly. An LLM judge is only as reliable as the rubric it’s given. Calibration is the systematic measurement of agreement between judge scores and expert human rater scores on a calibration dataset, and the rubric gets refined until agreement meets your deployment threshold.

Tune a rubric until it agrees with a pool of raters who agree with each other at 0.25, and you haven’t measured design quality. You’ve picked a side in an argument those raters were having. Hierarchy over density, or density over hierarchy. Familiar over faster. The rubric that ships is the one whose weighting got rewarded, and the threshold makes that look like convergence.

Then the ruling runs on every screen the system renders, at a volume no review meeting has ever touched.

Who makes that call

On most teams the rubric belongs to whoever runs evals, with designers brought in as raters. That’s backwards, and the research is already saying so. Hyewon Lee and colleagues, writing in September, start from the position that designers stay responsible for what a generative interface produces while being unable to specify or even see every interface it might produce. Their system trains its scoring model on designers’ own edits.

That responsibility is the job description now. What the scorer rewards is the last design call a person makes before a million screens get rendered without one.

When a team brings me a generated surface, I ask two things before I look at any screens. Which comparison produced the number. And whose judgment is in the label set the scorer was tuned against. If the answer to the second one is the eval team plus whichever raters were available, then the product’s taste got set by a staffing accident.

The trade I work from is that the team owns the method and I own the standard. A scoring rubric is a standard with a number bolted to it, so it’s mine to sign, the same way a design review was. It’s slower than approving screens. It’s also the only part of the work that survives the screens being different every time.

Those 20 designers were not split because one group was wrong. A judge tuned to 0.25 agreement reports one of their weightings as the standard, and the rest get outvoted by a dataset.