Judging novelty with an LLM:93.5% → 40.9% on prompt wording.

The real problem (beyond the announcement)

An LLM judge is becoming infrastructure. Teams use them to score answers, triage R&D ideas, filter content, compare generators. You pick a model, write a prompt, sanity-check it on ten examples — then let the verdict decide.

On October 27, 2026, the Allen Institute for AI (Ai2) published Old Ideas, Novel Problems (arXiv 2610.02022) with a public benchmark. The finding should cool that trust: the same pair of ideas, judged by the same model, does not get the same verdict when you reword the prompt. Not a two-point drift. Up to fifty-two.

To measure this cleanly, Ai2 built its data from ICLR 2026 peer reviews rather than paper acceptance: reviewers explicitly describe a contribution's originality, which is a different signal — a paper can be accepted without being novel and rejected despite an original idea. The result: a pool of 154 high-novelty human ideas (accepted papers in the top 10% of their area, novelty praised and never disputed) and a pool of 145 lower-novelty ones (rejected papers in the bottom 10%, originality concerns and no positive verdict).

A second setting, Human+Generated, pairs those 154 validated ideas with 154 ideas produced by a plain LLM generator — no retrieval, no tools, no iteration. Why the shift? Because judges are validated on human papers and then deployed on generated text. An expert Ai2 commissioned re-checked 30 pairs sampled from the best judge's errors: they preferred the human idea 30 times out of 30, and judged none of the generated ideas novel.

What the benchmark measures (and the cost table)

Six judges were tested — GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, Claude Opus 4.6 — in pairwise comparisons, varying one factor at a time: prompt wording, reasoning budget, access to the literature, idea formatting, and how many judgments get aggregated.

The factor that crushes all others is wording. First prompt: “reviewers found exactly one novel idea in each pair, which one?”. Second: “which of these two ideas is novel?”. Same pairs, same models:

GPT-5.4 collapses from 93.5% to 40.9% — below chance — and flips its verdict on more than half of the pairs. The “generous” prompt does not just measure the judge's skill: it leaks the label structure. That 93.5% is a score for reading priors, not for judging.

Two caveats. First, on human-only pairs (Human-Only), the same prompt change moved accuracy by only about 7 points (21 points for Claude Opus 4.6): a judge can look stable on human writing and turn chaotic on generated content. Second, there is no universal prompt — dropping the instruction “do not treat trivial or incremental contributions as novel” helped some models and hurt others. A prompt tuned on one model does not transfer.

The temptation is to buy your way out. Ai2 tested that too: an agentic judge (GPT-5.6 Sol, high reasoning budget, iterative arXiv search) peaks at 0.78 macro-F1, against 0.89 for Claude Opus 4.6 with five pre-retrieved abstracts — at roughly ten times the cost per decision.

The most expensive system finishes last. And this is not a model gap: both dedicated evaluators also ran on Claude Opus 4.6. The difference is methodology — a simple prompt with majority voting beats a retrieval + reranking chain when that chain is tested on the wrong terrain.

One more number for the road: when Ai2 pitted generators against each other (Claude Opus 4.5 vs Claude Sonnet 4.5, GPT-5.4 vs GPT-5.1), the tested prompt rewordings alone flipped a quarter to a third of verdicts. If your LLM judge tells you which generator is better, your ranking contains a slice of wording lottery.

When it changes your decision (and when to skip the worry)

The underlying lesson fits in one sentence: a judge score is not a measurement, it is the output of a model × prompt × format triplet. Treat it like a p95 latency: publish it with its pinned configuration, or don't publish it.

Same reasoning on budget: literature retrieval is often sold as the answer to judging reliability. Here it cost 12 to 30 times more for a lower score. The bottleneck is not the judge's intelligence — it is the quality of the corpus it faces and the wording of the question.

Production checklist

1. Version the judge. Model + prompt + format + reasoning budget, stored in the metadata of every evaluation run. A score without its configuration is unusable — and impossible to compare month over month.
2. Measure the flip rate. Have the same pairs judged with three paraphrases of your prompt. Ai2 sees 25 to 33% of verdicts flip across generators. Above roughly 10%, your judge is ranking wording noise.
3. Beware the prompt that “knows” the answer. Announcing “exactly one idea is novel” inflates accuracy by injecting the label structure. Measure with the prompt you actually run in production, not the one that flatters the model.
4. Start from the cheapest baseline. One strong model, a single pass, majority vote over three calls: 0.854 macro-F1 for $2.59 in the study. Add retrieval and reranking only if they beat that baseline on your own data.
5. Keep a human on the last step. The judge triages, the human decides. Ai2's expert got 30 pairs out of 30 right where the best judge erred on some of them — they were sampled from its mistakes.

Code and data are public (allenai/novelty-eval, novelty-judge-bench): the same procedure applies to your own judges, on your own data.