Methodology

What the agreement rate means

A sample of comparisons is put in front of a person with the model names removed and the judge's own verdict hidden. They pick a winner. The agreement rate is how often that choice matched the judge's. Cohen's kappa is the same figure corrected for chance: on a suite where one model dominates, always naming the favourite scores well on raw agreement while carrying no information, and kappa removes that.

A high number means the ranking is worth reading. A low one is not a failure of the experiment. It is the result.

The tasks

Twenty-five Python code generation tasks, hand-written. Each targets a specific way a plausible-looking answer goes wrong rather than testing whether a model can write Python: cancellation that must not be swallowed, half-open interval semantics, truncating UTF-8 without splitting a grapheme cluster, a context manager whose cleanup must not mask the original exception. Tasks every model passes tell you nothing, so they were written to separate rather than to be fair.

Generation

One output per task and model, all through the same interface with the same token cap and the same suite-level context. Every response is cached on disk keyed by a hash of the model, prompt, context and parameters, so a repeat run reproduces the exact inputs the ranking was built from. A model that fails permanently has the failure recorded against it rather than being quietly dropped from the comparison.

The judge, and its four controls

• Blind. The judge sees "Answer 1" and "Answer 2". No model name reaches the prompt, so it cannot prefer a brand.
• Both orderings. Every pair is judged twice, once each way round, as two separate calls. If the two disagree, that is position bias rather than an opinion, and the pair is recorded as a tie. The share of pairs that flipped is published as the order-flip rate; a high one means the judge is deciding on position rather than content.
• Different provider family. A judge drawn from the same family as any ranked model is refused outright. Self-preference is not a bias this tool can correct for, so it is excluded rather than adjusted.
• Strict parsing. A response without a well-formed verdict token is retried once, then recorded as a failure with the raw text kept. It is never coerced into a winner.

The exact judge prompt

Published in full, because a leaderboard whose grading instructions are private is not reproducible. This is not a copy kept in step with the tool: it is stored alongside every verdict and read back out of the run below, so what appears here is the prompt those verdicts were actually produced under. A suite says what "better" means; the response format and the instruction to ignore length belong to the tool and no suite can change them.

Labelling

The sample is stratified across tasks and model pairs, so the agreement rate describes the suite rather than whichever corner was convenient. Which answer appears first is randomised per item: a labeller who always saw the same model on the left would acquire the very position bias the judge is being measured for. The judge's verdict is never shown, because an agreement rate measured against a visible answer would record suggestibility.

From comparisons to a ranking

Ratings are fitted with Bradley-Terry over all pairwise results rather than counting wins. Raw win counts reward an easy schedule: a model that only faced weak opponents can outscore a better one that faced strong ones. Ties count as half a win to each side, and a small prior keeps the fit finite when a model wins every one of its matchups.

Win rates carry 95% percentile bootstrap intervals over at least 1,000 resamples with a recorded seed, so any published interval can be reproduced exactly. Models whose intervals overlap share a rank number and are shown as tied. Ordering within a tie would assert a difference the evidence does not support, which is the most common way a leaderboard misleads.

What is not measured

Deterministic checks such as compilation and linting inside a sandbox are specified but not yet implemented, so no answer here has been executed. Every judgement on this site is a reading of the code, not a test of it. Cost figures come from the provider's published pricing; where a model could not be priced, it is flagged rather than reported as free.