Incorporating Evaluations and Refinement Loops for Non-Regressive Agentic Workflows

TL;DR

Standard Environments (SE) helps individuals and teams evaluate and refine AI agents and workflows for the outcomes they want. This article is one case study: the code review agents I built for SE itself, the tests that grade them, and what running those tests on ten models taught me.

• Price does not predict quality. One round of the ten review tasks cost about $2 with the cheapest model and effort combination and $414 with the most expensive, at API-equivalent rates. The most expensive did not score highest.
• A cheaper model came close on the answer key. On Part 3’s five selected tasks, Luna at xhigh scored 0.42 against Haiku high’s 0.46; five rounds could not resolve that gap. Luna cost about a quarter as much per trial: $0.42 against $1.63.
• Extra context showed no measurable overall advantage. Vector found 14 recorded bugs in VoiceOver, compared with 9 from the harness-tools-only arm, but its six-case F1 advantage was within the uncertainty; these studies do not establish a general benefit.
• One attempt per task can misorder models. With five attempts at every task and then a second judge from another model family, the ranking changed.
• Reading the runs exposed a tool defect. SE Analyze read every trial and found a step that threw away valid findings. The merge is fixed; its effect still needs a rerun.

The terms these use are explained just below the four parts. To try it yourself, start at Reproduce it.

Standard Environments (SE) is the application I have been building to evaluate and refine AI agents and workflows for specific outcomes, whether they support a business process or an individual’s work. That includes agents for coding, research, computer use, conversation and other knowledge-work tasks. SE ships 258 open-source benchmarks across 12 domains, including software, cybersecurity, finance, law and medicine. It answers whether a change of model, prompt or harness made an agent better or worse, and what the change costs.

Building SE quickly meant reviewing every change for code and security regressions. Paid review bots helped, but their subscriptions include only a handful of reviews, and usage-based overflow gets expensive. After I hit my monthly cap, I built a code-review pipeline for the platform.

Evaluating that pipeline exposed a failure: two reviewers could report different problems on the same lines, and the merge treated them as duplicates and discarded one. SE Analyze traced the loss to the harness. I changed the merge to preserve both checked findings in one comment; the next evaluation has not run, so the recall effect remains an estimate. This case study shows how to evaluate an agent workflow, find a defect and measure a refinement.

Agentic workflows run inside SE as plugins. This article follows one code-review workflow in four parts:

• Part 1: A hand-run study with Luna and Sol on a small project, scored by answer-key recall and a weighted LLM judge.
• Part 2: The evaluation moves to SE, with more models and harnesses, repeated trials and an LLM judge on every task.
• Part 3: Five attempts on half the tasks measure score stability, the rules fix and a second judge from another model family.
• Part 4: SE Analyze reads the runs and guides changes to the review team’s prompts.

Terms used here

• Task: one test case. Here it is a real pull request frozen before review, the instructions the agents get, and a grader that scores their review.
• Benchmark: a set of tasks run together.
• Harness: the command-line agent that drives a model: Codex, Claude Code, Grok Build or Antigravity. CLI means command-line interface.
• Reasoning effort: how long a model may think before it answers, from low to max.
• Run group: one model at one effort on one harness. Each row of a leaderboard is one.
• Trial and round: a trial is one attempt at one task. A round is one attempt at every task.
• Answer key and recall: a task’s answer key records known bugs a review should find; it is not necessarily a complete inventory. Recall is the share of recorded bugs it found.
• LLM judge: a second model that reads the review and the code, then rates the review on four 1-5 scales.
• Mean score and pass@k: the mean score averages every trial, so a review that finds half the key counts as 0.5. pass@k counts only perfect scores: the share of tasks where at least one of k attempts scored 1.0.
• Refinement loop: reading the results, changing the agents’ prompts or code, and rerunning to check that the change helped. Unlike reinforcement learning, it retrains no model.

Running every eval on every model gets expensive. Group 1 alone cost about $1,434 at Harbor’s API-equivalent rates across 319 trials, not in subscription charges; one round of its ten tasks ranged from about $2 to $414:

The most expensive run group was not the best, and the best was not the cheapest. None comes near 1.0, and none should be expected to. A 1.0 means matching every issue recorded in the answer key, not proving that every bug in the code was found. Each SE key holds comments several paid review bots left on that pull request, limited to the ones I accepted and fixed; it is a record of known issues, not a complete inventory. Haiku high matched 0.45 of them for $1.89 a trial.

The review action and Part 1’s experiments, evals and results are in the code_review repository. Part 2 uses private Standard Environments code and bot comments, so its task snapshots and raw runs are not public; aggregate numbers are reported here.

Some of the strongest examples come from Standard Environments, a fast-moving Python and TypeScript monorepo reviewed by six paid bots and my pipeline on every pull request. Its source snapshots, bot comments and raw eval runs are private; this article reports aggregate results. VoiceOver, written in Rust, Python, Svelte and TypeScript, is the public counterpart: every case, key and score is in the repository, so you can check the work and rerun it. The reviewers are not tied to one language; these are the two repositories where they were measured.

Building the code review pipeline#

Before measuring the full review pipeline, I compared harness tools, a vector index and OKF notes to see whether extra context improved reviews. The studies below report that comparison.

The team#

Over about two weeks I tried a handful of open-source code and security review plugins on my own pull requests and kept the parts that did best. The result is one orchestrator running seven reviewers, each its own agent. A validator re-checks every serious finding before it is kept, and the orchestrator merges everything into one de-duplicated list. The skills live in src/plugin/, the orchestration and merging in src/review_core/.

Each skill’s CITATIONS.md traces every passage, line by line, to the plugin it came from and says whether it was changed. The full source list includes Every’s Compound Engineering, Addy Osmani’s agent-skills, brooks-lint, Cursor’s plugins, the antigravity-awesome-skills collection, Claude-BugHunter, Anthropic’s official Claude Code plugins, and Seth Hobson’s claude-code-workflows.

The two studies below were the initial experiments. Both used GPT-6 Luna (gpt-6-luna) at high reasoning effort through the Codex CLI: an affordable, capable model that Part 1 later compares with a stronger one.

Three kinds of context#

SE runs on a graphed version of Google’s Open Knowledge Format (OKF): curated notes about the architecture, interfaces, domains and deployment, linked into a graph. I also wanted to know how a vector database holds up as an application keeps growing. So every review ran in one of three arms. Each arm’s reviewers got the same review instructions plus one paragraph on how to gather context, kept in src/review_core/context/providers.toml:

• Harness tools only: no separate context engine or index. Each reviewer uses its harness’s file and search tools to read every changed line and gather the surrounding code: the whole function or class, its callers and callees (found by searching the repository for their names), and the tests that cover it. The agent gathers this code context in the model’s available context window.
• Vector: the reviewers also get a search command over a vector database of the repository: search "<words>" for a question and related <path> for a file. They are told to look for how the changed code is used elsewhere, for code that resembles it and for the codebase’s conventions, then open the files the results name. The index uses LanceDB, text-embedding-3-large (3,072 dimensions), full-text search and reciprocal-rank fusion, with code chunked along syntax boundaries into pieces of at most 80 lines (src/context_engine/).
• OKF: the curated notes sit in an okf/ folder at the top of the repository, and the reviewers read them with their ordinary file tools: start at its README, find the notes that cover the changed files, and follow them to the code they name (research/okf-notes/ has the VoiceOver set).

In every arm a finding has to come from the code itself, never from a search result or a note alone.

How to read the numbers#

The formulas, and how each interval is computed, are at the end in How the numbers are computed.

Study 1: 28 real pull requests on Standard Environments#

The ground truth came from the bots I was already paying for. Six commercial reviewers (Codex, Greptile, CodeRabbit, Devin, Graphite and Gitar) had left comments on 28 merged SE pull requests. The 111 comments I had accepted and acted on became the golden responses. An LLM judge, following the Martian Code Review Bench method, asked one question per finding: does it state the same underlying issue as a golden response? That made 84 reviews: 28 pull requests, three arms, one run each.

The harness-tools-only arm beat vector on 5 pull requests, lost on 2 and tied on 20 (sign test p = 0.45). OKF cost about 19% more billed input than the harness-tools-only arm for similar recall.

Three other things stood out:

• Variance beats context. Only 17 of the 111 issues were found by all three arms, but 46 (41%) were found by at least one. Running the same harness-tools-only review twice and merging the results may buy more than any context store.
• Most unmatched findings came from one reviewer. Compliance kept 168 findings that matched only 4 golden responses. Dropping it alone lifts key precision from 15% to 23% and costs one golden response. The correctness and language reviewers had the highest key-match rates, at 32 to 57% key precision.
• Billed tokens are the honest number. A run reads 27 to 35 million tokens, but about 97% of that is cached re-reading of the agents’ own growing conversations. Billed input is under 1 million per run. Quoting total tokens overstates the cost about thirtyfold.

Against the paid bots, the team matched about a quarter of the issues all six raised together. That is not a replacement yet; it did surface some accepted bugs the paid reviewers had not raised.

Study 2: a pre-registered replication on VoiceOver#

One private repository proves little, so I repeated the study on VoiceOver, an open-source app of mine. This time I froze the answer key on 2026-10-06, before any arm was scored. Six real changes (480 to 3,227 changed lines, each reviewed whole) hold 44 counted bugs. A deterministic matcher scored the findings, with macro F1 as the headline and a paired bootstrap over cases.

Key precision, recall and F1 are macro averages over the six cases. The key is not a complete inventory, so an unmatched finding may still be valid; the matcher only measures whether it agrees with a recorded issue. On the five development cases, used only for tuning, F1 was harness tools only 0.352, vector 0.487 and OKF 0.399.

On the six holdout cases, vector matched the most recorded bugs (14 of 44) and produced the most findings (59, against 36 from the harness-tools-only arm); harness tools only had the highest key precision. The frozen key is not complete, and the matcher cannot establish whether findings outside it were valid, so the extra findings cannot be called false positives. Everything is public and reproducible in research/: the exact trees the reviewers saw, the frozen keys, the OKF notes, every run’s findings and the scores.

Study 1: known issues found

1. Harness tools only32 of 111
2. vector29 of 111
3. OKF31 of 111

Data table

arm	found	total
Harness tools only	32	111
vector	29	111
OKF	31	111

Study 2: F1 change against harness tools only, 95% interval

1. vector+0.033
2. OKF−0.003
3. −0.2−0.10+0.1+0.2

Data table

arm	delta	low	high
vector	0.033	-0.109	0.144
OKF	-0.003	-0.152	0.136

Neither study found a measurable advantage for extra context in this workflow. Harness tools alone were the simpler choice; the vector index also takes about 25 million embedding tokens to build for SE, plus storage, serving and upkeep.

Building evals#

The studies above answered one question once. But the review tool changes every week: a reworded prompt, a new reviewer, a different model. I needed evals that rerun on every change and say whether the change helped. Building them is environment creation, and it took a lot more than the studies: answer keys, tasks that grade against them, a way to run the whole team as one task, graders that read the code, and a loop that rebuilds the tasks when the tool changes. Standard Environments exists to make this part easy, and an environment advisor, an agent that helps you build each of these pieces, is coming to it.

Answer keys from real bugs#

Every case is a real change with bugs someone confirmed. There are four sources:

• Standard Environments: comments from the paid bots that I accepted and fixed. These stay private.
• VoiceOver, scored: the pre-registered key from the study (vo-t01 to vo-t06), frozen with sha256 lock files so scores before and after any change are against the same bugs.
• VoiceOver, keyed from bot reviews: six more changes (vo-x1, vo-x4, vo-x7, vo-x8, vo-x17, vo-x20). Each key is the pull request’s first bot review, keeping only the findings the reviewed commit confirms on reading.
• The tool itself: cr-self-01, the 18 bugs the action shipped in its first commit.

Rubric tasks#

Each case becomes a Harbor task. The agent gets the code at the base and head of the change and is told to review it. A separate grader container holds the key, so the answers never enter the agent’s environment. A judge checks each key entry as one yes-or-no criterion: does any finding state the same underlying issue? The agent, model and judge are all picked at run time.

No task is trusted until it passes two gates. The oracle writes the reference answer and must score 1.0. The no-op agent writes nothing and must score 0.0. Building a Harbor task like this from scratch will be a post of its own.

Team tasks: the real pipeline as a multi-step task#

Harbor’s multi-step tasks run ordered steps in one container, each with a fresh agent session and its own instructions. Each team task has eight steps:

• Seven reviewer steps, one per reviewer in the team, each given that reviewer’s skill and the same message the orchestrator would send. Each writes its own findings file.
• A final validator step runs the shipped code: the first-pass merge, the validator and finalize. A separate grader scores the merged findings against the key.

It is one multi-step task rather than seven separate ones because the reviewers share the same checkout and the same message from the orchestrator, and only the merged list would ever be posted. So one grader scores that merged list, from its own container that no step can reach, and the reward measures what the pipeline would actually ship. The trial’s reward is the final step’s score, and each reviewer’s step result is kept beside it. So one run tells you both how the pipeline did and which reviewer pulled its weight. The oracle passes the golden responses through the real merge and validator path, so a 1.0 proves the plumbing as well as the grader. There are 22 team tasks: the 12 VoiceOver cases, public, and 10 SE cases, private.

Two graders: recall, and a judge that reads the code#

Recall answers one narrow question: how much of a known list did the review catch? It cannot tell a sharp review that found a real bug outside the key from a noisy one, and it says nothing about cost. So six of the public cases also run as -llm-judge tasks: the same eight steps with a second grader.

That judge works in a sandbox holding the review, the known issues and the code at the reviewed commit. It opens files to check each finding, then rates four 1-5 scales, each with its own scoring guide:

The weights sum to 1, and the reward is their weighted mean. They live in one table in the task bundle, so a new case or a new weighting is a data change, not a code change.

Evals that rebuild themselves#

An eval suite built once goes stale the first time the tool changes. The tasks still run, but they score code that no longer exists, and nothing says so. Three things stop that here:

• A pipeline digest. One sha256 covers every file that shapes a review: skills, prompts, the merge code, the generator and the grader. Every review run stamps it into its manifest, so every score names the exact code that produced it.
• A generator. It rebuilds every task from the current skills, gates it, and writes it back. Run it twice and nothing changes.
• A staleness test. A test regenerates the tasks and compares them with the committed ones. If someone edits a prompt and forgets to regenerate, the push fails.

In SE all of this lives in packages/plugins/code-review/: src/ for the plugin, evals/ for the cases and generator/ for the code that builds them. A small se.toml file tells Standard Environments where the skills are, how to regenerate and where the tasks land. That is what lets the app take over: import the repository, and the platform knows how to run the benchmark.

Part 1: by hand#

A stronger model found about 1.7 times as many of the known bugs, at about 19 times the cost per task. Everything before this used gpt-6-luna, an affordable model. That was a cost decision, not a finding, so I ran the team tasks on a stronger model too.

Luna against Sol#

Two full runs on 2026-10-12, one trial per task: the review team on the Codex CLI with gpt-6-luna (reasoning medium) and with gpt-6.1-sol (high), each graded by a Codex judge on gpt-6-luna. Cost is Harbor’s API-equivalent figure for the review team; the runs actually used a ChatGPT subscription, and the judge is not counted.

Sol finds 1.7 times as many known issues as Luna, at 19 times the cost

Recall

0.00.20.40.6

Luna, 12 public: 0.31Sol, 12 public: 0.54Luna, all 22: 0.22Sol, all 22: 0.47

$0$1$2$3$4$5

Cost per task (API-equivalent)

• 12 public
• all 22

Data table

model	set	recall	cost
Luna	12 public	0.31	0.24
Sol	12 public	0.54	4.65
Luna	all 22	0.22	0.23
Sol	all 22	0.47	4.62

• Sol’s recall is about 1.7 times Luna’s, at about 19 times the cost. It also writes 2.4 times as many findings. On the 12 public tasks Sol led on 8, Luna on 1 (vo-x20, 0.80 to 0), and 3 were tied.
• The SE cases are harder for both. Adding the 10 private SE cases pulls recall down for both models, and Sol’s lead holds.
• Single tasks are noisy. With one trial each, a key of one or two issues swings a whole point (vo-x7, vo-x1). Read the totals, not one task.

For a team on a budget, that is the economic question in one table: Luna runs a full 22-task sweep for about $5, Sol for about $100.

Picking the economical six#

A full sweep on every model adds up fast, so I picked six public tasks that still mean something: a spread of change sizes and bug types that make good stories.

The second grader on those six#

The -llm-judge versions of the six ran on 2026-10-13, one trial each, with the team and the judge both on Codex with gpt-6-luna (medium). Mean reward 0.41, at $0.24 per task for the review team.

The judge rates two of the six reviews above 0.5, and vo-x4 near zero

1. vo-x70.71
2. vo-t060.66
3. vo-t020.43
4. vo-t040.43
5. vo-t010.18
6. vo-x40.04
7. 0.5

Data table

task	agent_model	judge_model	reward	known_issues	finding_validity	severity_and_fixes	economic_value	findings	input_tokens	cost_usd
vo-t01-llm-judge	gpt-6-luna (medium)	gpt-6-luna	0.175	1	2	2	2	6	17005322	0.28
vo-t02-llm-judge	gpt-6-luna (medium)	gpt-6-luna	0.425	2	3	3	3	9	18129685	0.29
vo-t04-llm-judge	gpt-6-luna (medium)	gpt-6-luna	0.425	2	3	3	3	7	22732566	0.35
vo-t06-llm-judge	gpt-6-luna (medium)	gpt-6-luna	0.6625	3	5	4	3	6	11128751	0.18
vo-x4-llm-judge	gpt-6-luna (medium)	gpt-6-luna	0.0375	1	1	2	1	1	8366331	0.15
vo-x7-llm-judge	gpt-6-luna (medium)	gpt-6-luna	0.7125	5	1	4	5	2	8797148	0.16

• The judge checks the code. Its reasons name the functions behind each verdict. On vo-t04 it confirmed that the keychain write logs the error and returns, while the save still reports success.
• Requests for tests are not defects. Findings that only ask for more tests score low on validity. On vo-x7 the one known issue scores 5 on known issues and 5 on economic value, yet validity is 1 because the review’s only other finding was a test request.
• These were new runs of the team, not the same reviews the recall grader scored, so the two graders are not yet compared on identical findings.

Part 2: on the platform#

Part 2 moved the evaluations onto Standard Environments, compared more models, and used cases from SE’s own pull requests. Any eval suite gets onto the platform the same way, as long as its evals are Harbor tasks. Each task is a folder of its own files: a task.toml, an instruction, an environment and its tests. A se.toml file in the repository then tells SE where those tasks are and how to rebuild them, even when they share the repository with the code they test. These share the Standard Environments monorepo. SE imports each se.toml as a benchmark of its own and runs it across models. A prompt changed in the Prompts tab goes back to the repository as a pull request, and merging it starts the next round.

Two of the most useful results came from setups that went wrong.

How the runs are named#

Every number below comes from a leaderboard group in the SE Code Review benchmark, exported with SE Analyze’s Export CSV. A leaderboard group is a saved view: a set of tasks, a range of task versions, and the harness versions it admits. Two groups matter here:

• Group 1, the answer-key benchmark. The ten private SE cases (ge-d02 to ge-x766), each graded on recall against its answer key. Task versions 1 to 8 and every harness version are combined.
• Group 2, the judge benchmark. The same ten cases as -llm-judge tasks, graded by a judge on the four weighted scales above. Task versions 1 to 8 are combined here too.

Each row below is one run group: one model at one reasoning effort on one harness, gathering every run of it the group admits, whatever the launch date or harness version. The judge on every trial in both groups was Claude Code with claude-haiku-5-5, and every agent ran on my own subscriptions. Costs are Harbor’s API-equivalent figures, not what I was billed.

Each run below names the task version it ran on.

Group 1: the answer-key benchmark#

How to read it:

• Click a column header to sort. Each dollar amount is coloured against its column’s median. Cheaper is green, deepening toward the cheapest; dearer is pink, deepening to full red at the most expensive. The median cell has no colour. The distance is measured in ratios, not dollars, so ten times the median is as far above as a tenth is below.
• The interval is over the ten cases. Each case’s trials are averaged, then the ten case means are resampled (the math). It is wide because the cases differ far more than the models do: ge-x540 is solved by almost everyone, ge-x632 by almost no one.
• Cost per 0.1 recall is the column that matters. The advertised price per token says nothing about cost per known issue found. A model that runs more turns, reads more files or writes more tokens can cost more per trial than a model with a higher price, and a model that finds no known issues has an infinite cost per known issue found.
• The two time columns use different units. Median agent time is one trial’s agent phase in minutes and seconds, so 61:18 is just over an hour. Summed wall time is hours and minutes for every trial run one after another, which is how SE’s leaderboard reports it. grok-4.7-build-fast at xhigh took 11 hours 36 minutes for ten trials.

Haiku and Luna hold the frontier; the most expensive runs buy little recall

Data table

config	vendor	trials	reward	low	high	cost	per_point	minutes	wall_hours	frontier
Haiku high	Anthropic	10	0.454	0.265	0.642	1.89	0.42	22	5.2	1
Grok 4.7 fast xhigh	xAI	10	0.368	0.193	0.568	41.42	11.25	61.3	11.6	0
Grok 4.7 xhigh	xAI	10	0.344	0.147	0.564	19.51	5.67	93.6	16	0
Sonnet high	Anthropic	10	0.34	0.14	0.554	5.46	1.6	12.7	3.5	0
Sol medium	OpenAI	10	0.306	0.128	0.5	3.19	1.04	15	3.5	0
Terra xhigh	OpenAI	10	0.304	0.129	0.507	10.08	3.32	38.4	8.1	0
Luna xhigh	OpenAI	86	0.299	0.166	0.456	0.43	0.14	33.3	62.4	1
Luna max	OpenAI	50	0.29	0.159	0.457	0.47	0.16	32.7	34.8	0
Sonnet medium	Anthropic	10	0.282	0.117	0.483	3.44	1.22	7.1	2.3	0
Grok 4.7 low	xAI	10	0.278	0.128	0.475	7.1	2.56	28.1	5.3	0
Grok 4.7 fast low	xAI	10	0.276	0.118	0.48	13.93	5.05	18.3	4.1	0
Argon high	Google	10	0.223	0.06	0.43	13.01	5.84	123.9	21.3	0
Luna high	OpenAI	20	0.176	0.096	0.268	0.27	0.16	13.3	6.5	1
Luna medium	OpenAI	10	0.172	0.033	0.346	0.21	0.12	8.7	2.5	1
Flash high	Google	10	0.117	0.02	0.244	7.79	6.65	86.2	14.8	0
Luna low	OpenAI	30	0.092	0.038	0.148	0.26	0.29	9.5	8.4	0
Argon low	Google	10	0.035	0	0.09	9.01	25.75	72.2	14.1	0
Nemotron 30B	NVIDIA (local)	3	0	0	0	0		68.4	3.8	0

Grok and Gemini take an hour or more per trial

Data table

config	vendor	trials	reward	low	high	cost	per_point	minutes	wall_hours	frontier
Haiku high	Anthropic	10	0.454	0.265	0.642	1.89	0.42	22	5.2	1
Grok 4.7 fast xhigh	xAI	10	0.368	0.193	0.568	41.42	11.25	61.3	11.6	0
Grok 4.7 xhigh	xAI	10	0.344	0.147	0.564	19.51	5.67	93.6	16	0
Sonnet high	Anthropic	10	0.34	0.14	0.554	5.46	1.6	12.7	3.5	0
Sol medium	OpenAI	10	0.306	0.128	0.5	3.19	1.04	15	3.5	0
Terra xhigh	OpenAI	10	0.304	0.129	0.507	10.08	3.32	38.4	8.1	0
Luna xhigh	OpenAI	86	0.299	0.166	0.456	0.43	0.14	33.3	62.4	1
Luna max	OpenAI	50	0.29	0.159	0.457	0.47	0.16	32.7	34.8	0
Sonnet medium	Anthropic	10	0.282	0.117	0.483	3.44	1.22	7.1	2.3	0
Grok 4.7 low	xAI	10	0.278	0.128	0.475	7.1	2.56	28.1	5.3	0
Grok 4.7 fast low	xAI	10	0.276	0.118	0.48	13.93	5.05	18.3	4.1	0
Argon high	Google	10	0.223	0.06	0.43	13.01	5.84	123.9	21.3	0
Luna high	OpenAI	20	0.176	0.096	0.268	0.27	0.16	13.3	6.5	1
Luna medium	OpenAI	10	0.172	0.033	0.346	0.21	0.12	8.7	2.5	1
Flash high	Google	10	0.117	0.02	0.244	7.79	6.65	86.2	14.8	0
Luna low	OpenAI	30	0.092	0.038	0.148	0.26	0.29	9.5	8.4	0
Argon low	Google	10	0.035	0	0.09	9.01	25.75	72.2	14.1	0
Nemotron 30B	NVIDIA (local)	3	0	0	0	0		68.4	3.8	0

A tenth of the key costs $0.14 on Luna xhigh and $25.75 on Argon low

Data table

config	vendor	trials	reward	low	high	cost	per_point	minutes	wall_hours	frontier
Haiku high	Anthropic	10	0.454	0.265	0.642	1.89	0.42	22	5.2	1
Grok 4.7 fast xhigh	xAI	10	0.368	0.193	0.568	41.42	11.25	61.3	11.6	0
Grok 4.7 xhigh	xAI	10	0.344	0.147	0.564	19.51	5.67	93.6	16	0
Sonnet high	Anthropic	10	0.34	0.14	0.554	5.46	1.6	12.7	3.5	0
Sol medium	OpenAI	10	0.306	0.128	0.5	3.19	1.04	15	3.5	0
Terra xhigh	OpenAI	10	0.304	0.129	0.507	10.08	3.32	38.4	8.1	0
Luna xhigh	OpenAI	86	0.299	0.166	0.456	0.43	0.14	33.3	62.4	1
Luna max	OpenAI	50	0.29	0.159	0.457	0.47	0.16	32.7	34.8	0
Sonnet medium	Anthropic	10	0.282	0.117	0.483	3.44	1.22	7.1	2.3	0
Grok 4.7 low	xAI	10	0.278	0.128	0.475	7.1	2.56	28.1	5.3	0
Grok 4.7 fast low	xAI	10	0.276	0.118	0.48	13.93	5.05	18.3	4.1	0
Argon high	Google	10	0.223	0.06	0.43	13.01	5.84	123.9	21.3	0
Luna high	OpenAI	20	0.176	0.096	0.268	0.27	0.16	13.3	6.5	1
Luna medium	OpenAI	10	0.172	0.033	0.346	0.21	0.12	8.7	2.5	1
Flash high	Google	10	0.117	0.02	0.244	7.79	6.65	86.2	14.8	0
Luna low	OpenAI	30	0.092	0.038	0.148	0.26	0.29	9.5	8.4	0
Argon low	Google	10	0.035	0	0.09	9.01	25.75	72.2	14.1	0
Nemotron 30B	NVIDIA (local)	3	0	0	0	0		68.4	3.8	0

Run group by run group

One entry per row of the table, in the same order: open to read all 18.

• claude-haiku-5-5 high · claude-code 2.1.296 · v4 · 10 trials · 0.45 · $1.89 a trial. The best in the group, on the harder v4 key. It solved ge-x540 outright, found 0.79 of ge-d04, and scored 0 on only two cases. It is also a heavy writer: 268 thousand output tokens a trial, more than any Claude or OpenAI run. With one trial per case, its interval runs from 0.27 to 0.64. The judge that checked its recall was also Haiku, so the same family graded itself.
• grok-4.7-build-fast xhigh · grok-build 1.0.62 · v8 · 10 trials · 0.37 · $41.42. Second in the group and the most expensive run group in this article: $414.24 for ten trials, 11 hours 36 minutes end to end. Haiku scored higher for $18.92.
• grok-4.7 xhigh · grok-build 1.0.62 · v8 · 10 trials · 0.34 · $19.51. Tied with Sonnet high on recall at 3.6 times the cost, with a median trial of 94 minutes. The ten trials took 16 hours end to end.
• claude-sonnet-5-5 high · claude-code 2.1.296 · v4 · 10 trials · 0.34 · $5.46. Nearly three times Haiku’s cost per trial for a lower score. The bigger Claude did not buy more recall here.
• gpt-6.1-sol medium · codex 0.162.1 · v4 · 10 trials · 0.31 · $3.19. The reference point for Luna. Sol is the most frugal writer in the group, 30 thousand output tokens a trial. It still costs seven times what Luna at xhigh costs, because its tokens are priced higher.
• gpt-5.6-terra xhigh · codex 0.163.0 · v8 · 10 trials · 0.30 · $10.08. Luna xhigh’s score at 23 times its cost. Planned as 100 trials across both groups, it has 13 graded: a runner-image preparation failure ended most of its early jobs, and ten of the 13 are here.
• gpt-6-luna xhigh · codex 0.162.1 and 0.163.0 · v4 and v8 · 86 trials · 0.30 · $0.43. Similar average recall to Sol on the same tasks, at about a seventh of the cost. Luna uses 386 turns a trial against Sol’s 192. Its 86 trials span two task versions and two Codex releases, more trials than any other run group in the benchmark.
• gpt-6-luna max · codex 0.162.1 and 0.163.0 · v8 · 50 trials · 0.29 · $0.47. The same recall as xhigh for a few cents more a trial: on this key, max effort buys nothing over xhigh. It also holds the one perfect review outside ge-x540, below.
• claude-sonnet-5-5 medium · claude-code 2.1.299 · v4 · 10 trials · 0.28 · $3.44. The fastest trial in the group (7 minutes median). Dropping Sonnet from high to medium cost 0.06 of recall and saved 37% per trial.
• grok-4.7 low · grok-build 1.0.62 · v8 · 10 trials · 0.28 · $7.10. Sonnet-medium territory at twice Sonnet medium’s cost.
• grok-4.7-build-fast low · grok-build 1.0.62 · v8 · 10 trials · 0.28 · $13.93. The same recall as grok-4.7 at low effort, at twice the cost, ten minutes faster per trial. “Fast” buys those ten minutes at twice the price.
• gemini-4-argon high · antigravity 1.4.2 · v8 · 10 trials · 0.22 · $13.01. The slowest in the group, at a median trial of just over two hours and 57.5 million input tokens. High effort lifts Argon off the floor, at $13 a trial.
• gpt-6-luna high · codex 0.162.1 · v4 and v8 · 20 trials · 0.18 · $0.27. No better than medium, on either task version. On this key, Luna needs more than high effort to pay off.
• gpt-6-luna medium · codex 0.162.1 · v4 · 10 trials · 0.17 · $0.21. The cheapest in the group. It found most of ge-d04 (0.64) and ge-x540 (0.60) and nothing at all on six of the ten cases. It sets the floor.
• gemini-3.8-flash high · antigravity 1.3.5 · v5 and v6 · 10 trials · 0.12 · $7.79. Flash is the model sold as the cheap one, and only two run groups paid more per 0.1 recall ($6.65): Argon at low effort and grok-4.7-build-fast at xhigh. It ran up to 734 turns and 67 million input tokens a trial, the most turns of any model and as many tokens as Argon at high effort. Its one good case was ge-x540 (0.60).
• gpt-6-luna low · codex 0.162.1 · v4 · 30 trials · 0.09 · $0.26. 19 of 30 trials found nothing. Three trials per case make this one of the firmest numbers in the group, and it says low effort mostly misses.
• gemini-4-argon low · antigravity 1.4.0 and 1.4.2 · v8 · 10 trials · 0.04 · $9.01. Eight of the ten trials at zero. At $25.75 per 0.1 recall it is the least efficient run group in the article, and its larger cases cost the most per trial.
• Nemotron 3.5 Lightning 30B, self-hosted (UD-Q4_K_XL) · claude-code 2.1.304 · v8 · 3 trials · 0.00 · $0. Planned for 20 trials across both groups. I cancelled 13 because its trials ran about an hour each and none of the earlier ones had found anything. The three that finished found nothing either.

What Group 1 says

• Luna xhigh and Sol have similar average recall. Paired by case, their mean difference is −0.01 (95% CI −0.16 to +0.13); Luna costs $0.43 versus Sol’s $3.19 per trial. The comparison has uneven evidence: Luna’s 86 trials span task versions v4 and v8 and two Codex releases, while Sol has ten trials on v4.

Luna and Sol win different cases and tie overall

• Luna xhigh ($0.43 a trial)
• Sol medium ($3.19 a trial)

Data table

case	luna_xhigh	sol_medium
ge-d02	0.185	0
ge-d04	0.621	0.571
ge-d07	0.128	0.4
ge-d11	0.083	0.5
ge-x540	0.844	1
ge-x543	0.278	0.167
ge-x599	0.189	0
ge-x632	0.056	0
ge-x666	0.222	0.125
ge-x766	0.341	0.3

• Haiku has the highest observed recall, but only one trial per case. Its mean is 0.45, with a 95% interval of 0.27 to 0.64; Part 3 tests that lead with five attempts on selected tasks.
• Token price and effort do not translate linearly into cost or recall. For example, fast Grok costs twice as much as Grok at the same effort for the same recall; Luna’s largest gain is from high to xhigh, with no further gain at max.
• Time is a practical cost. Grok xhigh and Argon high each take more than 90 minutes per trial, too long for many CI review loops.

Group 2: the judge benchmark#

Sol did not run in Group 2, so the Luna-against-Sol comparison exists only on the answer key.

Luna max reaches the expensive models' scores for under half a dollar

Data table

config	vendor	trials	reward	low	high	cost	per_point	minutes	wall_hours	frontier
Haiku high	Anthropic	10	0.628	0.554	0.708	1.86	0.3	23.8	5.4	1
Terra xhigh	OpenAI	13	0.612	0.512	0.713	8.84	1.45	38.1	10.4	0
Luna max	OpenAI	55	0.556	0.489	0.64	0.46	0.08	32	41.5	1
Argon high	Google	10	0.525	0.436	0.611	12.69	2.42	104.9	20.1	0
Grok 4.7 xhigh	xAI	10	0.524	0.459	0.59	20.12	3.84	85.5	16.6	0
Grok 4.7 fast xhigh	xAI	10	0.491	0.431	0.552	33.24	6.77	61.9	12.6	0
Flash high	Google	2	0.487			9.08	1.86	82.9	3.1	0
Luna xhigh	OpenAI	50	0.479	0.428	0.53	0.41	0.09	23.2	27	1
Grok 4.7 low	xAI	10	0.473	0.407	0.545	7.17	1.52	28.3	5.2	0
Luna high	OpenAI	50	0.467	0.41	0.529	0.28	0.06	14.7	21.4	1
Luna medium	OpenAI	45	0.426	0.365	0.493	0.23	0.05	9.4	14.6	1
Grok 4.7 fast low	xAI	10	0.406	0.339	0.466	14.63	3.6	17	4.2	0
Luna low	OpenAI	50	0.398	0.324	0.471	0.23	0.06	8.7	15.8	0
Argon low	Google	12	0.348	0.246	0.439	8.52	2.45	73.6	16.2	0
Nemotron 30B	NVIDIA (local)	5	0.085	0.015	0.17	0		60.9	4.7	0

Luna max spends a little more time than xhigh, and scores higher

Data table

config	vendor	trials	reward	low	high	cost	per_point	minutes	wall_hours	frontier
Haiku high	Anthropic	10	0.628	0.554	0.708	1.86	0.3	23.8	5.4	1
Terra xhigh	OpenAI	13	0.612	0.512	0.713	8.84	1.45	38.1	10.4	0
Luna max	OpenAI	55	0.556	0.489	0.64	0.46	0.08	32	41.5	1
Argon high	Google	10	0.525	0.436	0.611	12.69	2.42	104.9	20.1	0
Grok 4.7 xhigh	xAI	10	0.524	0.459	0.59	20.12	3.84	85.5	16.6	0
Grok 4.7 fast xhigh	xAI	10	0.491	0.431	0.552	33.24	6.77	61.9	12.6	0
Flash high	Google	2	0.487			9.08	1.86	82.9	3.1	0
Luna xhigh	OpenAI	50	0.479	0.428	0.53	0.41	0.09	23.2	27	1
Grok 4.7 low	xAI	10	0.473	0.407	0.545	7.17	1.52	28.3	5.2	0
Luna high	OpenAI	50	0.467	0.41	0.529	0.28	0.06	14.7	21.4	1
Luna medium	OpenAI	45	0.426	0.365	0.493	0.23	0.05	9.4	14.6	1
Grok 4.7 fast low	xAI	10	0.406	0.339	0.466	14.63	3.6	17	4.2	0
Luna low	OpenAI	50	0.398	0.324	0.471	0.23	0.06	8.7	15.8	0
Argon low	Google	12	0.348	0.246	0.439	8.52	2.45	73.6	16.2	0
Nemotron 30B	NVIDIA (local)	5	0.085	0.015	0.17	0		60.9	4.7	0

Every Luna setting costs under a dime per 0.1; Grok's fast xhigh costs $6.77

Data table

config	vendor	trials	reward	low	high	cost	per_point	minutes	wall_hours	frontier
Haiku high	Anthropic	10	0.628	0.554	0.708	1.86	0.3	23.8	5.4	1
Terra xhigh	OpenAI	13	0.612	0.512	0.713	8.84	1.45	38.1	10.4	0
Luna max	OpenAI	55	0.556	0.489	0.64	0.46	0.08	32	41.5	1
Argon high	Google	10	0.525	0.436	0.611	12.69	2.42	104.9	20.1	0
Grok 4.7 xhigh	xAI	10	0.524	0.459	0.59	20.12	3.84	85.5	16.6	0
Grok 4.7 fast xhigh	xAI	10	0.491	0.431	0.552	33.24	6.77	61.9	12.6	0
Flash high	Google	2	0.487			9.08	1.86	82.9	3.1	0
Luna xhigh	OpenAI	50	0.479	0.428	0.53	0.41	0.09	23.2	27	1
Grok 4.7 low	xAI	10	0.473	0.407	0.545	7.17	1.52	28.3	5.2	0
Luna high	OpenAI	50	0.467	0.41	0.529	0.28	0.06	14.7	21.4	1
Luna medium	OpenAI	45	0.426	0.365	0.493	0.23	0.05	9.4	14.6	1
Grok 4.7 fast low	xAI	10	0.406	0.339	0.466	14.63	3.6	17	4.2	0
Luna low	OpenAI	50	0.398	0.324	0.471	0.23	0.06	8.7	15.8	0
Argon low	Google	12	0.348	0.246	0.439	8.52	2.45	73.6	16.2	0
Nemotron 30B	NVIDIA (local)	5	0.085	0.015	0.17	0		60.9	4.7	0

Run group by run group

One entry per row of the table, in the same order: open to read all 15.

• claude-haiku-5-5 high · claude-code 2.1.303 · judge v3 · 10 trials · 0.63 · $1.86 a trial. The best in the group. Every case scored at least 0.42, and ge-d04 scored 0.81. Haiku was also the judge, so this is the result most exposed to a judge favouring its own family. Haiku also led the answer key, where the judge only decides whether a finding matches a known bug, so the lead is not only the judge’s taste.
• gpt-5.6-terra xhigh · codex 0.163.0 · 13 trials · 0.61 · $8.84. Second in the group, 0.05 above Luna max at 19 times its cost per trial. It is the run group planned for 100 trials across both groups.
• gpt-6-luna max · codex 0.163.0 · judge v3 · 55 trials · 0.56 · $0.46. Third in the group and first per dollar among the strong run groups: 0.1 of judge reward for $0.08. No trial scored 0, and it reached 0.83 on ge-x540. Each trial took a little longer than xhigh (32 minutes against 23 at the median) and scored 0.08 higher.
• gemini-4-argon high · 10 trials · 0.53 · $12.69. Fourth in the group, at 105 minutes a median trial.
• grok-4.7 xhigh · grok-build 1.0.62 · 10 trials · 0.52 · $20.12. Effort buys Grok 0.05 for 2.8 times the cost.
• grok-4.7-build-fast xhigh · grok-build 1.0.62 · 10 trials · 0.49 · $33.24. Below grok-4.7 xhigh, for more money.
• gemini-3.8-flash high · antigravity 1.3.5 · judge v2 · 2 trials · 0.49 · $9.08. The judge rated Flash’s two reviews mid-table, the same week the answer key scored it 0.12. 811 turns a trial, the most in the article.
• gpt-6-luna xhigh · codex 0.162.1 · judge v3 · 50 trials · 0.48 · $0.41. Run on the same tasks and judge as Luna medium and low, which makes the three the cleanest effort comparison in the article. Each step up in effort adds 0.03 to 0.05.
• grok-4.7 low · grok-build 1.0.62 · 10 trials · 0.47 · $7.17.
• gpt-6-luna high · codex 0.162.1 · judge v1 and v2 · 50 trials · 0.47 · $0.28. No trial scored 0, the first sign the judge grades on a gentler curve than the key.
• gpt-6-luna medium · codex 0.162.1 · judge v3 · 45 trials · 0.43 · $0.23. ge-d02 did not finish, so nine cases.
• grok-4.7-build-fast low · grok-build 1.0.62 · 10 trials · 0.41 · $14.63. Twice grok-4.7 low’s cost for a lower score.
• gpt-6-luna low · codex 0.162.1 · judge v3 · 50 trials · 0.40 · $0.23. Four trials scored 0, more than any run group but Nemotron.
• gemini-4-argon low · antigravity 1.4.0 then 1.4.2 · 12 trials · 0.35 · $8.52. On the key this run group scored 0.04, and the judge gave it 0.35.
• Nemotron 3.5 Lightning 30B, self-hosted (UD-Q4_K_XL) · codex 0.163.0 and claude-code 2.1.304 · 5 trials · 0.09 · $0. One trial on the Codex harness and four on Claude Code. The only run group near the floor.

What Group 2 says

• Haiku’s top judge score needs an independent check. It leads at 0.63, but it was also the judge; Luna max scores 0.56 at a quarter of the cost. Part 3 regrades the reviews with a judge from another model family.
• The judge compresses scores while mostly preserving rank. Across the 15 run groups present in both groups, reward and recall have a rank correlation of 0.80. Excluding Nemotron, the score spread narrows from 0.04–0.45 on the key to 0.35–0.63 on the judge, which raises the weakest run groups most.

The judge scores every run group above its recall, and lifts the weakest most

Data table

config	key	judge
Haiku high	0.454	0.628
Terra xhigh	0.304	0.612
Luna max	0.29	0.556
Argon high	0.223	0.525
Grok 4.7 xhigh	0.344	0.524
Grok 4.7 fast xhigh	0.368	0.491
Flash high	0.117	0.487
Luna xhigh	0.299	0.479
Grok 4.7 low	0.278	0.473
Luna high	0.176	0.467
Luna medium	0.172	0.426
Grok 4.7 fast low	0.276	0.406
Luna low	0.092	0.398
Argon low	0.035	0.348
Nemotron 30B	0	0.085

What the judge actually rewards

The judge’s reward is a weighted sum of four scales, and the platform keeps each one per trial. Splitting the reward back into its scales shows where the lift over the key comes from.

Each scale is rescaled to 0 to 1 and averaged over cases, the same way as the reward. For 11 run groups the weighted scales reproduce the reward exactly. Luna low, Luna medium and Argon low each have trials that scored 0 with no scale scores (four, one and one), so their scale means sit slightly above their reward. Nemotron’s trials carry no scale scores at all, so it is left out.

On the question the key asks, the judge agrees with the key

• Judge: known issues scale
• Answer key: recall

Data table

config	key	known_issues	finding_validity	severity_and_fixes	economic_value	judge
Haiku high	0.454	0.4	0.8	0.6	0.725	0.628
Terra xhigh	0.304	0.379	0.842	0.683	0.617	0.612
Luna max	0.29	0.372	0.775	0.575	0.547	0.556
Argon high	0.223	0.2	0.825	0.575	0.575	0.525
Grok 4.7 xhigh	0.344	0.4	0.625	0.45	0.6	0.524
Grok 4.7 fast xhigh	0.368	0.3	0.6	0.475	0.6	0.491
Flash high	0.117	0.125	0.75	0.5	0.625	0.487
Luna xhigh	0.299	0.23	0.73	0.565	0.475	0.479
Grok 4.7 low	0.278	0.25	0.75	0.5	0.45	0.473
Luna high	0.176	0.205	0.765	0.61	0.41	0.467
Luna medium	0.172	0.131	0.757	0.608	0.385	0.426
Grok 4.7 fast low	0.276	0.125	0.65	0.425	0.475	0.406
Luna low	0.092	0.157	0.723	0.618	0.357	0.398
Argon low	0.035	0.083	0.778	0.5	0.306	0.348

• On the key’s own question, the judge is not lenient. Its known-issues scale lands within 0.09 of answer-key recall for 13 of the 14 run groups. Grok 4.7 fast low is the exception, 0.15 below its recall. Two very different graders, a yes-or-no check per golden response and a 1-to-5 rating of the whole review, roughly agree on how much of the key a review found.
• The lift comes from the other three scales, and they have a high floor. Known issues spans 0.08 to 0.40 across run groups. Finding validity never drops below 0.60, severity and fixes below 0.42, or economic value below 0.31. Weighted, those lowest values alone add about 0.31 before a single known issue is found. That is why the weakest paid run group still scores 0.35, and why the judge compresses the field.
• Validity rewards saying little. A review with few findings has few findings to be wrong. Argon at low effort found 0.08 of the known issues and still scored 0.78 on validity, close to Haiku’s 0.80. What drags validity down is a finding that does not hold up, such as a bare request for more tests, as vo-x7 showed in Part 1.
• The scales separate run groups the total hides. Grok 4.7 at xhigh ties Haiku on known issues (0.40 each) but scores 0.62 on validity against Haiku’s 0.80: more of Grok’s extra findings did not hold up in the code. Haiku’s lead over Luna max comes mostly from economic value (0.72 against 0.55).
• Haiku also judged Haiku. The judge on every trial was claude-haiku-5-5, so the top result is the one most exposed to a judge favouring its own family. Its known-issues lead agrees with the key, which asks Haiku only whether each finding matches a known bug, but the softer scales leave more to the judge. Part 3 tests this with a second judge, and the lead does not survive it.

How consistent the judge is stays open here: only two trials were ever regraded, so the platform’s flip-rate columns are empty. Part 3 takes that up.

The $80 caching trap#

One run put Claude Haiku on the Codex harness, through OpenRouter. On paper it was the cheap option. In practice it cost about $80 and finished little: 30% of its gateway requests failed (403s and rate limits), and most steps ended in errors.

The bigger problem was prompt caching. A provider that caches prompts keeps the start of a prompt it has already seen, and bills a repeat of that prefix at a fraction of the normal input price. Agents lean on it, because every turn re-sends the whole conversation so far. Across the 11 trials that recorded tokens, it sent about 103 million input tokens, and not one was read from cache. Not one was written to cache either. Sol, on the same Codex harness, had about 92% of its input cached.

The cause is a mismatch between the two tools. Codex was built for OpenAI, which caches a repeated prompt prefix without being asked. OpenRouter caches Claude only when the request asks for it, with a cache_control field and a session_id so every turn reaches the same server. Codex sends neither. With about a hundred turns per trial here, every turn paid full price for the whole conversation.

The platform’s own estimate said $11.51, because it priced Haiku from OpenRouter’s catalog at a fraction of what was billed. The check takes a minute on any provider: if a multi-turn agent run shows zero cache writes, caching was never switched on. Match the harness to the model family (Codex with OpenAI models, Claude Code with Claude models), or make sure your gateway is told to cache.

A 30B local model isn’t there yet#

Nemotron 3.5 Lightning is my daily driver locally for small, menial tasks: the bash command I forgot, a syntax slip. I run NVIDIA’s 30B model (3B active) as Unsloth’s UD-Q4_K_XL quant (unsloth/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF). In my use this quant has been the fastest, with the best tool calling. So I ran it on my own hardware through the Claude Code harness, at about 30 tokens a second. On these eight-step review tasks it isn’t ready. Each tool call took about 50 seconds, and each reviewer step took 16 to 28 minutes, which puts a trial at about three hours. Sonnet’s median trial was 13 minutes. Its first three reviewers wrote empty findings, and it re-wrote the same file several times.

Eight trials finished and were graded. On the answer key all three scored 0. The judge gave the four on Claude Code a mean of 0.11, and the one on Codex 0. A local model this size can run the team, but it neither finds the bugs nor finishes at a speed anyone would wait for.

SE Analyze found the bugs in my own evals#

After a round, SE Analyze reads every trial: the transcripts, each reviewer’s findings and the grader’s verdicts. It writes a report on what went wrong and what to change. Six reports on the ten private SE cases came back with 150 suggestions.

I checked each one against the code before acting. Most were habits rather than bugs, but three were bugs in the evals themselves, and between them they reached every task:

• Answers no reviewer could reach. 5 of the 62 known issues sat on files the snapshot strips before the review. One case topped out at 3 of 5, however good the review was.
• One bug counted twice. Two paid bots often reported the same defect, and both comments landed in the answer key. One finding then earned, or lost, two criteria. Folding the duplicates took the key from 62 criteria to 48.
• A reviewer that could not find the rules. The compliance reviewer looked for the project’s rules under the usual file names. The case snapshots ship them under another name, so it audited against nothing.

The rest were mostly habits: checks the reviewers skipped on several tasks, such as what happens when a new setting is empty, or what a multi-step write leaves behind when it stops halfway. Those go into the reviewer skills in general words. Suggestions that named a specific bug in a specific task were rejected, because writing the answer key into the prompt teaches to the test.

Part 3: testing what Part 2 found#

Part 2 left me with a shortlist and four doubts. The shortlist was two models: Luna on Codex, which had similar average recall to Sol at a seventh of the price and was the cheapest per point at every effort, and Haiku on Claude Code, which had the highest observed recall from one trial per case. The doubts were about how far to trust that:

• The ranking. Most run groups ran once per task, which gives a first ordering, not a settled one. Haiku high’s lead rested on ten trials, one per case.
• The effort step. Luna’s recall jumped between high and xhigh and went flat at max, but most effort levels had only one to five trials per task.
• The fix. The platform’s audit had just corrected the tasks, so nothing yet said whether the fix moved any score.
• The judge. Haiku judged every trial on the judge benchmark, its own reviews included.

Part 3 narrows to those two models and tests each doubt. It sweeps their efforts, Haiku at low, medium and high and Luna at medium, high, xhigh and max, and gives every task five attempts: seven run groups, all on the fixed tasks. To keep that affordable, it runs half the tasks, the ones that best tell reviewers apart.

Choosing the tasks#

The obvious pick is the hardest tasks. The data says otherwise. A task is only useful if run groups score differently on it, so I ranked every task by separation: how far apart the run groups’ average scores sit, measured against how much one run group’s scores bounce around on that task. A task where everyone scores near zero separates nothing, however hard it is.

I ranked on every trial Part 2 produced (319 answer-key trials, 342 judge trials) and kept the top five for each grader:

Two things stood out. The hardest answer-key task, ge-x632, averages 0.06 recall and 78% of its trials score zero, but it separates run groups at only 0.44, so it was left out. And the two graders disagree on what is hard: ge-x632 and ge-d11 are among the weakest tasks under the answer key, yet the judge separates run groups on both.

One caveat. On the answer key, only three run groups had three or more trials per task, so its ranking rests on less evidence than the judge’s (four to six run groups). Part 3’s own runs will show whether the choice holds.

The results#

Each row is one run group: one model at one effort, five attempts at every task, on all five answer-key tasks and all five judge tasks. The tables after it break both scores down by task.

Per task, each cell is the mean of five attempts. On the answer key:

On the judge:

Answer key: xhigh is where Luna's score steps up

1. Luna medium0.297
2. Luna high0.307
3. Luna xhigh0.425
4. Luna max0.423
5. Haiku low0.291
6. Haiku medium0.396
7. Haiku high0.46
8. 0.00.10.20.30.40.50.60.7

Data table

config	mean	low	high
Luna medium	0.297	0.228	0.366
Luna high	0.307	0.248	0.366
Luna xhigh	0.425	0.371	0.478
Luna max	0.423	0.254	0.592
Haiku low	0.291	0.225	0.357
Haiku medium	0.396	0.308	0.484
Haiku high	0.46	0.286	0.633

Judge: every run group lands between 0.49 and 0.60

1. Luna medium0.489
2. Luna high0.497
3. Luna xhigh0.556
4. Luna max0.561
5. Haiku low0.513
6. Haiku medium0.561
7. Haiku high0.6045
8. 0.40.50.60.7

Data table

config	mean	low	high
Luna medium	0.489	0.426	0.553
Luna high	0.497	0.442	0.552
Luna xhigh	0.556	0.496	0.617
Luna max	0.561	0.519	0.604
Haiku low	0.513	0.481	0.546
Haiku medium	0.561	0.499	0.623
Haiku high	0.6045	0.568	0.641

What effort buys#

More effort raises the score, but in one step rather than a slope. Luna on the answer key sits at 0.30 and 0.31 for medium and high, then jumps to 0.42 at xhigh. That jump is +0.12, about four times its standard error over five rounds each. Going on to max adds nothing: it lands at 0.42 as well. Haiku’s step comes lower: low to medium is +0.10 (t = 2.6), and medium to high is not distinguishable.

Effort labels do not line up across labs: each sets its own dial, so “high” on Haiku and “high” on Luna do not buy the same amount of thinking. What can be compared is what a setting spends. The chart puts each run group’s answer-key score against the output tokens its trials produced on those tasks, which count the thinking along with the review itself.

Luna's xhigh step and Haiku's medium step buy about the same per token

• Luna (Codex)
• Haiku (Claude Code)

Data table

config	model	effort	out_tok_k	mean	low	high
Luna medium	Luna	medium	46.8	0.297	0.228	0.366
Luna high	Luna	high	77.5	0.307	0.248	0.366
Luna xhigh	Luna	xhigh	162.6	0.425	0.371	0.478
Luna max	Luna	max	196.3	0.423	0.254	0.592
Haiku low	Haiku	low	138	0.291	0.225	0.357
Haiku medium	Haiku	medium	208.1	0.396	0.308	0.484
Haiku high	Haiku	high	319.3	0.46	0.286	0.633

On that scale the two models’ best steps look alike. Luna’s high-to-xhigh step adds about 85k output tokens a trial and 0.12; Haiku’s low-to-medium step adds about 70k and 0.10. Haiku’s next step adds 111k for 0.06, and Luna’s step to max adds 34k for nothing.

The judge rises at both ends, Luna from 0.49 to 0.56 and Haiku from 0.51 to 0.60, but no single step between neighbouring efforts clears the noise; the largest, Luna high to xhigh, reaches t = 2.02.

Across the two models, the answer-key means are close: Haiku high is 0.46 against Luna xhigh at 0.42, a difference five rounds cannot resolve. Luna costs a quarter as much per trial ($0.42 against $1.63). On the judge, Haiku high leads by about 0.05, which five rounds cannot separate from Luna max (t = 2.13) or Luna xhigh (t = 1.88), and a second judge reverses that lead (see “A second judge” below).

Did the fix move the scores?#

The fix itself was small. Part 2’s audit found the compliance reviewer looking for the project’s rules under a file name the case snapshots did not use, so it audited against nothing. Renaming the file in every snapshot and importing the newest commit moved the answer-key tasks from v8 to v9 and the judge tasks from v3 to v4. The V9 Latest and V9-latest-llmjudge leaderboard groups collect the runs on the new versions.

It did not move the scores measurably. The cleanest check is the judge, because Group 2 ran the same five judge tasks at v3 with five rounds for every Luna effort:

The judge scores barely move across the fix

• Before, task v3
• After, task v4

Data table

config	before	after	delta	ci95	before_trials
Luna medium	0.434	0.489	0.055	0.08	25
Luna high	0.483	0.497	0.013	0.063	25
Luna xhigh	0.519	0.556	0.037	0.085	25
Luna max	0.608	0.561	-0.046	0.133	25
Haiku high	0.605	0.6045	-0.001	0.085	5

The answer key agrees. Every V9 task also ran before the fix, at earlier task versions, so each run group can be scored on the same five tasks on both sides: Luna medium went from 0.25 to 0.30, high from 0.28 to 0.31, xhigh from 0.42 to 0.42 and max from 0.43 to 0.42. Haiku high fell from 0.66 to 0.46, on only five trials before. The keys were corrected in the same pass, so before and after are not quite the same test.

Luna’s Part 3 scores still sit well above its Group 1 scores (0.30 against 0.17 at medium, 0.42 against 0.30 at xhigh). That gap comes from the tasks, not the fix: Part 3 kept the five tasks that best separate run groups, and every one of these run groups scores higher on them.

That is a result in itself. The reviewer now reads the rules, but neither grader’s score moved by more than five rounds can detect.

How steady is one run group?#

A task’s five attempts run side by side, so “round three” is just the third attempt. The observed run-to-run spread is how far one task’s five scores scatter. On the answer key that is 0.12 to 0.19; on the judge, 0.06 to 0.09. In these trials, judge-score spread was about half as large. No trial was graded twice by the same judge, so the spread includes both judge and review variation; the two cannot be separated here.

That spread sets what five rounds can resolve. Two run groups need to differ by roughly 0.16 on the answer key, or 0.08 on the judge, before five rounds each tell them apart. Most gaps between neighbouring efforts are smaller than that.

One attempt per task misleads. In Group 1, Haiku high scored 0.63 on the three tasks it shares with V9 Latest, from one attempt per task; with five attempts on the same three tasks it scores 0.35. Part 2’s ranking put it first partly on luck.

Retries against consistency#

pass@k asks how often at least one of k attempts earns full marks; pass^k asks how often all k do. Only the answer key can be read this way. The judge has never given full marks: its best score in 521 trials is 0.96, and 0.89 in Part 3, so its pass@k is zero at every k.

Luna gains from retries; Haiku medium is either always right or never

• Luna xhigh
• Haiku medium
• Haiku high

Data table

config	k	pass_at	pass_hat
Luna medium	1	0	0
Luna medium	2	0	0
Luna medium	3	0	0
Luna medium	4	0	0
Luna medium	5	0	0
Luna high	1	0.04	0.04
Luna high	2	0.08	0
Luna high	3	0.12	0
Luna high	4	0.16	0
Luna high	5	0.2	0
Luna xhigh	1	0.12	0.12
Luna xhigh	2	0.22	0.02
Luna xhigh	3	0.3	0
Luna xhigh	4	0.36	0
Luna xhigh	5	0.4	0
Luna max	1	0.12	0.12
Luna max	2	0.18	0.06
Luna max	3	0.2	0.02
Luna max	4	0.2	0
Luna max	5	0.2	0
Haiku low	1	0.16	0.16
Haiku low	2	0.2	0.12
Haiku low	3	0.2	0.08
Haiku low	4	0.2	0.04
Haiku low	5	0.2	0
Haiku medium	1	0.2	0.2
Haiku medium	2	0.2	0.2
Haiku medium	3	0.2	0.2
Haiku medium	4	0.2	0.2
Haiku medium	5	0.2	0.2
Haiku high	1	0.16	0.16
Haiku high	2	0.2	0.12
Haiku high	3	0.2	0.08
Haiku high	4	0.2	0.04
Haiku high	5	0.2	0

The two models fail differently. Luna xhigh reaches full marks on 12% of tasks in one attempt and 40% in five, but never on every attempt: different tasks come good on different tries. Haiku medium sits at 20% whatever k is, because it solves ge-x540 every time and nothing else ever. More attempts help a model whose misses are random, not one whose misses are fixed. pass@k for k above 1 rewards a bigger sampling budget, so it describes reliability here and does not rank anything.

A second judge#

The last doubt is the judge. Haiku judged every judge trial, so Haiku’s lead was the result most exposed to a judge favouring its own family. I regraded all 175 of its judge trials with a second judge, gpt-6-luna on the Codex CLI. The rubric and the reviews it read stayed the same. The judge changed as a whole, model and CLI: Haiku graded in Claude Code, Luna in Codex.

Swap the judge and the Haiku run groups drop furthest

• Haiku judge
• Luna judge

Data table

config	haiku_judge	luna_judge	delta	ci95
Haiku high	0.604	0.461	-0.144	0.054
Luna max	0.561	0.469	-0.092	0.025
Haiku medium	0.561	0.411	-0.15	0.035
Luna xhigh	0.556	0.488	-0.068	0.021
Haiku low	0.513	0.367	-0.146	0.084
Luna high	0.497	0.432	-0.064	0.07
Luna medium	0.489	0.461	-0.029	0.069

• Luna is the stricter judge. It scored 137 of 175 attempts below Haiku and 0.10 lower on average. Attempt by attempt, the two judges’ scores correlate at 0.74, but much of that is the tasks’ shared difficulty: within one task the correlation runs from 0.18 (ge-x632) to 0.76 (ge-d11).
• The ranking does not survive the swap. Haiku’s judge puts Haiku high first and Luna xhigh fourth; Luna’s judge puts Luna xhigh first and Haiku high fourth, level with Luna medium. Only 11 of the 21 pairs of run groups keep their order.
• The judges split along family lines. Swapping the Haiku judge for the Luna judge costs Haiku’s reviews 0.15 and Luna’s 0.06. The 0.08 difference between those two drops is far outside the noise (Welch’s t = 5.4 over the run groups’ rounds). Two judges cannot say which one is off: Haiku may favour Haiku’s reviews, Luna may mark them down, or both. Each judge is also a model inside its own CLI, so the split belongs to the whole judge setup, not to the model family alone.
• The answer key does not resolve the gap. It asks the narrower question of whether each finding matches a known bug. Haiku high’s mean is slightly higher than Luna xhigh’s (0.46 against 0.42), but five rounds cannot separate them, and Haiku’s answer-key trials were not regraded with Luna. Its lead on the broader judge score depended on which judge did the grading.

So one judge, from the same family as one of the reviewers it ranks, cannot be checked against itself. Two judges from different families, with their disagreement reported beside the score, at least show when a ranking depends on who grades.

At five rounds, Luna xhigh is not distinguishable from Haiku high on the answer key, the second judge ranks Luna xhigh first, and it costs $0.42 per trial. That makes it a practical run group to refine in Part 4.

Part 4: refining the harness#

Part 4 reads Luna xhigh’s 50 trials, alongside the three other Luna efforts, to identify what to change in the review team.

Reading the runs with SE Analyze#

SE Analyze reads every trial in a run group: each reviewer’s transcript, the findings the team kept, and the grader’s verdict on each known issue or criterion. It writes a page per attempt, a summary per pull request under review, and a run-wide report with suggestions. A suggestion is a fix to the environment or grader, a change to how the run is set up, or a habit the reviewers should pick up. A habit has to recur on at least two different pull requests before it becomes run advice.

Four models read the Luna xhigh round: Sol, Claude Opus, Gemini and Grok. Opus wrote the most useful report, so the rest of this part works from it.

What Opus found in the team#

Opus’s most useful finding was in the harness, not the reviewers. Before the validator sees anything, the team merges findings that point at the same lines and keeps the most severe one. It never compares what the findings say. On one task, the language reviewer found a known issue. The correctness reviewer reported a different problem on the same three lines. The merge kept the correctness finding, and the judge marked the known issue missed: “Same code, different problem.”

I counted every merge across the 50 trials. It dropped 59 findings in 38 of them. 41 were the same problem reported twice, so the merge did its job. 18 were a different problem on the same lines, and 6 of those were known issues on the answer key. Five trials lost credit for one, which by my estimate costs the team about three points of recall, 0.425 against 0.454. Gemini and Grok named the same flaw. Sol saw the lost findings too, but blamed the reviewers for them.

That is a fix to the harness, not to a prompt. No prompt edit would have recovered a finding the reviewer had already made. The team now keeps every finding that passes its checks, even when another finding overlaps its lines, and groups the ones on the same lines into one comment, so a repeat shows as a second section, not a second comment.

Applying what Opus recommends#

The rest of Opus’s report is five habits for the reviewers. Each recurred on at least two problems, and each would read the same on any code review:

• Trace the failure path. For every external call or routine a change adds, trace how it fails as carefully as how it succeeds.
• Review configuration like code. For every new configuration field or task entry, trace how it is validated and how it is used at runtime.
• Rate severity by harm. Rate a finding by the harm a user would see, and anchor it on the narrowest lines that show it, so a docstring or coverage note never outranks a defect.
• Name the cause in the title. Put the root cause and its trigger in the title and first sentence, so the finding cannot be mistaken for a different problem on the same lines.
• Walk every state. For anything stateful, long-running or split across processes, list the states each side can be in and trace the code through each one.

The habits belong in the seven specialist skills, the files se.toml lists as the benchmark’s prompts. The app offers two ways to make the edit.

With Task QA. Open Task QA on the benchmark and attach the Opus report as evidence, with the run group it read. Choose Refine skills, tick the seven specialist skills, and give the five habits as the objective. Task QA prepares a proposal: a diff for each file it changes, written by the trace-to-skill plugin from the report and the runs behind it. Open each diff, write your review, and Apply to Prompts saves the candidates as drafts.

By hand in Prompts. Open the benchmark’s Prompts tab, edit the seven skill files, and save. The edits stay on the platform until you open a pull request.

Either way the drafts end up in the Prompts tab. There they run against the committed prompts as one linked comparison, so the five habits together get a number before anything reaches the repository. The pull request then writes each skill into the repository and into the copy baked into every eval task, so the evals never fall behind the prompts.

Opus’s last two suggestions change how the team runs. One is half done already. The validator only checks P0 and P1 findings, and the CI review skips it when there are none, but the eval tasks still run it as a fixed step. The other is that the team could pick its specialists by the kind of change instead of always running all seven. Opus framed that one as an A/B test, since every step re-reads a large prompt: 28.2M input tokens a trial on average, 97% of them cached.

What comes next#

The merge fix ships without a rerun, so its recall gain stays an estimate here. The next Luna xhigh round measures it. The five habits then run as a Prompts comparison, edited prompts against committed ones in the same launch, so the merge fix and the habits each get their own number.

Fine-tuning is the obvious next step, and not this article’s. The per-attempt failure types and the habits these reports recorded are the training signal it would start from, so a later article picks up from this analysis.

Turning it into a CI pipeline#

The review team now runs on every pull request as a GitHub Action. It posts its findings the way the commercial bots do: one review with inline comments on the lines they concern, plus a summary comment updated on every push. A review takes 2 to 6 minutes on a standard GitHub runner (median 3.5 over the last ten runs), with gpt-6-luna at high effort.

Installing it#

One script copies the tool into your repository:

That writes .code-review/ and .github/workflows/code-review.yml. By default Codex signs in with your ChatGPT subscription, so a review has no per-token cost:

To bill an OpenAI API key instead, store OPENAI_API_KEY and set the repository variable CODE_REVIEW_AUTH to api-key. The model, effort and Codex version are inputs in the workflow file. Files marked linguist-generated or linguist-vendored in .gitattributes are skipped, so regenerated fixtures and lockfiles do not burn a review. Pull requests from forks never run it: GitHub withholds secrets from forks, and the job skips them before it starts, so a stranger cannot spend your account. Subscription Codex also hands its agents every app connected to the ChatGPT account, so switch those off before you put a subscription login on a CI runner.

Where it runs now#

The action reviews every ready pull request on Standard Environments, run from the package packages/plugins/code-review, so a prompt edited in the SE app is reviewed by the changed prompt. It reads its own earlier threads before posting, so a finding someone answered stays settled unless its lines change. The public repository’s pull requests show it at work, with every finding and reply left in place: #3, #5 and #7. The 18 bugs that reviews found in the action’s first commit became the answer key of evals/cases/cr-self-01, which asks whether the tool can find the bugs in itself.

The loop#

Put together, the pieces form a loop that grades the review tool on its own bugs:

1. A change to the review logic is detected. The pipeline digest moves on any edit to a skill, prompt or merge code.
2. The evals regenerate. The generator rebuilds every task from the current code, gated by oracle and no-op.
3. Trials run across models and harnesses. The tasks take any agent, model and judge at run time.
4. The tool reviews the pull request. The CI action runs on every ready pull request.
5. Bugs found become eval cases. Confirmed bugs join the answer keys, as with cr-self-01.
6. Analysis says what to change next. Each trial records the task digest, agent, model and judge, so results compare cleanly.

Steps 1, 4 and 5 are proven. Today a person runs steps 2, 3 and 6. Making those automatic is the work in progress: the app writing the regenerated tasks into the same pull request as a prompt edit, and the platform rerunning the benchmark when the digest moves.

Reproduce it#

1. Import the code_review repository into Standard Environments. Every task folder under evals/ becomes part of one benchmark.
2. Pick the agents, harnesses and models to compare, and ideally a judge from a different model family. Part 2 used Haiku as the main judge, including for Haiku reviews; Part 3 also regraded 175 attempts with Luna from another model family and reports that comparison.
3. Launch the team tasks (evals/team-tasks/) on your subscription or API keys. Start with the six above, or their -llm-judge versions.
4. Compare recall, the llm-judge scales, tokens and cost per task, and keep the cheapest setup that holds up.

To run one task with Harbor directly:

Every trial’s numbers are in the repository: research/results/team-evals-2026-10-12.csv and research/results/llm-judge-2026-10-13.csv.

How the numbers are computed#

Every score in this article comes from a few formulas. They are here so you can check them, not so you have to read them first.

Part 1: recall, key precision and F1#

A matcher pairs each finding with at most one key entry. For these calculations, a matched finding is a true positive (TP), an unmatched finding is counted as a false positive (FP), and a key entry nothing matched is a false negative (FN). Because the key is incomplete, an unmatched finding is not necessarily a false bug report; it is simply unconfirmed by this measure. A second finding on an entry already matched counts as a false positive, so repeating a bug earns nothing.

A ratio with nothing to divide by scores 0. A case with no key entries and no findings scores 1 on all three.

Part 1: the sign test and the paired bootstrap#

Study 1’s sign test ignores ties and asks whether the wins and losses could be a coin flip. The harness-tools-only arm beat vector on 5 pull requests and lost on 2, so p is the chance of a split at least that lopsided in 7 fair flips: 2 × (1 + 7 + 21) / 2⁷ = 58/128 = 0.45.

Study 2’s intervals are a paired cluster bootstrap. For each case, take the arm’s F1 minus the harness-tools-only arm’s F1. Then draw six cases with replacement, average their differences, and repeat 10,000 times (seed 0). The 95% interval runs from the 2.5th to the 97.5th percentile of those 10,000 averages. Pairing matters: both arms reviewed the same cases, so the case-to-case difficulty cancels out of each difference. An interval that spans zero means the data cannot distinguish the arm from the harness-tools-only arm. The code is paired_bootstrap in src/review_core/scoring/. Running it on research/results/scores.csv reproduces both intervals to within 0.001, the rounding of the stored F1 scores.

Part 2: the two rewards#

• Answer-key recall (Group 1). The judge checks each key entry as one yes-or-no criterion: does any finding state the same issue? Recall is the yes count over the number of entries.
• Judge reward (Group 2). Each scale is scored 1 to 5 and rescaled to 0 to 1 as (score − 1) / 4. The reward is the weighted sum: 0.30 × known issues + 0.25 × validity + 0.15 × severity and fixes + 0.30 × economic value. vo-x7 above scored 5, 1, 4 and 5, which gives 0.30 × 1 + 0.25 × 0 + 0.15 × 0.75 + 0.30 × 1 = 0.71.

Part 2: the interval on each run group#

A run group’s mean reward is the mean over the cases it ran, ten for most, where each case’s trials are averaged first. That stops a run group with 50 trials on easy cases from looking better than one with 10. The 95% interval resamples those case means: draw as many as there are cases, with replacement, average them, repeat 4,000 times (seed 7), and take the 2.5th and 97.5th percentiles. Because the cases differ so much in difficulty, these intervals are wide.

Part 2: Luna against Sol#

The Luna-against-Sol comparison pairs by case. For each case c, take d_c = Luna's mean − Sol's mean. The estimate is the mean of the ten differences, d̄, and its 95% interval is d̄ ± t × s / √10. Here s is the standard deviation of the differences, and t = 2.262 is the Student’s t value for 9 degrees of freedom. Over the two run groups, d̄ = −0.01 and the interval is −0.16 to +0.13. The interval spans zero, so the two are indistinguishable on recall.

Part 2: cost, the frontier and the rank correlation#

• Cost per 0.1 of reward is cost per trial / (10 × mean reward): what a tenth of the key, or a tenth of the judge’s scale, costs on average. A run group with a mean of 0 has no finite value.
• The cost-efficient frontier holds every paid run group that no other run group beats on both counts, cheaper or equal per trial and higher in mean reward. Self-hosted Nemotron is left out because its $0 is my hardware, not a price.
• Summed wall time adds each trial’s wall time, the time a run takes when its trials run one after another.
• The rank correlation between recall and judge reward is Spearman’s ρ. Each list of 15 run groups is replaced by its ranks (ties share the average rank), and ρ is the ordinary correlation of the two rank lists: 1 means the same order, 0 means no relation. Here ρ = 0.80.

Part 3: choosing tasks by separation#

A run group here is a harness, a model and a reasoning effort. For each task, take only the run groups with three or more scored trials on it. Between is the standard deviation of those run groups’ mean scores. Noise is the square root of the average variance of scores within one run group. Separation is between divided by noise: above 1, run groups differ by more than one run group varies with itself. Regrades are left out, so each agent output counts once.

Part 3: rounds, intervals and the fix#

• A run group’s score is the mean over its five rounds, where a round’s score is the mean of one attempt at each task. Void attempts (the agent never ran) are left out. The 95% interval is mean ± t × s / √5, with s the standard deviation of the five round scores and t = 2.776 for 4 degrees of freedom. Attempts are independent, so the interval holds however attempts are assigned to rounds.
• Run-to-run spread is the standard deviation of one task’s five scores, averaged over the run group’s tasks.
• Comparing two run groups uses Welch’s t on their five round scores each. The smallest gap five rounds can detect, at 95% confidence and 80% power, is about (2.31 + 0.84) × √(2/5) × s, with s the average round-score standard deviation (0.08 on the answer key, 0.04 on the judge). That comes to about 0.16 and 0.08, the gaps quoted under “How steady is one run group?”.
• Before and after the fix pairs by task: for each shared task, the after mean minus the before mean, then d̄ ± t × s_d / √n over the n tasks.
• The second judge grades the same attempts, so its comparison pairs by round: for each run group, the Luna judge’s round score minus the Haiku judge’s, then d̄ ± t × s_d / √5. The family split compares those round differences for the Haiku run groups (15 rounds) against the Luna run groups (20) with Welch’s t. Rank agreement counts the 21 pairs of run groups that keep their order under both judges.
• pass@k for a task with n attempts, c of them at full marks, is 1 − C(n−c, k) / C(n, k), and pass^k is C(c, k) / C(n, k); both are averaged over tasks. Full marks means a reward of 1, as the leaderboard counts it.

About Standard Environments#

SE is at standardenvs.com. Beyond the benchmark library, you can team up with others on campaigns, run evals cheaply on your own subscription or API keys, and fork tasks into custom benchmarks. A company can write its own evals (or have me or another expert write them), connect its workflow from GitHub, and run that benchmark against any harness or model to compare results, token economics and more.

The review action, public studies, public eval tasks and their result files are on GitHub at davidbzyk/code_review. SE task snapshots and raw run records remain private; this article reports their aggregate results. Import the public repository into SE, or run its tasks with Harbor, and check those numbers against your own models.