Validation
How accurate is it? Here’s what we know so far
We have run one small, preliminary test. It is encouraging but it is not proof of accuracy, and we are not presenting it as one. This page shows exactly what has and hasn’t been measured.
Preliminary — not a claim of accuracy
- Only a 10-paper smoke test has been completed.
- Its 95% confidence interval is roughly −0.07 to 0.90, which includes zero.
- All 10 papers were rejected submissions, so the sample covers a narrow quality range.
- It ran on the previous model ensemble. The product now uses newer models, so the result needs to be re-measured.
Smoke test · 1 May 2026
Deep Audit scores vs ICLR 2025 human reviewers
We ran 10 ICLR 2025 submissions through the production Deep Audit pipeline and compared its scores with the mean human reviewer ratings published on OpenReview, using Spearman rank correlation (ρ). A ρ of 1 would mean identical rankings; 0 would mean no relationship.
Why our 0.61 is not “better than Stanford”: their figure comes from 147 papers; ours comes from 10, all of them rejected submissions. With a confidence interval that runs from below zero to 0.90, our result is compatible with anything from no correlation to a strong one. The Stanford numbers are useful context for what human reviewers themselves achieve — two humans agreed at ρ = 0.41 — not a scoreboard we have beaten.
Roadmap
Validation runs
A 30-paper pilot, which adds accepted papers to the sample, was started in late May 2026 but failed on a server configuration error before any paper was scored. It has not yet been re-run. The next run will also re-measure the smoke test on the current model ensemble.
| Run | Papers | Status | Spearman ρ |
|---|---|---|---|
| Smoke test | 10 | Complete (1 May 2026) | 0.61 |
| Pilot | 30 | Not yet completed | — |
| Half run | 75 | Planned, no fixed date | — |
| Full run | 150 | Planned, no fixed date | — |
Not yet measured
Consensus precision
What it will mean
Our review tools flag issues when multiple models agree. Consensus precision is the share of those flags — raised when two or more models agree — that a human expert confirms as real problems.
Why it matters
Correlation with reviewer scores tells you whether the overall ranking is sensible. Precision tells you whether an individual flag is worth your time. That is the number researchers actually need.
Where it stands
It has not been measured yet. We will publish the method and the result here when it is, including if the number is unflattering.
Full methodology
The technical overview has the per-dimension results, the scoring rubric and the full caveats.
- Jiang & Ng, Stanford Agentic Reviewer technical overview (147 held-out ICLR 2025 test papers): https://paperreview.ai/tech-overview