The Reproducibility Bill: 918 AI Papers Map the Cost of Missing Artifacts
A 918-paper study maps the cost of reproducing AI research. Use its findings to budget code, data, compute, and reviewer time.
A review of 918 empirical AI papers found that 6.72% of the implementation URLs supplied by authors were empty or returned an error. That is the earliest possible failure in a reproduction attempt: before anyone rents a GPU, obtains data, or checks a benchmark, the advertised route to the code ends.
The study, published in Patterns on August 27, turns that friction into a 1–10 score across five kinds of missing documentation. Its “cost” is an estimate of extra labor, not a cloud invoice, and the researchers did not execute all 918 papers. The result is best read as a map of documentation debt.
That map becomes more practical when placed beside the ICML 2026 agent reproduction challenge, where public logbooks were judged claim by claim. One participant reports pushing 147 papers through a heavily gated pipeline in 16 days. Together, the two projects show where a reproduction budget starts, which cheap checks should precede compute, and when missing material should halt the attempt.
Five kinds of documentation create the first bill
Thijs Snelleman, Holger Hoos, and Odd Erik Gundersen began with 1,061 papers published from 2022 through 2024 at AAAI, IJCAI, ICLR, ICML, NeurIPS, JAIR, and JMLR. They excluded 143 theoretical papers from the effective analysis, leaving 918 empirical studies.
Each paper received a score in five dimensions:
- Implementation: the effort needed to rebuild the method when code, pseudocode, designs, or practical descriptions are incomplete.
- Data: the effort needed to retrieve the same data, defend a comparable substitute, or document a new acquisition.
- Configuration: the work required to recover hyperparameters, semantic settings, seeds, and the budget under which those settings were selected.
- Experimental procedure: the missing steps between a runnable method and the paper’s reported metric, baseline, analysis, and conclusion.
- Expertise: the specialist knowledge an independent investigator must acquire because the documentation does not supply it.
The study deliberately leaves hardware requirements outside this rubric. A model can therefore score well on documentation and still be too expensive for a small lab to train. Conversely, an inexpensive CPU experiment can carry a high score because reproducing it requires guesswork. “Low cost” means a clearer path through the authors’ materials, not cheap execution.
Across the sample, 70.52% of papers made an implementation public and 94.01% used public data. The broken-link result has a narrower denominator than the 918-paper headline: 6.72% of provided implementation URLs, rather than 6.72% of papers, were empty or errored.
The venue comparison was also relative. The authors report that papers from the three machine-learning conferences were, on average, 16.52% less costly to reproduce than papers from the general AI conferences and 12.91% less costly than journal papers in this sample. Those percentages compare ordinal documentation scores across groups. They cannot price a particular reproduction or establish that its result will survive a rerun.
One reviewer carries most of the measurement
The primary review for every paper came from the study’s first author. Fourteen independent volunteers supplied a second review for 46 papers—5.01% of the 918-paper set.
Agreement was rated excellent for implementation, data, and configuration, good for experimental procedure, and moderate for expertise. That pattern is informative: a reviewer can usually verify whether a repository or parameter table exists, while estimating how much domain knowledge another researcher must acquire depends more heavily on judgment.
The study’s authors published their data and analysis code in a versioned Zenodo record, so readers can inspect the rubric and calculations. The archive improves auditability, but the limited second-review sample still constrains how confidently small venue differences should be interpreted.
Award-winning papers did not receive lower average costs than other papers from the same venue. Prestige, in other words, was not a reliable shortcut around artifact inspection.
The ICML campaign paid in gates before GPUs
The ICML challenge supplies a different unit of evidence. Its organizers collected public Space logbooks for 2026 conference papers and used an LLM judge to classify individual claims as verified, falsified, supported only by a reduced-scale “toy” experiment, or inconclusive.
AI News downloaded the final public verdicts dataset on August 31 and counted 6,885 judged Space logbook records covering 2,176 unique OpenReview paper IDs and 35,908 claim verdicts. Multiple logbooks can target the same paper, and an LLM verdict is not equivalent to independent human review. The counts establish operational scale and repeated attempts, not 6,885 confirmed scientific results.
Nate Mauer’s account of one campaign within that challenge reports 147 judged papers, 1,390 points, and 76 perfect scores over 16 days. The work ran on one workstation with a 20 GB GPU, apart from one rented-GPU fine-tuning job. Its first hand-run wave cost about USD 52; Mauer reports about USD 2.69 in metered paid inference for the later automated campaign.
The eye-catching inference bill omits the workstation, electricity, engineering time, and the rented training job. It is also internal bookkeeping from a participant’s retrospective, not an independently audited price. The transferable finding is the order of operations. The campaign ran cheap gates before paid model calls or long experiments:
- Probe the official repository and reject empty releases.
- Run each workspace twice and compare output to expose nondeterminism.
- Simulate the judge’s reading window so the required evidence remains visible.
- Require every number in claim-bearing prose to appear in a raw result file.
- Spend on execution and judging only after those checks pass.
That sequence does not validate the Patterns venue comparison. It shows how one operator converted many of the study’s documentation dimensions into intake controls.
| Source | Unit and scope | “Cost” or outcome measured | Legitimate use |
|---|---|---|---|
| Patterns study | 918 empirical papers from seven venues, 2022–2024 | Ordinal documentation burden across five dimensions | Estimate where missing material adds labor; compare groups cautiously |
| ICML verdict dataset | 6,885 judged Space logbook records for 2,176 paper IDs in the downloaded snapshot | LLM verdicts on 35,908 claim records | Inspect challenge throughput, repeated attempts, and claim-level outcomes |
| Mauer campaign report | One participant’s 147 judged papers over 16 days | Pipeline throughput, internal spend, failures, and gate design | Study one concrete operating strategy; do not treat its bill as a market rate |
Build the budget before opening the expensive queue
A real reproduction budget needs both reducible documentation labor and irreducible execution resources. The worksheet below is intentionally blank where local rates belong. Cloud prices, salaries, data fees, and acceptable uncertainty vary too widely for the study’s scores—or one challenge participant’s spending—to supply honest defaults.
| Budget line | Evidence to capture before execution | Estimate | Continue when | Pause or stop when |
|---|---|---|---|---|
| Claim definition | Exact claim, table or figure, metric, baseline, tolerance, and required scale | Evaluator hours | The acceptance test is explicit | The target claim or decision rule cannot be reconstructed |
| Artifact access | Canonical repository, immutable revision, checksums, weights, and release status | Triage hours + access fees | The reviewed artifact resolves and can be pinned | The official path is dead and reimplementation falls outside scope |
| Data and permission | Dataset version, split, license, access terms, transformations, and sensitive-data controls | Acquisition hours + fees | The team can lawfully obtain or defend a comparable dataset | Permission is unresolved or substitutions would change the claim |
| Environment | OS, runtime, library, driver, container, seed, and installation steps | Engineering hours | A clean environment reaches a deterministic smoke test | Setup requires undocumented patches or unbounded dependency archaeology |
| Configuration | Hyperparameters, search procedure, seeds, stopping rule, and original tuning budget | Search hours + candidate runs | The published setting or a predeclared search can be reproduced | Success depends on guessing configurations without a capped protocol |
| Execution | Hardware class, wall time, GPU or CPU hours, storage, egress, and energy where relevant | Resource hours × current rate | The expected run fits the approved cap | The minimum faithful scale exceeds money, time, or hardware limits |
| Evaluation and review | Raw outputs, baseline reruns, statistical checks, failure labels, and independent reviewer | Reviewer hours + reruns | Results can be traced from command to conclusion | The evaluator cannot separate an artifact failure from a claim result |
| Contingency | Maximum repair cycles, deadline, spending ceiling, and evidence required for another attempt | Explicit reserve | Each retry adds new diagnostic information | The cap is reached or retries repeat the same unexplained failure |
The resulting estimate is simple enough to audit:
Total budget = access and data fees + engineering hours × loaded rate + compute hours × hardware rate + storage and egress + reviewer hours × loaded rate + contingency.
The important field is the stop condition. Without it, a broken environment can consume the compute allocation, a missing license can strand completed work, and repeated repairs can quietly turn “one reproduction” into open-ended research.
A dead link ends access, not the scientific claim
Suppose a paper’s official repository returns 404. The correct intake result is artifact unavailable at the cited location. It is not “claim falsified.” A team may still reimplement the method from the paper, locate an author-maintained archive, or test a narrower hypothesis, but each choice changes the evidence path and the budget.
A failed rerun also needs classification. The cause may be code drift, a changed dataset, an unavailable checkpoint, nondeterministic training, a mismatched evaluator, insufficient scale, or a genuine disagreement with the reported result. Only the last possibility approaches scientific falsification, and even then the experimental conditions and statistical test have to support it.
This is where adjacent evaluation work becomes useful. A benchmark score depends on the pipeline that produced it, while model selection begins with permission, a runnable artifact, and an immutable revision. Reproduction inherits both obligations, then adds the burden of matching another team’s claim closely enough to draw a comparable conclusion.
The study’s 6.72% URL failure rate looks small beside a training budget, yet any one dead official link can make that entire budget premature. The next revealing audit is longitudinal: revisit the cited repositories after a year, record which revisions and datasets still resolve, and test whether venues with checklists preserve a stable path from claim to command to result.