ASI-Bench Finds AI Agents Struggle Without a Research Procedure
ASI-Bench tests scientific agents as human guidance disappears. Its largest score drop reveals a procedure-building problem, not proof about superintelligence.
AI agents lost almost half their average score when ASI-Bench removed a detailed human-written research procedure. The new preprint tests 18 model-and-agent configurations on 60 computational research projects, repeating each project with progressively less methodological guidance.
The headline result is real within the authors’ experiment, but the most useful interpretation is easy to miss. Average performance fell from 50.91 with a complete procedure to 29.10 when agents received only the method name. Removing even that name lowered the average by another 2.48 points, to 26.62. In this test, turning a named scientific method into a working sequence was a larger obstacle than choosing the method.
That is evidence about a controlled benchmark of scientific execution. It is not evidence that the systems discovered new science, a forecast of artificial superintelligence, or an independently reproduced measure of research productivity.
Four versions of one project move responsibility to the agent
The ASI-Bench paper starts from computational projects rather than isolated questions. Its 60 tasks span 11 domains, including physics, chemistry, biology, medicine and biostatistics, robotics, and electrical engineering. Each task has input data, an executable environment, required artifacts, and scoring criteria.
The key design choice is to hold those elements fixed while changing what the prompt reveals. The benchmark calls its four conditions B1 through B4.
| Condition | Human guidance supplied | Responsibility left to the agent |
|---|---|---|
| B1 | Scientific background, method, equations, and full procedure | Implement and execute the prescribed approach |
| B2 | Method and relevant constraints, without the procedure | Convert the named method into an executable workflow |
| B3 | Objective, data, constraints, and required outputs | Select a method, build the workflow, and validate the output |
| B4 | B3 plus factually correct but irrelevant information | Do the B3 work while ignoring distractions |
A representative task asks the agent to infer a nonlinear dynamical system from observed space-and-time data. B1 supplies the governing equation, numerical formulation, and solver procedure. B2 names the class of equation and suitable numerical approaches. B3 leaves the system and method for the agent to infer. The scientific objective and expected artifacts do not change.
This matched design is more informative than comparing unrelated “easy” and “hard” tasks. A score difference can be connected to information that was deliberately removed, although it still reflects every model, harness, prompt, runtime, and grader choice in the experiment.
The authors report an extensive construction process: more than 1,300 candidate ideas, five review rounds, more than 1,500 sandbox runs, and checks for scientific consistency, reproducibility, leakage, unintended shortcuts, and scoring behavior. Those are benchmark-development controls reported by the same team, not an outside audit.
The largest gap appears before method selection
The paper’s main table macro-averages scores over the 60 tasks and 18 agent-and-model configurations. Most configurations were run three times; one Claude Opus 5 result was a single run. The evaluation excluded external tool access.
| Guidance change | Author-reported mean | Difference | What the comparison isolates most directly |
|---|---|---|---|
| Full procedure (B1) | 50.91 | — | Execution with a human-specified recipe |
| Method name only (B2) | 29.10 | −21.82 | Building the missing procedure |
| Agent chooses method (B3) | 26.62 | −2.48 from B2 | Method selection after the procedure is already absent |
| B3 plus irrelevant facts (B4) | 26.99 | +0.36 from B3 | Robustness to this benchmark’s distractors |
The 50.91-to-26.62 comparison is a 24.29-point drop, or about 48% of the B1 mean. Describing that entire difference as a “method-selection gap” would be wrong. Roughly nine-tenths of the point loss occurred when the full procedure disappeared but the intended method remained available.
The cost results reinforce that reading. The authors report that B1 used an average 4.35 million tokens and 37.8 minutes per task. B2 was the most expensive condition at 6.91 million tokens and 49.7 minutes. A method name can constrain the direction without supplying enough detail to implement it, leaving the agent to reconstruct a procedure through search and retries.
B4 also deserves restraint. Its overall mean was effectively unchanged from B3, but that does not prove scientific agents are generally immune to distraction. It shows that the factually correct, task-irrelevant additions used here did not lower the aggregate score. Other irrelevant context, misleading evidence, tool output, or literature noise would require separate tests.
The result belongs to the model and harness together
ASI-Bench evaluates configurations, not bare language models. A harness controls the files an agent sees, the commands it can run, the way it records progress, and how it handles timeouts and errors. The paper reports different scores for the same backbone model in different harnesses, although those comparisons do not isolate a single harness feature.
The strongest reported B3 configuration, Codex with GPT-5.6 Sol at the ultra reasoning setting, scored 51.60. It was the only evaluated configuration above 50 in B3. That result is author-reported and does not establish that the model independently produces valid new scientific knowledge. It establishes how one configured system performed against the benchmark’s artifact scorers.
The distinction matters for procurement and research claims. A team cannot copy the model name from the leaderboard and assume it will obtain the same result with a different prompt, tool policy, context strategy, execution environment, or budget. Our NVIDIA AVO analysis reaches the same conclusion from an interactive benchmark: the system boundary travels with the score. The HarnessRisk analysis adds another boundary that ASI-Bench does not measure—whether a capable research agent can be contained and recovered when its execution goes wrong.
A computational benchmark is not a working laboratory
ASI-Bench expands evaluation beyond academic question answering. Its agents must work across multiple steps, create artifacts, diagnose failures, and satisfy project-specific checks. The official repository publishes a runner, two fixed-seed datasets, task infrastructure, and a route for submitting results. One seed exposes references for local scoring; the official route uses private references for the other seed.
Even so, the benchmark remains a designed computational environment. Its main result does not include external tool access, the table does not report a human baseline, and the paper is an August 18 version-one preprint. The authors built the tasks, validation process, and scoring system, then ran the evaluated configurations. No independent team has yet reproduced the reported score transitions.
Scientific autonomy outside that environment adds problems the score does not cover: choosing a worthwhile research question, locating incomplete or conflicting evidence, negotiating access to instruments or data, performing physical experiments, recognizing tacit laboratory constraints, and deciding whether an apparent result deserves belief.
Other evaluations make the scope problem visible. AstaBench covers more than 2,400 problems across stages of scientific work and argues that tool access, cost, and agent configuration are confounders that an evaluation must control. A separate independent case study tested eight open-source research frameworks on two real computational projects and reported substantial setup and debugging work, with the systems more useful as supervised assistants than autonomous researchers. Neither study validates ASI-Bench’s figures. Together, they show why “scientific agent” is not one stable task category.
An independent rerun needs more than the final score
The public code makes a serious rerun possible, but a leaderboard number alone cannot reveal whether a gain came from the model, harness, budget, task exposure, or scorer. A useful evidence bundle would preserve:
- the exact dataset seed, task and reference revisions, development access, and contamination review;
- the model endpoint, model revision, reasoning setting, sampling configuration, and provider date;
- the harness revision, prompts, tools, sandbox boundary, context policy, retry rules, timeout, and token budget;
- results for B1 through B4 on the same task set, with repeated runs, invalid runs, dispersion, and task-level failure categories;
- raw output artifacts, scorer versions, scoring traces, and adjudication for disputed or partial results; and
- a separate human or expert-assisted baseline when the claim compares the system with scientific practice rather than another agent.
That record would let an outside team test the paper’s most consequential finding: whether the sharp B1-to-B2 decline survives a pinned rerun. Follow-up experiments could then change one boundary at a time—for example, supplying a structured planning tool while holding the model, tasks, prompts, and budgets fixed—to learn which part of procedure construction is failing.
The benchmark’s name reaches toward artificial superintelligence, but its strongest contribution is more practical. ASI-Bench turns human methodological guidance into an experimental variable and exposes a large gap between receiving a complete procedure and having to build one. Independent reruns, task-level failure evidence, and comparisons beyond self-contained computational projects will determine how far that result travels.