NVIDIA AVO's 100% ARC-AGI-3 Result Measures a Harness, Not a Model
NVIDIA AVO cleared ARC-AGI-3's public set, but the benchmark does not isolate the harness contribution or test generalization on private tasks.
NVIDIA reports that its Agentic Variation Operators system, or AVO, paired with Claude Opus 5 to complete all 183 levels in ARC-AGI-3’s 25 public environments. The run used 6,624 environment actions and received a 100.00 Relative Human Action Efficiency score. It is an impressive result for the configured agent system.
It is not a 100% score for Claude Opus 5 by itself, a controlled estimate of the benefit from AVO, or evidence that the system generalizes to ARC-AGI-3’s private tasks. NVIDIA states each of those boundaries in its report. The useful lesson is narrower: model choice is only one variable in a long-horizon agent evaluation, and a benchmark record has to identify the harness, interface, task set, budget, and metric with it.
The score belongs to an end-to-end system
An agent harness is the operating layer around a model. It decides what observations enter the context, which tools and actions are available, how memory survives between steps, how failures are detected, and when a supervisor redirects work. Change that layer and the same model can behave like a different system.
For its public-set run, NVIDIA says AVO used persistent memory, a supervisor, tools, and its own execution loop. Claude Opus 5 received each observation as an exact 64-by-64 text grid rather than the rendered 512-by-512 image used by VISTA’s primary configuration. The agent received the available actions but no description of each environment’s rules or goal.
Those details are part of the result, not implementation trivia. A text grid changes the perception problem. Persistent memory changes what the model can reuse after its immediate context fills. A supervisor changes how the run responds to stagnation. Tool descriptions, retry policy, and action budget change the paths the system can explore.
- ModelClaude Opus 5 plus its reasoning setting
- HarnessMemory, supervisor, tools, context, and recovery loop
- Interface and tasksText-grid observations, available actions, and the public set
- Run recordRHAE score, completed levels, actions, cost, and failures
- Observed
- The configured end-to-end system completed the public-set run.
- Not isolated
- The causal contribution of the model, memory, supervisor, or interface.
- Not measured
- Generalization to ARC-AGI-3's semi-private or fully private tasks.
This is why subtracting one headline score from another does not reveal “the harness contribution.” Independent coverage correctly focused attention on the harness, but its shorthand that the model scored about 30% “without the harness” compresses materially different conditions into a single switch. NVIDIA says the lower ARC Prize figure used a different reasoning setting and a substantially different agent and evaluation setup.
RHAE rewards completion and efficient action
ARC-AGI-3 presents interactive, game-like environments without instructions, stated rules, or stated goals. A system must explore, infer what its actions do, discover the objective, and then complete progressively harder levels. The test is meant to capture adaptation through interaction rather than answer accuracy on a static question set.
The ARC Prize technical report defines Relative Human Action Efficiency, or RHAE, from the number of actions a system takes on each completed level compared with a first-time human baseline. The ratio is squared, so an agent that takes ten times as many actions as the baseline receives 1% credit for that level. Later levels receive more weight, incomplete levels cap the environment score, and the final benchmark score averages across environments.
A 100.00 RHAE score therefore records complete, action-efficient performance under that formula. It does not mean every action matched a person, every internal hypothesis was correct, the run was cheap, or the system achieved “100% general intelligence.” NVIDIA separately reports the raw completion and action counts so the metric can be interpreted rather than repeated alone.
The closest same-model system comparison in NVIDIA’s post is VISTA. Both configured systems completed the same 183 public levels with Claude Opus 5. NVIDIA reports 6,624 environment actions for AVO and 7,542 for VISTA, about 12% fewer for AVO.
| Run or reference | Task set | Observation and system boundary | Reported result | Valid conclusion |
|---|---|---|---|---|
| AVO with Claude Opus 5 | 25 public environments, 183 levels | Text-grid observations; AVO memory, supervisor, tools, and execution loop | 100.00 RHAE; 6,624 actions | This configured system completed the public set efficiently |
| VISTA with Claude Opus 5 | Same 183 public levels | Different backend, observation representation, memory, context management, and implementation | 7,542 actions | AVO used fewer actions in a cross-system comparison |
| ARC Prize model evaluation cited by NVIDIA | Public set | Same model family, different reasoning and evaluation setup | Approximately 30% | The model-level reference is not an AVO ablation |
NVIDIA explicitly says the AVO-versus-VISTA comparison is not a controlled ablation. Memory may have reduced repeated exploration, but the experiment did not measure memory alone. The 12% action difference could reflect several interacting changes, so it should motivate narrower tests rather than a causal claim.
Public-set completion is not private-set generalization
The word “public” carries more weight here than the word “perfect.” ARC Prize designed the 25 public environments as an accessible demonstration of the format. Its report says they are intentionally easier, emphasize clarity and engagement, and do not comprehensively represent the mechanics in the private set.
The actual holdout is much larger: 55 semi-private environments for testing models behind external APIs and 55 fully private environments reserved for the competition. Those tasks are intended to be harder and out of distribution relative to the public demonstrations. ARC Prize says it will not put public-set results on its official leaderboard because teams can inspect or target those environments and because public performance is not a valid measure of progress toward artificial general intelligence.
NVIDIA’s report does not claim a result on either holdout. An independent scope analysis reaches the same practical conclusion: keep the task set, metric, reasoning setting, and run conditions attached to the number. The public result can demonstrate a capable harness while leaving generalization untested.
This does not make the run meaningless. Completing 183 interactive levels with fewer actions than another successful configured system is useful evidence about system engineering. It simply answers a different question from the official private benchmark.
The public result does not isolate AVO’s contribution
AVO and VISTA both completed the same 183 public levels with Claude Opus 5, while AVO used about 12% fewer environment actions. That comparison is suggestive, but the report does not hold memory, supervision, recovery, prompts, and every other harness choice fixed one at a time. The marketed component therefore cannot inherit the whole system’s gain.
The public set also served as a development surface that teams could inspect. It can demonstrate a capable end-to-end system, but only the semi-private and fully private environments can test whether that system transfers beyond tasks available during tuning. This is the same scope problem identified by the HarnessRisk analysis: a model score does not travel independently of its harness.
The next evidence should isolate the mechanism
The strongest follow-up would be a controlled AVO ablation that holds the model, reasoning setting, text-grid interface, public environments, and action budget fixed while separately testing memory, supervision, and recovery. A semi-private or fully private evaluation would answer a different question: whether the system transfers beyond tasks its designers could inspect.
Cost and latency also belong beside the score. Long-horizon systems can trade more model calls, tool executions, supervisor passes, or stored context for better completion. The public report gives environment actions, but not enough information to compare full serving cost across the cited configurations.
Until those results arrive, NVIDIA’s run should be described precisely and credited for what it demonstrates. AVO with Claude Opus 5 completed the ARC-AGI-3 public set with a 100.00 RHAE score and 6,624 environment actions. That is evidence that the surrounding system matters. It is also a reminder that the next serious benchmark question is not “which score is larger?” but “which part of the system changed, and did the result survive a holdout?”