September 13th, 2026: Evaluators, private benchmarks, and managed agents
Anthropic opens its lab to embedded evaluators, Real-SWE tests agents on private code, and OpenAI turns its Codex harness into an API.
Today’s releases make the model look less like the whole product than the component that must be governed, measured, and operated. Anthropic is offering outside evaluators a seat inside the lab, a private-code benchmark is testing agents on work public leaderboards cannot see, and OpenAI is selling its agent harness as infrastructure. The connective question is no longer only what a model can do, but who controls the evidence, execution, and recovery around it.
Lead developments
1. Anthropic offers outside evaluators a permanent seat inside the lab
Dario Amodei’s three-stage proposal begins with an immediate Anthropic commitment: selected independent evaluators would receive employee-like access, including desks, badges, laptops, and direct contact with researchers, while retaining the ability to publish. He also calls for shared industry safety standards, an antitrust waiver to coordinate them, and international agreements to pace frontier development. OpenAI’s Sam Altman said his company would follow the evaluator commitment, according to AP, but neither lab has published selection rules, access terms, redaction rights, or a start date. The proposal therefore matters less as proof that oversight works than as a testable promise to move verification inside the development cycle. Amodei’s capability and catastrophe timelines remain forecasts, not established facts.
Watch: The first named evaluators, their publication rights, disclosed conflicts, and whether reports appear before deployment decisions rather than after them.
Sources: Amodei’s full proposal and Anthropic commitment, AP’s independent report and Altman response, the same-day practitioner discussion
2. Real-SWE moves coding-agent evaluation behind company firewalls
Specific’s Real-SWE benchmark replaces public issue repositories with completed engineering work from licensed private codebases: billing, tax, migration, and other production tasks. Its first release covers only 10 tasks, but runs each model-and-harness combination eight times for 640 scored rollouts. The reported leader, Fable 5.1 with Claude Code, solved 38.8%; GPT-6 Astra with Codex CLI reached 33.8%, and Gemini 3.8 Flash with Gemini CLI reached 31.2%. Hidden code reduces obvious training contamination and makes the work more representative of enterprise maintenance. It also limits reproducibility, while pairing each model with a different harness prevents clean model-only comparisons. The useful result is the benchmark design, not a durable winner from a ten-task sample.
Watch: More donated codebases, confidence intervals by task, repeated tests with shared harnesses, and rules governing whether closed providers retain private evaluation data.
Sources: Specific’s benchmark, scores, and methodology, the launch-day Hacker News scrutiny, the LocalLLaMA discussion of privacy and reproducibility
3. OpenAI turns its agent harness into a managed control plane
OpenAI’s Agents API exposes the Codex harness as a public beta for sessions that can run for hours or days, with context management, tools, multi-agent orchestration, and an OpenAI-hosted sandbox. Developers can instead run tools on their own infrastructure or through partners. Vercel’s integration shows the resulting split clearly: OpenAI maintains the loop and session state, while Vercel runs the application and sandbox, reconnecting work through signed webhooks and queues. That can remove substantial orchestration work, but it does not outsource tool permissions, data boundaries, recovery policy, or vendor dependency. The API adds no separate service fee; model tokens, tools, and hosted environments still incur their normal costs. OpenAI’s launch metrics are selected customer testimonials, not independent benchmarks.
Watch: Public reliability data for long-running sessions, resumable failures, permission audits, end-to-end bills, and equivalent self-hosted harness comparisons.
Sources: OpenAI’s Agents API architecture and pricing notes, Vercel’s implementation pattern, the independent developer discussion
More signals
4. RubyGems separates a verified spam campaign from uncertain AI attribution
Ruby Central says it removed more than 500 malicious packages from a May campaign, but cannot confirm researchers’ claim that OpenAI agents published them or whether attempts to obtain API keys succeeded. The packages’ RubyDoc configuration could execute code during documentation builds. The practical lesson is attribution-independent: registries need upload throttles, build isolation, and anomaly response even when the operator is unknown.
Sources: Ruby Central’s verified findings and caveats, Simon Willison’s review of the evidence, the high-attention security discussion
5. OpenAI’s Habitat makes constrained storage an AI-scale feature
OpenAI says Habitat now handles more than 70 million requests per second across roughly 40 regions and 500 petabytes, using a deliberately narrow NoSQL interface while offloading complex queries. A two-engineer, Codex-assisted Rust rewrite now serves 95% of production traffic; vendor measurements claim sixfold CPU and fifteenfold memory efficiency over Python. The architecture is instructive, but no public latency distribution, error rate, or code supports independent comparison.
Sources: OpenAI’s Habitat architecture and reported measurements, an independent technical reading of the rewrite
6. AgentZip targets the memory duplicated across agent sandboxes
AgentZip compresses sandbox memory against shared templates and neighboring sandboxes, then restores likely-needed pages ahead of use. Its authors report 8.7× compression of sandbox-owned memory, versus 2.1× for Linux compression, and reduce aggressive-mode slowdown from 3.1× to 1.40× through prefetching and work scheduled during model waits. The paper is an unreviewed, author-run evaluation; independent implementations must test whether the gains survive real workloads and failures.
Sources: the AgentZip paper and experimental setup, an independent roundup placing the result in the week’s systems research
7. GPT-5.3-Codex-Spark gets a retirement notice without a migration map
Codex product lead Thibault Sottiaux says GPT-5.3-Codex-Spark will retire “next week,” citing declining use and better models. No exact cutoff, affected surfaces, or designated replacement accompanied the notice, and the official deprecations page had not yet added a row at review time. Teams with pinned Spark workflows should inventory them now, but should not assume API and product availability end together.
Sources: Sottiaux’s direct retirement notice, Modelyard’s timestamped record of the announcement, the Codex community’s migration concerns
What to watch next
The durable advantage is moving outward from model access into the surrounding operating system: evaluators who can inspect work before release, private tests that resemble actual repositories, orchestration that can resume safely, and infrastructure that exposes its real limits. The next useful disclosures will make those controls auditable—named reviewers, reproducible private-task methods, failure and cost distributions, and migration deadlines precise enough to plan against.
Sources
- Dario Amodei’s We Must Pace the Frontier essay
- AP report on Anthropic’s evaluator commitment
- Hacker News discussion of Amodei’s proposal
- Specific’s Real-SWE benchmark and methodology
- Hacker News discussion of Real-SWE
- LocalLLaMA discussion of Real-SWE’s private tasks
- OpenAI announcement of the Agents API public beta
- Vercel’s Agents API deployment integration
- Hacker News discussion of the Agents API
- Ruby Central update on the May spam campaign
- Simon Willison’s review of the RubyGems attribution
- Hacker News discussion of the RubyGems investigation
- OpenAI’s Habitat storage architecture report
- Independent technical review of the Habitat rewrite
- AgentZip paper on memory compression for agent sandboxes
- Independent research roundup covering AgentZip
- Codex product lead’s GPT-5.3-Codex-Spark retirement notice
- Modelyard record of the Spark retirement notice
- Codex community discussion of the Spark retirement