OpenAI's Jalapeño Chip Benchmark: What the Numbers Actually Show

OpenAI’s first published results for its Jalapeño inference chip show a real systems achievement, not a procurement verdict. In company-run tests using the public SemiAnalysis InferenceX framework, Jalapeño delivered a better combination of throughput per kilowatt and response latency than the NVIDIA Blackwell systems selected for comparison.

The headline ratios are substantial. OpenAI reports 1.5 to 1.9 times more AI work per watt at peak throughput and 1.7 to 3.6 times lower end-to-end latency across GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T. Those results support the claim that OpenAI has working first-generation silicon with a competitive serving architecture.

They do not establish a lower customer price, lower data-center energy use, or a reason for another company to replace its accelerator fleet. The charts cover specified models, sequence lengths, precisions, software configurations, and package power ratings. Jalapeño is not for sale, and OpenAI expects only a small initial deployment late in 2026.

Jalapeño improved both sides of the serving trade-off

An inference service rarely maximizes one number. Adding concurrent requests can increase total token throughput, but each user may wait longer. Reducing batching can make a single stream feel faster while leaving expensive hardware underused. The useful comparison is therefore a curve: how much aggregate work can the system complete at each acceptable latency?

OpenAI used three fixed-sequence InferenceX workloads. The published appendix labels them as nominal 8K-input/1K-output tests and reports results across operating points rather than at one hand-picked concurrency level. GPT-OSS 120B was compared with GB200; DeepSeek R1 and Kimi K2.5 were compared with GB300. The DeepSeek and Kimi tests used MXFP4, a compact 4-bit floating-point format.

At peak throughput, OpenAI reports these ratios:

Tested model	Comparison system	Jalapeño peak throughput per kW	Jalapeño end-to-end latency
GPT-OSS 120B	NVIDIA GB200	1.9× higher	1.7× lower
DeepSeek R1 670B	NVIDIA GB300	1.7× higher	3.6× lower
Kimi K2.5 1T	NVIDIA GB300	1.5× higher	3.4× lower

These are OpenAI’s measurements, not an independent reproduction. Still, the framework is more informative than a raw chip peak. InferenceX defines interactivity as the streaming token rate per user and end-to-end normalized interactivity as output tokens divided by the full request latency, including time to first token. Its published methodology also varies sequence length, concurrency, model, precision, framework, and hardware—the conditions required to make an inference number interpretable.

The result matters because the three models stress different system shapes. GPT-OSS 120B is smaller than the other two; DeepSeek R1 and the trillion-parameter Kimi model require work to be distributed across many accelerators. Reporting from the Hot Chips presentation says OpenAI tested Jalapeño with single-token prediction while the NVIDIA baselines used multi-token prediction in relevant comparisons, and that OpenAI also presented a narrower direct comparison for DeepSeek. Those configuration details belong beside the ratios.

Throughput per kilowatt is not the electricity bill

OpenAI normalized the charts with each accelerator’s published package thermal design power, or TDP: 700 watts for Jalapeño, 1,200 watts for GB200, and 1,400 watts for GB300. TDP is a design rating used for power and cooling planning; it is not the energy a complete service actually consumed during a run.

OpenAI says Jalapeño stayed at or below 550 watts on the tested workloads. Data Center Dynamics independently reported both the 700-watt rating and that measured ceiling from the Hot Chips presentation. Yet the comparison uses the 700-watt published rating, not the lower observed draw. That is conservative for Jalapeño at the package level, but it still leaves several bills outside the denominator:

• host CPUs, memory, switches, storage, and power conversion;
• cooling and other facility overhead;
• idle power when request volume falls below the benchmark load;
• failed or retried requests, model loading, and cache movement; and
• the energy and time needed to produce an accepted task result.

The distinction cuts both ways. A package-TDP comparison avoids giving one accelerator credit for a favorable momentary reading while using a rating for another. It also cannot support a claim about measured joules per token or total facility power. The InferenceX documentation treats those as separate metrics and exposes both joules-per-token and custom power calculations where runs provide the necessary telemetry.

The procurement number is usually total cost of ownership, not tokens per package watt. Hardware price, networking, memory, rack density, reserved capacity, utilization, software labor, downtime, and financing can outweigh the direct electricity difference. OpenAI has not published enough of those inputs to calculate a customer-facing cost advantage, and there is no Jalapeño price because the chip is intended for OpenAI’s own infrastructure.

The architecture explains the direction, not every ratio

Large-language-model serving has at least two different hardware phases. During prefill, the system processes the prompt and performs substantial parallel computation. During decode, it generates tokens step by step and repeatedly reads model weights and the key-value cache—the stored attention state from earlier tokens. Decode is often constrained by memory bandwidth and data movement rather than arithmetic alone.

OpenAI says Jalapeño co-designs compute, high-bandwidth memory, networking, kernels, and rack-scale software around those phases. Model state can remain local while the system changes which resources are active, reducing transfers between separately specialized pools. The Hot Chips presentation described a low-latency local network joining 128 accelerators and a larger domain spanning 2,048 chips.

That mechanism makes the benchmark’s direction plausible: less waiting for model state can improve response latency and keep more of the power budget doing useful work. It does not isolate which feature produced each gain. There is no public ablation that holds the workload fixed while separately removing Jalapeño’s local memory placement, network topology, compiler optimizations, or model-specific kernels.

OpenAI also says each model family needs new kernels and optimization work. That is normal for accelerator software, but it means “supports three models” is evidence of portability across those ports, not proof that every architecture will reach the same frontier. The strongest follow-up would publish exact benchmark recipes and raw power traces for the Jalapeño points, then let independent operators repeat the runs when hardware access exists.

Four questions turn the chart into a deployment decision

Teams evaluating any inference platform—not just Jalapeño—can keep the benchmark honest by recording four layers of evidence.

Decision layer	Minimum record	Why the headline ratio is insufficient
Workload	Exact model revision, precision, input/output lengths, prompt mix, concurrency, batching, and speculative-decoding policy	A frontier-model agent and a short chatbot response stress different bottlenecks.
User experience	Time to first token, time between tokens, end-to-end latency, and p50/p95/p99 values under normal and burst load	An average can hide the slow requests that determine whether a service feels reliable.
Capacity and energy	Accepted tokens or tasks per second, measured accelerator and system power, idle power, utilization, and joules per accepted task	Package TDP cannot show the energy of the rack or the cost of failed work.
Economics and operations	Acquisition or rental cost, software support, uptime, failure rate, staffing, rollout capacity, and cost per accepted task	A faster unavailable system—or one that needs extensive porting—cannot serve today’s demand.

Plot throughput against latency rather than comparing one peak. Repeat the test at the context lengths and concurrency levels the service actually sees. Measure tail latency, because sequential agent steps compound slow outliers. Then calculate energy and cost for accepted outputs, not merely generated tokens.

This is the same discipline needed for smaller deployments. Our guide to Qwen3.8-27B on a 16GB GPU separates weight capacity, context memory, task quality, throughput, and total operating cost. The scale is different, but the error is the same: a hardware-friendly benchmark can answer its own question correctly while leaving the buyer’s question untouched. Our broader AI evaluation explainer shows how to begin with the decision and choose a measurement that can support it.

Deployment, not another ratio, is the next test

OpenAI and Broadcom announced an initial deployment by the end of 2026. OpenAI hardware chief Richard Ho gave TechCrunch a more restrictive schedule: very small volumes at the end of 2026, followed by more significant deployment in 2027. OpenAI also says it will continue deploying accelerators from NVIDIA and other partners.

That makes Jalapeño strategically important to OpenAI without making it a current purchasing option for anyone else. The next decisive evidence is whether the chip can hold its latency and efficiency advantage across production traffic, changing models, sustained utilization, failures, and a maturing software stack. Independent measurements would strengthen the case; service pricing and reliability would show whether the hardware gain reaches customers.

For now, the precise conclusion is already useful. Jalapeño moved the published inference frontier for three specified workloads under OpenAI’s test conditions. It has not moved the procurement boundary, because availability, whole-system energy, operational reliability, and total cost remain unmeasured or undisclosed.

Sources

1. OpenAI's first Jalapeño benchmark results
2. OpenAI and Broadcom Jalapeño announcement
3. SemiAnalysis InferenceX benchmark methodology and metrics
4. TechCrunch report on Jalapeño performance and deployment timing
5. Data Center Dynamics report from the Hot Chips presentation
6. ServeTheHome report from the Hot Chips presentation