Kubernetes Inference Perf: Benchmark the Serving Stack, Not Just the Model
Kubernetes Inference Perf makes model-server comparisons more consistent, but its results still depend on prompts, load patterns, token counts, and clusters.
Kubernetes’ Inference Perf project now has a peer-reviewed software paper. The Journal of Open Source Software published it on August 27, giving practitioners a citable description of a tool built to send realistic generative-AI traffic through different model servers and serving stacks.
Its most useful promise is consistency, not a universal leaderboard. Inference Perf can hold one load generator and metric contract steady while a team changes a model server, accelerator, router, or Kubernetes policy. It cannot make two runs comparable when their prompts, output lengths, load patterns, token counters, warmup windows, or server settings differ.
That boundary turns “model-server-agnostic” from a marketing adjective into an engineering requirement: freeze the workload, record the whole serving stack, and compare only the dimension the experiment intended to change.
Inference Perf measures a deployment, not model intelligence
The JOSS paper describes a modular benchmark with data generation, load generation, server clients, metric collection, JSON reports, and analysis. The current project documentation lists verified integrations for vLLM, SGLang, and Hugging Face TGI, along with support for compatible OpenAI-style endpoints. It can generate constant-rate, Poisson, concurrent, burst, saturation, shared-prefix, multi-turn, and trace-replay workloads.
Those features answer serving questions. They can show how quickly a user sees the first token, how steadily later tokens arrive, how many requests meet a latency objective, where a system saturates, and whether a router or autoscaler recovers from a traffic change. They do not show whether the model’s answer is correct, safe, grounded, or useful.
The original Kubernetes WG Serving proposal made that scope unusually clear. It wanted a benchmark-as-code tool that could exercise model servers, accelerators, and orchestration without being tied to one stack. Recommending a winning server or hosted offering was explicitly a non-goal.
The working group itself has since completed its charter. A February CNCF project update says its work moved into Kubernetes special-interest groups and that Inference Perf is sponsored by SIG Scalability. The software is active: v0.6.1, released July 23, is the latest tagged version, while the main branch has continued to change. Pinning a version is therefore part of the result, not housekeeping.
The serving question determines the useful metric
One throughput number hides several different systems. A benchmark should begin with the user or operator promise, then select the metric that can falsify it.
| Question the run should answer | Primary evidence | Keep beside the result | What can fool the reader |
|---|---|---|---|
| How long before a user sees progress? | Time to first token (TTFT), especially p50 and p95 | Prompt-length distribution, cache state, concurrency, warmup treatment | A good mean can hide a slow tail or a cold-start penalty |
| How smooth is a streaming response? | Time per output token (TPOT) and inter-token latency (ITL) | Output length, streaming mode, token-count source | Shorter outputs or different tokenizers can manufacture an apparent gain |
| How much work can the stack complete? | Requests and input/output tokens per second | Offered rate, achieved rate, failures, request lengths | Counting only successes can hide overload and dropped work |
| What capacity meets the product objective? | Request or token goodput under declared latency constraints | Exact service-level thresholds and error policy | Raw throughput can rise after useful latency has already collapsed |
| Did routing or autoscaling help? | Stage-level latency, throughput, goodput, errors, and infrastructure telemetry | Replica count, routing policy, scale events, queue depth, GPU and server metrics | A steady-state snapshot can miss the transition the feature was meant to improve |
The project’s metric definitions distinguish end-to-end request latency, TTFT, TPOT, normalized TPOT, and ITL rather than collapsing them into “latency.” Its goodput calculation counts only successful requests that meet every configured latency constraint. That makes goodput a stronger capacity measure when a product has an actual service-level objective.
Even a correct metric can have the wrong denominator. Inference Perf records server-reported token usage and client-side re-tokenization because the two can disagree through chat-template overhead, tokenizer revisions, tool schemas, or streamed text. Its reports expose fallback counts and token mismatches. If one run normalizes TPOT with client counts and another uses server counts, the comparison has already changed two things.
Three layers separate server speed from cluster behavior
A useful Kubernetes serving evaluation separates local server behavior from cluster behavior. Combining everything in one run may produce an impressive chart without revealing which component caused it.
The cleanest baseline holds the model revision, quantization, server image, accelerator, replica count, routing path, prompt distribution, output behavior, and offered load steady. A rate sweep can then reveal the point where latency or errors make additional throughput useless. Router and autoscaler tests answer a different question again: they expose scale-up delay, queue growth, cache disruption, failures, and recovery during a transition rather than measuring only the final steady state.
Repeated runs in alternating order make that separation more credible. Raw stage and per-request reports, the generated config.yaml, server and client logs, and cluster events reveal instability that a median or one best run would hide.
Inference Perf is also capable of testing multi-turn and shared-prefix traffic, but those workloads require another control: cache admission and routing affinity. A request that lands on a warm prefix cache is not comparable with one routed to a cold replica. Record cached and uncached prompt tokens when the server exposes them, and treat a routing change that alters cache hits as a full-stack result rather than a pure server-speed result.
Identical tooling does not guarantee identical work
The project’s new cross-tool comparability guide is also a warning about comparisons made with Inference Perf alone. Default input and output distributions are not fixed lengths. Open-loop rate tests and closed-loop fixed-concurrency tests measure different behavior. Sampling parameters, tokenizer behavior, and early-stop settings change the amount of work sent to the server.
Warmup is a particularly sharp edge. The guide says Inference Perf has no dedicated excluded warmup phase: every request it sends is measured. A tool that discards warmup requests is observing a different window, especially when compilation, caches, or autoscaling make early requests slow. Teams can warm the server first or use a short leading stage and exclude it from the comparison, but the chosen treatment must be recorded.
Cross-tool comparisons need even more care. The guide documents version-dependent flag meanings and defaults in other benchmark clients, then recommends checking total input and output tokens before comparing rates. Equal averages are not enough when the minima, maxima, arrival spacing, or token source differ.
A server-agnostic harness removes one source of variation. It does not remove experimental design.
A sponsored Kubernetes comparison shows both sides of the method
A May Principled Technologies methodology report provides a concrete example. The testers used Inference Perf with Llama 3.1 8B Instruct, vLLM 0.11.0, streaming responses, shared-prefix prompts, and a Poisson rate sweep. They published cluster manifests, tool configuration, software versions, multiple load stages, and the rule used to select a throughput-latency trade-off point.
That detail makes the experiment inspectable. It also shows why a result is inseparable from its setup: the report compares configured GKE and EKS systems, not abstract clouds, generic Kubernetes, or every workload. The appendix says the testing concluded on April 14 and the project was commissioned by Google. It also refers to older Inference Perf versions than the current v0.6.1 release.
The study is useful as a worked measurement record, not neutral proof that one platform will be faster for another model, region, accelerator, server build, prompt distribution, or routing policy. Reproducing its YAML without reproducing its whole environment would create a new experiment.
A defensible result is a bounded claim
Before choosing a server or cluster policy, compare the intended delta against an unchanged baseline. If the model server changes, keep the model, hardware, request stream, and Kubernetes path fixed. If the router changes, keep the server images and replica pool fixed. If the accelerator changes, disclose every associated software and topology change that could not be held constant.
Then state the conclusion at the same scale as the experiment: “policy B sustained the declared workload at the p95 gate in this cluster,” not “policy B is faster.” Pair the serving result with a separate quality evaluation if the change can alter outputs, and calculate cost from measured resource use rather than accelerator list price alone.
The general AI evaluation explainer shows why the metric becomes part of the product decision. The Gemini pricing analysis adds the same warning from another direction: token throughput is not the business outcome when retries and rejected work consume the bill.
Peer review makes Inference Perf easier to cite. The stronger reason to use it is more practical: one repeatable harness can expose where the serving stack bends under real traffic. Its numbers become portable evidence only when the workload, counter, window, and environment travel with them.
Sources
- JOSS: Inference Perf, a benchmarking tool for GenAI inference
- Kubernetes SIGs Inference Perf repository and documentation
- Inference Perf v0.6.1 release
- Inference Perf metric definitions and token-count provenance
- Inference Perf goodput documentation
- Inference Perf cross-tool comparability guide
- Kubernetes WG Serving Inference Perf proposal
- CNCF: Kubernetes WG Serving concludes its work
- Principled Technologies GKE inference study methodology