Cross-Model KV Cache Transfer Is Fast—and Surprisingly Pair-Specific
NVIDIA researchers report a 25× component speedup for one KV-cache mapping, while several other model pairs lost much of the target model’s task accuracy.
NVIDIA researchers report that a linear mapper can convert one language model’s key-value cache into the format expected by a larger relative, avoiding a second pass over a long conversation. In their best latency example, mapping a 32,768-token cache from Qwen3 14B to Qwen3 32B took 278 milliseconds; running the 32B model’s prefill took 6,975 milliseconds. That is a measured 25.1× component-level gap.
It is not a 25× application speedup.
The August 4 preprint tested only dense, full-attention models in three families, with matching key-value geometry. Two of six small-to-large pairs lost most of the target’s benchmark accuracy under the linear method. Even two pairs that looked successful on a five-benchmark average retained only 18.2% and 36.6% of the target’s GSM8K result, respectively. The latency experiment also excluded one part of an end-to-end deployment: delivering the mapped cache to the target process.
The result is therefore pair-specific, not a feature switch. A matched cache shape makes an experiment possible; it does not predict whether task quality will survive.
The cache saves work inside one model
An autoregressive language model produces text one token at a time. Before that loop begins, the model processes the input tokens in a prefill pass. Each attention layer writes key and value tensors that represent the context. During decode, the model reads those stored tensors instead of recalculating every earlier token on every step.
Hugging Face’s cache documentation describes the ordinary case: dynamic, fixed-size, quantized, or offloaded caches that belong to one model. It also shows prefix caching, where one model precomputes a shared prompt and reuses it for several continuations.
Switching models breaks that assumption. A Qwen3 14B cache contains the internal representations produced by Qwen3 14B; Qwen3 32B expects its own. A router can always send the accumulated text to the 32B model and let it prefill normally. Cross-model transfer tries to replace that work with a learned conversion.
Our vectors and embeddings explainer develops the underlying idea: coordinates are useful only relative to the representation system that gives them meaning. The mapper is an attempt to translate between two related systems without rerunning the model that would normally produce the target coordinates.
Several related systems use the same word, “transfer,” for different operations:
| Technique | What remains the same | What changes | Main question |
|---|---|---|---|
| Prefix caching | Model and prefix | Continuation or request | Can identical earlier tokens be reused? |
| Cache offloading | Model and cache meaning | Memory location or precision | Can memory pressure be traded for movement cost? |
| KV-cache transmission | Compatible cache layout | Serving component or device | Can the same cache reach the decoder efficiently? |
| Cross-model mapping | Tokens and intended context | Model representation | Can an approximate target cache preserve behavior? |
NVIDIA’s TensorRT-LLM transmission guide, for example, moves cache data from a context phase to a generation phase and handles layouts across tensor- and pipeline-parallel configurations. That is serving infrastructure, not evidence that the new paper’s cross-model approximation is integrated or production-ready.
The mapper is simple; its contract is not
The paper defines a matched-KV pair as two models with the same number of KV heads and the same dimension per head. Its experiments also stay within a model family, where source and target share a tokenizer, and use dense full attention. Model depth and total parameter count may differ.
For each target layer and attention head, the method selects the source layers whose caches best predict the target’s cache. It concatenates the top k source features and solves a separate ridge regression for keys and values. Ridge regression is a linear fit with a small penalty that helps keep the solution numerically stable.
Keys also contain rotary position embeddings, or RoPE: position-dependent rotations that tell attention where a token sits in the sequence. The mapper reverses the source rotation, fits in that position-free space, and applies the target rotation afterward. Values do not carry RoPE and are mapped directly.
The researchers fitted each pair with 500 FineWeb-Edu sequences of 1,024 tokens. Depending on the pair, the resulting mapper contained 1.01 billion to 3.36 billion parameters and occupied 4–12 GB in the paper’s configuration. Fits took roughly 47–87 minutes on one eight-H100 node. Each direction is separate, so a 14B-to-32B artifact cannot be assumed to work in reverse.
None of those details is a deployment threshold. The paper used one calibration domain, chose k partly on benchmarks later reported, did not test mismatched-KV pairs, and did not cover cross-family or hybrid-attention transfer. A model revision can also change the representations the fitted mapper expects even when a marketing name stays constant.
That makes the correct unit of approval an exact tuple:
source revision → target revision + tokenizer + direction + mapper artifact + workload evaluation
The accuracy average hides the decisive failures
The paper reports raw retention as mapped-cache accuracy divided by the target model’s normal-prefill accuracy. It also reports floor-normalized retention, which sets a benchmark’s chance score to zero. That second number matters: raw retention can make a result near or below chance look less severe.
The six small-to-large pairs did not form one reliable class:
| Source → target | Five-task average | Floor-normalized average | GSM8K retention |
|---|---|---|---|
| Qwen3 14B → 32B | 97.6% | 96.3% | 95.6% |
| Qwen3 8B → 32B | 87.5% | 80.7% | 68.8% |
| Llama 3.1 8B → 70B | 72.8% | 62.9% | 18.2% |
| Ministral 3B → 8B | 76.2% | 65.9% | 36.6% |
| Ministral 3B → 14B | 44.2% | 14.7% | 3.2% |
| Ministral 8B → 14B | 41.6% | 11.1% | 1.6% |
GSM8K was the study’s chain-of-thought math-generation benchmark; the other four tasks in the average were classification-style evaluations. The Llama pair’s five-task average was 72.8%, but its mapped GSM8K score was 14.78 against the target’s 81.12. That does not establish that every reasoning workload will fail. It establishes that an average dominated by other task shapes cannot admit a mapper for math reasoning, tool use, coding, or the application behind a production router.
The two Ministral-to-14B linear mappings failed much more broadly. A two-hidden-layer nonlinear mapper recovered 24.3 and 36.8 percentage points of HellaSwag retention for those pairs, but it also replaced the paper’s closed-form, gradient-free advantage with a trained model. “Use a bigger mapper” is a new system to evaluate, not an automatic repair.
The study’s multi-turn result needs the same boundary. It tested Qwen3 14B and 32B on 100 CoQA conversations across ten turns. Drift stayed small in that one pair and task. VentureBeat’s August 21 report brought useful attention to the long-session use case, but its stronger wording about proving the system would not cascade into failure goes beyond that experiment. Ten turns on one compatible pair cannot guarantee another model, task, or session length.
Recalculate the 25× result around the whole handoff
At 32,768 tokens, the Qwen3 14B-to-32B result supports this paper-local calculation:
reported avoided prefill = target re-prefill - mapper application
= 6,975 ms - 278 ms
= 6,697 msThe timing used one eight-H100 node with NVLink, bfloat16 forward passes, 50 warmups, and 30 timed trials per cell. Re-prefill ran the target transformer body with FlashAttention 2 and excluded the language-model head. Mapper timing included cross-GPU transfers needed to move the source cache to the target, but the authors say an end-to-end system would also need to deliver the mapped cache to the target process, which they did not measure.
A complete handoff result would instead be:
net handoff saving
= normal target prefill
- mapper application
- source/cache transport not already included
- mapped-cache delivery
- amortized artifact loading
- rejected-transfer and fallback penaltyThe distribution matters across context lengths and concurrency, as does cost per accepted handoff, including mapper fitting, artifact storage, failed mappings, shadow comparison, and fallback prefill. A 25× component result can still be valuable, but only where that saving survives contact with the serving topology.
That denominator follows the same discipline as our cost-per-accepted-task evaluation: rejected work and retries still consume resources. The AlphaEvolve matrix-multiplication analysis provides the broader systems lesson—the improved operation matters only when it is large enough in the real workload to change the whole-system result.
Cross-model KV-cache transfer is a promising result precisely because the avoided work is concrete. The paper mapped a target cache much faster than it recomputed one, and it preserved behavior on several pairs. It also supplied the warning label: compatible shapes can fail, broad averages can hide task collapse, and the published latency is not the application’s latency. The next credible milestone is not a larger speedup headline. It is a versioned production integration that publishes task-level acceptance, complete handoff cost, and safe fallback results for each direction.