Pipette's On-Device AI Results Are Deployment Tests, Not Chip Rankings
Pipette publishes latency, throughput, memory, and quality results across many configurations, but different phones, runtimes, and quantizations are not one clean ranking.
Liquid AI and Artificial Analysis released Pipette on August 24 as an open benchmark for small language models running on phones and computers. Its useful unit is not a model name. It is the complete deployment: model, quantization, runtime, device, runtime settings, and workload.
That design fixes a familiar problem in on-device AI comparisons. A full-precision quality score from a server does not say whether a quantized model fits on a phone, responds before a user loses patience, or slows sharply as the prompt grows. Pipette puts those dimensions in one dashboard—but putting them beside each other does not make every row directly comparable.
The central limitation is simple: cross-device charts are deployment observations, not chip rankings, and the quality score shown beside phone performance was measured through a separate evaluation path.
Pipette measures a deployment configuration
The Pipette launch announcement says the initial public dataset contains more than 1,000 combinations across over 30 models, multiple quantizations, llama.cpp builds for macOS, iOS, Windows, and Android, and prompt lengths from 256 to 8,192 tokens. It reports latency, prompt-processing speed, token-generation speed, and peak memory on target devices, then pairs compatible rows with task-quality evaluations.
The extra dimensions are not metadata to ignore after finding a model. They can change the result:
| Dimension | What it changes | What must stay fixed for a clean comparison |
|---|---|---|
| Model artifact and quantization | Weight memory, output quality, and sometimes speed | Exact model artifact and quantization format |
| Runtime | Kernels, accelerator use, measurement boundaries, and supported features | Runtime name, build, backend, and relevant flags |
| Device and operating system | Available memory, processor path, counters, power behavior | Exact device class and OS build |
| Workload shape | Prefill work, generation work, context pressure, and key-value cache memory | Input tokens, output tokens, benchmark definition, and context allocation |
| Device conditions | Thermal throttling and background load | Power, cooling, readiness gate, and comparable idle state |
| Quality evaluation | Which capability the score represents | Dataset version, scorer, thinking mode, model, and quantization |
Pipette’s result-comparison guide is stricter than a leaderboard glance: runtime flags, Flash Attention, thread count, GPU-layer count, thinking mode, quantization, and token configuration are part of the configuration. A changed setting is a changed experiment.
This is why “Model A gets 80 tokens per second” is incomplete. A useful statement sounds more like: “this quantized artifact, on this device and runtime build, generated 100 output tokens at this rate after a 2,048-token prompt under the documented lab conditions.”
Latency, prefill, and decode answer different questions
Language-model inference has two main phases. Prefill processes the input prompt before the first generated token. Decode generates the response one token at a time. A long document and a short answer stress prefill; a short prompt and a long answer put more emphasis on decode.
Pipette’s performance methodology reports these phases separately. Its standard decode benchmark generates 100 output tokens, while end-to-end latency generates 256. Timing runs use greedy decoding, discard startup and warm-up behavior, and report a mean plus sample standard deviation across five measured repetitions.
Choose the metric that matches the product question:
| Product question | Pipette metric | Read it with |
|---|---|---|
| How fast can the system absorb a prompt? | Prefill throughput, in input tokens per second | The exact input length and runtime path |
| How quickly does text appear after generation starts? | Decode throughput, in output tokens per second | The output length and model’s response behavior |
| How long does the complete fixed request take? | End-to-end latency | Both token counts and included client overhead |
| Does the deployment fit? | Peak memory | The platform-specific counter and context length |
| Does the artifact retain useful capability? | IFBench, GPQA Diamond, or MATH-500 | The task represented and the separate evaluation path |
Do not estimate user-visible response time by adding two rounded throughput numbers. Pipette notes that end-to-end paths may include tokenization and local request overhead that the isolated phase rates exclude. Measure the actual app as well if startup, model loading, prompt construction, streaming, or UI work matters.
Variance also belongs in the decision. Pipette advises treating a standard deviation above 5% of the mean as evidence of a noisy or unstable run. Five close repetitions do not prove population-wide performance, but they are more informative than a rounded average hiding one throttled run.
The phone score and the quality score come from different machines
The easiest Pipette chart mistake is to assume every plotted value was produced on the selected phone. Performance is measured on the displayed device. The current published quality results were generated separately with llama.cpp on NVIDIA H100 80GB systems, then matched to device results when the model and quantization are compatible.
That matching is useful. It lets a team ask whether a smaller quantization saves enough memory and time without losing too much task performance. It does not prove that the phone ran MATH-500, GPQA Diamond, or IFBench at the speed shown on the performance axis. Pipette’s publication methodology also says changing the performance device does not select a new quality row.
Quality coverage is narrower than “good at mobile AI.” The launch set tests instruction following, competition mathematics, and science reasoning. Pipette’s documented limitations say it is not comprehensive for agentic behavior, knowledge-heavy tasks, multimodal work, or other device-oriented uses. A team shipping summarization, extraction, tool calls, speech, or vision still needs an evaluation built from that feature’s inputs and failure costs.
Quantization makes this separation worth keeping. It reduces the precision used to store model weights, often lowering memory use and sometimes improving speed, but the quality loss depends on the artifact and task. Compare quantizations within one model and target deployment, then test the application task. A general benchmark is evidence, not an acceptance test for a product promise.
A cross-device chart is not a processor shootout
Pipette explicitly recommends comparisons within the same device. The initial Android and iOS paths are not hardware-controlled equivalents. The coverage methodology says public Android results use a CPU-only llama.cpp command-line path with Flash Attention disabled and no GPU layers. Published iOS measurements run inside the app with Metal and settings that have no Android equivalent.
The platform also affects measurement. Peak-memory counters do not mean exactly the same thing everywhere, and the iOS public thermal signal is too coarse for Pipette’s lab process. Published iOS runs use an internal thermal-aware build that can read the chip temperature, a capability the public app cannot reproduce exactly.
Pipette controls some environmental noise. Its device-conditions methodology checks thermal and load signals before timing repetitions, keeps fleet phones on mains power, and uses external cooling. Those controls improve comparisons within its lab. Cooling remains operator-managed and is not stored on each result row, so the dataset cannot by itself prove that two arbitrary phone rows had identical physical conditions.
An independent GIGAZINE hands-on shows the public iPhone app can download a model and run local benchmark workloads. That is useful confirmation of the client experience, not an independent reproduction of Liquid AI’s full lab dataset or private iOS thermal instrumentation.
If the decision is “which configuration should our iPhone app use?”, filter to the supported iPhone and compare configurations there. If the decision is “which phone chip is faster?”, Pipette’s current cross-platform paths do not isolate the chip from the runtime, backend, flags, operating system, power, cooling, and counter differences. A larger number can describe the complete published deployment without identifying the component that caused it.
What Pipette can establish today
Pipette makes on-device model selection more auditable by publishing versioned workloads, configuration-level results, variance, and traceable submissions. It can show that one tested configuration is faster, smaller, or stronger on a covered evaluation than another configuration under the recorded conditions.
It cannot turn different platform paths into a controlled processor ranking. It cannot make three server-run quality tests represent every mobile feature. And because the current verified results come from Liquid AI’s lab while community publication remains beta, it does not yet describe the distribution of ordinary users’ hot, busy, battery-powered devices.
The next meaningful evidence will be independently reproducible submissions with clear condition labels and broader device coverage. Until then, Pipette is a strong shortlist and comparison surface whose rows still describe complete deployments—not isolated models or processors.