Pipette's On-Device AI Results Are Deployment Tests, Not Chip Rankings
Pipette publishes latency, throughput, memory, and quality results across many configurations, but different phones, runtimes, and quantizations are not one clean ranking.
Tag
#benchmarks
7articleswith this tag.
ASI-Bench Finds AI Agents Struggle Without a Research Procedure
ASI-Bench tests scientific agents as human guidance disappears. Its largest score drop reveals a procedure-building problem, not proof about superintelligence.
OpenAI's Jalapeño Chip Benchmark: What the Numbers Actually Show
OpenAI's Jalapeño inference chip leads its published tests, but latency, power normalization, availability, and workload fit limit the buying conclusion.
NVIDIA AVO's 100% ARC-AGI-3 Result Measures a Harness, Not a Model
NVIDIA AVO cleared ARC-AGI-3's public set, but the benchmark does not isolate the harness contribution or test generalization on private tasks.
llama.cpp v0.2.0 Makes Stable and Nightly Different Promises
llama.cpp now has semantic releases beside nightly builds. Its first stable tag shares code with a build tag, showing that channel names describe policy rather than quality.
Grok 4.6's Low Token Price Does Not Make It the Cheapest Model
Grok 4.6 combines low headline pricing with strong reported benchmarks, but retries, failed tasks, latency, and tool charges can reverse the apparent advantage.
Why evaluations shape the AI products we eventually use
A practical look at how evaluation choices influence model claims, product decisions, and what readers should inspect in AI announcements.