Google HEIR Makes Private Inference Compilable, Not Automatically Practical

Google has presented HEIR as a route from an ordinary machine-learning model to inference over encrypted inputs. The open-source compiler is real, its four public demos cover useful workloads, and its design reaches across multiple cryptographic schemes and software backends.

It is not a switch that makes any model private. HEIR’s current documentation still requires engineering choices about operators, numerical approximations, ciphertext layout, encryption parameters, backends, keys, and hardware. Google’s showcase reports single-threaded CPU measurements for its demos and says accelerator latency results will come later. That makes HEIR a credible prototype path, not evidence that an arbitrary model will meet a production service-level objective.

The important split is between privacy necessity and system feasibility. Fully homomorphic encryption (FHE) matters only when it removes an input exposure the application actually has; HEIR matters only when the model fits its compilation path. Encrypted evaluation time, memory, ciphertext traffic, accuracy, key handling, and failure recovery then determine whether that privacy gain is viable on the intended workload. Treating those as separate questions prevents a privacy requirement from quietly becoming an open-ended infrastructure project.

HEIR changes the compiler problem, not the cost equation

Ordinary transport and storage encryption protects data while it moves or rests, but a service usually decrypts that data before computing on it. FHE uses special encryption schemes that let a server evaluate supported operations on ciphertext. The client decrypts the returned ciphertext to recover the corresponding result, while the evaluating server does not need the secret key or plaintext input.

That property can remove a consequential point of exposure. A recommender could score encrypted user features, for example, without the scoring service seeing those features. It does not protect every part of the product. Client endpoints, keys, logs, model updates, cleartext preprocessing, returned answers, access control, and side channels still need their own design. The output itself may reveal sensitive information if the application asks an unsafe question.

HEIR addresses the hard translation in the middle. Its machine-learning design takes model representations from PyTorch, TensorFlow, and ONNX-related paths into MLIR, a compiler framework with multiple levels of intermediate representation. The pipeline marks secret data, selects a cryptographic scheme and backend, replaces or approximates operations that do not map cleanly to encrypted arithmetic, chooses ciphertext layouts, manages noise growth, selects parameters, and emits code for a library or lower-level hardware path.

The compiler can automate much of that work, but the application owner still supplies decisions that change correctness and cost. HEIR’s own example configuration includes the approximation degree and input range for a nonlinear activation. Choose a range that does not match production data and an otherwise valid encrypted program can lose model quality. Choose a deeper or more expensive circuit and latency, ciphertext size, key material, or memory can cross the deployment budget.

Four demos establish scope, not universal readiness

Google’s August 14 announcement points to four compiled demonstrations: content recommendation, credit-card fraud detection, network anomaly detection, and hotword recognition. The public demo repository makes the boundary more concrete.

The fraud example compiles a small multilayer perceptron with sigmoid activations to CKKS, a scheme for approximate-number arithmetic, and exposes Lattigo and OpenFHE evaluation paths. The recommendation and hotword examples warn that their runs can need at least 96 GiB of RAM. The hotword model is a compact temporal convolutional network, not a general speech or language model. The network example includes five- and 50-feature anomaly detectors and a timing harness that separates encryption, evaluation, and decryption.

Those are useful artifacts because a team can inspect the models, data paths, build targets, and timing code. They are still demonstrations chosen for compatible structures. Google’s post says the published latency numbers use a single CPU thread and that the team plans to show accelerator latency benefits later. It does not publish a cross-backend production benchmark, concurrent serving test, availability target, or total cost comparison.

The wording matters. These examples prove that HEIR can compile and run several nontrivial encrypted-inference programs. They do not prove that the same pipeline supports every operator in a transformer, that a demonstrated backend fits a team’s hardware, or that an FHE service is cheaper than keeping the workload local. A current HEIR getting-started note also says an end-to-end binary for workflows such as converting a precompiled Torch model to a backend is still being developed; the documented route uses lower-level heir-opt and heir-translate tooling in the meantime.

The arithmetic constrains the backend

An FHE scheme defines the encrypted arithmetic available to the program. A backend implements that scheme. They are linked decisions, not interchangeable package names.

Workload property	Candidate scheme family	Current documented HEIR library paths	Prototype question that can stop the work
Exact integer or modular arithmetic	BGV or BFV	OpenFHE or Lattigo	Can every comparison, lookup, and nonlinear step be represented without silently changing the model?
Approximate real-number inference	CKKS	OpenFHE or Lattigo	Does approximation error stay inside the task-quality gate across real input ranges?
Boolean or short-integer circuits	CGGI	tfhe-rs or Jaxite	Do circuit depth and bootstrapping frequency fit the latency and hardware budget?
Hardware-oriented research path	Scheme-level or lower-level HEIR IR	CPU, GPU, FPGA, ASIC, or photonic integrations vary by target	Is the target reproducible and available, or only a research or partner demonstration?

This table is a starting map derived from HEIR’s current pipelines and backend documentation, not a compatibility promise. A scheme name does not establish operator coverage, parameter security, compiler maturity, or equivalent results between libraries. Pin the exact compiler, frontend, scheme, backend, parameter set, model, and generated artifact used in the test.

Version choice needs the same discipline. As checked on August 25, GitHub’s latest-release endpoint identifies v2026.08.11.dev0, despite the development-style suffix; the repository also offers an August 1 monthly release and newer dated nightlies. That naming makes “stable” an unsafe shortcut. Select one tested tag and commit deliberately. “Latest” is not a reproducible dependency.

Privacy has to justify the ciphertext bill

FHE is most compelling when an external service must compute on data it must not see, local execution is unavailable or unacceptable, and a hardware enclave or organizational control does not satisfy the threat model. That case depends on which fields stay encrypted, where plaintext returns, who holds keys, and what request timing, metadata, errors, and decrypted outputs can still reveal.

If those boundaries are vague, no favorable latency result can make the system worthwhile. On-device execution, data minimization, a smaller local model, confidential computing, or a split workflow may satisfy the same need with less complexity. The Qwen local-deployment analysis shows why the memory printed on a device is not the same as usable inference capacity; FHE adds a different but equally important layer of representation and runtime overhead.

One operator cannot represent the encrypted system

The second gate starts from a small public or synthetic workload that preserves the production model’s shapes and ranges without exposing sensitive data. Compile the smallest useful path first. Record unsupported operations and model changes as results, not inconveniences to hide.

Then measure the whole request lifecycle:

Measurement	Minimum record	Decision it protects
Task quality	Plaintext metric, encrypted metric, per-slice failures, approximation settings	Prevents a fast encrypted path from shipping a materially different model
Compile and setup	Frontend export time, compile time, parameter generation, key generation, artifact size	Exposes work that a steady-state inference number omits
Latency	Client encryption, transfer, server evaluation, return transfer, client decryption; p50 and p95	Separates the expensive phase and preserves the user-visible budget
Throughput	Requests per second at declared concurrency and batch size	Prevents a single-thread demo from becoming a capacity forecast
Memory	Peak resident memory, evaluation keys, rotation or bootstrapping keys, model artifacts, ciphertext buffers	Tests whether the target host can run and recover the service
Data expansion	Plaintext bytes, ciphertext bytes sent and returned, temporary bytes	Makes network and storage costs visible
Operations	Counts of additions, multiplications, rotations, comparisons, bootstraps, and layout conversions	Locates the circuit feature that controls cost
Reliability	Timeouts, malformed inputs, precision failures, retries, key rotation, rollback time	Tests whether the system is operable rather than merely executable

Memory deserves its own gate. The independent 2026 KeyMemRT preprint identifies rotation keys as a major memory consumer in complex FHE applications and tests a compiler/runtime approach for managing their lifetimes. Its reported improvements belong to its own setups and should not be transferred to a HEIR service. The broader finding is still useful: key material and runtime memory are part of the system design, not negligible cryptographic metadata.

Hardware results need equally tight attribution. An accelerator benchmark can show what one circuit, scheme, parameter set, compiler revision, device, batching strategy, and precision achieved. It cannot supply the latency row for a different model. The Waymo edge-inference analysis makes the parallel point for TOPS: a peak arithmetic number does not settle memory, transfer, thermal, software, or workload fit.

A successful prototype can still end with “do not deploy”

There are several useful outcomes short of an FHE production service. The privacy gate may show that the protected computation is small enough to isolate while the rest stays local or uses ordinary encryption. The compiler test may identify one unsupported activation that can be replaced and retrained without harming quality. The benchmark may show that the model works but only a batch workflow, not an interactive API, fits the budget. It may also show that memory, ciphertext expansion, or key operations overwhelm the value.

That last result is not a failed experiment. HEIR’s contribution is to make more of this trade-off inspectable in a shared compiler framework. The 2025 HEIR preprint presents the project as a platform for implementing, combining, and comparing homomorphic-encryption techniques across the stack. Google’s newer showcase demonstrates how that platform can reach recognizable machine-learning workloads.

The remaining uncertainty belongs to the application, not the announcement. HEIR earns a production path only when the encrypted result still solves the task and the complete system—not one favorable operator—fits the quality, latency, memory, security, and recovery constraints.

Sources

1. Google: How Google Is Making Private AI Practical with Homomorphic Encryption
2. HEIR machine-learning compiler design
3. HEIR getting-started guide
4. HEIR compiler pipelines
5. HEIR v2026.08.11.dev0 release
6. Google fully homomorphic encryption demo repository
7. HEIR: A Universal Compiler for Homomorphic Encryption
8. KeyMemRT: Unlocking Memory-Scalable FHE