Google Agent Evaluations Are GA. Offline and Live Scores Still Need Version Control

Google has made Agent and Model Evaluations in Gemini Enterprise Agent Platform generally available. Teams can now reuse registered metrics across offline experiments and online monitors, inspect traces behind failures, generate synthetic cases, simulate users and tool environments, and watch sampled production scores for drift.

That shared engine removes a lot of evaluation plumbing. It does not make two scores comparable by itself. An offline suite and live traffic can use a metric with the same name while testing different agent revisions, case populations, trace fields, judge models, or sampling rules. A falling chart may reflect the product, the evaluator, or the traffic mix.

What Google has made generally available

Google’s July 31 announcement describes more than 20 pre-built metrics covering task success, tool use, trajectory quality, grounding, safety, and reference-based language tasks. Teams can also register Python metrics and custom large-language-model judges. Server-side experiments retain artifacts in Cloud Storage, while online monitors can score collected production traces and send drift alerts.

The current agent-evaluation documentation, updated August 26, breaks the process into cases, inference, traces, scoring, analysis, and optimization. It supports both local development and assessment of deployed agents. That lifecycle is broader than a model benchmark because the trace can include tool selection, arguments, intermediate events, and multi-turn state rather than only the final answer.

One interface boundary remains easy to miss. Google’s GenAI Client guide still labels that recommended interface Preview and subjects it to pre-GA terms. The service overview labels the older vertexai.evaluation.EvalTask module GA and maintained for compatibility, but says it does not support newer methods such as adaptive rubrics. The evaluation service can be GA while the exact client a team adopts is not.

The independent AgentPedia implementation guide reaches the same operational conclusion: pin the SDK and hide Preview-dependent calls behind a replaceable adapter. A release decision should record the lifecycle status of the interface actually used, not inherit the broadest status in a launch headline.

A comparable score needs a versioned measurement bundle

Google’s Metric Registry documentation says registered definitions can be applied consistently to offline assessments and continuous monitors. That is the stable center of the loop. The surrounding measurement still needs its own record.

Measurement component	Record across the offline-to-live boundary
System under test	Pin the agent, model, instruction, retrieval corpus, and tool-schema revisions offline. Record the exact deployed revision and traffic allocation online. A system change can move the score even when the metric is unchanged.
Cases	Name the offline dataset revision, held-out status, scenario source, and expected outcomes. For production, record the trace filter, cohort, language, tool path, region, and time window. A fixed suite and changing user population answer different questions.
Metric	Use the same registry resource and definition revision in both places. Edited criteria can look like product drift.
Judge	Pin the judge model ID, parameters, prompt or rubric revision, and retry policy for the comparison window. A judge update can change verdicts without any agent change.
Evidence	Define the required trace events, redaction rules, missing-field policy, and artifact location. Confirm that production exposes the same observable fields; a judge cannot grade a tool action that the live trace omitted.
Sampling and summary	Record offline repetitions, seeds, invalid-run rules, slices, and statistic. Pair them with the live sampling percentage, maximum samples, schedule, invalid traces, slices, and statistic. A sampled average can move because volume or composition changed.

This table is a publication recommendation, not a claim that Agent Platform automatically captures every field. Google’s registry supplies reusable metric resources. The team remains responsible for connecting those resources to immutable application, data, telemetry, and analysis revisions.

A practical identifier can be a hash of that bundle rather than a hand-written label such as quality-v3. Store the readable fields beside it. When the judge model or rubric changes, score a shared calibration set and a recent production-trace sample with both versions. Only splice the chart if the overlap shows that any score shift is understood; otherwise begin a new series and annotate the change.

Adaptive rubrics improve fit, but they add another generated artifact

An adaptive rubric asks a judge model to create case-specific pass/fail criteria from the evaluation case, developer instruction, and tool declarations, then grade the resulting trace. For a refund agent, one case might require checking shipment state before cancellation, making at most one mutation, and reporting the actual tool result. A generic “helpfulness” score would miss those state and authorization boundaries.

The extra specificity is useful only if the generated criteria are inspected. A rubric can demand behavior the task never requested, reward a verbose explanation over a correct tool outcome, or omit a critical authorization invariant. Treat the generated rubric group as part of the evidence: version it, sample it for human review, and test whether the judge can distinguish known pass, fail, and ambiguous fixtures.

Research predating Google’s release shows why calibration matters. The peer-reviewed MT-Bench and Chatbot Arena study found that strong language-model judges could approximate human preferences in its chat-assistant setting, but it also documented position, verbosity, self-enhancement, and reasoning biases. Those results do not measure Google’s current managed raters. They establish the narrower point that agreement must be tested for the judge, task, labels, and conditions a team actually uses.

Use programmatic checks wherever the expected property is exact: allowed tool names, required JSON fields, arithmetic totals, permission boundaries, transaction state, or a prohibited side effect. Use a language-model judge for semantic properties that need interpretation. Neither score should override a failed deterministic safety invariant.

Our general AI evaluation explainer covers the broader rule that a metric shapes the product decision. The HarnessRisk analysis extends the scope to configuration, capability installation, persistent state, action control, and recovery—areas a response-quality score cannot certify.

Simulation expands coverage; production supplies a different distribution

Agent Platform can generate cases, simulate multi-turn users, and intercept tool calls with controlled data, errors, or latency. Those features make rare and dangerous paths easier to test without breaking a real backend. They can show that an agent handles a synthetic timeout, permission denial, or malformed result under defined conditions.

A simulator does not establish the prevalence of those failures in production, the fidelity of a real service’s side effects, or the behavior of users the simulator did not represent. Generated cases may also repeat assumptions from the agent instruction and tool schema used to create them. Keep synthetic, human-authored held-out, and production-derived cases as separate slices rather than blending them into one pass rate.

Google’s online-monitor documentation lets operators filter traces, set a sampling percentage, and cap samples per run. That controls cost, but it makes the sampling rule part of the result. A monitor restricted to slow, long, or high-token traces is useful for diagnosis; its score is not an estimate of all traffic unless the analysis accounts for that selection.

The cleanest bridge is a shadow comparison. Before launch, run the release candidate on the held-out suite. After launch, send a small reviewed sample of eligible traces through the identical metric bundle while retaining a random baseline. If a live score moves, break it down by agent revision, tool path, language, cohort, and trace completeness before calling it product drift.

Cost, telemetry, and security controls belong in the result

Two current Google pages disagree on computation-metric charges. The launch post says code-based and computation metrics add no additional cost. The dedicated Agent Platform pricing page lists character-based charges for computation metrics and separately bills model-based metrics through the underlying autorater. Treat the pricing page and the billing account as authoritative for budgeting, and timestamp the rates used in any cost-per-evaluated-trace calculation.

The announcement also says datasets and traces remain in the customer’s project. That statement should not be expanded into a guarantee that every desired control is available. Google’s current enterprise-security table marks Agent evaluation as supporting HIPAA, while listing VPC Service Controls, customer-managed encryption keys, at-rest data residency, Access Transparency, and Access Approval as unsupported for that service. Product support can change, so verify the table, region, target model, Cloud Trace path, and Cloud Storage configuration for the exact deployment.

Evaluation traces can contain prompts, responses, tool definitions, arguments, session identifiers, and failure examples. A production monitor therefore needs a reviewed field allowlist, redaction before export, access and retention rules, deletion behavior, and an explicit policy for regulated or contractual data. A higher sampling percentage is not automatically better if it copies more sensitive content into a less controlled evidence path.

One product family does not make offline and live scores identical

Google’s GA release makes experiments, traces, reusable metrics, and online monitors easier to operate together. The credible benefit is continuity of evidence, not continuity of a metric name.

A live score still depends on which traces were sampled, which fields survived logging and redaction, which judge and rubric version ran, and whether production users resemble the offline cases. When any of those change, the number may move even if the agent does not—or stay flat while an important failure slice worsens.

The product closes an operational gap. It does not remove the analytical work required to decide whether an observed change belongs to the agent, the evaluator, or the traffic.

Sources

1. Google announcement: Agent and Model Evaluations are generally available
2. Google Cloud agent evaluation overview
3. Google Cloud metric registry documentation
4. Google Cloud continuous online evaluation documentation
5. Google Cloud GenAI Client agent evaluation documentation
6. Google Cloud Gen AI evaluation service overview
7. Gemini Enterprise Agent Platform pricing
8. Google Cloud Agent Platform enterprise security support table
9. AgentPedia independent implementation guide to Gemini agent evaluations
10. NeurIPS paper: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena