A Human Reviewer Who Cannot Overturn the Model Is Not a Control

A person clicking approve after an automated system has effectively settled the outcome is not meaningful human review. The reviewer needs enough evidence and time to make an independent judgment, authority to pause or change the result, a recorded reason, and an appeal path that can still correct the decision.

That is the practical lesson from a disputed enforcement decision against Uber. On August 21, the Dutch Data Protection Authority said it had fined Uber €824.99 million for fully automated driver-account decisions and insufficient notice between 2018 and 2022. The French data protection authority, CNIL, which cooperated on the case, likewise describes temporary and permanent deactivations made without human intervention. Uber disputes the findings and says the relevant policies are historical.

The challenge status needs a caveat. The Dutch authority’s Dutch announcement says Uber announced an administrative objection. Its English version says an appeal has been filed, while AP reports that Uber said it would file one. The public sources reviewed on August 24 do not identify a challenge decision or court ruling. The enforcement decision should therefore be treated as issued, disputed, and procedurally unresolved—not as a final judicial judgment.

This engineering and governance test does not prove legal compliance. Applicable rights and obligations depend on the jurisdiction, decision, organization, and people affected.

Human contact and human control are different system properties

The Dutch authority says the historical Uber software tracked driving behavior and ratings, then automatically deactivated accounts after suspected fraud or low ratings. The authority says the violations ended. Uber’s current public account-review page separately describes manual review, the ability to submit additional evidence, and an in-app Review Center that is expanding around the world.

The enforcement summary and company page describe different periods and evidence types. The findings concern 2018–2022. Uber’s page describes its current stated approach and is not an independent audit of present behavior. Neither source proves how every deactivation works today.

Any automated-decision workflow can contain a person yet leave the result functionally untouched. Review becomes weak when the system withholds contrary evidence, presents its score as authoritative, gives the reviewer only seconds, punishes overrides, or lets an appeal return to the same person with the same evidence and no greater authority.

The Dutch authority’s 2025 guidance on meaningful human intervention organizes the problem around the person, technology and interface, process, and governance. It asks whether reviewers understand the system and its limits, can reject its output, have enough time, receive organizational support, and are monitored without making them solely responsible for a poorly designed process.

The UK’s Information Commissioner’s Office reaches similar operational tests in its human-review audit framework. It calls for qualified and independent reviewers, documented test methods, override logs with reasons, manageable caseloads, and processes for re-review or overturning. The ICO warns that this guidance is being reviewed after changes to UK law, so it should not be treated as a current rulebook for another jurisdiction.

The authority matrix exposes a rubber stamp before deployment

Write down what each role can actually do. A role name such as human reviewer proves nothing unless the workflow grants the corresponding evidence access and decision power.

Capability	First-line reviewer	Appeal reviewer	System owner	Evidence that the capability is real
See the automated recommendation and confidence or uncertainty	Required	Required	Read-only for support	Rendered case view plus access log
Inspect the underlying evidence and relevant contrary evidence	Required	Required	Maintains source connections	Evidence manifest, missing-data flag, and retrieval log
Ask for additional information	Required	Required	Implements request path	Dated request and response receipt
Pause the outcome before it takes effect	Required for consequential cases	Required	Defines a bounded safe hold	Hold event with expiry and owner
Accept, reject, or replace the recommendation	Required	Required	Cannot silently rewrite a decided case	Signed decision event with reason
Reopen a completed decision	Escalation only	Required	Provides controlled state transition	Reopen event linked to the prior decision
Change the policy or model	No	No	Controlled release process only	Versioned change record and separate approval
Delete or rewrite audit evidence	No	No	No direct edit; append corrections only	Immutable event history and correction link

The separation between case review and system change matters. A reviewer should correct an individual outcome without altering the model. A system owner should improve the policy or model without retroactively rewriting the evidence of an inconvenient decision.

The matrix also makes a common failure visible: a reviewer may be formally allowed to disagree but unable to stop the action before it takes effect. That is not the same authority as a pre-decision hold. Timing belongs in the control definition.

The evidence of review is not a generic human-reviewed: true flag. It is the sequence connecting the automated recommendation, the evidence available at the time, the reviewer’s authority, the reason for the decision, and the state that downstream systems finally enforced. Timing matters: a comment added after an account is already deactivated is not the same control as a hold that prevents the action.

Versioning matters for the same reason it does in an AI evaluation. A review can describe only the policy, automated system, evidence, and interface that produced that case. Logs can help reconstruct the sequence, but they do not make it true or complete; access controls, correction rules, retention limits, and independent sampling remain separate questions.

An appeal should create a new decision path, not replay the first one

A working appeal trace preserves the original events and adds a new review with a different decision-maker, any new evidence, and a fresh reason. It does not overwrite decision: accepted with decision: rejected and erase how the first outcome occurred.

Sequence	Required event	Failure that the log should expose
1. Automated recommendation	Versioned recommendation, inputs, uncertainty, and evidence manifest	Only the final label survives; the evidence cannot be reconstructed
2. First review	Assignment, evidence access, time spent, authority used, reason, and outcome	Reviewer clicked accept with no evidence view or reason
3. Notice	What was communicated, when, and how to seek review	Notice omits the basis, deadline, or usable challenge path
4. Appeal request	Request time, grounds, new evidence, and immediate effect such as a temporary hold	Appeal exists in policy but cannot prevent ongoing harm
5. Independent re-review	Different reviewer, complete original record, new evidence, reason, and authority	The same queue repeats the first result without examining the challenge
6. Correction and recovery	Changed outcome, restoration or remedy, notification, and downstream updates	The case changes in one database while connected systems keep enforcing the old result

The European Data Protection Board’s AI-auditing project offers a broader methodology and checklist for inspecting AI systems. That resource can help organize an audit, but it is not an EDPB finding about Uber and does not certify any particular review process.

Real authority appears when the reviewer disagrees

A process can look complete until contrary evidence appears, a required document is missing, or a reviewer rejects a high-confidence recommendation. Those cases reveal whether the interface exposes the source material, whether the reviewer can pause the consequence, and whether downstream systems obey the human decision rather than the original automated result.

Appeal adds an even stronger test because it must reopen a completed case, admit new evidence, assign independent judgment, and restore access or another affected state across connected systems. One successful override is not enough to prove good judgment, but a system that cannot execute an override has not delegated meaningful authority in the first place.

Diagnostic metrics need denominators and investigation

Metrics can reveal where to look; none is a universal compliance threshold.

• Override rate: overridden recommendations divided by reviewed recommendations, segmented by outcome type, reviewer group, policy version, and time. A near-zero rate may indicate an excellent system, weak reviewer authority, or a broken log.
• Appeal reversal rate: appealed decisions changed after re-review divided by completed appeals. A high value may reveal useful correction, poor first review, or selective appeals.
• Review latency: time from evidence readiness to a reasoned decision. Report the distribution, not just the average, and separate waiting time from active review time.
• Evidence completeness: reviewed cases with every required evidence class available divided by reviewed cases. The metric is only meaningful if requirements are versioned and missing evidence cannot be represented as an empty value.
• Reviewer disagreement: different conclusions on a blinded sample divided by double-reviewed cases. Investigate ambiguous policy, evidence gaps, training, and case mix before blaming individuals.
• Downstream correction rate: changed decisions whose effects were corrected in every connected system divided by changed decisions. A good appeal that leaves the old restriction active is still a control failure.

The review history needs enough context to investigate those rates without turning employee performance or affected-person data into an unrestricted analytics feed. Bounded access, pseudonymous operational identifiers, documented retention, and a correction path remain part of that trade-off.

Review must also come after the last material automated change. The Claude watermark and EU AI Act analysis shows the same sequencing problem in publishing: a substantive AI rewrite after sign-off makes the earlier review evidence describe the wrong version. For long-lived case records, the data-governance boundaries in the Spirit enterprise-data analysis are a useful reminder that preserving auditability does not justify retaining every underlying record forever.

Meaningful review is not proved by a button, job title, or appealing policy statement. It is proved when a qualified person can inspect the evidence, stop the automated path, give an independent reason, change the real outcome, and leave a record that another reviewer can reconstruct. The strongest final test is the appeal that corrects every downstream effect without erasing the mistake that made the appeal necessary.

Sources

1. Dutch Data Protection Authority: Uber fined for automated driver blocking
2. Dutch-language authority announcement of the Uber decision
3. CNIL account of the coordinated Uber enforcement decision
4. AP report on the Uber fine and response
5. Dutch authority guidance on meaningful human intervention
6. ICO human-review audit framework
7. European Data Protection Board AI-auditing resources
8. Uber account-deactivation review description