Audit payer AI the way you audit a reviewer. Every health plan that uses clinical criteria in utilization review already has a way to check whether its reviewers apply them consistently. Nurses and physician reviewers take inter-rater reliability tests built from standard cases, and a reviewer who falls below a passing score loses the right to review unsupervised. The AI tools that now score requests and draft recommendations are usually tested once, when the plan buys them.
That asymmetry is backward. A reviewer who drifts affects the cases on one desk. A tool that drifts affects every case it touches, and nobody notices until the appeals come back.
How plans already audit reviewers
The methods are well established. NCQA’s utilization management standards ask accredited plans to evaluate how consistently their reviewers apply criteria, and most plans do it with inter-rater reliability testing. A Blue Shield of California policy for its Medi-Cal plan, revised in December 2024, is typical. Clinicians who apply criteria must pass at least 2 tests a year built from hypothetical cases, and new hires must pass before reviewing without supervision. A score of 90% is required to review unsupervised. Staff who fail get targeted training and a retest. A second failure leads to a corrective plan.
Plans watch outcomes too. Overturned denials are the clearest signal that a review went wrong. In Medicare Advantage, KFF found that 80.7% of appealed denials in 2024 were overturned in full or in part, although only 11.5% of denials were appealed. A reviewer whose denials are reversed far more often than a colleague’s gets a closer look.
These checks share a premise. A clinician’s judgment is trusted only as far as it has been tested against a standard and against outcomes, and the testing repeats because judgment drifts.
What the same audit looks like for a tool
An AI tool in utilization review is, in effect, another reviewer. It reads the record and applies criteria to reach a view, even when a person signs the decision. The audits that work for people translate directly.
- A reliability test on standard cases. Run the tool on the same case set reviewers use, scored against the same answer key. Require the same passing score before it touches live requests.
- Blind re-review of live cases. Sample a share of the tool’s recommendations each month and have a reviewer decide them without seeing the tool’s output. Agreement on live cases is the number that matters.
- Overturn rates for tool-assisted decisions. Track appeals and reversals separately for decisions where the tool recommended the outcome, and compare them with decisions made without it.
- Override rates. Record how often reviewers disagree with the tool. A rate near zero can mean the tool is excellent, or that reviewers have stopped checking.
- A retest after every change. A new model version or a new criteria release is a new reviewer. Test it before it goes live.
The case set matters as much as the score. It should include the hard cases, such as requests that sit near a criterion’s threshold or records with conflicting documentation, not only the clean ones that any tool gets right. A plan can draw many of them from its own appeals, since an overturned denial is a case where someone already found the first answer wrong. Weighting the set toward the services the tool actually sees keeps the score honest.
Regulators are moving toward the same expectations. The NAIC adopted a model bulletin on insurers’ use of AI in December 2023. It expects insurers to validate and retest AI systems, including by measuring performance against expert review. It also expects them to evaluate drift, which it defines as the decay of a model’s performance over time. Insurers are also expected to confirm that vendors’ systems comply. Nearly half of the states had adopted the bulletin by March 2025, according to the law firm Quarles. As we described in May, California’s SB 1120 requires AI tools used in utilization review to be reviewed periodically for performance and outcomes.
A tool that touches every case deserves at least the scrutiny a plan gives a nurse who touches a few hundred.
The number to watch is disagreement
Of the measures above, the override rate deserves special attention, because it tests the human safeguard as well as the tool. Every state law that bars AI-only denials rests on the idea that a person exercises real judgment. A plan cannot show that without measuring how often the person disagrees, and why.
A healthy program should expect some disagreement. If reviewers accept nearly every recommendation, there are 2 possible explanations. Either the tool is narrowly scoped to cases where it is almost always right or the review has become a formality. The data will tell a plan which, if it looks.
Say a tool makes recommendations on 10,000 requests in a quarter. Reviewers change its recommendation on 40 of them. That rate, 0.4%, says little by itself. Broken down, it might show that most of the overrides came from 2 services, which points to criteria the tool reads poorly. It might also show a reviewer who made no changes at all across 2,000 cases, beside colleagues who made several. That reviewer is the one to sample first.
Disagreement also teaches. Each override, recorded with the reviewer’s reason, is a labeled example of where the tool went wrong. A plan that collects them builds a test set for the next version that reflects its own members and its own criteria.
What we build for
We build tools health plans use in utilization review, and we think they should be audited like the reviewers they assist. A clinician makes every denial. A machine can say yes, but only a person can say no. Every recommendation shows its reasons, so a reviewer can check the criteria and the facts behind it. Every decision can be reconstructed.
Those rules make the audit cheap. When the system records each recommendation alongside the reviewer’s decision and reason, override rates and blind re-review become reports rather than projects. We would rather a plan know exactly how often its reviewers disagree with us than trust a validation study from the day it signed the contract.