One Average Can Mislead an AI Review Pilot: Lessons from 740 Pre-Reviews

A 740-material pre-review dataset recorded a 3.73-minute blended average—but that number covered three very different workloads. For teams evaluating an AI review pilot, the lesson is simple: segment before comparing performance.

A single average can hide the workload mix behind it.

By June 16, an internal AI-assisted pre-review project had recorded 740 medical content materials, 128 users, and a cumulative average processing time of 3.73 minutes per material. The headline number was useful. The more important finding was that it combined three materially different workloads.

A buyer could treat 3.73 minutes as a universal speed benchmark. A project team could compare it with an earlier cumulative average of 5.20 minutes and conclude that the system became 28% faster. A leader could assume that faster AI processing must translate directly into faster formal MLR approval.

The dataset does not support any of those conclusions on its own.

What it does show is more operationally valuable:

For MLR operations and digital transformation leaders evaluating a pilot, the practical rule is:

Segment before you compare. AI review performance must be interpreted by material type, complexity, measurement boundary, and reviewer usefulness.

A snapshot of the operational data

The workbook's usable cumulative series runs from March 4 through June 16, 2026. By the end of that period, it records:

• 740 medical content materials processed;
• 128 cumulative users served;
• 342 case-based materials;
• 366 agenda materials;
• 32 short-form social materials;
• 3.73 minutes as the cumulative average AI pre-review processing time.

The cumulative material count increased from 168 on March 4 to 740 on June 16. Over the same recorded series, the cumulative average processing time moved from 5.20 minutes to 3.73 minutes.

This is encouraging operational evidence that the workflow was used at growing scale. It is not a controlled performance study. The material mix changed, new users entered the workflow, a new material category appeared later, and the underlying rules or operating practices may also have evolved.

The right question is therefore not simply, "Did the average go down?"

It is: What changed inside the average?

One average was covering three different workloads

The project did not process one uniform document type.

Agenda materials were generally faster to process in the recorded weekly data. Case-based materials usually required several minutes and showed greater variability. Short-form social materials appeared later in the project and also took longer than agenda materials in the observed weeks.

That difference matters because a blended average changes whenever the workload mix changes.

If one week contains more agenda materials, the overall average may fall even when the system's performance for each material type is unchanged. If another week contains more complex case-based or short-form social materials, the average may rise even when the system is operating normally.

This is a classic composition effect. It can make a workflow appear to improve or deteriorate when the main change is what entered the queue.

The same problem appears when teams compare different business units, markets, or vendors. A system reviewing short, structured materials should not be benchmarked directly against one reviewing long, image-heavy, evidence-dense presentations unless the comparison controls for complexity.

A fair evaluation compares like with like.

Processing time is not the same as approval time

The 3.73-minute figure refers to AI pre-review processing recorded in the project workbook. It is not the duration of human review, correction, resubmission, workflow routing, or formal MLR approval.

Those stages may influence one another, but they are not interchangeable metrics.

An AI system can process a material quickly and still create little workflow value if its findings are noisy, difficult to verify, or disconnected from the applicable SOP. Conversely, a complex material may require more processing time while producing a better evidence package for reviewers.

This is why an enterprise pilot should define its timing boundary before collecting results:

• When does the clock start?
• Does processing include file conversion, OCR, retrieval, and rule execution?
• When is a result considered complete?
• Is reviewer verification included?
• Are queue time and system wait time separated?
• Is formal MLR approval measured independently?

Without these definitions, two teams can report the same metric name while measuring different processes.

Three rules for evaluating an AI review pilot

A useful pilot does not need the largest possible dashboard. It needs a small set of metrics that support a defensible decision.

1. Segment the workload before comparing performance

Report processing time separately by material type and complexity band. Where possible, include the median and a high-percentile measure such as P90—not only the mean.

Useful segments may include:

• document length and file size;
• native versus flattened presentation files;
• number of charts, tables, images, and references;
• OCR dependence;
• number and type of review points executed;
• language, market, audience, and channel.

Coverage and adoption still matter. Teams should track the number of materials, active users, participating teams, and eligible materials entering pre-review. But those measures describe the pilot footprint; they do not establish performance quality.

Segment before you compare. Otherwise, a change in the queue can be mistaken for a change in the system.

2. Define where the clock starts and stops

The timing boundary must remain consistent across every comparison. Teams should state whether the measure includes file conversion, OCR, retrieval, rule execution, queue time, reviewer verification, correction, resubmission, or formal MLR approval.

Any before-and-after claim should use comparable cohorts. Material type, complexity, team, period, and workflow stage should be matched as closely as possible. Processing time, reviewer effort, and end-to-end approval time can all be useful, but they answer different questions.

3. Pair speed with reviewer usefulness

Speed has value only when the output helps reviewers find and assess relevant issues. A pilot should therefore examine:

• agreement with qualified human reviewers;
• false-positive and false-negative rates by review point;
• page- or object-level location accuracy;
• completeness of evidence and rule references;
• frequency and reason for human overrides;
• routing of unresolved or low-evidence cases;
• time spent verifying routine issues;
• rework, resubmission, and escalation patterns under consistent definitions.

These measures require a defined evaluation set and reviewer ground truth. They cannot be inferred from processing-time data.

For a pre-MLR review preparation layer, reviewer usefulness means making the issue, location, evidence, rule context, and rationale easier to inspect before formal review. AI can prepare and connect that record; qualified reviewers still interpret the evidence and own the decision.

How different teams should interpret the same pilot results

A pilot result has different meanings depending on the team's responsibility. Those perspectives should be evaluated together, not collapsed into one headline metric.

Reading the same pilot through these three perspectives reinforces the central principle: no single convenient metric can represent workflow value, technical reliability, and decision quality at the same time.

How to read the 740-material result responsibly

The project dataset provides a useful operational signal. It shows sustained use across hundreds of materials, expansion to 128 users, and meaningful variation between material categories. It also shows a lower cumulative average processing time at the end of the recorded period than near the beginning.

What it does not establish is causation.

The reduction in the cumulative average could reflect workflow learning, rule optimization, infrastructure changes, material mix, user behavior, or several factors at once. The workbook does not contain a randomized control, a locked cohort, or complete quality labels. It also does not prove that formal MLR approval became faster or more accurate.

That distinction is not a weakness. It is the basis of a trustworthy measurement approach.

Operational data should first be used to understand volume, variation, and workflow behavior. Controlled evaluation should then test quality and matched outcomes. Only after those layers are connected should a team make broader performance claims.

The real benchmark is decision usefulness

An enterprise AI review system should not be selected because it produces the lowest isolated processing-time number.

It should be evaluated on whether it can handle the organization's actual material mix, surface relevant issues, show the evidence behind its findings, route uncertainty correctly, and reduce avoidable search work without weakening human accountability.

The useful unit of measurement is not merely minutes per file. It is reviewable decision support for a defined material, review point, and workflow stage.

Segment before you compare. That is what turns a promising speed metric into an evidence-based decision about review readiness and AI-assisted medical content review.

Data note

The figures in this article are derived from an anonymized internal project workbook through June 16, 2026. Only aggregate AI pre-review operational data is used. Product-specific records are excluded. The dataset is observational, contains some missing early-period fields, and should not be interpreted as a controlled study or a universal industry benchmark.