Review Point Explainer: Statistical Significance Claims in Medical Content Materials

A claim is not supported merely because a slide contains a P-value. Review must establish what the statistic evaluates and whether it supports the exact claim being made.

"Drug A significantly prolonged survival."

The sentence is short. The evidence check behind it is not.

A slide may show two survival curves, two median values, a hazard ratio, a confidence interval, an asterisk, and a P-value in a footnote. None of those elements automatically proves that the headline is supported. The reviewer still has to establish which endpoint was analyzed, which comparison the statistic belongs to, whether the analysis was planned and appropriately interpreted, and whether the wording stays within what the evidence can support.

The review question is not simply, "Is there a P-value?" It is, "What does this P-value support?"

Why "significant" is not just a stronger adjective

In medical content, words such as "significant," "significantly," and "statistically significant" can imply a formal statistical conclusion. They should not be used as decorative substitutes for "large," "important," or "noticeable."

The American Statistical Association notes that a P-value does not measure effect size or the importance of a result, and that scientific conclusions should not rest only on whether a value crosses a threshold. In clinical research, interpretation also depends on the study design, endpoint, analysis population, statistical model, multiplicity strategy, and prespecified analysis plan.

This matters for review because statistical significance, effect magnitude, and clinical relevance are related but different questions:

• A small P-value does not tell the reviewer whether an effect is large or clinically meaningful.
• A visible numerical difference does not prove statistical significance.
• A hazard ratio or confidence interval may be relevant evidence, but it must be interpreted in the context of the endpoint, model, and claim.
• P<0.05 is a common convention, not a universal stand-alone compliance rule. The applicable threshold and inferential framework depend on the study and analysis plan.

The purpose of an MLR pre-review is therefore not to reduce statistical interpretation to one number. It is to determine whether the claim and its supporting evidence form a coherent, reviewable chain.

Risk pattern 1: A numerical difference is presented as proof

Imagine a slide that states "significant improvement" and displays event rates of 1% versus 10%. The difference looks large. But if the slide and its cited source provide no statistical test connected to that comparison, the visual contrast alone does not establish statistical significance.

This is a common form of false confidence: the material contains numbers, so the claim appears evidence-based. Yet the numbers may be descriptive, based on a small subgroup, drawn from a post hoc analysis, or otherwise unsuitable for the intended inference.

A useful review flag should not merely say "P-value missing." It should explain the relationship:

The slide uses a statistical-significance claim for the 1% versus 10% comparison, but no inferential result has been identified that is clearly linked to this comparison. Add the applicable evidence or revise the claim to an appropriately descriptive statement.

Difference in magnitude is not a substitute for evidence of statistical significance.

Risk pattern 2: Evidence exists, but the connection is unclear

The second risk is more subtle. A P-value may appear somewhere on the slide, but it may not support the claim in the headline.

For example, a survival chart may contain:

• a headline claiming a significant improvement in overall survival;
• an asterisk placed on one curve;
• a table containing a hazard ratio and 95% confidence interval;
• a footnote stating P=0.002;
• several endpoints or subgroup comparisons on the same page.

Keyword search can find "significant" and P=0.002. That is not enough. The reviewer needs to know whether the P-value belongs to overall survival, another endpoint, a within-group comparison, a subgroup, or a different time point. The evidence may also come from a secondary or exploratory analysis whose status affects the permitted wording.

Co-occurrence is not evidence binding. A claim and a statistic can appear on the same slide and still be unrelated.

Risk pattern 3: Partial statistics are treated as a complete conclusion

A slide may state "statistically significant" while showing only an effect estimate, such as HR=0.85, or an estimate with a 95% confidence interval. These elements can be informative, and in some correctly specified analyses a confidence interval can convey inferential information. But the material must still allow a reviewer to understand the estimand, reference group, interval limits, model, direction of benefit, and connection to the exact claim.

The right response is not to impose a rigid rule that every slide must display the same statistic in the same format. It is to identify whether the evidence package is complete enough for the applicable SOP and source.

Where the support is incomplete, the system can recommend one of two paths:

1. provide the missing or correctly linked statistical evidence; or
2. revise the statement to a neutral, descriptive formulation that does not imply an unsupported inferential conclusion.

A valid non-trigger: No significance claim, no forced P-value check

Over-enforcement creates its own review burden.

If a slide makes a descriptive statement such as "the observed curve remained above the comparator curve during follow-up," it may still require accuracy and balance review, but it does not necessarily claim statistical significance. A pre-review system should not force every numerical or graphical comparison into a P-value workflow.

This distinction helps reduce false positives:

• Trigger: wording explicitly or implicitly claims statistical significance.
• Do not trigger automatically: wording describes an observed trend without asserting statistical significance.
• Escalate: the language is ambiguous, promotional context amplifies the implication, or the evidence structure cannot be resolved.

The objective is not to maximize flags. It is to surface the right decision points with the evidence needed to review them.

How ZENO supports this review point

ZENO can support statistical-significance review as an AI-assisted pre-review layer before formal MLR approval.

The workflow can:

1. detect significance-related claims in headings, body text, tables, captions, and callouts;
2. identify the endpoint, comparison, population, and direction of the claim where the material provides them;
3. retrieve candidate evidence across chart labels, legends, tables, footnotes, references, and neighboring content objects;
4. evaluate whether the evidence is plausibly linked to the same endpoint and comparison;
5. identify missing, ambiguous, or conflicting support;
6. produce a structured explanation with page and object locations for human review.

A reviewer-facing result might state:

The headline claims a statistically significant improvement in overall survival. A hazard ratio and confidence interval were found in the chart table, but no inferential result could be reliably linked to the headline claim. Please verify the cited source and applicable statistical analysis before approval.

Or:

The slide describes an observed trend and does not make a statistical-significance claim. No significance-specific evidence flag was triggered. Accuracy, balance, and source verification may still apply.

These outputs are deliberately narrower than "compliant" or "noncompliant." They show what was claimed, what evidence was found, how the two were connected, and what a qualified reviewer still needs to decide.

What remains a human judgment

Qualified medical, legal, regulatory, and statistical reviewers remain responsible for determining:

• whether the source and analysis are appropriate for the intended use;
• whether the endpoint and analysis were prespecified;
• whether multiplicity, subgroup, sensitivity, or missing-data considerations affect the interpretation;
• whether effect size and clinical relevance are represented accurately;
• whether the wording is balanced and consistent with approved materials and company SOP;
• whether the material should proceed, be revised, or be escalated.

The FDA's guidance on multiple endpoints illustrates why this context matters: analyzing multiple endpoints can increase the risk of false conclusions if multiplicity is not appropriately managed. A small number on a slide cannot carry that full context by itself.

From "number found" to "claim supported"

A reliable significance review does not stop when it detects the word "significant," a P-value, or an asterisk.

It asks whether the claim identifies a reviewable endpoint and comparison; whether the statistical evidence belongs to that claim; whether the interpretation stays within the evidence; and whether unresolved context is made visible to the accountable reviewer.

That shift—from detecting numbers to validating evidence relationships—is what turns a statistical keyword check into a meaningful MLR pre-review capability.