When Capture-Recapture Meets Frontier AI

A New Hypothesis About Model Diversity, Coverage, and Orchestration

By Noopur Davis, Executive Vice President, Chief Information Security and Product Privacy Officer, Comcast; and Sean McRae, Senior Principal Security Engineer, Software Development ＆ Engineering, Comcast

This blog is a part of Comcast's Frontier AI Field Notes Series

Executive Summary

As part of Project Glasswing, we applied capture-recapture techniques to findings generated by five frontier AI models assessing the same production software system. The statistical analysis was interesting. The overlap patterns were more important.

We found that frontier models behaved less like interchangeable inspectors and more like specialists. Many findings were unique to individual models, suggesting that model diversity expands coverage in ways repeated execution of a single model may not.

This observation has important economic implications. If no single model provides sufficient coverage, organizations may increasingly require orchestrated portfolios of models rather than reliance on a single provider. In this environment, the challenge shifts from selecting the best model to balancing coverage, cost, and operational complexity. These dynamics may increase the strategic importance of both model orchestration and open-weight models.

The central lesson from this analysis is simple: repeated execution increases saturation; model diversity increases coverage. If this finding holds at larger scale, model orchestration may become as important as model capability itself.

The Question

During Project Glasswing, Anthropic's initiative to help secure critical software for the AI era through access to Claude Mythos Preview, one of the questions we asked was deceptively simple:

After all the assessments, how many vulnerabilities remain undiscovered?

For decades, software engineering has borrowed a statistical technique from ecology known as capture-recapture. The idea is straightforward. If two inspectors review the same system, the amount of overlap between their findings can be used to estimate how many defects neither inspector found.

Watts Humphrey popularized this approach in software quality, treating independent reviewers as "capture sources." If reviewer A finds 100 defects, reviewer B finds 100 defects, and they share 50 in common, statistical models can estimate the total defect population and, by extension, how many defects remain undiscovered.

Project Glasswing gave us an opportunity to apply the same concept in a new context: frontier AI models used as separate vulnerability discovery sources. Using a representative production system as the test case, we compared findings from five different models: Mythos 5, Mythos Preview, Opus 4.6, Opus 4.7, and Sonnet 4.6. Across those models, we observed 166 unique findings.

We then analyzed the overlap patterns using a multi-source log-linear capture-recapture model. The model estimated a total findings population of approximately 177, implying that roughly 11 findings remained undiscovered and that the assessment had uncovered about 94% of the estimated population.

At first glance, this looks like a success story for capture-recapture.

But the more interesting story is what happened when we looked at the assumptions.

The Models Did Not Behave Like Human Inspectors

Traditional capture-recapture works best when inspectors are interchangeable and largely independent. Each inspector is assumed to draw from the same underlying population of defects with roughly similar probabilities of detection.

Our data suggested otherwise.

Different models consistently surfaced different findings. Certain findings appeared readily discoverable by one model and nearly invisible to another. Rather than behaving like five versions of the same inspector, the models behaved more like specialists approaching the same code from different perspectives.

This observation was consistent with other findings from Project Glasswing. Repeated runs of a single model increased saturation, but introducing model diversity often produced entirely new findings. The overlap patterns suggested not merely random variation but meaningful differences in how models reasoned about code, attack surfaces, and exploitability.

^{Figure A. Distribution of findings across five frontier models. Many findings were discovered by only a subset of models, while comparatively fewer were detected universally. This pattern suggests that different models contribute distinct perspectives rather than functioning as interchangeable inspectors. Overall, 78 findings were found by only one model, 31 were found by two, and only 17 were found by all five.}

The Most Interesting Result Wasn't the Estimate

The natural temptation is to focus on the number.

Was the answer 177 findings?

Maybe. Maybe not.

The more important observation is that the assumptions behind classical software-inspection capture-recapture appear strained when applied to frontier AI systems.

In traditional inspections, capture-recapture assumes inspectors help estimate the size of a fixed defect population.

In frontier AI assessments, the models themselves may be helping define what that population looks like.

Different models appear to expose different regions of the vulnerability landscape. A model optimized for exploit development may not search the same way as a model optimized for code reasoning. A model with stronger attention to authentication flaws may behave differently than a model that excels at attack chaining. As a result, overlap tells us less about redundancy and more about specialization.

A New Interpretation of Coverage

This leads to a hypothesis that deserves further study:

Model diversity may increase coverage in ways that repeated execution of a single model cannot. In other words, repeated executions increase saturation, model diversity increases coverage.

If true, the future of AI-powered security assessment may not be a race toward a single dominant model. Instead, it may look more like portfolio construction - combining models with different strengths, costs, perspectives, and operating characteristics to maximize discovery while minimizing validation effort.

That conclusion aligns with another major lesson from Project Glasswing: the future is likely about model orchestration rather than model monoculture.

^{Figure B. Incremental coverage from model diversity. In this figure, models were added in order of maximum incremental contribution. The first model discovered the majority of findings, but each additional model continued to contribute unique findings. Although marginal returns decreased over time, coverage continued to expand as model diversity increased.}

The Economic Case for Open Source / Open Weight Models

This analysis also makes the case for open source / open weight models. If no single model is sufficient, if multiple models are needed, if saturation and orchestration are pre-requisites, then the cost of only using proprietary models can become prohibitive. A balanced approach to the combination of proprietary/open models is needed. There are policy considerations for the United States.

The Bigger Lesson

Capture-recapture gave us an estimate.

More importantly, it gave us a new question.

For decades, software organizations assumed that adding more reviewers primarily increased confidence. Frontier AI suggests that adding different reviewers may actually change what can be discovered.

If that observation holds across larger datasets, the implications extend far beyond vulnerability management. It would suggest that the most effective cybersecurity architectures of the future will not rely on a single model, no matter how capable. They will combine multiple models that contribute distinct perspectives, and success will depend as much on orchestration and validation as on model capability itself.

In that sense, the most valuable insight from our capture-recapture experiment was not the estimate of findings remaining. It was the realization that frontier AI systems may not behave like traditional inspectors at all. And that ultimately, it is about the economics - the balance between efficacy and cost.

Appendix - Our Methodology - and Why We're Skeptical of the Findings Estimate

Terminology

Throughout this paper, the term finding refers to a potential security issue identified by an AI model during code analysis. Findings have not necessarily undergone human validation and therefore should not automatically be interpreted as confirmed vulnerabilities.

A vulnerability is a finding that has been subsequently validated and determined to represent a genuine security weakness.

The capture-recapture analysis presented in this paper estimates the population of discoverable findings within the analyzed codebase. It does not estimate the total number of confirmed vulnerabilities. Because the same validation process was applied consistently across all models included in the study, the analysis remains useful for estimating the completeness of model-generated discovery and the magnitude of remaining undiscovered findings.

Dataset

We analyzed findings from five frontier models against the same application: Mythos 5, Mythos Preview, Opus 4.6, Opus 4.7 and Sonnet 4.6

Across all models, the assessment identified 166 unique findings.

These counts alone immediately suggest that the models are not behaving identically. Some models discovered substantially more findings than others. The variance in numbers suggest that each model-class took different approaches in this experiment, but they should not be interpreted as a definitive ranking of model capability. Because frontier AI systems are stochastic and this dataset is limited, a model that found more findings in one run or small set of runs may not continue to outperform others as iterations increase, particularly if it reaches diminishing returns sooner.

Method

Classical capture-recapture techniques estimate the size of an unseen population by examining overlap patterns among multiple observers.

For every vulnerability, we recorded whether each model identified it:

This creates a capture-history matrix where each unique pattern represents a detection history.

For five models there are 32 possible capture histories:

00000

00001

00010

...

11111

The only history we cannot directly observe is:

00000

which represents findings that none of the models discovered.

The objective of capture-recapture is to estimate the size of this missing cell.

Results

Using a Poisson log-linear capture-recapture model across all five detection sources produced:

• Observed findings: 166

• Estimated unseen findings: ~11

• Estimated total findings population: ~177

This implies:

and suggests approximately 6.2% of the estimated population remained undiscovered.

^{Figure C. Capture history distribution across five frontier models. Each bar represents a unique detection pattern across the five models. The large number of populated capture histories indicates that findings are distributed across many different model combinations rather than concentrated in a small number of shared findings.}

Why We Are Skeptical of the Coverage Estimate

The result itself is less interesting than the assumptions required to produce it.

Capture-recapture assumes that:

1. Observers are reasonably independent.

2. Detection probabilities are relatively stable.

3. Overlap provides information about the unseen population.

Our observations suggest these assumptions may not fully hold. The models appeared to exhibit distinct search behaviors:

• Some findings were common across multiple models.

• Some findings were concentrated in only one or two models.

• Different models appeared to excel at different classes of findings.

This behavior is consistent with a hypothesis that frontier models function more like specialists than interchangeable inspectors.

Another consideration is that vulnerabilities are not always a fixed population in the same way traditional software defects are. Code that appears safe today can become vulnerable later as exploit techniques, chaining patterns, dependencies, deployment assumptions, or threat models evolve. Any capture-recapture estimate should be framed as estimating the currently discoverable findings under the assessment condition.

The estimate is therefore useful as an exploratory benchmark, but the overlap patterns are the more important result. They suggest that model selection and orchestration may materially affect coverage, not merely cost or speed.