Calibration

What the pipeline actually produced when run on Linux mainline at the snapshot recorded in each dossier’s meta.json.

Verdict distribution (top-1000 sweep)

864 total dossiers (858 from the top-1000 shortlist plus 6 from earlier calibration runs).

The shape is what we want from a conservative pipeline:

• ~6% of probed drivers flagged for any change (52 actionable)
• remove (1%) reserved for the strongest signal — model found an active upstream removal patch series
• unsure is empty because the prompt’s confidence-cap rule diverts low-confidence answers into not-a-driver (when the evidence says it’s content) or keep-annotate (when the evidence is ambiguous)

URL fidelity

Across all 3,552 URLs cited in the 864 dossiers — and a focused spot-check of 159 URLs from the deprecate/remove subset — the pattern was consistent:

Zero genuine fabrications. When the model can’t cite real URLs, the prompt forces sources: [] and confidence ≤ 0.3.

The validator script scripts/spot_check.py reproduces this check at any time. Scaling it across all 3,552 URLs takes ~5 minutes with parallelism.

Cost (top-1000 sweep)

Empirical totals across 858 fresh probes:

Concurrency caveat: this is single-threaded. A 4-way asyncio-gather runner would drop wall-clock to ~4 hours without significantly increasing cost (cache hit rate stays high across parallel calls).

Where the evidence comes from

Tool-call breakdown across the 858 probes:

Despite the lore_search failure rate, the surviving lore_activity

• lore_file_timeline calls produced the strongest evidence in the corpus — both remove verdicts driven by lore_file_timeline discovering active 2026-04-22 removal patch series.

Subsystem coverage

The 858 probed dirs span 122 distinct top-level subsystems:

The remaining 75 subsystems have 1-3 probes each — the long-tail legacy infrastructure (rapidio, ipack, hsi, parport, sbus, ps3, ssb, bcma, isdn, etc.) where deprecation candidates concentrate.

Diminishing returns past rank 500

The deprecate count between top-500 and top-1000 was unchanged at 42. The model found exactly 3 new remove verdicts in ranks 504-858 (caif, hamradio, isdn/mISDN). Signal density is concentrated in the top ~500.

This is the empirical justification for the top-1000 cutoff. Going to top-2000 would burn ~$140 more for at most a few additional deprecates.

Confidence-vs-verdict shape

Among the 9 remove dossiers:

• 5 have confidence ≥ 0.90, all backed by lore patches
• 3 have confidence in 0.83-0.89, backed by mixed lore + web evidence
• 1 has confidence 0.78 (the weakest remove — the model ranked the older mISDN modular driver alongside the stronger isdn/hardware/mISDN evidence)

Among the 43 deprecate dossiers:

• 15 have confidence ≥ 0.80, the strongest tier
• 22 are in 0.70-0.80
• 6 are in 0.60-0.70 (the model is hedging — the dossier explicitly says “deployment is plausible in $niche but evidence is thin”)

Among keep-annotate:

• median confidence ~0.78
• the 401 in this bucket are mostly “old hardware, plausible niche use, no active maintenance, but no strong removal case either”

Independent cross-checks

Five high-leverage verdicts spot-verified by hand:

Zero of five verdicts were overturned. Caveat: spot-checks are not random sampling. The actionable subset (51 drivers) is small enough to be auditable individually before any disclosure.

Re-run reproducibility

The pipeline is reproducible-ish:

• Phase 1 (no LLM) is bit-identical given the same kernel SHA, ref, and since date.
• Phase 2 (LLM) is approximately reproducible — re-running the same prompt with the same model and effort produces verdicts that match in ~95% of cases (rough estimate from spot-checks). Source URLs cited can vary from run to run; the verdict and confidence track closely.

Each meta.json records the model + reasoning effort + kernel SHA used, so corpora built across multiple model versions can be audited.