Medical AI / Threshold calibration
A stable threshold.
A useful operating point?
Testing how positive-case availability shapes local threshold reproducibility in breast ultrasound and chest radiography.
- Breast panel size
- 250 patients
- Controlled positive counts
- 25 / 50 / 100
- Panels per design
- 1,000
The question.
A service needs an operating threshold, not only a model score. When local positive cases are scarce, how much does their number affect the reproducibility of that threshold—and does greater stability translate into a useful sensitivity and alert burden?
Hold the model fixed.
Vary the local evidence.
The study held breast-ultrasound and chest-radiograph scorers fixed. A controlled breast-ultrasound experiment compared panels of 250 patients containing 25, 50 or 100 positive cases.
Thresholds targeting 95% empirical sensitivity were evaluated on fixed held-out cohorts across 1,000 panels per design. Sampling with and without replacement separated positive-count effects from finite-pool sampling effects. Pneumothorax experiments compared natural sampling with positive-case enrichment.
Study design at a glance
- 01
Fix the scorer
Breast ultrasound and chest radiography
- 02
Sample locally
Control positive cases and panel size
- 03
Choose a cutoff
Target 95% empirical sensitivity
- 04
Evaluate
Held-out stability, sensitivity and alerts
Study findings
More positive cases.
Less threshold variation.
Two settings show why the composition of a calibration sample matters.
Breast panels contain 250 patients, sampled without replacement for these results. With replacement, increasing positives from 25 to 100 narrowed the width from 15.79 to 12.28 percentage points.
At the same 250-examination panel size, natural and enriched sampling gave ranges of 59.8–100.0% and 93.9–97.6%, respectively.
Does the sampling design change the pattern?
Without replacement
With replacement
Sensitivity-range width · percentage points
Breast-ultrasound panels of 250 patients. Increasing positive cases narrowed the 95% resampling range under both sampling designs; sampling without replacement produced a larger reduction.
Stability is one part
of the decision.
Across six chest-radiograph models, attainment of the 95% sensitivity target in an external cohort ranged from 20.1% to 99.5%. A reproducible threshold did not guarantee that its target would transfer.
In the breast-ultrasound evaluation, the full-calibration cutoff detected 168 of 171 malignancies while flagging 287 of 361 benign lesions. Sensitivity and the resulting flagging workload must be read together.
Manuscript
Under reviewLocal Threshold Calibration for Imaging AI with Limited Positive Cases
Zheyuan Zhang, MSc · Sole author · PhyZio, Oxford, UK
Study conception, methodology, analysis and manuscript preparation.
Manuscript under review. Status updated 29 October 2026.