Medical AI / Model selection
More models.
A better choice?
Testing whether extra target-validation labels improve model selection beyond reusing a strong source-selected recipe.
- Candidate pools
- 2 × 21 recipes
- Target tasks
- 9
- Training trajectories
- 1,476
The question.
Automated model development can produce many candidate recipes. The next problem is choosing one: does collecting additional target-validation labels reliably improve on a recipe selected using source tasks?
Equal training updates.
Different selection rules.
Two AI-generated pools each contained 21 model recipes. The study used three source tasks and nine target tasks. Each candidate was trained using the target training split; selection methods differed in how they used additional validation information.
Target-specific selection was evaluated with 16, 32, 64 and full validation sets, against a strong source-selected recipe with matched training updates per candidate. Training one reused recipe and all 21 candidates required different total computation.
Study design at a glance
- 01
Generate
Two pools of 21 recipes
- 02
Train
Target data; matched updates per candidate
- 03
Select
Source recipe or target-validation choice
- 04
Evaluate
Nine held-out target-task tests
Study findings
Selection leaves much
of the improvement unused.
Compare the mean gain with the fraction of available improvement recovered.
Mean test AUROC difference
95% CI -0.01163 to 0.02159
95% CI -0.00981 to 0.02246
Full target-validation selection versus the source-selected recipe. Mean gains across the nine target tasks; both confidence intervals include zero.
Using only 16 or 32 validation examples reduced average performance in both pools.
What this changes.
Model selection is a practical constraint on what automated development delivers. The availability of better candidates does not ensure that a small validation sample will identify them.
The study separates the value of generating candidates from the value of selecting among them, and examines the value of additional local validation labels.
Manuscript
Under reviewTarget-specific model selection versus source-recipe reuse in medical image classification
Zheyuan Zhang, MSc · Sole author · PhyZio, Oxford, UK
Study conception, methodology, analysis and manuscript preparation.
Manuscript under review. Status updated 29 October 2026.