Evaluation Principles for Life Sciences AI
A biology AI claim is only as strong as the evaluation that produced it. Most disappointments in the field, including ones that have cost programs and shaped public skepticism, trace to evaluations that look rigorous in a paper but do not match the conditions under which the model is then used. The fix is not more metrics; it is a small number of evaluation choices that respect biology rather than treat it as generic tabular data.
Use this chapter to:
- Separate real biological capability from benchmark artifacts, weak splits, leakage, poor calibration, and retrospective fit.
- The evaluation design has to match the biological question: sequence-family splits, donor splits, scaffold splits, tissue splits, time splits, and prospective validation answer different claims.
Summary: Separate real biological capability from benchmark artifacts, weak splits, leakage, poor calibration, and retrospective fit. Blind benchmarks and biology-aware splits are widely accepted in some areas, but many newer model classes still rely on narrow or self-reported evaluations.
Key point: The evaluation design has to match the biological question: sequence-family splits, donor splits, scaffold splits, tissue splits, time splits, and prospective validation answer different claims. Open question: whether prospective, decision-aware validation becomes routine outside a few mature benchmark cultures.
Bottom line: Evaluation is the bridge between all domains in the handbook because every model class eventually has to survive a decision-relevant test.
What is this field trying to solve? Separate real biological capability from benchmark artifacts, weak splits, leakage, poor calibration, and retrospective fit.
What is the core idea? The evaluation design has to match the biological question: sequence-family splits, donor splits, scaffold splits, tissue splits, time splits, and prospective validation answer different claims.
What is the current state of the field? Blind benchmarks and biology-aware splits are widely accepted in some areas, but many newer model classes still rely on narrow or self-reported evaluations.
What do we know, and what remains open? Known reference points include CASP, CAMEO, PoseBusters, MoleculeNet, Therapeutics Data Commons, OpenProblems, scIB, DOME, calibration metrics, and prospective validation studies. What remains open is whether prospective, decision-aware validation becomes routine outside a few mature benchmark cultures.
Why does this matter? Evaluation is the bridge between all domains in the handbook because every model class eventually has to survive a decision-relevant test.
Introduction
Evaluation in life sciences AI must respect biology, not only machine-learning convention. A model can score well on a random hold-out from a public dataset and fail completely the first time it sees an unfamiliar protein family, an unfamiliar scaffold, or an unfamiliar assay protocol. The cost of that failure is not abstract: it is a months-long internal program betting on an external claim, or a regulatory submission whose performance claims fall apart under external audit.
The discipline of evaluation in biology has been worked out over thirty years by specific communities solving specific problems. The structural-biology community built CASP starting in 1994; the docking community built PoseBusters to test physical and chemical validity beyond geometric scores; the molecular-property-prediction community built MoleculeNet to standardise datasets and evaluation; the broader biology-ML community published DOME in 2021 as a reporting checklist. Their shared lesson is not a universal ranking of designs. It is that the split, comparator, validity checks, uncertainty analysis, and prospective evidence must match the claim.
The rest of this chapter walks each principle and then assembles them into an operational checklist. The intent is to give a non-specialist reader, a vendor evaluator, or a program lead a framework that travels across subdomains so the same questions apply whether the model in question is for protein structure, small-molecule generation, single-cell perturbation, or clinical-trial enrolment.
Demonstrated capability
Match each claim to an evidence design
Evaluation starts by stating the claim precisely. A study can be well executed and still be unable to support the conclusion later attached to it. The minimum useful design rises as the claim moves from prediction to biological, operational, or translational benefit.
| Claim | Evidence that can support it | What that evidence does not establish |
|---|---|---|
| A model predicts a defined endpoint on held-out data | Prespecified metric, relevant comparator, biology-aware split, uncertainty, and leakage review | Performance on a different family, scaffold, assay, site, time period, or decision |
| A representation transfers across tasks | Multiple prespecified downstream tasks with matched baselines and evaluation rules | General biological understanding or superiority on untested tasks |
| Generated candidates are physically or chemically plausible | Modality-specific validity checks applied to the complete generated set | Whether they can be made, expressed, bound, functionally validated, used safely, or translated into therapeutic value |
| Model-ranked candidates improve experimental selection | Prospective experiments, full denominator, prespecified hit definition, relevant comparator, failed-run reporting, and uncertainty | Reproducibility across targets, laboratories, assay systems, or later development stages |
| A tool improves a research workflow | Comparative workflow study with configured-system details, time and quality endpoints, failure recovery, and user context | Scientific or clinical benefit unless those outcomes were measured |
| A system improves translational or clinical outcomes | Prospective comparative evaluation with patient- or program-relevant endpoints and appropriate oversight | Transport to populations, institutions, or indications outside the evaluated setting |
Evidence should be read at the level of the measured endpoint. Retrospective discrimination is not prospective utility; a prospective workflow improvement is not automatically a biological or clinical benefit; and experimental validation of selected candidates does not establish the performance of every generated candidate.
Blind benchmarks: the credibility floor
A blind benchmark withholds evaluation targets during method development or before a prediction deadline, reducing opportunities to tune directly on the test cases. The structural-biology community institutionalised this in 1994 with the Critical Assessment of protein Structure Prediction (Kryshtafovych et al., 2024). CASP runs biennially, uses newly solved targets, employs independent assessment, and publishes category-specific results. Blinding strengthens the test, but benchmark design, target selection, participation, and reporting still require scrutiny.
CAMEO (CAMEO, 2026) extends the idea to continuous evaluation: each week, predictors submit models for newly released PDB structures, and results are scored as the experimental structures appear. Continuous evaluation catches degradation and regression that a biennial format misses.
PoseBusters (Buttenschoen et al., 2024) is the docking community’s equivalent: not only a new benchmark, but a set of physical and chemical validity checks (bond lengths, stereochemistry, intermolecular clashes, ring geometry) that surface failures geometric metrics miss. The PoseBusters paper documented that a substantial share of low-RMSD docking poses from leading systems failed at least one basic validity check.
The credibility lesson is that capability claims supported only by self-reported test-set numbers sit several rungs below blind-benchmark results. The handbook’s Benchmarks for Bio AI chapter catalogues the blind benchmarks that exist in each subdomain.
Biology-aware splits
The single most common evaluation failure in life sciences ML is using a random train/test split when a biology-aware split would be appropriate. Random splits assume independent and identically distributed examples; biological data are rarely either.
Scaffold splits group molecules by their Bemis-Murcko scaffold and assign whole scaffolds to either train or test. MoleculeNet helped standardize their use in molecular-property benchmarks (Wu et al., 2018). Scaffold splits reduce close-analogue overlap but do not guarantee transport to novel chemistry, assays, or development programs.
This is not only a cheminformatics convention. Wallach and Heifets showed that many ligand-based classification benchmarks favor models that recognize chemotypes already present in training rather than models that generalize to new chemical series (Wallach and Heifets, 2018). Any claim about compound generalization should therefore disclose the split axis and the nearest-neighbor relationship between test compounds and the training set.
Time splits assign examples by date, with older data in training and newer data in test. They can approximate temporal transport to newer compounds or measurements, but they do not reproduce prospective deployment, changing assay practice, or model-guided selection.
Sequence-family or homology splits assign related protein sequences to the same partition or cap similarity across partitions. A 30% sequence-identity threshold is used in some settings, but the appropriate clustering rule depends on the biological task, sequence length, domain architecture, and intended use. Random splits can place close relatives in training and test and therefore overstate performance on remote sequences.
A 2026 preprint analyzing 41 viral and 33 cellular deep-mutational-scanning datasets found that pooled mutation splits allowed supervised protein-language-model predictors to exploit site effects, inflating apparent generalization; site-aware splits reduced performance, and a site-mean baseline matched or outperformed supervised models on many datasets (Vieira et al., 2026, preprint). The result is specific to mutational-effect prediction on existing single-substitution datasets and does not show that protein language models generally fail on viral biology.
Cell-line splits, target splits, and assay splits apply the same logic for cellular models, target-based drug discovery, and high-throughput screens: whichever biological axis the deployed model must generalise across should be the axis it has not seen at training time.
The operational rule: pick the split that matches the distribution the model will face in deployment. A model evaluated under a strictly weaker split has not been evaluated for its intended use; it has been evaluated for an easier version of its intended use.
Evaluate the configured system
Biology workflows rarely deploy an untouched base model. They add prompts, retrieval, post-training, adapters, tool calls, preprocessing, filters, confidence thresholds, and human review. The evaluation unit is the configured system that produces the decision-relevant output. A base-model benchmark cannot establish the performance of a later fine-tuned or tool-using workflow.
For post-trained systems, preserve the base-model identifier, training-data lineage and permissions, filtering and deduplication, objective, checkpoints, prompt or tool configuration, and evaluation-set separation. Re-run the central tests after any material change. This matters when a system is described by a model-family name even though its behavior depends on local data or orchestration.
Validity checks beyond geometric metrics
Geometric and statistical metrics are necessary but not sufficient. PoseBusters showed this for docking. The same lesson applies across the field.
Physical and chemical validity for docked poses, generated molecules, and predicted structures: realistic bond lengths, valid stereochemistry, no severe clashes, reasonable ring geometry.
Biological plausibility for generated sequences and structures: sequence constraints, host-expression assumptions, construct boundaries, model confidence, and predicted biophysical liabilities are stated before experimental testing.
Manufacturability and synthetic accessibility for designed molecules: computational flags are reviewed against route feasibility, reagents, stereochemistry, purification, scale, safety, cost, and medicinal-chemistry judgment.
GuacaMol (Brown et al., 2019) extended this principle into the generative-molecule literature with a multi-objective benchmark covering distribution-learning quality, goal-directed generation, and several validity dimensions; goal-directed scores that ignore validity collapse to optimising whatever metric the generator can game.
For claims about valid structures, poses, generated molecules, or phenotypes, a benchmark without modality-appropriate validity checks is incomplete.
Calibration and uncertainty
A model that reports its uncertainty correctly is often more useful than a slightly more accurate model that does not. Calibration is the property that, when a model reports 80% confidence, it is right roughly 80% of the time. Modern deep models in biology are often poorly calibrated by default and require explicit calibration (Platt scaling, isotonic regression, temperature scaling, or conformal prediction) before their confidence outputs are trustworthy.
The structure-prediction community made this concrete with per-residue and pairwise confidence outputs. AlphaFold 2’s pLDDT and predicted aligned error (Jumper et al., 2021) and AlphaFold 3’s confidence outputs (Abramson et al., 2024) help identify locally confident and uncertain regions. Confidence is model-specific and does not validate side chains, interfaces, ligands, dynamics, or function.
For decisions that use model scores as probabilities or confidence estimates, point-prediction metrics alone are insufficient. Calibration, uncertainty, and failure-mode analyses should be selected for the intended decision and data-generating process.
Statistical uncertainty in benchmark scores
Calibration addresses a model’s per-prediction confidence. A separate question is uncertainty in the benchmark score itself: accuracy, AUC, or hit rate is estimated from a finite test set and is not a fixed property of the model. A 2024 preprint describes standard errors, paired comparisons, clustering for correlated test cases, and power analysis for model evaluations (Miller, 2024, preprint). The uncertainty method must match the sampling and dependence structure; repeated variants from one protein, compounds from one scaffold, or targets from one assay family should not be treated as independent by default.
Prospective experimental validation
When the claim is that a model improves experimental selection, running the selected experiments is the most direct validation. Retrospective evaluation cannot fully reproduce the selection process, resource constraints, assay failures, or adaptations that occur after predictions enter a workflow.
This matters most for generative methods and for active-learning loops. A generator that produces 1,000 candidate binders is interesting only insofar as some non-trivial fraction validate experimentally. The RFdiffusion programme (treated in detail in the Protein Design and Engineering chapter) is methodologically notable in part because the papers reported experimental hit rates, not just in-silico scores.
For variant interpretation, the analogue is functional validation. AlphaMissense (Cheng et al., 2023) is positioned as a research tool partly because the large-scale functional evidence to back its classifications at clinical-decision quality does not yet exist for most variants; using it in care without that evidence would be premature.
For an AI system that drives experimental decisions, ask for prospective evidence, a prespecified comparator, denominators, hit definition, failed runs, cost, and uncertainty. A hit rate without those elements can still mislead.
Reporting discipline: DOME and adjacent guidance
The DOME (Data, Optimization, Model, Evaluation) recommendations (Walsh et al., 2021) provide a useful checklist for supervised-ML reporting in biology. A report using DOME addresses:
- Data: where the data came from, how it was preprocessed, how it was split, what was held out
- Optimization: how hyperparameters were chosen, what was searched, what budget was used
- Model: the architecture, the training objective, the loss function, the regularisation
- Evaluation: which metrics, which baselines, which significance tests, which failure modes are surfaced
A paper that omits material data, optimization, model, or evaluation details is difficult to appraise. The Reproducibility and Open Science chapter treats the institutional layer, including access to code, weights, data, model cards, and protocol records. DOME is one reporting aid at the study level.
No checklist substitutes for domain-specific appraisal. Use DOME where it fits, and require enough detail to reproduce and interpret the claim even when a study follows different reporting guidance.
Summary table
| Evidence Anchor | What It Supports | Practical Constraint |
|---|---|---|
| CASP | Blinded community assessment for structure prediction | Targets and categories change across rounds |
| CAMEO | Continuous server evaluation | Automated evaluation depends on target release and criteria |
| PoseBusters | Physical and chemical plausibility for docking | RMSD-only evaluation rewards incomplete success |
| MoleculeNet | Standard scaffold-split benchmarks for molecular property prediction | Datasets age; absolute numbers are not always comparable across years |
| GuacaMol | Multi-objective generative-molecule evaluation | Goal-directed metrics can be gamed if validity is not enforced |
| DOME | Reporting checklist for supervised ML in biology | Not a universal professional standard |
| Prospective wet-lab validation | Direct test of model-guided experiment selection | Needs a prespecified comparator, denominators, and failure reporting |
Theoretical capability
Several evaluation regimes are plausible but not yet routine, and would change the field’s quality bar if adopted.
Cost-aware evaluation. Current benchmarks rank methods by accuracy or by a single quality metric. A more useful evaluation would rank methods by expected discovery yield per dollar, integrating model accuracy, prospective hit rate, and the cost of the experimental steps the model selects. This is the metric an R&D programme actually cares about; it is rarely the metric a paper reports.
Cross-modality benchmarks. A model that simultaneously handles sequence, structure, and small-molecule chemistry should be evaluated on tasks that require all three. The benchmark suite for biology-wide foundation models is still emerging; current evaluations mostly stitch together per-modality benchmarks rather than testing integrated reasoning.
Active-learning benchmarks. Many real workflows use the model to choose the next experiment, retrain, repeat. Benchmarks that evaluate the full loop, not just the model in isolation, would better reflect deployment. The single-cell perturbation literature is closer to this than most subfields, and the Perturbation Prediction and Virtual Cells chapter discusses the current state.
External-validation registries. A registry that connects vendor-reported performance to independently reproduced results could improve comparison across settings. Its governance, incentives, scope, and maintenance remain design questions.
An evaluation regime is plausible only when the data, comparator, governance, resources, and analysis are specified. The gaps above combine technical, organizational, economic, and incentive constraints.
Beyond current capability
Two evaluation goals remain beyond current capabilities.
A universal score across all of biology. Structure prediction, compound screening, cellular response, and clinical translation require different ground truth and different error costs. A model that excels on CASP targets says nothing about its cellular-perturbation performance, and a model that excels at single-cell perturbation prediction says nothing about its docking accuracy. The dream of a single benchmark that orders methods across the entire field is incompatible with the heterogeneity of biological evaluation.
Fully simulated deployment-equivalence evaluation. Even the best retrospective evaluation is a model of deployment, not deployment itself. A simulator that fully predicts how a method will behave in a real laboratory, with real assay variability, real reagent batches, and real human handlers, would be tantamount to solving the underlying biology. It is the kind of capability that, if it existed, would itself transform the field.
The rule: claims that promise either of these should be treated with deep skepticism. They are conceptually beyond what the current generation of methods and infrastructure can support, regardless of training compute.
Evidence that would change the assessment
Evaluation practice becomes more promising if the field adopted prospective, cost-aware, and externally reproduced assessments as routine reporting rather than special studies. Better evidence would include registries that compare vendor-reported performance with independent reproduction, active-learning benchmarks that score the full experiment-selection loop, and calibration reports tied to the decision the model supports. A universal biology score would require evidence that one metric predicts deployment behavior across molecules, structures, cells, organisms, and translation, and that evidence does not currently exist.
Implications for research and program decisions
For practitioners evaluating any AI system in the life sciences:
- Match the benchmark to the claim. CASP, CAMEO, PoseBusters, MoleculeNet, and GuacaMol test different objects and questions. Ask what the selected benchmark establishes, what it does not, and whether the test distribution matches the intended use.
- Match the split to the use. Ask which biology-aware split was used (scaffold, time, sequence-family, cell-line, target, assay) and whether the deployment distribution looks like the training distribution.
- Require validity checks, not only geometric metrics. For docking, PoseBusters-style physical and chemical validity. For generative molecules, synthetic accessibility and goal-aligned validity. For predicted structures, confidence-aware geometry, not raw RMSD.
- Read the calibration plot when scores drive thresholds or prioritisation. A confidence score should be evaluated against the decision it informs.
- Require prospective validation for claims about model-guided experimental yield. Report the comparator, denominator, hit definition, failed runs, costs, and uncertainty.
- Apply DOME or appropriate domain guidance. Require enough information about data, optimization, model, and evaluation to appraise and reproduce the central claim.
- Look for failure-mode breakdowns. Predictable failure (in disordered regions, novel scaffolds, induced fit, large complexes) is a workable constraint to design around. Unreported failure modes are not; they surface as deployment surprises.
- Treat vendor performance figures with the same standard as published ones. Performance reported in a slide deck, marketing site, or webinar is not stronger than performance reported in a paper; usually it is weaker because there is no version of the manuscript to scrutinise.
- Verify by independent reproduction wherever possible. A capability is real to the field when an independent group reproduces it. The Benchmarks for Bio AI and Reproducibility and Open Science chapters detail what to look for.