Appendix C — Case Studies
Short applied cases for interpreting life sciences AI claims. Each case maps a published result or capability claim to the three-tier evidence framework (Demonstrated, Theoretical, Beyond Current Capabilities) and the validation discipline the handbook applies elsewhere in detail. Cross-references to chapters in parentheses.
Case 1: Structure Prediction as a Research Input
A team without an experimental structure uses AlphaFold DB to generate hypotheses about a protein domain. The correct use is hypothesis generation, followed by confidence review (pLDDT distribution, PAE matrix), domain inspection, and assay planning. The high-pLDDT regions are usable for docking and mutagenesis design; the low-pLDDT regions are predictions of disorder and should be treated as such.
The failure mode this case avoids: treating the structure visualization as if it were a measured structure. AlphaFold confidence metrics are decision aids; reading the visualization without reading the confidence is misuse.
References: AlphaFold Protein Structure Database, 2026; Jumper et al., 2021. See Protein Structure Prediction chapter.
Case 2: Protein Binder Design
A protein engineering group uses RFdiffusion for backbone proposals and ProteinMPNN for sequence design. The design is not a success until expression, binding, specificity, and stability data support it. In-silico filtering (per-residue pLDDT distribution, predicted developability, basic biophysical filters) reduces the candidate set before bench testing.
Evidence tier placement: backbone generation and inverse-folding sequence design are Demonstrated. Function on a specific target is Theoretical until the assay confirms it.
References: Watson et al., 2023; Dauparas et al., 2022; Dauparas et al., 2025. See Protein Design and Engineering chapter.
Case 3: Docking Benchmark Failure
A docking method looks strong by RMSD but generates physically implausible poses. PoseBusters shows why chemical validity checks belong beside geometric metrics: bond lengths, stereochemistry, severe clashes, ring geometry. Many low-RMSD deep-learning docking poses fail at least one basic validity check.
The lesson generalises: a single metric can hide invalid outputs in many AI domains. Validity filtering is part of the evaluation, not an optional add-on.
References: Buttenschoen et al., 2024. See Small Molecule Generation and ADMET, and Translational Evidence and Failure Modes chapters.
Case 4: Single-Cell Perturbation Forecasting
A group uses GEARS to rank perturbations for a follow-up experiment. The result supports prioritisation, not general causal certainty. The Ahlmann-Eltze 2025 Nature Methods critique shows that for many perturbation-prediction tasks, PCA plus linear regression matches or beats deep methods. The right discipline is always to run the linear baseline first.
Evidence tier placement: GEARS-style perturbation prediction is Demonstrated for bounded settings (in-distribution gene pairs); generalisation to novel-context settings is Theoretical or weaker.
References: Roohani et al., 2024; Ahlmann-Eltze et al., 2025. See Perturbation Prediction and Virtual Cells chapter.
Case 5: Closed-Loop Experimentation in Chemistry
A self-driving lab chooses the next experiment based on prior measurements. Coscientist uses GPT-4 as planner on cloud robotic platforms and reproduces Pd-catalysed cross-coupling chemistry. The system needs protocol versioning, instrument logs, objective functions, and stopping rules. Counting loop iterations is the right discipline: a platform that runs once is automation, not autonomy.
Evidence tier placement: closed-loop optimisation for bounded chemistry tasks is Demonstrated. Open-ended autonomous biology beyond bounded settings is Beyond Current Capabilities.
References: Boiko et al., 2023; Burger et al., 2020. See Self-Driving Laboratories chapter.
Case 6: Multi-Agent Biology Research
Virtual Lab (Stanford) coordinates a PI agent and specialist agents to design 92 SARS-CoV-2 nanobodies. Two designs showed improved binding to JN.1 and KP.3 variants while retaining ancestral spike binding in experimental validation. The contribution is the multi-agent workflow with experimental followthrough, not the absolute binding numbers.
Evidence tier placement: multi-agent biology research with wet-lab validation for one target is Demonstrated. Generalisation across biology classes is Theoretical.
References: Swanson et al., 2025. See Agentic Science Workflows chapter.
Case 7: A-Lab and the Novelty-Claim Discipline
A-Lab reported 41 new inorganic materials autonomously produced in 17 days. A PRX Energy critique from solid-state chemists argued that the novelty claim did not hold because many entries were misclassified or were known phases. The episode is a clear case study of how autonomous-discovery novelty claims can outrun validation: throughput is not discovery.
Lesson: automated characterisation can be wrong in characteristic ways; independent expert review is part of validation; novelty claims should pass scrutiny from the relevant subdiscipline before being treated as discoveries.
References: Szymanski et al., 2023; Leeman et al., 2024. See Self-Driving Laboratories chapter.
Case 8: AI-Discovered Drug in Phase IIa
Insilico Medicine’s ISM001-055 (rentosertib) is an AI-discovered TNIK inhibitor for idiopathic pulmonary fibrosis. The peer-reviewed Nature Biotechnology paper covers the discovery and Phase I narrative. Positive Phase IIa topline was reported in November 2024 by the company. Phase IIa is a meaningful signal that the compound has measurable activity in the indication; it is not registration-grade and not yet peer-reviewed at Phase IIa quality.
Evidence tier placement: an AI-discovered candidate reaching Phase IIa is Demonstrated for this specific development path. Clinical success at registration scale is Theoretical for this specific candidate and Beyond Current Capabilities as a general AI-discovered-drug claim.
References: Ren et al., 2025; Insilico Medicine, November 2024. See Small Molecule Generation and ADMET, Translational Evidence and Failure Modes chapters.
Case 9: Variant Interpretation Inside the ACMG Framework
A clinical laboratory adds AlphaMissense outputs to its variant-interpretation pipeline. The correct use is as PP3 (computational evidence of pathogenicity) or BP4 (computational evidence of benignity) input alongside population data, functional studies, and segregation. The score is not a classification; the framework is.
Lesson: variant predictors are evidence inputs into the ACMG/AMP framework, not substitutes for it. Per-laboratory validation of score thresholds is mandatory before clinical use.
References: Cheng et al., 2023; Richards et al., 2015. See Variant Effect Prediction chapter.
Case 10: Single-Cell Foundation Model in Practice
A research group uses scGPT for cell-type annotation and considers extending it to perturbation prediction. For annotation, the model is Demonstrated. For perturbation, the published evidence is contested by Ahlmann-Eltze 2025: deep methods do not consistently beat PCA plus linear regression. The right discipline is to use scGPT for what it is demonstrated to do (representation, integration, atlas search) and run a linear baseline before adopting it for perturbation prediction.
References: Cui et al., 2024; Ahlmann-Eltze et al., 2025. See Single-Cell Foundation Models chapter.
Case 11: AlphaFold 3 Restricted Release and Open-Source Response
AlphaFold 3 was published in Nature in May 2024 but released as a server with rate limits and no training code. The community responded with open-source AF3-class models: Boltz-1 (MIT, November 2024) and Chai-1 (October 2024). For research programs that need on-premise inference, batch screening at scale, or commercial-use rights, the open-source alternatives are the practical implementations.
Lesson: licensing and code availability are first-class evaluation criteria, not afterthoughts. A model that cannot run in your environment is not usable for your program.
References: Abramson et al., 2024; Wohlwend et al., 2024, preprint; Chai Discovery et al., 2024, preprint. See Protein Structure Prediction, Reproducibility and Open Science chapters.