Reproducibility and Open Science
Reproducibility in AI-biology has two linked meanings: computational reproducibility (code, weights, data, random seed) and experimental reproducibility (protocols, reagents, instruments, operators, environment). A notebook that reruns is necessary but not sufficient if the assay cannot be repeated. DOME is a community reporting checklist for supervised machine learning in biology (Walsh et al., 2021); model cards and dataset cards provide additional documentation layers. Bridge2AI and ARPA-H IGoR provide federal infrastructure examples. AlphaFold 3’s initial restricted release, its November 2024 academic code-and-weights release, and separately developed Boltz and Chai models illustrate distinct reproducibility and licensing choices.
Use this chapter to:
- Make AI-biology work repeatable across code, data, weights, protocols, assays, instruments, and wet-lab execution.
- Computational reproducibility and experimental reproducibility are different requirements and both matter for life sciences AI.
Prerequisites: Biological Data Infrastructure for the data layer; Evaluation Principles for Life Sciences AI for DOME context.
Summary: Make AI-biology work repeatable across code, data, weights, protocols, assays, instruments, and wet-lab execution. Open code, model cards, dataset cards, and benchmark reports have improved, but restricted models and incomplete experimental protocols remain common barriers.
Key point: Computational reproducibility and experimental reproducibility are different requirements and both matter for life sciences AI. Open question: whether released artifacts allow independent computational reruns and independent experimental checks.
Bottom line: Reproducibility links every chapter because confidence in AI biology depends on whether others can rerun, inspect, and experimentally test the work.
What is this field trying to solve? Make AI-biology work repeatable across code, data, weights, protocols, assays, instruments, and wet-lab execution.
What is the core idea? Computational reproducibility and experimental reproducibility are different requirements and both matter for life sciences AI.
What is the current state of the field? Open code, model cards, dataset cards, and benchmark reports have improved, but restricted models and incomplete experimental protocols remain common barriers.
What do we know, and what remains open? Known reference points include DOME, model cards, dataset cards, GitHub releases, Zenodo, protocols.io, AlphaFold open-source responses, OpenFold, Boltz, Chai, and data repositories. What remains open is whether released artifacts allow independent computational reruns and independent experimental checks.
Why does this matter? Reproducibility links every chapter because confidence in AI biology depends on whether others can rerun, inspect, and experimentally test the work.
Introduction
Bridge2AI and IGoR both point toward data and protocol infrastructure as prerequisites for AI-supported science (NIH Common Fund Bridge2AI, 2026; ARPA-H IGoR, 2026). Open models such as Boltz-1 (Wohlwend et al., 2024, preprint) show demand for inspectable biomolecular modelling stacks. The DOME framework (Walsh et al., 2021) is a useful reporting checklist, not a universal professional standard.
Demonstrated capability
Demonstrated capability includes public datasets, open-source model releases, protocol repositories, and reproducible benchmark scripts. AlphaFold DB and the ESM Metagenomic Atlas demonstrate the research value of large public predicted-structure resources (AlphaFold Protein Structure Database, 2026; ESM Metagenomic Atlas, 2026). Boltz-1 and Chai-1 provided separately developed, inspectable biomolecular interaction models with reported benchmark performance (Wohlwend et al., 2024, preprint; Chai Discovery et al., 2024, preprint). Their preprint benchmarks do not by themselves establish full reproduction of AlphaFold 3 across use cases. DOME provides a community reporting checklist (Walsh et al., 2021).
Documentation standards fill different gaps. Model cards describe intended use, evaluation, limitations, and ethical considerations for a trained model (Mitchell et al., 2019). Dataset datasheets document provenance, collection methods, recommended uses, and limitations for training and evaluation data (Gebru et al., 2021). FAIR principles address whether data can be found, accessed, combined, and reused (Wilkinson et al., 2016). None of these substitutes for wet-lab reproducibility, but together they make computational claims auditable.
A 2026 Nature Methods Perspective mapped open and sustainable AI recommendations across the life-sciences AI ecosystem, linking reproducibility, reusability, model and data documentation, archival infrastructure, and environmental considerations (Farrell et al., 2026). Its contribution is a structured implementation resource, not evidence that following a checklist guarantees reproducible science. Openness has to be specified at the level of data, code, weights, environment, documentation, and permitted use; a system can be open in one dimension and restricted in another.
| Evidence Anchor | What It Supports | Practical Constraint |
|---|---|---|
| AlphaFold DB | Public predicted protein structure resource | Predictions require confidence-aware use |
| Boltz-1 and Chai-1 | Open biomolecular interaction models | Open release does not replace independent validation |
| ESM Metagenomic Atlas | Open predicted metagenomic structures | Predicted structure does not imply organismal function |
| DOME (Walsh 2021) | Community reporting checklist for supervised ML in biology | Not a universal professional standard |
| OSAI (Farrell 2026) | Open and sustainable AI recommendations and ecosystem map | Guidance requires workload- and institution-specific implementation |
| Bridge2AI / ARPA-H IGoR | Institutional infrastructure for reproducible AI-biology | Programmatic, takes years to mature |
Theoretical capability
Theoretical capability includes reproducible research networks where protocols, agents, data, and instruments share interoperable records. That goal requires common schemas, incentives, and institutional support. Cross-laboratory reproducibility for AI-assisted experimental work is partial; cloud-lab infrastructure improves the situation for chemistry but biology lags. Production-grade cross-institutional reproducibility is future work.
Beyond current capability
Beyond current capabilities includes fully reproducible biology from computational artefacts alone. Physical experiments require materials, instruments, environments, and local expertise that cannot be fully captured in computational artefacts.
Evidence that would change the assessment
Reproducibility claims become more promising when released artifacts allow independent groups to rerun the central computational claim and repeat the relevant experimental measurement with comparable conclusions. Stronger evidence would include failed-run records, reagent and instrument context, and cross-laboratory protocol execution, not only open code.
Implications for research and program decisions
- Release code, model version, data version, and filtering logic when possible.
- Archive protocol details, reagent lots, instrument settings, and failed runs.
- Use model cards and dataset cards for external readers.
- Use access controls when openness conflicts with privacy, contracts, or safety.
- Consider DOME when reporting supervised machine-learning studies in biology, while following journal- and domain-specific requirements.
- For restricted releases, document the explicit reasoning and reproducibility-vs-risk tradeoff.