Reproducibility and Open Science
Reproducibility in AI-biology has two linked meanings: computational reproducibility (code, weights, data, random seed) and experimental reproducibility (protocols, reagents, instruments, operators, environment). A notebook that reruns is necessary but not sufficient if the assay cannot be repeated. The DOME framework (Walsh et al., Nat Methods 2021) is the per-paper reporting standard; model cards and dataset cards provide the documentation layer; Bridge2AI and ARPA-H IGoR provide the institutional infrastructure. The AlphaFold 3 restricted release and the Boltz/Chai open-source response illustrate that open-source releases can fill capability gaps when the field judges restrictions excessive.
Use this chapter to:
- Make AI-biology work repeatable across code, data, weights, protocols, assays, instruments, and wet-lab execution.
- Computational reproducibility and experimental reproducibility are different requirements and both matter for life sciences AI.
Prerequisites: Biological Data Infrastructure for the data layer; Evaluation Principles for Life Sciences AI for DOME context.
Summary: Make AI-biology work repeatable across code, data, weights, protocols, assays, instruments, and wet-lab execution. Open code, model cards, dataset cards, and benchmark reports have improved, but restricted models and incomplete experimental protocols remain common barriers.
Key point: Computational reproducibility and experimental reproducibility are different requirements and both matter for life sciences AI. Open question: whether released artifacts allow independent computational reruns and independent experimental checks.
Bottom line: Reproducibility links every chapter because confidence in AI biology depends on whether others can rerun, inspect, and experimentally test the work.
What is this field trying to solve? Make AI-biology work repeatable across code, data, weights, protocols, assays, instruments, and wet-lab execution.
What is the core idea? Computational reproducibility and experimental reproducibility are different requirements and both matter for life sciences AI.
What is the current state of the field? Open code, model cards, dataset cards, and benchmark reports have improved, but restricted models and incomplete experimental protocols remain common barriers.
What do we know, and what remains open? Known reference points include DOME, model cards, dataset cards, GitHub releases, Zenodo, protocols.io, AlphaFold open-source responses, OpenFold, Boltz, Chai, and data repositories. What remains open is whether released artifacts allow independent computational reruns and independent experimental checks.
Why does this matter? Reproducibility links every chapter because confidence in AI biology depends on whether others can rerun, inspect, and experimentally test the work.
Introduction
Bridge2AI and IGoR both point toward data and protocol infrastructure as prerequisites for AI-supported science (NIH Common Fund Bridge2AI, 2026; ARPA-H IGoR, 2026). Open models such as Boltz-1 (Wohlwend et al., 2024, preprint) show the pressure for transparent biomolecular modelling stacks. The DOME framework (Walsh et al., 2021) is the per-paper reporting standard the field has converged on.
What is demonstrated?
Demonstrated capability includes public datasets, open-source model releases, protocol repositories, and reproducible benchmark scripts. AlphaFold DB and the ESM Metagenomic Atlas demonstrate the research value of large public predicted-structure resources (AlphaFold Protein Structure Database, 2026; ESM Metagenomic Atlas, 2026). The Boltz-1 and Chai-1 releases demonstrated that open releases can reproduce similar capability when restricted releases occur (Wohlwend et al., 2024, preprint; Chai Discovery et al., 2024, preprint). DOME provides a community baseline for reporting (Walsh et al., 2021).
Documentation standards fill different gaps. Model cards describe intended use, evaluation, limitations, and ethical considerations for a trained model (Mitchell et al., 2019). Dataset datasheets document provenance, collection methods, recommended uses, and limitations for training and evaluation data (Gebru et al., 2021). FAIR principles address whether data can be found, accessed, combined, and reused (Wilkinson et al., 2016). None of these substitutes for wet-lab reproducibility, but together they make computational claims auditable.
| Evidence Anchor | What It Supports | Practical Constraint |
|---|---|---|
| AlphaFold DB | Public predicted protein structure resource | Predictions require confidence-aware use |
| Boltz-1 and Chai-1 | Open biomolecular interaction models | Open release does not replace independent validation |
| ESM Metagenomic Atlas | Open predicted metagenomic structures | Predicted structure does not imply organismal function |
| DOME (Walsh 2021) | Per-paper reproducibility reporting standard | Adoption is uneven across journals |
| Bridge2AI / ARPA-H IGoR | Institutional infrastructure for reproducible AI-biology | Programmatic, takes years to mature |
What is theoretical?
Theoretical capability includes reproducible research networks where protocols, agents, data, and instruments share interoperable records. That goal requires common schemas, incentives, and institutional support. Cross-laboratory reproducibility for AI-assisted experimental work is partial; cloud-lab infrastructure improves the situation for chemistry but biology lags. Production-grade cross-institutional reproducibility is future work.
What is beyond current capability?
Beyond current capabilities includes fully reproducible biology from computational artefacts alone. Physical experiments require materials, instruments, environments, and local expertise that cannot be fully captured in computational artefacts.
What would make this more promising?
Reproducibility claims become more promising when released artifacts allow independent groups to rerun the central computational claim and repeat the relevant experimental measurement with comparable conclusions. Stronger evidence would include failed-run records, reagent and instrument context, and cross-laboratory protocol execution, not only open code.
What should researchers, biotech teams, funders, and program leaders do with this?
- Release code, model version, data version, and filtering logic when possible.
- Archive protocol details, reagent lots, instrument settings, and failed runs.
- Use model cards and dataset cards for external readers.
- Use access controls when openness conflicts with privacy, contracts, or safety.
- Cite DOME (Walsh 2021) explicitly in methods sections.
- For restricted releases, document the explicit reasoning and reproducibility-vs-risk tradeoff.