flowchart TD
A["Name the user, input,<br/>output, and decision"] --> B["Freeze data, versions,<br/>comparator, and endpoint"]
B --> C["Run representative cases<br/>under the same rules"]
C --> D["Record outputs, abstentions,<br/>failures, and interventions"]
D --> E["Assess uncertainty, validity,<br/>transport, cost, and burden"]
E --> F{"Prespecified acceptance<br/>and stop rules met?"}
F -- "Yes" --> G["Adopt for the tested use<br/>and schedule monitoring"]
F -- "Partly" --> H["Bound use or revise<br/>and retest"]
F -- "No" --> I["Reject and preserve<br/>the result"]
Practical Evaluation Workflows and Worksheets
Evaluation becomes useful when a team can repeat it, inspect failures, and connect the result to a decision. Short workflows and downloadable records make the handbook’s evidence principles operational. The files are plain CSV so they can be used in a spreadsheet, version-controlled repository, electronic laboratory notebook, or governed data system without a new software dependency.
Learning Objectives
Use this chapter to:
- Freeze an intended use, comparator, evaluation set, acceptance rule, and stopping rule before running a candidate system.
- Preserve the evidence chain from model input through human review, downstream decision, failure analysis, and final disposition.
- Apply task-specific records to single-cell models, laboratory loops, clinical-trial AI, and tool adoption.
Chapter Summary (TL;DR)
Summary: A credible evaluation is a decision record, not a leaderboard screenshot. Write the intended use first, freeze the comparison, record every failure and intervention, then choose adopt, bounded use, revise and retest, or reject.
Key framework: Decision, claim, data, comparator, endpoint, uncertainty, failures, consequence, and stop rule.
Critical takeaways: Use paired cases where possible. Keep full denominators. Separate immediate model performance from downstream scientific or operational value. Preserve versions and evidence links. Do not extend a result beyond the task and distribution tested.
Introduction
The handbook’s chapters describe what current evidence supports. A local adoption decision requires one additional layer: a record of the exact configured system, cases, comparator, failures, and consequence. Without that record, later teams cannot tell whether a result applies to their workload or whether an apparent gain came from a changed dataset, model, prompt, assay, or reviewer.
The workflow is deliberately small. It does not replace a statistical analysis plan, validation protocol, quality-management system, institutional review, or regulator-facing documentation when those are required. It provides the minimum information environment from which those artifacts can be built.
The universal decision record
Every evaluation should preserve the same chain before domain-specific fields are added.
The protocol should be frozen before candidate outputs are reviewed. If an endpoint, threshold, or exclusion changes after results are known, preserve the original analysis and label the revised analysis exploratory.
Downloadable worksheets
| Worksheet | Use | Download |
|---|---|---|
| AI evaluation record | General claim, split, comparator, endpoint, uncertainty, failure, and disposition record | Download CSV |
| Tool-adoption record | Compare a candidate with the current workflow, including cost, integration, reproducibility, and exit | Download CSV |
| Single-cell evaluation | Preserve donor, tissue, assay, split, baseline, failure-stratum, and adoption-boundary fields | Download CSV |
| Self-driving laboratory loop log | Distinguish a closed loop from automation and record interventions, quality control, exceptions, and validation | Download CSV |
| Clinical-trial AI validation | Connect context of use, immediate endpoint, downstream endpoint, human action, denominator, and consequence | Download CSV |
Store the completed record beside the data snapshot, analysis, model or software identifier, and decision memo. A blank cell should mean “not recorded,” not “not applicable.” Use an explicit value such as not_applicable when a field genuinely does not apply.
Workflow 1: Evaluate a configured biology AI system
- Write the intended use. Name the user, input, output, decision, environment, and consequence of a wrong result.
- State one testable claim. Separate predictive performance, biological validity, workflow improvement, and translational benefit.
- Freeze the evaluation set. Record provenance, version, inclusion and exclusion rules, independence unit, leakage review, and the biological axis held out.
- Choose credible comparators. Include the current workflow, a simple baseline, and a domain-standard method where each is relevant.
- Prespecify endpoints and uncertainty. Identify the primary endpoint, supporting endpoints, analysis unit, dependence structure, and uncertainty method.
- Run the configured system. Preserve preprocessing, prompts, tools, filters, thresholds, model and software versions, random seeds when relevant, and every output.
- Review failures before averages. Inspect abstentions, invalid outputs, false negatives, out-of-distribution cases, manual corrections, and important strata.
- Apply the decision rule. Adopt, retain for bounded exploratory use, revise and retest, or reject. Record the evidence and the next review trigger.
Use the AI evaluation record as the index for the supporting artifacts.
Workflow 2: Compare a tool with the current process
A tool trial should compare work, not feature lists. Freeze representative cases and run the current process and candidate on the same inputs. Record machine time, active staff time, corrections, failed handoffs, output quality, reproducibility, and whether the result changed the next scientific decision.
The adoption record should also include data boundaries, license and access terms, integration with existing identifiers and systems, exportability, support burden, and an exit test. A candidate that performs well but cannot preserve its inputs, outputs, parameters, or provenance has not passed a reproducibility or continuity test.
Use the tool-adoption record, then apply the detailed criteria in the Toolkit for AI-Augmented Bio Research.
Workflow 3: Evaluate a single-cell foundation model
Start with the exact task: annotation, integration, atlas search, perturbation prediction, or another named use. Record whether the pretraining corpus covers the intended tissue, donor population, disease state, assay, species, and label structure. Choose a hold-out that represents the transport claim, then compare the configured model with relevant PCA, Seurat, Harmony, scVI, scANVI, or task-specific baselines.
The result should include decision-relevant endpoints and failures by donor, tissue, batch, cell rarity, disease state, and confidence. An embedding visualization is supporting evidence, not the adoption endpoint. Use the single-cell evaluation worksheet with the decision path in Single-Cell Foundation Models.
Workflow 4: Audit a self-driving laboratory claim
Record every loop iteration from proposal through physical execution, measurement, quality control, and next-experiment selection. Distinguish automated execution from a closed loop, and preserve every human correction, restart, interpretation, and exception. Platform throughput and successful runs should never erase failed or invalid loops.
The final disposition should state whether the system demonstrated automation, lab-in-the-loop research, bounded closed-loop optimization, or independently validated discovery. Use the self-driving laboratory loop log and the classification test in Self-Driving Laboratories.
Workflow 5: Validate clinical-trial AI
Write the context of use before selecting a metric. Preserve the chain from input and model output through human review, immediate endpoint, downstream trial endpoint, and operational, scientific, or regulatory consequence. Retrieval, criterion review, completed screening, referral, consent, enrollment, retention, endpoint validity, and patient outcomes are separate endpoints.
The validation record should keep the full denominator, false-negative review, missing-information pathway, protocol and model versions, overrides, subgroup and site performance, and failure consequences. Use the clinical-trial AI validation worksheet with Clinical Trial AI for Translational Research.
Illustrative decision record
In this illustrative case, a team is deciding whether to replace its current single-cell annotation workflow with a foundation model. The intended use is cell-type annotation for a defined tissue and assay, with a scientist reviewing low-confidence cases. The frozen comparison uses donor-held-out data, the current workflow and a relevant probabilistic baseline, a prespecified annotation endpoint, and failure review for rare cell types and disease-state samples.
The record does not assume that the foundation model wins. If it fails the acceptance rule or creates worse rare-cell failures, the disposition is retain the current method. If it improves the prespecified endpoint without an unacceptable failure pattern, the disposition is adopt for that tissue, assay, and version, with a review trigger for distribution or model changes. The result remains local to the tested use and does not become a general ranking of single-cell models.
Report the result without overclaiming
A short decision memo should contain:
- the exact intended use and configured system;
- the frozen data, comparator, endpoints, and decision rules;
- the primary result with uncertainty;
- failures, abstentions, invalid outputs, manual interventions, and important strata;
- the disposition and its boundary;
- the monitoring or retest trigger; and
- links to the completed worksheet and supporting artifacts.
Use unverified when a required check was not run, blocked when a dependency prevents a valid result, and rejected when the prespecified bar was not met. An incomplete run is not a success.