📄 / / profiles / r-and-d / raw_evidence / scientific-agent-skills / skills / peer-review / references / statistical_reproducibility.md

Statistical, Methods, and Reproducibility Review

This guide supports structured questions; it does not replace a statistician, methodologist, domain expert, or independent reanalysis. Sources verified on 2026-07-23 are recorded in assets/source_ledger.csv.

Evidence hierarchy for the review

Use, in order:

The stated research question and target population
Protocol, registration, analysis plan, and amendments
Reported design and data-generating process
Methods, code, tables, figures, supplements, and repository records
Applicable primary method or regulatory guidance
Current reporting guidance
Target venue policy

Do not reject a method merely because another method is more familiar. Explain the estimand, assumption, error, or interpretation at stake.

Core review sequence

1. Define what is being estimated

Write down:

Unit of inference
Population
Intervention, exposure, test, or predictors
Comparator or reference condition
Outcome and time horizon
Target effect, association, accuracy, or predictive performance
Intercurrent events, censoring, and missing observations where relevant

Then trace whether design, data collection, analysis, result, and claim target the same quantity.

For applicable clinical trials, ICH E9(R1) provides a framework for estimands and sensitivity analyses. It is not a universal rule for every study.

2. Reconstruct the design

Identify:

Prospective, retrospective, cross-sectional, longitudinal, experimental, or observational structure
Recruitment or sampling frame
Experimental/observational unit
Pairing, nesting, clustering, repeated measures, sites, batches, and time
Allocation, concealment, blinding, matching, or weighting
Primary and secondary outcomes
Prespecified versus exploratory analyses

If the design cannot be reconstructed, first request missing reporting. Do not label the design invalid solely because details are absent.

3. Trace every denominator

Reconcile:

Eligible, enrolled, assigned, treated/exposed, followed, measured, and analyzed units
Outcome-specific denominators
Exclusions before and after allocation or measurement
Missing values and reasons
Complete-case, imputed, weighted, or model-based analysis populations
Figure, table, abstract, text, and supplement totals

Report the exact mismatch and locations; do not infer why counts differ.

4. Assess analysis–design alignment

Ask whether the method accounts for:

Outcome scale and distribution
Pairing and repeated measures
Clustering and multilevel structure
Unequal follow-up, censoring, or competing events
Sampling weights or matched designs
Baseline adjustment and prespecified covariates
Multiplicity and outcome hierarchy
Model tuning and validation
Missingness assumptions

The name of a statistical test is not enough. The report should state inputs, model form, uncertainty method, software/version, and relevant diagnostics.

5. Assess estimates and interpretation

Prefer:

Effect or performance estimates with units
Compatible uncertainty intervals
Absolute as well as relative quantities when decision-relevant
Exact denominators and analysis sets
Assumption and sensitivity context
Clinical, biological, policy, or practical relevance distinct from statistical compatibility

The ASA’s six p-value principles include:

A p-value is about incompatibility with a specified model, not the probability a hypothesis is true.
Threshold crossing alone should not determine scientific conclusions.
Transparent reporting of all relevant analyses is required.
Statistical significance does not measure effect size or importance.
A p-value alone is not a good measure of evidence.

SAMPL provides concise biomedical statistical reporting guidance. Apply it as reporting guidance, not a universal analysis recipe.

Topic-specific checks

Sample size and precision

Look for:

Prospective calculation or precision rationale
Target effect or interval width
Variance, event rate, prevalence, or accuracy assumptions
Type I error, power, sidedness, and multiplicity when applicable
Design effect, clustering, attrition, noncompliance, and missingness
Model complexity and effective sample size
Simulation details for complex designs

Do not request observed/post hoc power as a remedy for an imprecise result. Examine the estimate and uncertainty.

Randomized trials

Check:

Allocation sequence and concealment
Prespecified estimand and analysis population
Protocol/registry/outcome consistency
Baseline adjustment and stratification factors
Intercurrent events, adherence, treatment switching, and missing data
Harms and unintended effects
Sensitivity and supplementary analyses
Non-inferiority/equivalence margin and interpretation if relevant

Use CONSORT 2025 and SPIRIT 2025 for reporting. Use ICH E9/E9(R1) only when its scope and decision context fit.

Observational causal analyses

Check:

Causal question and target contrast
Time zero, eligibility, treatment/exposure assignment, follow-up, and outcome timing
Confounder rationale and measurement timing
Positivity/overlap
Exchangeability and consistency assumptions
Missingness, censoring, selection, and measurement error
Model specification and balance diagnostics
Sensitivity to unmeasured confounding or alternative specifications

Avoid judging causal identification from adjusted versus unadjusted p-values.

Diagnostic accuracy

Check:

Intended use, setting, and participant spectrum
Index test and reference standard
Threshold prespecification
Blinding and timing
Indeterminate/missing results
Verification and incorporation bias
Two-by-two denominators and uncertainty
External applicability

STARD/STARD-AI describe reporting; use an appropriate risk-of-bias framework separately.

Prediction models

Check:

Intended use, prediction time, outcome, and target population
Data source and participant flow
Predictor availability at intended use
Missing-data and preprocessing leakage
Sample size relative to outcome information and complexity
Internal validation and optimism correction
Independent evaluation and dataset shift
Calibration, discrimination, decision utility, and uncertainty
Hyperparameter tuning separated from evaluation
Reproducible model specification and preprocessing
Subgroup performance tied to plausible use and harms

TRIPOD+AI replaces TRIPOD 2015 for regression and machine-learning prediction model reporting. STARD-AI is more appropriate when diagnostic accuracy of an index test is the primary aim.

Systematic reviews and meta-analyses

Check:

Protocol and registration
Eligibility criteria and information sources
Reproducible search dates and strategies
Duplicate screening/extraction processes or justified alternatives
Risk-of-bias assessment
Effect measure and synthesis model
Heterogeneity and prediction intervals when appropriate
Dependence among estimates
Small-study and reporting biases
Certainty assessment, if claimed
Transparent deviations and unavailable data

PRISMA 2020 assesses reporting. Do not substitute PRISMA coverage for review-conduct appraisal.

Clustered and longitudinal data

Check:

Level of assignment, measurement, and inference
Within-cluster/subject correlation
Number and distribution of clusters
Small-cluster corrections where needed
Time structure, nonlinear change, and irregular measurement
Informative visit, dropout, or censoring processes
Cluster-level versus individual-level covariates

Repeated observations do not increase independent sample size one-for-one.

Multiplicity

Identify the inferential family before recommending adjustment:

Multiple primary outcomes
Multiple intervention arms or contrasts
Repeated time points
Subgroups and interactions
Interim analyses
High-dimensional features
Model selection

Possible responses include hierarchical testing, family-wise control, false-discovery control, multilevel modeling, transparent exploratory labeling, or emphasis on estimates and uncertainty. The remedy depends on the claim and decision rule.

Missing data

Check:

Missingness by group, variable, outcome, and time
Reasons and relation to intercurrent events
Information used by imputation or weighting
Number of imputations and pooling when applicable
Compatibility of imputation and analysis models
Uncertainty propagation
Sensitivity to plausible departures from assumptions

Avoid demanding one preferred technique without considering the estimand and missingness process.

Reproducibility review

Materials and provenance

Check:

Stable identifiers for datasets, samples, models, protocols, and materials
Raw-to-processed provenance
Exclusion and transformation records
Versioned analysis inputs and outputs
Repository accession numbers
Data dictionary, units, and coding
Domain metadata standard where applicable

Legacy domain standards and their current status are summarized in references/reporting_standards.md.

Code and computational environment

Check:

Executable code for central analyses when sharing is permitted
Dependency versions or lock/environment file
Operating-system or hardware requirements that affect results
Random seeds and nondeterminism
Parameter, configuration, and model checkpoints
Run order and instructions
Tests or validation of custom code
License and access restrictions

Code availability does not prove that the code generated the reported result. Provenance and a reproducible run record are separate evidence.

Data and access

Open sharing may be limited by consent, privacy, indigenous/community governance, security, contracts, or licensing. A useful statement should identify:

What exists
Where it is held
Who can request access
Criteria and process
Expected timeline
Restrictions and rationale
Whether code or synthetic/aggregate alternatives are available

Do not request disclosure that would violate ethics, law, consent, or governance.

Independent reproduction

Claim independent reproduction only if the reviewer actually:

Obtained authorized inputs.
Recorded versions and environment.
Ran documented commands.
Preserved content hashes or equivalent provenance.
Compared prespecified outputs.
Recorded deviations and failures.

A static consistency audit is not reproduction.

When to request specialist review

Escalate when a central conclusion depends on methods outside competence, including:

Complex adaptive, Bayesian, causal, survival, multilevel, spatial, or longitudinal methods
High-dimensional omics or multiple-testing procedures
Diagnostic, prediction, or AI evaluation
Survey weighting or complex sampling
Economic modeling
Meta-analysis with dependent effects or network structure
Unfamiliar qualitative or mixed-methods methodology
Image forensics, biosecurity, privacy, or domain-specific ethics

Say what expertise is needed and which claim depends on it. Do not mask uncertainty with an automated score.

Using the local checklist

Copy assets/statistical_reproducibility_template.json, record evidence locations without pasting manuscript prose into report fields, and run:

python3 scripts/audit_statistics_reproducibility.py local-checklist.json

Statuses:

verified_present
partly_documented
missing
not_assessed
not_applicable with rationale

The tool reports item IDs and counts. It does not calculate merit, rerun analyses, or certify reproducibility.