← Về thư mục
📄 / / profiles / r-and-d / raw_evidence / scientific-agent-skills / skills / datalad / references / provenance.md

Computational provenance with DataLad

Why this is the interesting half

Version control tells you that a file changed. Provenance tells you what produced it, from what, and with which software. datalad run captures all three in the same commit that carries the change, which means the evidence cannot drift away from the result. datalad rerun then reads that record back and re-executes it, so "is this reproducible" becomes a command rather than an argument.

Principles: STAMPED, and the YODA layout it grew out of

YODA ("YODAs Organigram on Data Analysis") is the convention DataLad analyses were originally built on, and it is still what datalad create -c yoda configures. Its three principles (one thing one dataset, record where the data came from, record what was done to it) describe what a well-formed analysis looks like, but they were inspirational rather than operational: there is nothing in them to check a dataset against.

STAMPED (https://stamped-principles.org) is the operationalized successor, formalized by some of the original YODA authors. It states seven properties of a reproducible research object, each backed by normative MUST/SHOULD/MAY requirements:

Property Core requirement
Self-contained Everything essential to replicate the computation is reachable within a single top-level research object
Tracked Persistent content identification and provenance are recorded for every component and every modification, including the versions involved
Actionable The object carries enough instruction to reproduce all results, specified as executable specifications rather than prose
Modular Components are organized as independently versioned modules, included directly or linked as subdatasets
Portable Procedures depend on no undocumented host state; environments are explicitly specified and version controlled
Ephemeral Results are produced in disposable environments built only from the object's own contents
Distributable Every referenced module and component is persistently retrievable by others

Actionable is the property datalad run exists to satisfy. A README describing how a figure was produced is documentation; a run record is an executable specification, and datalad rerun is what makes it executable. The same commit satisfies Tracked, because the command, its inputs, its outputs, and the versions they were taken at are recorded next to the change rather than in a separate log that can drift away from it. Modular maps onto subdatasets, Portable and Ephemeral onto containers-run, and Distributable onto siblings and RIA stores (see publishing.md).

Two companion resources make this checkable rather than aspirational: https://checklist.stamped-principles.org walks the requirements as a MUST/SHOULD/MAY checklist, and https://examples.stamped-principles.org collects worked patterns, including ones built directly on datalad run and datalad rerun.

The YODA layout in practice

Apply the layout at creation time:

datalad create -c yoda "my_analysis"

That produces:

.
├── .gitattributes
├── CHANGELOG.md
├── code
│   ├── .gitattributes
│   └── README.md
└── README.md

The configuration matters more than the directories. Everything in code/, plus README.md and CHANGELOG.md, is tracked by Git rather than git-annex, so scripts stay directly readable and diffable in a clone that has fetched no data at all. Input data is then added as a subdataset:

datalad clone -d . https://github.com/OpenNeuroDatasets/ds000001.git inputs/raw

The -d . is what registers the clone as a subdataset of the analysis rather than leaving an unrelated repository sitting inside it.

datalad run

datalad run [-h] [-d DATASET] [-i PATH] [-o PATH] [--expand {inputs|outputs|both}]
    [--assume-ready {inputs|outputs|both}] [--explicit] [-m MESSAGE]
    [--sidecar {yes|no}] [--dry-run {basic|command}] [-J NJOBS]
    [--version] ...
Option Documented behaviour Practical consequence
-i/--input PATH "A dependency for the run. Before running the command, the content for this relative path will be retrieved." The command does not fail on an unfetched pointer, and rerun knows what to fetch elsewhere
-o/--output PATH "Prepare this relative path to be an output file of the command." Unlocks or removes the target so git-annex write protection does not block the write
--explicit "Consider the specification of inputs and outputs to be explicit. Don't warn if the repository is dirty." Lets a run proceed in a dirty dataset, and saves only the declared outputs
--assume-ready {inputs\|outputs\|both} "Assume that inputs do not need to be retrieved and/or outputs do not need to unlocked or removed." Skips preparation for speed; only safe when you have already done it
--expand {inputs\|outputs\|both} "Expand globs when storing inputs and/or outputs in the commit message." Records the concrete file list rather than the glob, which is what you want when the glob's meaning could change
--dry-run {basic\|command} "Do not run the command; just display details about the command execution." Check placeholder expansion before committing anything
--sidecar {yes\|no} Store the run record in a separate file rather than in the commit message Keeps long records out of git log output

Placeholders available in the command string: {pwd} (current working directory), {dspath} (dataset path), {tmpdir} (a temporary directory), {inputs} and {outputs} (the values of the corresponding flags), and {inputs[0]} for indexed access.

Globs are permitted in --input and --output, and multiple flags may be given:

datalad run -m "second-level model" \
  -i "derivatives/sub-*/func/*_bold.nii.gz" \
  -i "code/model.py" \
  -o "results/group_map.nii.gz" \
  --expand inputs \
  "python code/model.py {outputs}"

The run record

The commit message carries a machine-readable JSON block between the markers === Do not change lines below === and ^^^ Do not change lines above ^^^. It records the command, the dataset ID, the exit status, and the input and output specifications. The handbook is explicit that this section is "less for the human user" and exists "for DataLad, in particular for the datalad rerun command". Editing it by hand, including during an interactive rebase, breaks rerun silently.

Two behaviours that surprise people:

datalad rerun

datalad rerun [-h] [--since SINCE] [-d DATASET] [-b NAME] [-m MESSAGE] [--onto base]
    [--script FILE] [--report] [--assume-ready {inputs|outputs|both}] [--explicit]
    [-J NJOBS] [--version] [REVISION]

The reproducibility check worth building into a project:

datalad rerun --report --since <first-analysis-commit> HEAD    # inspect the plan
datalad rerun -b repro-check --since <first-analysis-commit> HEAD
git diff main repro-check -- results/                          # empty means reproduced

Rerunning onto a branch keeps the original results intact while the replay lands elsewhere, so a mismatch is a finding rather than a lost result.

Containers

datalad-container (PyPI datalad-container, currently 1.2.x) records the software environment alongside the command.

datalad containers-add [-h] [-u URL] [-d DATASET] [--call-fmt FORMAT]
    [-i IMAGE] [--update] [--extra-input FILE] [--version] NAME

Supported URL schemes:

For shub:// and docker://, a Singularity-based call format is configured automatically unless --call-fmt overrides it. --call-fmt is what you change to add bind mounts, environment variables, or GPU flags that a given image needs.

datalad containers-run [-h] [-n NAME] [-d DATASET] [-i PATH] [-o PATH] [-m MESSAGE]
    [--expand {inputs|outputs|both}] [--explicit] [--sidecar {yes|no}] [--version] ...

-n/--name selects "the name of or a path to a known container to use for execution, in case multiple containers are configured". With exactly one container configured it may be omitted. During execution the environment variable DATALAD_CONTAINER_NAME holds the name of the container in use, which is available to the command itself.

The image is tracked in the dataset like any other file, so it is annexed content: a collaborator gets it with datalad get and the provenance record points at a specific image rather than at a tag someone may have re-pushed.

Exporting provenance to a standard form

DataLad's run records are DataLad's own format. Converting them to an interoperable representation is an open area rather than a solved one, and this is worth stating plainly rather than implying a pipeline exists:

Until one of those is settled, the durable artifact is the DataLad history itself plus datalad rerun --script, which produces a plain, reviewable command sequence that does not depend on DataLad to read.

Further reading