workflows/The versioned science-workflow benchmark's manifests: thirteen workflows and 46
cases, each workflow JSON declaring what a run is supposed to do and what
counts as having done it. The root-level next-round-acceptance.json is a
separate Stage 0 field/safety pack, not a fourteenth science workflow.
They live in the repository rather than in a fixture directory for one reason:
a case change has to be a reviewable diff. The runner that executes them is
openai4s/benchmark/, and every step it
takes drives production code — the real Store, the real kernel manager, the
real host dispatcher, the real compute manager. What gets injected is only what
cannot run offline: the model, the network, and a package manager.
A declared outcome is part of the contract, not a status column. failure,
permission_denied, recovered and provenance cases fail when the run
succeeds, because a benchmark that scores "no exception" measures nothing
about the half of the system whose job is to refuse.
| Workflow | What it covers |
|---|---|
next-round-acceptance.json |
Strict Stage 0 pack for six field paths, seven safety actions, and denominator-explicit baseline metrics |
artifact-lineage/ |
A derived artifact carries its lineage |
codebase-mode/ |
A source deliverable, and whether the Host believes the claim about it |
delegation/ |
What the parent is told a child did |
environment-provenance/ |
An artifact's environment provenance |
environment-transaction/ |
plan -> apply -> rollback as a transaction |
evidence-package/ |
Exporting and verifying an evidence package |
permission-boundary/ |
The workspace boundary refuses a write outside it |
python-analysis/ |
Python analysis producing a traceable artifact |
r-analysis/ |
R is its own channel, not a wrapper |
remote-compute/ |
submit -> poll -> harvest against a real shell |
science-retrieval/ |
Scientific retrieval with source evidence |
telemetry-identity/ |
Revoking telemetry destroys the identity with it |
tool-bringup/ |
Tool bring-up: build, weights, canary, admission, frozen record |
Replay the strict Stage 0 pack, with a JSON report suitable for a gate, using
openai4s benchmark --acceptance --json. A matching baseline gap is printed as
BASELINE, never as a current capability. The loader freezes the manifest's
path IDs, claims, execution modes, assertion keys, expected values, and
canonical content digest under one pack_version; a semantic edit without a
reviewed version/digest is rejected rather than silently changing the baseline.
The current 2026-08-16.2 manifest requires the Reviewer observation to prove
that the formal workspace fingerprint remained unchanged.