Scenario test cases

中文

This directory owns evaluation examples, dataset provenance, installation, and private scoring. It is deliberately separate from ../pipelines/. The bundled JSON cases are CC0 synthetic protocol checks; they verify schemas, isolation, budgets, hashes, and evaluator wiring but make no model-accuracy claim.

Install one case in a single command:

uv run python skills/retrosynthesis_planning/scenarios/test_cases/install.py \
  --case skills/retrosynthesis_planning/scenarios/test_cases/01_single_step_retrosynthesis.json \
  --workspace /tmp/openai4s-retro-case

The installer creates disjoint public/, private_evaluator/, and results/ directories plus an installation.json containing every installed file hash. The GT pipeline checks the public hashes without opening private files. The evaluator verifies both boundaries before scoring; changed or missing files invalidate the installation instead of silently changing a frozen score. Run the matching public pipeline, freeze its output, and only then evaluate:

uv run python skills/retrosynthesis_planning/scenarios/gt_codebases/01_single_step_retrosynthesis.py \
  --workspace /tmp/openai4s-retro-case
uv run python skills/retrosynthesis_planning/scenarios/test_cases/evaluate.py \
  --scenario single_step --workspace /tmp/openai4s-retro-case

Each case's private_evaluator/references.json deliberately does not repeat the model's own rank-1 public output: a reference copied from the prediction makes every accuracy 1.0 by construction, and a scorer returning a constant would pass. tests/test_retrosynthesis_evaluator.py pins the resulting non-trivial values, so "correcting" a reference back to the model's answer is a red test, not a silent improvement.

database_sources.json is the production-data registry. Entries marked not_frozen must not be silently downloaded or republished. A maintainer must first freeze the source revision, license decision, split, and SHA256; until then, only the bundled protocol fixture is installable.

Production source verification (2026-09-09)

This read-only audit checked official documentation and small metadata responses, not dataset bytes. All six registry entries still had release_status: not_frozen and no revision, license, split, or sha256. The candidates below are preparation for maintainer review, not approved or frozen benchmark data. A code commit identifies the documentation or preprocessing implementation; it does not pin an external Dropbox, Box, or Google Drive archive.

The Lowe original USPTO metadata declares CC0, version 1, DOI 10.6084/m9.figshare.5104873.v1. This is useful upstream evidence, but the relationship and terms of each selected derivative still need review. USPTO_MIT identifies a dataset/split convention; the MIT code license is not a license conclusion for that derived data. The HF metadata at the admitted model revision declares MIT; that observation does not establish permission for Parrot's separate Google Drive training/test data.

PaRoutes file evidence

These are published metadata, not locally verified bytes. The six required benchmark files total 125,966,045 bytes; the rest of the 2.7 GB record need not be fetched just to acquire these targets, stocks, and references.

Decisions and verification still required

A maintainer must record the license basis and allowed use/redistribution for each selected dataset, including attribution obligations. If that basis is unresolved, distribute only acquisition instructions and evidence, and keep the dataset not_frozen. Atom mapping additionally needs a concrete independent truth source and reviewer; a mapper checkpoint or download landing page is insufficient. These are the remaining human decisions, not fields an agent may fill from repository license badges.

After approved acquisition, compute SHA256 from the actual input and generated benchmark files, build deterministic split and deduplication manifests, and verify private-reference isolation. Preserve upstream MD5 as supplementary integrity evidence; neither MD5 nor a Git commit/blob SHA1 substitutes for SHA256. Checkpoint training-overlap evidence is a separate prerequisite for formal scientific scores. The existing test_production_database_registry_fails_closed_until_frozen checks that the four fields are nonempty when a row is marked frozen; passing it alone proves neither licensing review nor file-hash or truth validation.

Files