← Về thư mục
📄 / / academic-research-skills / docs / migration / v3.7.3-contamination-signals-backfill.md

v3.7.3 contamination_signals backfill — migration guide

Tool: scripts/migrate_literature_corpus_to_v3_7_3.py Issue: #105 Spec: v3.7.3 §3.2 R-L3-2-B (the deferred batch operation)

When to run

After upgrading to ARS v3.7.3+ if your Material Passport's literature_corpus[] was populated by an adapter run before v3.7.3 was deployed.

How to tell: open the passport YAML and look for contamination_signals under any literature_corpus[] entry. If absent on entries you trust were produced by an adapter, you have a pre-v3.7.3 corpus and this tool is for you.

If contamination_signals is already present (because bibliography_agent v3.7.3+ produced the entry at ingest time), the tool will detect that and skip — idempotent re-run is safe.

What it does

For each pre-v3.7.3 entry in literature_corpus[], the tool computes both v3.7.3 advisory signals:

It then writes contamination_signals: {preprint_post_llm_inflection, semantic_scholar_unmatched} plus a scalar provenance timestamp contamination_signals_backfilled_at (ISO-8601 UTC) onto the entry.

Exemptions (per spec §3.2):

How to run

# Always dry-run first
python scripts/migrate_literature_corpus_to_v3_7_3.py --dry-run --verbose /path/to/passport.yaml

# Single passport
python scripts/migrate_literature_corpus_to_v3_7_3.py /path/to/passport.yaml

# Directory scan (non-recursive; only *.yaml at the top level)
python scripts/migrate_literature_corpus_to_v3_7_3.py /path/to/passports/

The tool prints a one-line report on stdout:

processed=42 patched=37 skipped_already_migrated=5 skipped_insufficient_data=0 dry_run=False

skipped_already_migrated = entries that already had contamination_signals set (idempotency guarantee). skipped_insufficient_data = entries missing year (a schema invariant; only reachable if the YAML was hand-edited).

Idempotency guarantee

Re-running the tool on a migrated passport produces zero changes. The detection rule is "skip if contamination_signals is present on the entry"; this rule is keyed on object presence, not on the contamination_signals_backfilled_at timestamp, so the first migration's timestamp is preserved across re-runs.

The tool uses ruamel.yaml round-trip serialization to preserve comments, key order, and quoting style. Diffs across an idempotent re-run are byte-identical.

Semantic Scholar rate-limit considerations

The unauthenticated Semantic Scholar API allows ~1 request per second. The migration tool retries on HTTP 429 per the existing protocol (deep-research/references/semantic_scholar_api_protocol.md — 2s backoff × 3). For a corpus of 5,000 entries this means ~1.5 hours of API time.

If the migration is interrupted (Ctrl+C, network drop, system restart), re-run the tool — the idempotency rule ensures already-migrated entries are skipped. Entries that got semantic_scholar_unmatched omitted due to API failure during the interrupted run will have the field filled in on the next successful pass.

Resumable checkpoint state is not implemented in v3.7.3; see #105 design doc §8 for the deferred-scope rationale (will be reconsidered if a user reports a corpus > 10,000 entries).

What this tool does NOT do

Backward compatibility