← Về thư mục
📄 / / academic-research-skills / docs / design / 2026-05-15-issue-105-contamination-signals-backfill-design.md

Design — #105 v3.7.3 contamination_signals backfill migration tool

Date: 2026-05-15 Issue: #105 (v3.7.3 release completeness follow-up) Status: design — ready for TDD implementation Scope: minimal — single tool, directory-scan, scalar provenance, ruamel round-trip. No batch mode, no resumable checkpoint, no structured backfill provenance.

1. Problem

v3.7.3 spec §3.2 R-L3-2-B says: "bibliography_agent computes contamination_signals at ingest time, not at audit time. Re-running the check post-hoc on existing entries is a separate batch operation (deferred to user invocation; not part of v3.7.3 finalizer)."

Pre-v3.7.3 passport entries lack contamination_signals. Without a backfill tool:

2. Spec anchors (frozen, do not re-litigate)

3. Open-question resolutions (user-chosen 2026-05-15)

# Question Resolution Rationale
Q1 API rate-limit handling backoff-only (existing SS protocol: 429→2s×3); no resumable checkpoint YAGNI for minimal scope; large-corpus users add checkpoint in follow-up if needed
Q2 Schema field naming scalar contamination_signals_backfilled_at: ISO-8601 timestamp string issue body itself suggested "start with scalar"; strictly additive upgrade path to structured if v3.7.4 needs it
Q3 Multi-passport batch mode directory-scan only (no --input-list) YAGNI for minimal scope; user passport is typically 1 active session
Q4 YAML library ruamel.yaml passport is user-owned file; preserve comments + key order (memory feedback_toml_duplicate_table_corruption discipline spirit)

4. Design — three pieces

4.1 Resolver module

scripts/contamination_signals.py — pure functions:

PREPRINT_VENUES = frozenset({
    "arXiv", "bioRxiv", "medRxiv", "SSRN", "Research Square",
    "Preprints.org", "ChemRxiv", "EarthArXiv", "OSF Preprints", "TechRxiv",
})

def compute_preprint_signal(entry: Mapping) -> bool:
    """Signal 1 per spec §3.2 Vector 1."""

def compute_ss_unmatched_signal(entry, ss_client) -> bool | None:
    """Signal 2 per spec §3.2 Vector 2. Returns None on (a) manual exemption,
    (b) API degradation; returns True/False otherwise."""

The two signals are independently testable. SS API call is dependency-injected so tests use a mock; production uses a real client.

4.2 Migration script

scripts/migrate_literature_corpus_to_v3_7_3.py — CLI tool:

usage: migrate_literature_corpus_to_v3_7_3.py [--dry-run] [--verbose] <passport_or_dir>

Arguments:
  passport_or_dir   Path to a single passport YAML or a directory containing passports.
                    Directory scan finds *.yaml files non-recursively.

Options:
  --dry-run         Print proposed changes; write no files.
  --verbose         Per-entry logging.

Behavior:

  1. Load passport via ruamel.yaml (round-trip — preserve comments + key order)
  2. For each entry in literature_corpus[]:
  3. If contamination_signals already present → SKIP (idempotency)
  4. If entry's schema state is "insufficient-data" (no year, or no venue + no source_pointer hint) → SKIP, log reason
  5. Compute Signal 1
  6. Compute Signal 2 (skip on manual; omit on API degradation)
  7. Construct contamination_signals object per emission rules
  8. Set contamination_signals_backfilled_at to current UTC ISO-8601 timestamp
  9. If --dry-run, print diff and exit. Else, write back via ruamel.yaml round-trip.
  10. Emit migration report at stdout: processed=N patched=M skipped=K errored=E, plus per-skip-category counts.

SS API client: reuse existing references/semantic_scholar_api_protocol.md logic. Implementation: thin wrapper around requests with 429 backoff per protocol. Single shared client across the migration run (HTTP keepalive).

4.3 Schema extension

shared/contracts/passport/literature_corpus_entry.schema.json — add one optional field:

"contamination_signals_backfilled_at": {
  "type": "string",
  "format": "date-time",
  "description": "v3.7.3 backfill provenance. ISO-8601 UTC timestamp set by scripts/migrate_literature_corpus_to_v3_7_3.py when contamination_signals was computed post-hoc rather than at ingest. Absence means signals were either computed at ingest (bibliography_agent v3.7.3+) or not computed at all. Backward compat: pre-v3.7.3 entries lack both contamination_signals and this field; ingest-time entries lack only this field."
}

Pure additive — existing v3.7.3 ingest-time entries (which don't carry this field) remain valid.

4.4 Migration guide doc

docs/migration/v3.7.3-contamination-signals-backfill.md — user-facing:

5. Files touched

File Change kind
scripts/contamination_signals.py new — Signal 1 + Signal 2 resolvers
scripts/migrate_literature_corpus_to_v3_7_3.py new — CLI migration tool
scripts/test_contamination_signals.py new — unit tests for resolvers
scripts/test_migrate_literature_corpus_to_v3_7_3.py new — pytest for migration tool (mocks SS API)
shared/contracts/passport/literature_corpus_entry.schema.json additive — new optional contamination_signals_backfilled_at field
scripts/adapters/tests/test_literature_corpus_entry_schema.py extend — 2-3 tests for the new field (valid present / valid absent / invalid type)
docs/migration/v3.7.3-contamination-signals-backfill.md new — user-facing migration guide
requirements-dev.txt add ruamel.yaml>=0.18
CHANGELOG.md Unreleased entry under #105

Files explicitly NOT touched

File Reason
deep-research/agents/bibliography_agent.md v3.7.3 spec frozen; ingest-time computation unchanged
academic-pipeline/agents/pipeline_orchestrator_agent.md Finalizer behavior unchanged; migration is offline
Existing scripts/adapters/* Adapters produce ingest-time entries; migration is downstream

6. Test discipline

Per superpowers:test-driven-development:

Resolver tests (test_contamination_signals.py):

Migration tests (test_migrate_literature_corpus_to_v3_7_3.py):

Schema tests (extend test_literature_corpus_entry_schema.py):

7. Regression budget

8. Out of scope (defer)