This directory holds the v3.10 #184 generalized gold sets. Each subdirectory under gold/ is a self-contained gold set for one measurement target. The structure mirrors the v3.8 scripts/fixtures/claim_audit_calibration/ pattern but generalizes to multiple targets.
evals/
├── README.md # this file
├── gold/
│ ├── citation_extraction/ # Phase 1a/1b — baseline for #182
│ │ ├── README.md
│ │ ├── manifest.yaml
│ │ ├── tuples/
│ │ │ └── NNN-{kind-slug}-{discriminator}.json
│ │ └── expected_outcomes.json
│ ├── rq_framing_patterns/ # #257 Socratic wording advisory calibration
│ ├── status_classification/ # Phase 2 (lands post-#183)
│ └── summarization_adequacy/ # Phase 2 (lands post-#183)
The multi-task harness scripts/run_evals.py discovers every gold/<task>/manifest.yaml, measures each task, and emits a report shaped by shared/evals_lift_report.schema.json:
PYTHONPATH=. python -m scripts.run_evals # all tasks
PYTHONPATH=. python -m scripts.run_evals --task citation_extraction --output report.json
PYTHONPATH=. python -m scripts.run_evals --baseline before.json --compare after.json
--baseline + --compare produce a side-by-side report (lift_pre / lift_post). The ranking-lift gate scripts/check_ranking_lift.py reads those reports and blocks on un-acknowledged regressions; CI wires both via .github/workflows/eval-harness.yml (Delta 3 path filter). Tasks whose entrypoint module or gold set is not yet present are reported as pending/skipped rather than failing.
See each task's README.md for task-specific conventions (tuple naming, kind distributions, expected outcomes shape).
Run python -m scripts.check_evals_gold_set evals/gold/<task> to validate any gold set against its manifest. The same validator runs in CI on every PR that touches evals/gold/**.
run_evals.py harness + lift gate): v3.10 #184, spec docs/design/2026-05-21-v3.10-184-extend-eval-harness-spec.mddocs/design/2026-05-28-kong-257-idea-diversity-coverage-gap-advisory.md