Date: 2026-05-21
Issue: #184
Umbrella: #198 (v3.10 deterministic verification layer)
Target release: v3.10
Companion specs: 2026-05-21-v3.10-183-epistemic-status-spec.md (full, the §3.6 verification_gate shared contract), 2026-05-21-v3.10-182-promote-citation-gate-spec.md (lite, supplies the lookup_verified per-citation signal this spec's citation-extraction gold subset measures against)
Upstream v3.8 reference: academic-pipeline/references/claim_audit_calibration_protocol.md (the existing calibration runner protocol; this spec generalizes the pattern but does not duplicate it)
This is a lite spec. It documents the v3.10 extend delta on top of the existing v3.8 claim-audit calibration infrastructure. The umbrella issue gap analysis confirmed that ~60% of #184's original "Proposal" already ships (20-tuple gold set, FNR/FPR runner, T-C1/T-C2/T-C3 threshold tests, benchmark-report honesty schema); v3.10 #184 generalizes the calibration pattern, adds two more gold-set tracks, introduces a ranking-method lift gate, and ties CI path filters to a uniform eval harness.
| Capability | Existing path | First shipped |
|---|---|---|
| 20-tuple synthetic gold set (12 alignment + 8 constraint) | scripts/fixtures/claim_audit_calibration/gold_set.json |
v3.8 |
| Calibration runner with FNR/FPR aggregate + per-class reporting | scripts/claim_audit_calibration.py |
v3.8 |
| T-C1 (aggregate FNR<0.15, FPR<0.10) / T-C2 (per-class) / T-C3 (shape integrity) tests | scripts/test_claim_audit_calibration.py |
v3.8 |
| Calibration protocol documentation | academic-pipeline/references/claim_audit_calibration_protocol.md |
v3.8 |
| Benchmark-report honesty schema + disclosure lint (not a lift gate) | shared/benchmark_report.schema.json, scripts/check_benchmark_report.py |
v3.8 |
| Literature quality scoring rubric (potential baseline metric source for ranking-change comparisons; the exact section heading needs verification against repo state at implementation time before being wired as a baseline) | academic-paper/agents/literature_strategist_agent.md |
v3.5 |
The umbrella issue's reframing made the #184 scope clear: this child generalizes the v3.8 calibration runner from "FNR/FPR for the claim-audit judge" to "lift/regression for any ranking-method or generation change," and adds two new gold-set tracks (epistemic-status classification, summarization adequacy) on top of the existing claim-audit track. The v3.8 surface is the reference design; v3.10 builds the general harness using the same pattern.
Phase 1 (parallel with #182/#183 ship):
- Citation-extraction gold subset (this spec's deliverable 1) — required as baseline for #182 lookup_verified promotion (otherwise the promotion has no measurement target).
Phase 2 (sequential after #183 ships):
- Status-classification gold subset — depends on #183 epistemic_status schema field landing.
- Summarization-adequacy gold subset — depends on #183 prose-register inference landing (so the gold set can score writer outputs against extractor-assigned status).
- run_evals.py general harness — can land in Phase 1 or Phase 2; only depends on the calibration-runner pattern (already shipped).
- Ranking-method lift gate — independent of #182/#183 ship; can land any time.
- Human expert concordance rows — independent infrastructure; can land any time.
The phasing matters because #182's blocking-default promotion (per #182 spec Delta 3) needs a measurement target the day it ships. The citation-extraction subset is the only #184 deliverable on #182's critical path.
evals/gold/ directory + three core-task gold subsetsA new top-level directory evals/gold/ holds the v3.10 generalized gold sets. The directory mirrors the v3.8 scripts/fixtures/claim_audit_calibration/ pattern but is not constrained to claim-audit.
evals/
├── README.md # entry point for human contributors
├── gold/
│ ├── citation_extraction/ # Phase 1 — baseline for #182
│ │ ├── README.md
│ │ ├── manifest.yaml # task description, threshold contract, sample N
│ │ ├── tuples/
│ │ │ ├── 001-doi-resolves.json
│ │ │ ├── 031-fabricated-synthetic-genome.json
│ │ │ ├── ...
│ │ │ └── 050-fabricated-counterfactual-spectral-knapsack.json
│ │ └── expected_outcomes.json # ground-truth per tuple
│ ├── status_classification/ # Phase 2 — depends on #183
│ │ └── ... # mirror layout
│ └── summarization_adequacy/ # Phase 2 — depends on #183
│ └── ...
Target: measure the per-citation lookup_verified 3-class enum (true / false / unresolvable) classification accuracy produced by #182's verification_gate.verify_citation against ground-truth resolver outcomes. The metric is classification accuracy (all three classes treated symmetrically), not boolean accuracy — collapsing unresolvable into false would mis-attribute manual-skip cases as verification failures. Initial corpus: 50 hand-curated tuples covering:
lookup_verified=true, resolver_outcomes.crossref.status="matched", resolver_outcomes.arxiv.status="skipped" (arXiv resolver does not run on non-arXiv citations per #182 Delta 1 design; the .status enum value is "skipped", not a separate arxiv_unmatched boolean).lookup_verified=true, resolver_outcomes.arxiv.status="matched".obtained_via: manual); expected resolvers .status="skipped" per v3.7.3 exemption on all four resolvers, lookup_verified=unresolvable (every applicable resolver is skipped ⇒ the set of adjudicating resolvers is empty per #182 Delta 4).lookup_verified=false, all applicable resolvers .status="unmatched". Mirrors v3.8's fabricated-reference fixture. Per #182 Delta 4 anti-fabrication bias, partial outage on these tuples (some .status="unreachable" from a transient API outage during fixture replay) still yields lookup_verified=false if at least one resolver returned .status="unmatched".Threshold contract for citation-extraction: aggregate accuracy >= 0.90 against the 50-tuple expected outcomes. Per-class threshold (true / false / unresolvable): accuracy >= 0.85 per class. These are v3.10.0 binding default thresholds; changing them before v3.10.0 ship requires a spec amendment, while post-ship tuning happens via manifest version bump per E-V2 / OQ-1.
Amendment (2026-05-27, #250): The
valid_unresolvableclass (10 tuples) was removed before v3.10.0 ship. No stable, first-party-verifiable citation that is unmatched across all four resolvers was found under current index coverage; the class was unbuildable. Tuples 031–040 are reassigned tofabricated(total 15). Thefalseclass is carried entirely by fabricated tuples. The accepted coverage gap (no real-but-unindexed tuple) is tracked in #250. Thelookup_verifiedreducer (#182) is unchanged.
Target: measure #183's epistemic_status_extractor accuracy against human-labeled ground truth. Initial corpus: 30 source paper excerpts with hand-labeled epistemic_status ∈ {Established, Supported, Preliminary, Speculative, Contested} (6 per label).
Threshold contract: aggregate accuracy >= 0.75 across the 5 labels. Per-label accuracy >= 0.65. The labels are inherently softer than citation-extraction's binary, so thresholds are deliberately lower. Phase 2 schedule: gold set authoring begins after #183 ships.
Target: measure whether prose generated by synthesis_agent / draft_writer_agent / report_compiler_agent preserves the source register, given the extractor-assigned epistemic_status. Each tuple is (source_text, extractor_assigned_status, writer_emitted_prose, ground_truth_silent_upgrade). Initial corpus: 40 tuples covering all 6 upgrade-pair types (Speculative→Preliminary/Supported/Established, Preliminary→Supported/Established, Supported→Established) plus same-level and downgrade negative controls.
Threshold contract: aggregate silent_upgrade detection accuracy >= 0.85. Per upgrade-pair accuracy >= 0.75. Phase 2 schedule: depends on #183 prose-register inference landing.
scripts/run_evals.py general harnessGeneralize the v3.8 scripts/claim_audit_calibration.py pattern from a single-task FNR/FPR runner to a multi-task harness driven by evals/gold/<task>/manifest.yaml declarations.
python -m scripts.run_evals [--task <task_name>] [--baseline <path>] [--compare <path>] [--output <report.json>]. Default behavior (no --task): run all tasks declared under evals/gold/. With --task citation_extraction: run only that task.--baseline and --compare are provided, the harness emits a side-by-side report showing pre/post metrics per task per class. Without them, the harness emits a single-run report against the gold set.shared/evals_lift_report.schema.json (specified per #183 §3.6 forward declaration). The schema is purpose-built for multi-task harness output and is structurally distinct from shared/benchmark_report.schema.json (which is a single-benchmark-vs-human-baseline disclosure schema, not a multi-task per-class metrics shape). The v3.8 benchmark_report schema is unchanged. The new schema's required fields: harness_version, run_id, gold_set_version, per_task array, and caveats (the last mirrors the v3.8 honesty-disclosure convention). Each per_task[] entry carries task_name, manifest_version, aggregate_metric, per_class array, optional lift_pre / lift_post (when invoked in baseline/compare mode), and optional expert_concordance per class (when human-labeled tuples exist per Delta 5).run_evals.py calls claim_audit_calibration.run_calibration for the claim-audit task and a parallel runner for each new task. The v3.8 calibration runner is not modified; v3.10 adds new tasks that conform to the same I/O shape (tuple-in, per-tuple-verdict-out, aggregate-metric-emit)..github/workflows/eval-harness.yml triggers run_evals.py on PRs touching paths listed in Delta 3.Today the v3.8 calibration runner is invoked manually. v3.10 wires it into CI to fire automatically on PRs that change scoring or generation logic.
run_evals.py against the affected gold-set tasks:scripts/claim_audit_*.py → trigger claim-audit taskscripts/verification_gate/**, scripts/arxiv_client.py, scripts/verification_cache.py, scripts/crossref_client.py, scripts/openalex_client.py, scripts/citation_verification_summary.py, and the Semantic Scholar lookup module invoked from deep-research/agents/bibliography_agent.md (path determined at implementation time per #182 Delta 5 note) → trigger citation-extraction task (per #182 Delta 5). Rationale: any resolver implementation change can affect the lookup_verified truth signal that the citation-extraction gold set measures, so all four resolver clients plus the unified summary derivation module are on the same critical path as the verification_gate wrapper.scripts/epistemic_status_extractor.py, scripts/check_silent_upgrade.py, shared/references/epistemic_status_vocabulary.md → trigger status-classification + summarization-adequacy tasks (per #183 deliverable 6/7; the vocabulary doc is included because §5.3.3 confirms it carries the active inference patterns).deep-research/agents/synthesis_agent.md, academic-paper/agents/draft_writer_agent.md, deep-research/agents/report_compiler_agent.md → trigger status-classification + summarization-adequacy tasks (writer prompts that govern prose register)evals/gold/** itself → trigger all tasks (gold-set edits are eval-config changes and must regress-test against existing pipeline).github/workflows/eval-harness.yml (new). Concurrency group ${{ github.event_name }}-eval-harness-${{ github.ref }} per repo CI discipline. Runs python -m scripts.run_evals --task <triggered_task> and posts a markdown comment to the PR with the run metrics.[eval-regression-acknowledged] token AND a follow-up issue URL link. CI verification mechanism: .github/workflows/eval-harness.yml runs a small shell step that parses the PR description (via gh api /repos/{owner}/{repo}/pulls/{n} and a grep over the body) for the literal [eval-regression-acknowledged] token AND for at least one GitHub issue URL matching https://github.com/{owner}/{repo}/issues/\d+. The workflow then calls gh api on each found issue URL to verify the issue exists and is OPEN. Missing token, missing issue URL, closed issue, or 404 on the issue URL all fail the gate. This is the deterministic part of the lift gate; the LLM-as-judge regression detection lives downstream in #183.When a PR changes scoring or ordering logic — e.g., bibliography_agent quality scoring, literature_strategist_agent quality rubric, the new verification_gate.verify_citation heuristic, or future ranking-method experiments — v3.10 requires explicit pre/post comparison rather than "this should be better in theory" claims.
scripts/check_ranking_lift.py — new lint. Reads two run_evals.py outputs (baseline + compare) and computes per-task per-class lift values. Each metric in the manifest carries an explicit direction: "higher_is_better" | "lower_is_better" declaration (e.g., accuracy: higher_is_better, FNR: lower_is_better, FPR: lower_is_better).higher_is_better metrics: signed_lift = (compare - baseline) / |baseline| if baseline != 0; positive = improvement, negative = regression.lower_is_better metrics: signed_lift = (baseline - compare) / |baseline| if baseline != 0; positive = improvement, negative = regression. (The numerator inverts so the polarity convention "negative lift = regression" holds for every metric regardless of direction.)baseline == 0): the lift formula is undefined; the lint reports signed_lift = "+inf" when compare improves from 0 and signed_lift = "-inf" when compare regresses from 0. Manifest may declare a zero_baseline_policy for the metric (default: treat any non-zero compare as significant change and require explicit acknowledgement).signed_lift < -0.05 (5% relative regression on the chosen metric, polarity-corrected) OR any zero-baseline metric changes value, unless the PR carries [ranking-regression-acknowledged] + follow-up issue. Positive or zero signed_lift passes.Affected metric: <task>.<class>.<metric>). The check_ranking_lift lint validates the declaration matches the observed change..github/pull_request_template.md adds an "Eval impact" section instructing contributors how to declare ranking changes.The v3.8 gold set is synthetic (judge-written tuples with known-ground-truth verdicts). v3.10 #184 adds a parallel human expert concordance column to each gold-set task, recording domain-expert verdicts alongside the synthetic ground truth.
human_expert_verdict field. Initial population: 20% of each task's tuples (10 of 50 citation-extraction tuples, 6 of 30 status-classification, 8 of 40 summarization-adequacy) hand-labeled by domain experts.scripts/run_evals.py emits an expert_concordance field per task per class (named per #183 §3.6 contract vocabulary), computed as the agreement rate between the synthetic ground truth and the human expert verdicts on the subset of tuples that carry a human_expert_verdict.evals_lift_report output.Concordance scope: the 20% sample is the initial v3.10 ship target. Expansion of the human-labeled coverage is a future workstream; per-tuple expert verdicts are expensive to source. The infrastructure (schema field + concordance computation) lands at v3.10; the data populates over time.
evals/ top-level directory + evals/README.md (Delta 1)evals/gold/citation_extraction/ — Phase 1 corpus (50 tuples + manifest + expected_outcomes) (Delta 1)evals/gold/status_classification/ — Phase 2 corpus (30 tuples + manifest, Phase-2 schedule) (Delta 1)evals/gold/summarization_adequacy/ — Phase 2 corpus (40 tuples + manifest, Phase-2 schedule) (Delta 1)scripts/run_evals.py — general harness (Delta 2)shared/evals_lift_report.schema.json — new schema for multi-task harness output (Delta 2; aligned with #183 §3.6 forward declaration).github/workflows/eval-harness.yml — CI path-filter workflow (Delta 3)scripts/check_ranking_lift.py — ranking-lift gate lint (Delta 4).github/pull_request_template.md — Eval impact section (Delta 4)human_expert_verdict optional field on each tuple type (Delta 5)scripts/test_run_evals.py, scripts/test_check_ranking_lift.py, scripts/test_evals_citation_extraction.py, fixtures for Phase 2 tasks added when those tasks ship..github/workflows/spec-consistency.yml with the new test paths; .github/workflows/eval-harness.yml is the new workflow per Delta 3.CHANGELOG.md + docs/ARCHITECTURE.md + ROADMAP §3.10 anchor — v3.10 release-alignment 6-invariant set per repo discipline.scripts/claim_audit_calibration.py and its 20-tuple gold set remain in place; run_evals.py calls into the existing runner for the claim-audit task. v3.10 does not duplicate v3.8 calibration logic.evidence_type 6-tier biomedical refinement gold set. Per #198 vocabulary lock, v3.10 stays on the 5-tier cross-disciplinary enum. A 6-tier biomedical track would land alongside the deferred evidence_type follow-up if that schema field is ever added.The I-V1 through I-V15 from 2026-05-21-v3.10-183-epistemic-status-spec.md §9 and the C-V1 through C-V5 from 2026-05-21-v3.10-182-promote-citation-gate-spec.md §5 are not duplicated. The following are #184-specific:
E-V1 — Gold-set immutability per release. Once a evals/gold/<task>/manifest.yaml version ships in a release, the corpus tuples for that version are frozen. Patches may add tuples; modifying or removing existing tuples requires a manifest version bump. This protects calibration from silent drift.
E-V2 — Per-task threshold contract. Each task's manifest.yaml declares its aggregate threshold and per-class thresholds. The CI lint reads the manifest, not hard-coded values, so threshold tuning over time is a manifest-edit operation, not a code change.
E-V3 — Synthetic vs expert concordance. The synthetic ground truth (in expected_outcomes.json) is the source of truth for CI gates. Human expert verdicts in human_expert_verdict fields are advisory; mismatch between synthetic and expert verdicts surfaces as an expert_concordance metric but does not fail the CI gate. v3.10 does not gate on concordance; future versions may.
E-V4 — Lift-gate metric explicitness. PRs that touch ranking logic MUST declare Affected metric: <task>.<class>.<metric> in PR description. The check_ranking_lift lint cross-checks the declaration against observed metric changes; undeclared regressions block ship, declared regressions require explicit [ranking-regression-acknowledged] plus follow-up issue.
E-V5 — Phase-1-first dependency. The citation-extraction gold subset MUST ship at or before #182 ship cutoff. The status-classification and summarization-adequacy subsets MUST NOT block #182 ship; they are sequential Phase-2 work depending on #183 surfaces.
D1 — evals/ directory shipped. Top-level evals/ directory present with README.md, gold/ subdirectory, and at minimum the citation-extraction Phase-1 corpus.
D2 — Citation-extraction gold subset shipped (Phase 1). evals/gold/citation_extraction/ contains 50 hand-curated tuples per §3.1.1, manifest with 0.90 aggregate / 0.85 per-class thresholds, and expected_outcomes.json ground truth. The harness can run python -m scripts.run_evals --task citation_extraction and emit a report.
D1 — evals/ directory shipped. Top-level evals/ directory present with README.md, gold/ subdirectory, and at minimum the citation-extraction Phase-1 corpus.
D2 — Citation-extraction gold subset shipped (Phase 1). evals/gold/citation_extraction/ contains 50 hand-curated tuples per §3.1.1, manifest with 0.90 aggregate / 0.85 per-class thresholds (v3.10.0 binding defaults per OQ-1), and expected_outcomes.json ground truth. The harness can run python -m scripts.run_evals --task citation_extraction and emit a report.
D3 — run_evals.py general harness shipped. Module accepts --task flag, defaults to running all tasks. --baseline / --compare flags produce side-by-side reports. Output conforms to the new shared/evals_lift_report.schema.json.
D4 — evals_lift_report schema shipped. New shared/evals_lift_report.schema.json lands per §3.2 Delta 2 output-shape description. The v3.8 shared/benchmark_report.schema.json is unchanged by v3.10 #184. The two schemas serve different purposes and remain independent surfaces.
D5 — Eval-harness CI workflow shipped. .github/workflows/eval-harness.yml triggers run_evals.py on PRs matching the Delta 3 path filter. PR markdown comment with metrics generated. Concurrency group includes github.event_name suffix.
D6 — Ranking-lift gate shipped, direction-aware. scripts/check_ranking_lift.py operational. Each manifest metric declares direction: higher_is_better | lower_is_better. Polarity-corrected signed_lift < -0.05 blocks PR unless [ranking-regression-acknowledged] token + follow-up issue present. Zero-baseline metric changes also block unless acknowledged. .github/pull_request_template.md includes Eval impact section per §3.4 Delta 4.
D7 — Expert concordance field shipped. Each gold-set tuple schema includes optional human_expert_verdict field. run_evals.py emits expert_concordance metric per task per class (aligned with #183 §3.6 contract vocabulary). Initial 20% sample populated for the citation-extraction Phase-1 corpus.
D8 — v3.8 calibration runner preserved. scripts/claim_audit_calibration.py and its T-C1/T-C2/T-C3 tests are unchanged by v3.10 #184. The general harness calls into the existing runner for the claim-audit task.
D9 — Six-doc release alignment. At tag time, scripts/check_version_consistency.py reports zero drift across CHANGELOG, README, ARCHITECTURE, PERFORMANCE, bilingual mirror docs, and package metadata per repo release discipline.
D10 — Shared-contract claim verified. The §3.6 verification_gate contract per #183 spec is implemented for the lift-gate row: this spec's run_evals.py + check_ranking_lift.py produce lift_pre / lift_post / expert_concordance columns in the new evals_lift_report schema, matching the #183 §3.6 contract table's "output column" property for the lift gate row.
The following depend on #183 schema surface and ship in a follow-up PR after #183 lands. They are NOT required for v3.10.0 ship; v3.10.0 with only the citation-extraction track is a valid ship.
D11 (follow-up) — Status-classification gold subset. 30-tuple corpus per §3.1.2, manifest with 0.75 aggregate / 0.65 per-class thresholds. Lands in a follow-up PR after #183 epistemic_status schema field is in production.
D12 (follow-up) — Summarization-adequacy gold subset. 40-tuple corpus per §3.1.3, manifest with 0.85 aggregate / 0.75 per upgrade-pair thresholds. Lands in a follow-up PR after #183 prose-register inference is in production.
Per OQ-3, whether D11 / D12 ship within the v3.10 release window or slip to v3.10.1 depends on #183 implementation pace. The dependency is one-directional: D11 / D12 cannot ship before #183.
OQ-1 — Threshold tuning. The 0.90 / 0.85 (citation) / 0.75 / 0.65 (status) / 0.85 / 0.75 (summarization) numbers are placeholder thresholds. Implementation may tune based on first 30-60 days of operational data; tuning is a manifest edit, not a code change. Recommend re-tuning at v3.10.1 patch release once production data accumulates.
OQ-2 — Human expert sourcing. Who labels the human concordance verdicts? v3.10 ships infrastructure for the field; sourcing protocol is operational (e.g., maintainer self-label, recruited domain experts, paid annotators). Out of scope for the schema spec.
OQ-3 — Phase 2 timing. Phase-2 deliverables (D8, D9) depend on #183 ship. Whether they ship within the v3.10 release window or slip to v3.10.1 depends on #183 implementation pace. The dependency is one-directional: Phase 2 cannot ship before #183. CI-workflow sequencing constraint: the Delta 3 path filter triggers status-classification + summarization-adequacy tasks on PRs touching epistemic_status_extractor.py and writer prompts. If eval-harness.yml ships in v3.10.0 ahead of D8/D9, the workflow MUST guard each task invocation with a gold-set-present precondition (skip task with a non-failing "Phase 2 deliverable pending" notice rather than fail the gate) so PRs that touch upstream writer prompts before the Phase 2 gold sets land do not get blocked by missing harness tasks. Implementation choice between "guard with precondition" and "defer the workflow to ship alongside D8/D9" is open; either resolution is acceptable.
OQ-4 — Lift-gate threshold value. The -0.05 absolute-regression threshold is a guess. Implementation may tune based on observed metric volatility; tighter thresholds (-0.02 or stricter) for production-critical metrics, looser (-0.10) for experimental ranking explorations.
OQ-5 — Gold-set version bumping. When a manifest version bumps (e.g., 1.0 → 1.1 because a tuple was reformulated), the threshold contract resets. Whether to require a sandbox window (e.g., 7 days running new tuples without gating) before activating the new threshold is deferred.
verification_gate contract source of truth): docs/design/2026-05-21-v3.10-183-epistemic-status-spec.mdlookup_verified signal measured by citation-extraction track): docs/design/2026-05-21-v3.10-182-promote-citation-gate-spec.mdacademic-pipeline/references/claim_audit_calibration_protocol.mddocs/design/2026-05-15-issue-103-claim-alignment-audit-spec.mdscripts/claim_audit_calibration.pyshared/benchmark_report.schema.json