Status: v3.8
Parent agent: claim_ref_alignment_audit_agent
Mode name: calibration
Purpose: Measure the audit agent's own false-negative rate (FNR) and false-positive rate (FPR) on alignment judgments (SUPPORTED / UNSUPPORTED / AMBIGUOUS / RETRIEVAL_FAILED) AND negative-constraint judgments (VIOLATED / NOT_VIOLATED) against a synthetic gold set, then gate CI on FNR < 0.15 AND FPR < 0.10.
This protocol is modeled on academic-paper-reviewer/references/calibration_mode_protocol.md. The reviewer mode measures editorial decision accuracy on whole papers; this mode measures claim-to-source alignment accuracy on per-citation tuples. Both share the same FNR / FPR / threshold-gate vocabulary; the difference is unit of analysis.
The audit agent emits HIGH-WARN gate-refuse annotations on UNSUPPORTED rows and VIOLATED constraint matches. Those signals are only useful if the underlying judge is reliably distinguishing supported from unsupported claims. Without measurement, a judge that systematically over-flags AMBIGUOUS cases as UNSUPPORTED looks identical at the row level to a judge that nails the discrimination — but the former produces a flood of false HIGH-WARN refusals that erodes operator trust and eventually gets the agent ignored.
Calibration mode closes the measurability gap. It does NOT try to make the judge perfect; it makes the judge's error profile legible BEFORE the agent ships to a new operator's pipeline.
Gold set: 25 tuples (17 alignment incl. 5 #213 partial-support fixtures + 8 constraint per the canonical shipped fixture at scripts/fixtures/claim_audit_calibration/gold_set.json). Per-tuple shape and the four ingestion-gate rules (alignment-judgment coverage, ≥3 NOT_VIOLATED floor, required constraint fields, tuple_kind validity) are spelled out in Gold tuple schema below + spec §7.7 rules (a)-(d). The shipped fixture is the canonical example.
Judge function: a callable matching the v3.8 judge interface (claim_text, retrieved_excerpt, anchor_kind, anchor_value, active_constraints, judge_model kwargs). The shipped test fixture pairs the gold set with a perfect-judge stub so T-C1 / T-C2 / T-C3 exercise the calibration tooling end-to-end without requiring a live LLM call. Production deployment plugs a real judge_fn in place of the stub.
Thresholds (optional): {"FNR": 0.15, "FPR": 0.10} per spec §7.7 + §9 acceptance. Tightening these is a spec bump; loosening them is not permitted without a corresponding judge-quality-improvement justification recorded in the calibration report's thresholds block.
Two discriminated tuple_kind shapes, validated at ingestion by validate_gold_set (raises GoldSetValidationError on the first rule violation).
{
"tuple_kind": "alignment",
"claim_text": "<the prose claim under test>",
"ref_text_excerpt": "<source-passage excerpt the judge will evaluate against; null for RETRIEVAL_FAILED tuples>",
"anchor": {"kind": "page" | "section" | "quote" | "paragraph" | "none", "value": "<...>"},
"expected_judgment": "SUPPORTED" | "UNSUPPORTED" | "AMBIGUOUS" | "RETRIEVAL_FAILED"
}
Alignment tuples MUST NOT carry any of: constraint_under_test_id, constraint_under_test_rule_text, manifest_fixture_path (rule (b)). Mixing constraint fields onto an alignment tuple corrupts the per-class confusion matrix because the judge call shape differs between the two kinds.
{
"tuple_kind": "constraint",
"claim_text": "<the prose claim under test>",
"ref_text_excerpt": null,
"anchor": {"kind": "page" | "section" | ..., "value": "<...>"},
"expected_judgment": "VIOLATED" | "NOT_VIOLATED",
"constraint_under_test_id": "MNC-N" | "NC-CN-M",
"constraint_under_test_rule_text": "<inline rule text>", // EITHER this
"manifest_fixture_path": "scripts/fixtures/.../mnc.json" // OR this
}
Both rule sources are accepted by validate_gold_set because manifest-bound constraints sometimes carry domain-specific rule text that authors want versioned alongside the manifest. The v3.8.0 calibration runner supports only the inline constraint_under_test_rule_text form. A manifest-only tuple validates clean per rule (c) but run_calibration raises NotImplementedError at run time rather than reach the judge with an empty rule (R2 codex P1 closure on the silent-skip risk that T-C3 is meant to prevent). The manifest-fixture resolver is a post-v3.8 deliverable; until it ships, author calibration fixtures with inline rule text.
validate_gold_set(tuples). Any of the four spec §7.7 rules (a)/(b)/(c)/(d) failing raises GoldSetValidationError with the offending tuple index and the rule name. Do NOT catch the exception — fix the gold set.For each tuple, the calibration runner calls judge_fn(...) with kwargs matching the v3.8 judge interface:
- Alignment tuples (all four expected_judgment values including RETRIEVAL_FAILED) → retrieved_excerpt = tuple.ref_text_excerpt (may be null for RETRIEVAL_FAILED tuples), active_constraints=[]. The runner does NOT pre-filter RETRIEVAL_FAILED tuples — the judge stub is responsible for returning the matching label.
- Constraint tuples → retrieved_excerpt = tuple.ref_text_excerpt (typically null), active_constraints=[{"constraint_id": ..., "rule": ..., "scope": "MNC" | "NC"}]. Scope is derived from constraint_under_test_id: MNC-N → "MNC", NC-CN-M → "NC". Malformed ids are rejected by _derive_constraint_scope with a diagnostic naming the expected shape.
No ensembling at this layer. Spec §7.7 does not require N-run majority voting (reviewer mode does, but reviewer-paper unit-of-analysis is heavier and benefits more from variance reduction; per-tuple alignment is a simpler call). Re-running the same tuple is the operator's responsibility if a high-variance judge_model warrants it.
The runner accumulates two confusion matrices:
expected_judgment ≠ actual judgment. For alignment, every mismatched tuple counts as both FN (for the expected class) and FP (for the wrongly-picked class), so aggregate FNR and FPR are symmetric. For constraint tuples, VIOLATED is the positive class (gate-refuse signal); the matrix is the standard binary form.SUPPORTED, UNSUPPORTED, AMBIGUOUS, violated_constraint, the runner computes FNR / FPR with that class as the positive label. n_positive / n_negative denominators are surfaced so a "0.0 FNR on n_positive=0" entry is distinguishable from "0.0 FNR on n_positive=8".The fourth alignment class RETRIEVAL_FAILED is intentionally NOT surfaced in per_class: the pipeline sets it pre-judge at operational deployment, so judge-quality FNR/FPR against this label is uninformative for tooling validation. To keep the per-tuple call shape uniform, the runner still passes RETRIEVAL_FAILED tuples to judge_fn (no pre-filter) — the stub or real judge echoes the expected label, and the tuple contributes to aggregate FNR/FPR via the non-per_class aggregate path.
If report["FNR"] >= thresholds["FNR"] OR report["FPR"] >= thresholds["FPR"], the test (T-C1) fails. The report's thresholds block echoes the active gate values so CI failure messages can distinguish:
Both surface as the same CI failure; the report block makes the diagnosis explicit.
{
"FNR": 0.0,
"FPR": 0.0,
"per_class": {
"SUPPORTED": {"FNR": 0.0, "FPR": 0.0, "n_positive": 5, "n_negative": 12},
"UNSUPPORTED": {"FNR": 0.0, "FPR": 0.0, "n_positive": 8, "n_negative": 9},
"AMBIGUOUS": {"FNR": 0.0, "FPR": 0.0, "n_positive": 3, "n_negative": 14},
"violated_constraint": {"FNR": 0.0, "FPR": 0.0, "n_positive": 5, "n_negative": 3}
},
"partial_support": {"miss_rate": 0.0, "n_partial": 5},
"thresholds": {"FNR": 0.15, "FPR": 0.10},
"n_total": 25,
"n_alignment": 17,
"n_constraint": 8
}
The shape is the canonical contract scripts/test_claim_audit_calibration.py (T-C1 / T-C2) pins against. Adding fields is non-breaking; removing or renaming is a spec bump.
Partial-support subset (partial_support, #213). Gold tuples tagged expected_prompt_verdict=PARTIAL carry expected_judgment=UNSUPPORTED (the B1 normalization), so a judge that stops decomposing and emits a bare UNSUPPORTED matches the aggregate gate and hides the exact regression #213 closes. The subset metric counts a partial fixture as passed ONLY when the judge emits UNSUPPORTED AND a well-formed true-partial sub_claim_breakdown (≥2 items, ≥1 SUPPORTED AND ≥1 non-SUPPORTED — the INV-19 definition). miss_rate = misses / n_partial; a bare-UNSUPPORTED judge registers miss_rate=1.0 here while the aggregate FNR stays green. The block is additive — a gold set with no partial fixtures yields {"miss_rate": 0.0, "n_partial": 0}.
Calibration reports the judge's error profile on a specific gold set in a specific domain. It does not:
#89 / gold fixtures work tracked separately (per spec §2 out-of-scope).academic-pipeline/SKILL.md mode flags. v3.8.0 ships the audit agent dispatch as opt-in default-OFF; T-C1 passing on the canonical gold set is necessary but not sufficient evidence for default-ON.If a deploying operator brings a gold set that is itself biased (all tuples from one venue, all post-2024, all heavily-redacted), calibration reports a biased profile. Emit a warning during intake if the gold set looks clustered — this protocol does not currently auto-detect clustering, but the operator's pre-deployment review should.
Partial-evidence regression IS caught (#213). One failure the aggregate gate alone would NOT catch: a judge that regresses to emitting bare UNSUPPORTED on compound claims instead of decomposing them. Because partial fixtures carry expected_judgment=UNSUPPORTED, the aggregate FNR stays green on that regression. The partial_support subset metric exists precisely to surface it — miss_rate > 0 means the judge stopped producing inspectable sub_claim_breakdown decompositions even though it got the headline label right. Treat a non-zero partial_support.miss_rate as a judge-quality regression on the §F.3.2 partial-evidence trap, distinct from an aggregate FNR/FPR threshold breach.
| Mode | Interaction with calibration |
|---|---|
| Stage 4 → 5 audit dispatch | Calibration runs offline; results inform the ARS_CLAIM_AUDIT ramp-on decision. |
| Cite-Time Provenance Finalizer | No direct coupling — calibration evaluates the judge in isolation; the finalizer's matrix is downstream. |
/ars-mark-read clearance |
Calibration does not interact with mark-read state; HIGH-WARN-CLAIM-NOT-SUPPORTED stays non-clearable regardless of calibration outcome. |
Reviewer calibration mode |
Sibling, not coupled. Both compute FNR/FPR but on different units (papers vs claim-tuples); both can run in the same session. |
scripts/test_claim_audit_calibration as a CI gate. Re-calibration with a domain-specific gold set is the operator's call, not auto-triggered.ARS_CROSS_MODEL interacts with the audit agent dispatch path, not the calibration script. An operator wanting cross-model calibration runs the runner twice with different judge_model settings.PYTHONPATH=. python3 -m unittest scripts.test_claim_audit_calibration -v
The CI-equivalent invocation lives in .github/workflows/spec-consistency.yml (the v3.8 #103 unittest step). Both paths exercise T-C1 (threshold gates), T-C2 (per-class FNR/FPR reporting), and T-C3 (gold-set shape integrity).
docs/design/2026-05-15-issue-103-claim-alignment-audit-spec.md §7.7 (test contract) + §1 deliverable 7 (this doc) + §9 (acceptance criteria) + §13 step 10 (implementation order).academic-pipeline/agents/claim_ref_alignment_audit_agent.md — dispatches calibration mode and consumes the report's thresholds block. The agent prompt and this doc form a two-way pair following the v3.6.5 protocol-doc convention (literature_corpus_consumers.md ↔ bibliography_agent.md).academic-paper-reviewer/references/calibration_mode_protocol.md (same vocabulary, different unit of analysis).