Status: Design — brainstorm-driven, four rounds (Q1–Q4) settled before drafting; pending iterative codex review to zero finding before ship.
Date: 2026-04-30
Target release: v3.6.7 follow-up PR (Step 6 + Step 8 of v3.6.7 main spec §9)
Scope: Runtime enforcement of the v3.6.7 downstream-agent pattern protection layer via an artifact-as-contract audit gate at each pipeline_orchestrator_agent stage transition. Plus the Step 8 synthetic evaluation fixture set that demonstrates all 17 patterns trigger and protect end-to-end.
This spec implements two of the eight steps in the v3.6.7 main spec (2026-04-29-ars-v3.6.7-downstream-agent-pattern-protection-spec.md) §9 implementation table:
Steps 1–5 + Step 7 already shipped as v3.6.7 Step 1+2+7 (main b4fbffd, PRs #48 + #49). The runtime enforcement layer + the eval case are the remaining v3.6.7 work; this spec covers both because the eval case design is the survival test for the runtime layer and they share contract surfaces.
The v3.6.7 protection that already shipped is prompt-level + lint-level: each downstream agent prompt carries a PATTERN PROTECTION (v3.6.7) block, and a static lint script verifies the prompt has the right obligation phrasing in CI. That covers what the agent is told to do; it does not cover what the agent actually produces at runtime.
This spec adds runtime enforcement: at every stage transition where one of the three v3.6.7 downstream agents (synthesis_agent, research_architect_agent survey-designer mode, report_compiler_agent abstract-only mode) just produced a deliverable, the orchestrator must locate a matching audit artifact in Material Passport (Schema 9), verify the artifact passes Layer 2 (JSONL schema) + Layer 3 (sidecar metadata) anti-fake-audit checks, and read the verdict. PASS proceeds; MATERIAL blocks; MINOR escalates to user.
Audit execution itself sits outside the LLM session — codex CLI is not LLM-callable in any safe way, and feedback_subagent_tool_hallucination.md documents what happens when sub-agents claim audit completion they could not perform. The spec therefore treats the audit artifact as the boundary object: the deployment runs the audit (via the wrapper script in §4), the orchestrator reads the artifact. Neither side can ship without the other; faking either side is the failure this spec prevents.
The eval case (§7) is two-tier (17 micro + 1 integration), mirroring the existing 29-mutation lint suite's unit-vs-integration split that already ships in v3.6.7 Step 1+2.
Threat model (load-bearing — narrows the scope of what the anti-fake-audit guard claims to defend against):
The guard defends against inadvertent Pattern C3 over-claims: a sub-agent dispatched to produce a deliverable internalizes the multi-stage description of the audit pipeline in its prompt and emits output annotated with audit-passed metadata it never produced. This is the failure mode documented in feedback_subagent_tool_hallucination.md: the agent has no codex tool, never ran any audit, but the prompt's mention of a downstream audit step triggers hallucinated metadata. The guard is sufficient against this case because the hallucinated agent does not write Layer 2 JSONL or Layer 3 sidecar files at all — orchestrator finds no proposal file (§4.9), the gate blocks, the failure is loud.
The guard is not designed for motivated forgery: an actor with shell access who knows the spec can fabricate Layer 2 + Layer 3 + verdict files that pass all eleven gating checks. Layer 2 + Layer 3 raise per-attempt forgery cost (compose a syntactically-valid JSONL stream matching the §3.3 canonical event ordering — including an item.started / item.completed pair for any claimed tool call — with self-consistent canonical UUID thread_id, monotonic timestamps, real git_sha, matching bundle SHAs, exit_code 0, plus a verdict file whose Section 6 text passes parse_audit_verdict.py) but do not provide cryptographic non-repudiation. Defense against motivated forgery relies on §4.7 (wrapper not LLM-callable from the same session producing the deliverable) plus deployment-side discipline (audit runs in CI / SubagentStop hook / human-triggered out-of-band terminal). The motivated-forgery boundary is the rejected scope for v3.6.7 — Layer 5 (cryptographic signing) was considered and rejected as over-engineering for the inadvertent-Pattern-C3 threat surface declared above. §9 R2 carries the full risk discussion; the threat-boundary statement above is the load-bearing version that §1-§5 rely on.
This narrowed threat model — Pattern C3 inadvertent over-claim, not motivated forgery — is what justifies Layer 2 + Layer 3 as proportionate. Spec §3.3 / §3.4 / §4.7 / §5.2 / §9 R2 must all read as defending the same threat boundary; cross-section drift on this point would invite scope creep into signing or attestation infrastructure that the v3.6.7 timeline does not budget for.
The v3.6.7 main spec §2.2 lists "No schema change" as a non-goal. Step 6 amends that line to "No breaking schema change": Schema 9 gains one optional audit_artifact[] field, mirroring the v3.6.3 (reset_boundary[]), v3.6.4 (literature_corpus[]), and v3.4 (compliance_history[]) precedents documented in shared/handoff_schemas.md. All three precedents added optional append-friendly Schema 9 fields without breaking prior versions; Step 6 follows the same pattern.
The amendment is recorded here, not by editing the v3.6.7 main spec, because:
audit_artifact[] field is genuinely additive — the original "No schema change" framing was too absolutist given existing precedent.The v3.6.7 main spec §2.2 line should be read as: "No schema change beyond the optional audit_artifact[] add specified in Step 6 spec §3."
Two rounds of decisions converged before this spec started: a four-question brainstorm with the user (Q1–Q4) that fixed the major design axes, then a three-approach architecture comparison (Approach 1–3) that fixed the runtime model. The decisions below ride into §3–§7 as load-bearing assumptions; revisiting them requires re-opening the brainstorm.
| # | Question | Decision | Consequence |
|---|---|---|---|
| Q1 | Trigger scope — which agents auto-trigger audit? Just the three v3.6.7 downstream agents (A), Phase 2 + bibliography (B), or all stages (C)? | A — v3.6.7-only. Trigger only on synthesis_agent, research_architect_agent (survey-designer mode), report_compiler_agent (abstract-only mode). bibliography_agent patterns and other agents tracked separately. |
§5 stage-transition gate fires only at the three v3.6.7 agents' exit transitions. The five bibliography_agent hallucination patterns documented in feedback_ars_bibliography_agent_hallucination_patterns.md are out of scope for v3.6.7; they are candidates for v3.6.8+ on their own subline. Future scope expansion follows the same artifact-as-contract structure but is not a Step 6 deliverable. |
| Q2 | Execution model — where does audit run? Inline orchestrator dispatch (A), external hook script (B), sub-agent dispatch (C), or artifact-as-contract (D)? | D — artifact-as-contract. Orchestrator does not execute codex; it verifies that an audit artifact exists in the passport, that it passes Layer 2 + Layer 3 anti-fake-audit checks, and reads its verdict. Audit execution is a deployment concern. | §4 specifies the wrapper script contract; §5 specifies the orchestrator block protocol. The orchestrator agent prompt grows by one new "Audit Artifact Gate" subsection (§5.6) and does not internalize codex CLI semantics. Token cost of audit becomes a deployment-side operational cost, not a per-pipeline-run LLM cost. |
| Q3 | Anti-fake-audit guard — how does the orchestrator detect a forged audit artifact? Existence check only (Layer 1), JSONL schema (Layer 2), sidecar metadata (Layer 3), user attestation (Layer 4), or cryptographic signing (Layer 5)? | Layer 2 + Layer 3 + partial inversion. Layer 2 validates codex 0.125+ --json event-stream shape (opens with thread.started + turn.started; closes with turn.completed carrying non-zero usage integers; the body interleaves item.completed agent_message events — codex thinking aloud and the final verdict — with item.started / item.completed pairs for any tool calls, in arbitrary order; canonical UUID thread_id; parseable verdict text in the last agent_message item.completed per audit template Section 6 — see §3.3 for the full schema, the tool-call event pair, and §3.7 family A row A7 for the stream-shape pairing invariant). Layer 3 validates a sidecar <run_id>.meta.json written by the wrapper script (CLI version, hostname, cwd, git SHA, timestamps, exit code, jsonl_thread_id matching the JSONL stream's thread.started event). Partial inversion: agent prompts retain the "DO NOT simulate any audit step" prohibition but remove sentences that reveal the audit pipeline exists downstream (e.g., "The orchestrator runs codex audit afterward" — those sentences trigger Pattern C3 sub-agent fake-audit hallucination per feedback_subagent_tool_hallucination.md). |
§5.2 specifies the eleven gating verification checks (4 Layer 2 gates from L2-2/L2-3/L2-4/L2-5 + 7 Layer 3 gates from L3-2 through L3-8); two precondition slots L2-1/L3-1 are file/schema existence checks reported as "audit not conducted" rather than gating failures, so they are not counted in the eleven. §6 specifies the partial inversion sweep across the three downstream agent prompts. Layer 4 (user attestation) was rejected as violating the "automatic audit" goal; Layer 5 (signing) was rejected as over-engineering relative to the threat model. Two independent failure modes (Layer 2 + Layer 3) raise forgery cost to the point where running real codex is the economical path under the §1.2 threat model. |
| Q4 | Relation to v3.6.6 + Step 8 fixture shape. Is v3.6.6 generator/evaluator contract the same axis as Step 6 audit, and does Step 8 use single corpus (A), per-pattern micro-fixtures (B), or hybrid (C)? | Orthogonal + Hybrid (C). v3.6.6 (evaluator_full Schema 13.1 contract) is in-pipeline LLM self-discipline at the writer/evaluator pair; Step 6 audit is cross-model external codex verification at the deliverable boundary. They run in parallel: a deliverable can carry both an evaluator_full contract and an audit_artifact entry. Step 8 fixture: 17 per-pattern micro-fixtures + 1 chapter-level integration fixture. |
§8 records the orthogonality with a one-paragraph distinction and an explicit "do not merge" non-goal. §7 specifies both fixture tiers; §7.6 wires CI to run unit-level + integration-level test scripts separately. The 17 micro-fixtures align with the existing 29-mutation lint suite's unit philosophy already shipping in v3.6.7 Step 1+2. |
After Q1–Q4 settled the axes, the runtime model still had three plausible shapes. The block-or-warn axis is load-bearing: it determines whether v3.6.7's "ship-quality target" goal has runtime teeth or is only an aspiration.
| # | Approach | Block-or-warn | Decision |
|---|---|---|---|
| 1 | Strict Block | Orchestrator refuses stage transition without a passing audit artifact. MATERIAL verdict blocks; PASS proceeds; MINOR (≤3 P3) escalates to user. | Selected. |
| 2 | Soft Warn | Orchestrator displays audit verdict in checkpoint but allows user to proceed regardless. | Rejected — anti-fake-audit guard has no teeth without a block; "ship-quality 0 P1+P2" goal becomes display-only. |
| 3 | Tiered (high blast radius blocks; low blast radius warns) | User-configurable tier. | Rejected as YAGNI — Q1 fixed trigger to three high-blast-radius agents; no Tier B caller exists today. Reopen if v3.6.8+ adds low-stakes triggers. |
Strict Block is the only approach where Layer 2 + Layer 3 verification cost (Q3) is recovered. Soft Warn is internally inconsistent: spec §5.3 of the v3.6.7 main spec mandates anti-fake-audit verification, which is meaningless if the verification result does not gate progression. Tiered would be a future-proofing choice without a current Tier B consumer; revisit only when a low-blast-radius trigger is actually added.
These were raised during brainstorm or scope check but settling them needs schema or wrapper detail in §3–§7. Resolved in §9 as risks/open questions.
| # | Open question | Lands in |
|---|---|---|
| L1 | Round upper bound — does Strict Block cap at 3 rounds (matches audit template Section 1 default target_rounds) or 5 rounds (matches feedback_codex_iterative_spec_review_to_zero.md "iterate to one zero-finding round" guidance)? Both are defensible. |
§5.4 |
| L2 | Partial inversion sweep completeness — is the inversion rule applied only to the three v3.6.7 agents (Q1 scope) or across all ARS agent prompts that mention audit? Wider sweep is safer but is scope creep against Q1. | §6.3 |
| L3 | ARS_PASSPORT_RESET interaction — when a stage's audit artifact lives in passport at session A and the user resumes in session B, must session B re-verify the artifact or trust the passport? | §9 R4 |
| L4 | Wrapper script deployment friction — strict block means a user without the wrapper script installed cannot ship. How discoverable should this gate be? | §9 R1 |
Step 6 introduces one Schema 9 optional field plus four new standalone schemas under shared/contracts/ (entry, JSONL, sidecar, verdict). None of the existing schemas (Schema 9 base shape, Schema 13/13.1, literature_corpus_entry.schema.json, reset_ledger_entry.schema.json) is modified — every change is additive.
audit_artifact[] field (new optional)shared/handoff_schemas.md Schema 9 (Material Passport) gains one optional append-friendly field. Each entry records one audit run for one downstream agent's deliverable at one stage transition.
audit_artifact: # optional, added v3.6.7 Step 6
- stage: 2 # pipeline stage at which audit fired
agent: synthesis_agent # one of three v3.6.7 agents
deliverable_path: chapter_4/synthesis.md
deliverable_sha: a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2 # SHA-256 of deliverable file at audit time (full 64-hex per audit_artifact_entry.schema.json regex)
run_id: 2026-04-30T15-22-04Z-d8f3 # human-sortable run identifier
bundle_id: phase2-chapter4-2026-04-30 # optional, links to other entries in same logical bundle
bundle_manifest_sha: 9a8b7c6d5e4f3b2a1c0d9e8f7a6b5c4d3e2f1a0b9c8d7e6f5a4b3c2d1e0f9876 # SHA-256 over manifest of primary + supporting + template files (full 64-hex per schema regex)
artifact_paths:
jsonl: audit_artifacts/2026-04-30T15-22-04Z-d8f3.jsonl
sidecar: audit_artifacts/2026-04-30T15-22-04Z-d8f3.meta.json
verdict: audit_artifacts/2026-04-30T15-22-04Z-d8f3.verdict.yaml
verdict: # parsed from verdict.yaml; mirrored here for fast reads
status: MINOR # persisted enum: PASS | MINOR | MATERIAL (AUDIT_FAILED is proposal-arm only — see §3.2 Lifecycle-conditional fields). This example shows MINOR because p3=1; PASS requires all-zero per §3.2 cross-field rule
round: 2 # which audit round produced this verdict
target_rounds: 3 # cap; round must be <= target_rounds
finding_counts:
p1: 0
p2: 0
p3: 1
verified_at: "2026-04-30T15:23:11.847Z" # when orchestrator verified Layer 2+3 passed (persisted entry only); RFC 3339 UTC string, ms precision per §3.2 cross-section consistency rule (quoted in YAML so the schema's `string` + `rfc3339_ms_utc` regex match — unquoted ISO 8601 parses as a YAML datetime object and would fail the schema's `type: string` check)
verified_by: pipeline_orchestrator_agent
This example shows a persisted entry (verified_at / verified_by present). Proposal entries emitted by the wrapper are structurally identical but omit verified_at / verified_by; see §4.9 lifecycle and §3.2 schema oneOf for the proposal-vs-persisted distinction. Schema 9 only ever stores persisted entries; orchestrator validates passport-side audit_artifact[] entries with --mode persisted (proposal-mode entries in passport are treated as malformed and rejected by lint).
Field semantics:
stage is the integer destination stage the just-completed deliverable is about to enter (per §5.1 destination-stage convention). v3.6.7 hooks emit stage: 2 for synthesis_agent exit (gate fires before Stage 1→2 transition), stage: 2 for research_architect_agent survey-designer exit (gate fires before the survey instrument enters Material Passport at the Stage 2 boundary), and stage: 5 for report_compiler_agent abstract-only exit (gate fires before Stage 5 FINALIZE format conversion). The canonical destination-stage table lives in §5.1.agent is one of synthesis_agent | research_architect_agent | report_compiler_agent. Other agent names are reserved for future versions and rejected by Schema 9 lint.deliverable_sha is the SHA-256 of the deliverable file at the moment audit was dispatched. Used by orchestrator to detect post-audit deliverable mutation (audit pass on file X, then file X edited, then ship attempt — orchestrator must reject and require re-audit).run_id is the canonical handle the wrapper script (§4) emits. Format: <ISO-8601-Z>-<4-hex-suffix>. The suffix is required to disambiguate runs at sub-second timestamp resolution.artifact_paths.jsonl / sidecar / verdict are repo-relative paths to the three artifact files. They are co-located by run_id (same prefix); sidecar and verdict are written atomically via tmp-file-rename per §4.4, while JSONL is streamed by codex CLI directly to its final path (intentionally non-atomic — see §4.4 atomicity guarantee for why; the proposal-last lifecycle rule ensures orchestrator never observes JSONL mid-write). Missing any of the three is treated as audit-not-conducted. The fourth file the wrapper produces, <run_id>.audit_artifact_entry.json (the proposal entry consumed by the §4.9 lifecycle), is NOT recorded under artifact_paths because the entry IS the proposal once orchestrator has merged it; pointing at the proposal file from inside the persisted entry would create a self-reference loop.verdict is mirrored from the verdict file for fast read. Source of truth is verdict.yaml; if mirror disagrees with file, file wins (lint catches drift).Append-friendly semantics:
audit_artifact[] entries for the same (stage, agent, deliverable_sha) tuple ARE allowed — they represent multiple audit rounds. Orchestrator reads the latest entry by verified_at for verdict.deliverable_sha changes (deliverable mutated), prior entries become stale but are NOT deleted — they remain as audit history. Orchestrator only accepts entries whose deliverable_sha matches the current deliverable.reset_boundary[] append-only ledger pattern: history is preserved, freshness is computed.shared/contracts/passport/audit_artifact_entry.schema.json (new)Standalone schema for one audit_artifact[] entry. Referenced from Schema 9 via $ref. JSON Schema draft 2020-12.
The schema describes two lifecycle states for the same logical entry:
proposal state — emitted by the wrapper script (§4.9) at audit completion, before orchestrator verification. verdict.verified_at and verdict.verified_by MUST be absent; presence is treated as malformed (Pattern C3 attack surface).persisted state — appended to passport audit_artifact[] by the orchestrator after the eleven gating checks (§5.2) pass. verdict.verified_at and verdict.verified_by are now required and filled by the orchestrator.Both states are validated against the same schema file via JSON Schema oneOf. This mirrors the v3.6.6 spec §3.3 D1 allOf if mode startsWith reviewer_ conditional pattern: one schema, lifecycle-dependent constraints, idiomatic for state-machine entities.
Common fields (required in both states):
| Field | Type | Constraint |
|---|---|---|
stage |
integer | 1 <= stage <= 6 |
agent |
string | enum: ["synthesis_agent", "research_architect_agent", "report_compiler_agent"] |
deliverable_path |
string | repo-relative POSIX path (no .., no leading /) |
deliverable_sha |
string | ^[a-f0-9]{64}$ (SHA-256 hex) |
run_id |
string | ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}-[0-9]{2}-[0-9]{2}Z-[0-9a-f]{4}$ |
bundle_manifest_sha |
string | ^[a-f0-9]{64}$. SHA-256 over the canonical manifest of all bundle files (primary + supporting + template), per §3.6 freshness rule. |
artifact_paths.jsonl |
string | must end with .jsonl |
artifact_paths.sidecar |
string | must end with .meta.json |
artifact_paths.verdict |
string | must end with .verdict.yaml |
verdict.status |
string | enum in proposal arm: ["PASS", "MINOR", "MATERIAL", "AUDIT_FAILED"]. Three completion states + AUDIT_FAILED for wrapper-side aborts (§4.6). AUDIT_FAILED proposals must pass schema validation in §5.6 Path B4 so Path B5 can short-circuit on them; rejecting AUDIT_FAILED at the schema layer would deadlock the failure-signaling channel. enum in persisted arm: ["PASS", "MINOR", "MATERIAL"] (AUDIT_FAILED is excluded — see Lifecycle-conditional fields table below for the operational reason). Cross-field rule (lint-enforced) below limits AUDIT_FAILED to zero finding counts plus required failure_reason. |
verdict.failure_reason |
string | optional in proposal/persisted entries; required when verdict.status == "AUDIT_FAILED", forbidden otherwise. One-line human-readable reason (matches §3.5 verdict file rule). |
verdict.round |
integer | >= 1 |
verdict.target_rounds |
integer | >= 1, >= verdict.round (mirrors §3.5 verdict file target_rounds for cross-section consistency; resolves §5.4 cross-reference) |
verdict.finding_counts.{p1,p2,p3} |
integer | >= 0 |
bundle_id |
string | optional; opaque tag shared by all entries that belong to one logical multi-file audit bundle (see §4.5). When present, MUST match the bundle.bundle_id value in the corresponding sidecar. |
acknowledgement |
object | optional; present only on persisted entries appended via §5.4 ship_with_known_residue. When present, MUST contain finding_ids (non-empty array of strings, each matching one findings[].id in the companion verdict file), acknowledged_at (RFC 3339 datetime, UTC, ms precision), and acknowledged_by (const: "user"). Forbidden in proposal entries (wrappers cannot acknowledge residue). See §5.4 escalation mechanics for how this field is populated and §3.7 family A row A4 + family B rows B8/B10 + family C row C3 for the full invariant set. |
Lifecycle-conditional fields:
| Field | proposal state |
persisted state |
|---|---|---|
verdict.verified_at |
MUST be absent (not.required) |
required; RFC 3339 datetime, UTC (Z suffix), millisecond precision (mirrors §3.4 sidecar timing.* precision so latest-by-verified_at ordering at §3.1 / §5.6 Path A has resolution ≥ 1 ms — see §5.4 acknowledgement append for the monotonic-bump rule that closes the residual same-millisecond tie) |
verdict.verified_by |
MUST be absent (not.required) |
required; const: "pipeline_orchestrator_agent" |
verdict.status |
enum: ["PASS", "MINOR", "MATERIAL", "AUDIT_FAILED"] |
enum: ["PASS", "MINOR", "MATERIAL"] (AUDIT_FAILED excluded) |
Why persisted excludes AUDIT_FAILED: AUDIT_FAILED is not an audit result; it is evidence that the audit boundary failed to produce a trustworthy verdict (codex aborted, JSONL truncated, network timeout, etc.). The passport is a ledger of verified audit artifacts, not a log of attempted audit executions. Letting AUDIT_FAILED entries persist would force every downstream consumer (§5.3 ship/block, §5.6 Path A latest-by-verified_at selection, future analytics) to reason about a fourth durable state that lifecycle prose says must not exist. Forensic clarity for failed audits is preserved by leaving the failed proposal file plus its sidecar / verdict / JSONL in audit_artifacts/ (§4.9 step 9 also intentionally does not move AUDIT_FAILED proposal files into consumed/) and by surfacing failure_reason in the §5.6 block message. If a future ARS version needs durable forensic attempt history, the right shape is a separate audit_attempt[] ledger with weaker semantics, not overloading audit_artifact[].
JSON Schema sketch:
{
"$id": "audit_artifact_entry.schema.json",
"type": "object",
"required": ["stage", "agent", "deliverable_path", "deliverable_sha", "run_id", "bundle_manifest_sha", "artifact_paths", "verdict"],
"properties": { "...": "all common fields above" },
"oneOf": [
{
"title": "proposal",
"description": "Wrapper-emitted, pre-verification. acknowledgement is forbidden because wrappers cannot acknowledge residue — acknowledgement is exclusively orchestrator-written at §5.4 ship_with_known_residue.",
"not": {
"required": ["acknowledgement"]
},
"properties": {
"verdict": {
"type": "object",
"not": {
"anyOf": [
{"required": ["verified_at"]},
{"required": ["verified_by"]}
]
}
}
}
},
{
"title": "persisted",
"description": "Orchestrator-merged, post-verification. AUDIT_FAILED excluded — see Lifecycle-conditional fields rationale.",
"properties": {
"verdict": {
"type": "object",
"required": ["verified_at", "verified_by"],
"properties": {
"status": {"enum": ["PASS", "MINOR", "MATERIAL"]}
}
}
}
}
]
}
Validators called from §4.9 step 4 (proposal validation) and §5.2 / §5.6 (persisted validation) accept a --mode {proposal,persisted} flag that selects the corresponding oneOf arm. Lint script scripts/check_audit_artifact_consistency.py (added in §10 Phase 6.3) reads the mode flag and dispatches accordingly.
Cross-field rules (enforced by lint, not schema): these are the entry-side per-artifact rules; see §3.7 family A for the authoritative cross-artifact index and §3.7 families B / C / D / E / F for rules that span entry plus other artifacts.
verdict.status == "PASS" requires verdict.finding_counts.{p1,p2,p3} all zero AND failure_reason absent.verdict.status == "MINOR" requires p1 == 0 AND p2 == 0 AND p3 <= 3 AND failure_reason absent.verdict.status == "MATERIAL" requires p1 > 0 OR p2 > 0 OR p3 > 3 AND failure_reason absent.verdict.status == "AUDIT_FAILED" requires p1 == 0 AND p2 == 0 AND p3 == 0 AND failure_reason is a non-empty one-line string. (See §3.5 + §4.6.) AUDIT_FAILED is reachable only in the proposal arm (per the Lifecycle-conditional fields table); a persisted entry carrying status == "AUDIT_FAILED" is rejected at --mode persisted schema validation.verdict.round <= verdict.target_rounds (lifecycle invariant; see §5.4 escalation mechanics).acknowledgement is allowed only when verdict.status == "MATERIAL" AND the entry is in the persisted arm (proposal arm forbids acknowledgement entirely per the JSON Schema sketch). A persisted entry carrying acknowledgement with verdict.status of PASS / MINOR is rejected at --mode persisted validation. This closes the hand-edit attack surface where someone manually adds an acknowledgement block to a non-MATERIAL persisted entry to claim user residue acknowledgement that was never solicited (the §5.4 prompt only fires on round-3 MATERIAL).These rules live in scripts/check_audit_artifact_consistency.py.
shared/contracts/audit/audit_jsonl.schema.json (new — Layer 2)Schema for the codex CLI JSONL output that the wrapper script produces. This is the Layer 2 anti-fake-audit check from Q3: orchestrator validates the JSONL against this schema before reading any verdict.
Codex 0.125+ --json event-stream shape (load-bearing). Codex 0.125+ --json emits a typed event stream over stdout (verified compatible across 0.125 through 0.128): each line is a JSON object with a type field naming one of thread.started, turn.started, item.started, item.completed, turn.completed, or error. There is no per-row model field, no per-row reasoning_effort field, no session_id field, no final_message field, and no per-row usage field — those names belonged to a pre-0.125 draft and are retired here. The stable run identifier is thread_id carried on the opening thread.started event; usage lands on the closing turn.completed; the assistant verdict text lands inside an item.completed event whose item.type == "agent_message" and item.text carries the structured verdict (severity-bucket count summary per audit template Section 6).
Tool-call events. Whenever codex invokes a tool during the run (a git diff, a file read, etc.), the stream emits an item.started event when the tool call begins and a matching item.completed event when it returns. Both events carry an item block whose item.type names the tool kind (command_execution, file_change, etc.) — neither is an agent_message, so neither participates in verdict extraction. The audit prompt at shared/templates/codex_audit_multifile_template.md typically reads bundle files via tools, so most real audits emit at least one item.started / item.completed pair before the final agent_message; the no-tool case is reachable but rare (e.g., codex deciding the bundle context already in the prompt is sufficient and answering directly). L2-5 enforces only the pairing invariant — every observed item.started is matched 1:1 by a later item.completed sharing the same item.id (parallel tool starts complete in any order, see the schema-rule paragraph below) — not a minimum tool count, because requiring ≥1 tool pair would over-scope the anti-fake-audit guard (Q3 minimal-Layer-2 surface) and reject legitimate no-tool audits. Pre-0.125 drafts that enumerated only four events were wrong about the wire format; PR #52 round-4 surfaced the omission via empirical capture of codex exec --json against a live tool-using run.
A canonical no-tool run emits four events in order — this is the "minimum" pre-0.125 drafts described, and L2-4 validates against this minimum (the no-tool case is rare in practice but legitimate, per the preceding paragraph; the next paragraph shows the typical tool-using shape):
{"type":"thread.started","thread_id":"019de371-4c13-7521-8af7-fccf6bd23279"}
{"type":"turn.started"}
{"type":"item.completed","item":{"id":"item_0","type":"agent_message","text":"<verdict text>"}}
{"type":"turn.completed","usage":{"input_tokens":...,"cached_input_tokens":...,"output_tokens":...,"reasoning_output_tokens":...}}
A canonical tool-using run (the actual audit shape, since the audit template requires file reads) interleaves item.started / item.completed pairs around the assistant message:
{"type":"thread.started","thread_id":"019de371-4c13-7521-8af7-fccf6bd23279"}
{"type":"turn.started"}
{"type":"item.completed","item":{"id":"item_0","type":"agent_message","text":"<intermediate response>"}}
{"type":"item.started","item":{"id":"item_1","type":"command_execution"}}
{"type":"item.completed","item":{"id":"item_1","type":"command_execution"}}
{"type":"item.completed","item":{"id":"item_2","type":"agent_message","text":"<final verdict text>"}}
{"type":"turn.completed","usage":{"input_tokens":...,"cached_input_tokens":...,"output_tokens":...,"reasoning_output_tokens":...}}
Schema validates each event row by type. Required fields per event:
Event type |
Required additional fields | Constraint |
|---|---|---|
thread.started |
thread_id (string, UUID) |
canonical ^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$ (8-4-4-4-12 layout — ^[0-9a-f-]{36}$ was too loose; 36 dashes or hex without separators would have passed) |
turn.started |
(none additional) | |
item.started |
item.id, item.type |
tool-call begin marker (command_execution, file_change, etc.); carries no text; the matching item.completed also carries no text (only agent_message item.completed events carry text); does not participate in verdict extraction |
item.completed |
item.id, item.type, item.text (when item.type == "agent_message") |
item.text non-empty for agent_message items; non-agent_message completions (e.g. command_execution) carry no text |
turn.completed |
usage.input_tokens, usage.cached_input_tokens, usage.output_tokens, usage.reasoning_output_tokens (all integers) |
each >= 0; usage.input_tokens > 0 for non-trivial runs |
error |
error.message (string) |
only present on failed runs; companion verdict MUST be AUDIT_FAILED |
Schema-level rules:
type == "thread.started" and carries the run's thread_id. All subsequent events SHOULD NOT redeclare thread_id (codex emits it once at stream open).turn.completed. A failed run may end with error (or be truncated by SIGKILL — see §4.4 Step 2a JSONL placeholder).agent_message item.completed event in the stream carries the verdict text. Tool-using runs may emit intermediate agent_message events between tool-call item.started / item.completed pairs (codex thinking aloud); the verdict-bearing event is always the last agent_message item.completed, but it is NOT the final stream event — turn.completed (carrying usage) is always emitted after it as the closing event. The wrapper extracts this via parse_audit_verdict.py (§4.4 Step 4); the verdict text format is contracted by shared/templates/codex_audit_multifile_template.md Section 6 (severity-bucket count summary), NOT by this schema. (Pre-PR-#52 drafts said "exactly one" agent_message; that was correct for the no-tool minimum but rejected the actual tool-using audit shape.)item.started events appear only in tool-using runs and pair 1:1 with later item.completed events sharing the same item.id. Codex may start multiple tools concurrently (e.g., item_1, item_2, item_3 start in that order) and complete them in any order (e.g., item_1, item_3, item_2); pairing is by item.id only, not by completion-order FIFO. They are evidence that codex actually invoked tools (anti-fake-audit signal); verdict logic ignores them. Pairing is a stream-level invariant, not a per-row schema rule — Layer 2 (audit_jsonl.schema.json) validates each row's shape and §5.2 L2-3/L2-4 cover the canonical opening / closing / final-agent_message slots; the 1:1 item.started ↔ item.completed pairing check is enumerated in §3.7 family A row A7 and enforced by scripts/check_audit_artifact_consistency.py (Phase 6.3). Tool-event pairing was added to the canonical wire-format description here so implementers know the shape, but the gating burden stays per-row at Layer 2 by design (Q3 chose minimal Layer 2 surface; stream-level is Phase 6.3 territory). The §3.7 row makes the rule discoverable to the lint-script author without re-reading §3.3 prose.-m gpt-5.5 -c 'model_reasoning_effort="xhigh"' per §4.4 Step 2b. There is no event-stream field that records these — they are properties of the CLI invocation captured in the sidecar's prompt block, not Layer 2 evidence per se. Layer 3 therefore cannot cross-check the stream against model = "gpt-5.5"; this is a known limit of the 0.125+ --json shape and is captured in §9.1 R2 motivated-forgery boundary (a forger with shell access could invoke codex with a different model and the JSONL alone would not betray them; deployment-side discipline per §3.7 E9 is the defense).Why these specific fields: they are the events codex 0.125+ actually emits, validated for shape. A forger needs to fabricate a canonical UUID thread_id (8-4-4-4-12 layout), the canonical event ordering (with tool-call pairs if claiming a tool-using audit), usage integers on turn.completed, AND a syntactically-valid final item.completed.item.text agent_message that passes parse_audit_verdict.py (which itself enforces audit-template Section 6 schema). Layer 2 alone does not guarantee genuineness, but it raises the floor — echo '{}' >> fake.jsonl no longer passes; the canonical event sequence is the minimum forgery surface.
parse_audit_verdict.py contract (Phase 6.1 deliverable per §10): reads the JSONL stream, extracts the last item.completed event whose item.type == "agent_message" (intermediate agent_message events from codex thinking aloud between tool calls are skipped), and converts the audit-template Section 6 structured text into the <run_id>.verdict.yaml shape (§3.5). On parse failure (no agent_message event at all, malformed Section 6 text in the final agent_message, etc.), emits AUDIT_FAILED verdict with failure_reason: "JSONL parse error: <reason>" per §4.6 case (b). The wrapper invokes parser with --probe first to validate the JSONL has parseable shape; absence of parseable shape triggers the AUDIT_FAILED branch in §4.4 Step 4.
shared/contracts/audit/audit_sidecar.schema.json (new — Layer 3)Schema for the sidecar metadata file. This is the Layer 3 anti-fake-audit check: forging requires fabricating both the JSONL (Layer 2) AND a self-consistent sidecar.
# <run_id>.meta.json structure (validated by Layer 3 schema)
run_id: 2026-04-30T15-22-04Z-d8f3
codex_cli_version: 0.128.0 # from `codex --version`
runner:
hostname: runner.example.local # `uname -n`
cwd: /path/to/academic-research-skills
git_sha: b4fbffd # repo HEAD at audit start
git_dirty: false # uncommitted changes flag
timing:
started_at: "2026-04-30T15:22:04.123Z"
ended_at: "2026-04-30T15:22:58.471Z"
duration_seconds: 54.348
process:
exit_code: 0
stdout_path: audit_artifacts/2026-04-30T15-22-04Z-d8f3.stdout
stderr_path: audit_artifacts/2026-04-30T15-22-04Z-d8f3.stderr
stream:
jsonl_thread_id: 019de371-4c13-7521-8af7-fccf6bd23279 # MUST match the JSONL's `thread.started` event's `thread_id`
prompt:
audit_template_path: shared/templates/codex_audit_multifile_template.md
audit_template_sha: 7f4a8b2c3d5e6f1a9b0c8d7e6f5a4b3c2d1e0f9a8b7c6d5e4f3a2b1c0d9e8f7a
bundle:
bundle_id: phase2-chapter4-2026-04-30 # optional, opaque tag tying multi-file bundle entries
bundle_manifest_sha: 9a8b7c6d5e4f3b2a1c0d9e8f7a6b5c4d3e2f1a0b9c8d7e6f5a4b3c2d1e0f9876
primary_deliverables:
- path: chapter_4/synthesis.md
sha: a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2
supporting_context:
- path: chapter_4/bibliography.json
sha: e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6
- path: chapter_4/verification.md
sha: c2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3
Required field constraints:
| Field | Constraint |
|---|---|
run_id |
identical to file basename |
codex_cli_version |
semver ^[0-9]+\.[0-9]+\.[0-9]+$ |
runner.git_sha |
^[a-f0-9]{7,40}$ |
timing.started_at, ended_at |
RFC 3339 with millisecond precision |
timing.duration_seconds |
> 0, must equal ended_at - started_at (lint check) |
process.exit_code |
integer, 0 for accepted artifact (non-zero rejected by orchestrator) |
stream.jsonl_thread_id |
UUID format AND MUST match the JSONL's single thread.started event's thread_id field (cross-file lint) WHEN the companion verdict's status != "AUDIT_FAILED". WHEN status == "AUDIT_FAILED", jsonl_thread_id is allowed to be "" (empty string) because no usable JSONL thread exists; the UUID format constraint and cross-file rule 1 are both suspended for AUDIT_FAILED entries. Schema expresses this via JSON Schema conditional (if/then keyed on the companion verdict's status, resolved by Layer 3 lint when the two files are co-validated). Any other absence (missing key, null, non-string) is malformed regardless of verdict status. |
prompt.audit_template_path |
const "shared/templates/codex_audit_multifile_template.md" for v3.6.7 |
prompt.audit_template_sha |
SHA-256 of the template file as of audit time |
prompt.bundle.primary_deliverables[].sha |
MUST match audit_artifact_entry.deliverable_sha |
Cross-file rules (Layer 3 verification): these are the sidecar-anchored rules; see §3.7 family B for the full cross-artifact index (B1–B6 mirror rules 1–6 below; B7 mirrors rule 7 below; B8–B10 cover the acknowledgement / bundle_id / finding_ids rules that live elsewhere in §3 but apply to the same Layer 3 boundary).
The cross-file rules below apply only when the companion verdict's status != "AUDIT_FAILED". AUDIT_FAILED entries short-circuit at orchestrator §5.6 Path B5 before Layer 3 verification runs and are not subject to rules 1–7. The sidecar still gets written for AUDIT_FAILED runs (per §4.6) so that postmortem tooling can read runner / timing / exit_code, but the orchestrator does not feed those entries through Layer 3.
stream.jsonl_thread_id matches the JSONL stream's single thread.started event's thread_id. (Suspended for AUDIT_FAILED — jsonl_thread_id may be "" per the field constraint above.)prompt.bundle.primary_deliverables[].sha matches the corresponding audit_artifact_entry.deliverable_sha AND matches the current SHA-256 of that file at verification time.prompt.bundle.bundle_manifest_sha (per §3.6) matches the current bundle manifest computed from the union of primary_deliverables[], supporting_context[], and the audit template (prompt.audit_template_path + prompt.audit_template_sha).runner.git_sha is a valid commit in the current repo (not a fabricated hash).timing.ended_at - timing.started_at == duration_seconds within ±1 second tolerance.process.exit_code == 0. A non-zero exit code means audit was attempted but failed; the sidecar still gets written (per §4.6) for postmortem, but the companion verdict for any non-zero-exit run MUST be AUDIT_FAILED, and orchestrator MUST reject any non-AUDIT_FAILED audit_artifact whose sidecar carries non-zero exit_code (defense in depth against a forger pairing exit_code != 0 with a fabricated PASS/MINOR/MATERIAL verdict).sidecar.run_id equals the bare <run_id> basename of the sidecar file itself AND the bare <run_id> basename of every co-located artifact file (jsonl, verdict, plus the proposal entry file when observed at Path B before merge). The set of co-located artifacts differs by lifecycle — four files in proposal mode (jsonl + sidecar + verdict + proposal entry), three in persisted mode (proposal entry consumed at §4.9 step 9 and not recorded under artifact_paths); all co-located artifacts in the current lifecycle mode share one canonical bare <run_id> stem per §3.7 family F (F2/F3) — closes the "swap one co-located artifact for a different run's file" forgery seam (§3.7 family B row B7 carries the cross-artifact statement; this rule is the §3.4 sidecar-anchored half).These seven cross-file checks are the Layer 3 contribution beyond Layer 2's per-event schema. A forger needs to: produce a syntactically-valid JSONL stream that opens with thread.started + turn.started, closes with turn.completed, and interleaves agent_message item.completed events with any claimed tool-call item.started / item.completed pairs (matched by item.id; parallel tool starts may complete in any order) per §3.3 (Layer 2), produce a sidecar whose jsonl_thread_id matches the thread.started event's thread_id (rule 1), pin the deliverable SHA AND the live file SHA (rule 2), compute and pin the bundle manifest SHA correctly across all files (rule 3), use a real repo commit (rule 4), keep timestamps arithmetically self-consistent (rule 5), ensure exit_code is 0 (rule 6), AND keep sidecar.run_id consistent with the basename of every co-located artifact file in the current lifecycle mode (rule 7 — defeats per-file forgery by binding all co-located artifacts to one identifier; four files in proposal mode, three in persisted mode). Layer 2 + Layer 3 raise forgery cost to a level where running the real codex CLI is the economical path; this is sufficient against the threat model declared in §1.2 (Pattern C3 inadvertent over-claims), and known-insufficient against motivated forgery (the boundary discussed at §1.2 closing paragraph; §9.1 R2 carries the full risk write-up).
shared/contracts/audit/audit_verdict.schema.json (new — verdict file)Schema for the human-readable verdict file produced by the wrapper after parsing the JSONL stream's final item.completed event whose item.type == "agent_message" (per §3.3 event-stream contract — tool-using runs may interleave intermediate agent_message events between tool-call item.started / item.completed pairs; only the last such event is verdict-bearing). This is what the orchestrator reads for verdict status; the JSONL itself is treated as raw evidence for Layer 2 only, never directly parsed for verdict logic.
# <run_id>.verdict.yaml — clean-completion shape (status PASS / MINOR / MATERIAL)
run_id: 2026-04-30T15-22-04Z-d8f3
verdict_status: MINOR # one of PASS | MINOR | MATERIAL | AUDIT_FAILED
round: 2
target_rounds: 3
finding_counts:
p1: 0
p2: 0
p3: 1
findings:
- id: F-007
severity: P3
dimension: "3.7" # string enum value; one of audit-template §3.1-§3.7 or "4(f)"
file: chapter_4/synthesis.md
line: 482
description: "deictic temporal phrase 'currently' on reflexivity disclosure"
suggested_fix: "replace with 'as of 2026-04-30' or 'former director'"
generated_at: "2026-04-30T15:22:58.471Z"
generated_by: scripts/run_codex_audit.sh
generator_version: 1.0.0 # wrapper script semver
Required:
verdict_status enum: ["PASS", "MINOR", "MATERIAL", "AUDIT_FAILED"]. The fourth value AUDIT_FAILED exists for wrapper failure signaling per §4.6; semantics are "audit aborted, not completed", distinct from PASS/MINOR/MATERIAL (which all signify completed audit). Orchestrator BLOCKs on AUDIT_FAILED without running gating checks (per §5.6 Path B5).verdict_status consistent with finding_counts per §3.2 cross-field rules. AUDIT_FAILED requires finding_counts.{p1,p2,p3} == 0 AND failure_reason set.failure_reason (string, one line): required when verdict_status == "AUDIT_FAILED", forbidden otherwise. Human-readable reason ("codex exit 70: network timeout", "JSONL parse error at row 47").findings[].severity enum ["P1", "P2", "P3"]findings[].dimension enum: ["3.1", "3.2", "3.3", "3.4", "3.5", "3.6", "3.7", "4(f)"] matching audit templateround <= target_roundsCross-field rule — finding_counts MUST agree with findings[] (verdict-file rule per §3.7 family A row A5; load-bearing for §5.3 ship/block decisions; lint-enforced):
finding_counts.p1 == count(findings[severity == "P1"])finding_counts.p2 == count(findings[severity == "P2"])finding_counts.p3 == count(findings[severity == "P3"])verdict_status == "AUDIT_FAILED", findings MUST be [] (matches finding_counts.{p1,p2,p3} == 0 from the AUDIT_FAILED rule above; the empty list is the canonical representation, not a rule about absence).Without this invariant a verdict file could declare finding_counts: {p1: 0, p2: 0, p3: 0} while findings[] lists three P1 entries (or vice versa), and §5.3 ship/block decisions — which read finding_counts only — would silently disagree with the human-readable findings list. The invariant lives on the verdict file itself (the only artifact carrying findings[]); Schema 9 audit_artifact[].verdict is a counts-only mirror and is not in scope for this rule. Lint enforcement closes this gap on every wrapper-emitted verdict file; orchestrator §5.6 Path B verification re-runs the rule against the on-disk verdict file before merging the proposal entry into the passport.
AUDIT_FAILED variant (per §4.6 wrapper failure modes):
# <run_id>.verdict.yaml — failure-mode shape (status AUDIT_FAILED)
run_id: 2026-04-30T15-22-04Z-d8f3
verdict_status: AUDIT_FAILED
round: 2
target_rounds: 3
finding_counts:
p1: 0
p2: 0
p3: 0
failure_reason: "codex exit 70: network timeout after 600s"
findings: []
generated_at: "2026-04-30T15:22:58.471Z"
generated_by: scripts/run_codex_audit.sh
generator_version: 1.0.0
Both shapes share the same schema; the difference is verdict_status plus the failure_reason cross-field rule. Orchestrator §5.6 Path B5 short-circuits on AUDIT_FAILED without running the eleven gating checks.
The verdict file is the artifact orchestrator parses to make ship/block decisions. JSONL serves only as evidence that codex actually ran (Layer 2); sidecar serves as evidence that the runtime environment was real (Layer 3); verdict is the policy surface.
deliverable_sha (§3.1) detects mutation of the primary deliverable only. A multi-file audit bundle (§4.5) typically also depends on supporting context (bibliography.json, verification.md, prior-round verdict, etc.). If any of those files change between audit time and ship time, the audit becomes stale even though deliverable_sha matches.
To close this gap, the wrapper computes a bundle_manifest_sha covering every file the audit prompt references — primary deliverable plus all --supporting paths plus the audit template itself.
Manifest format (deterministic, line-oriented):
<role>:<repo-relative-path>:<sha256-hex>
One line per file. Roles: primary | supporting | template. Sort lines lexicographically by (role, path) for determinism. SHA-256 over the manifest text (UTF-8, LF line separator, trailing LF) is bundle_manifest_sha.
Example manifest text:
primary:chapter_4/synthesis.md:a1b2c3d4...
supporting:chapter_4/bibliography.json:e5f6a7b8...
supporting:chapter_4/verification.md:c2d3e4f5...
template:shared/templates/codex_audit_multifile_template.md:9a8b7c6d...
Where it lives:
prompt.bundle.bundle_manifest_sha (§3.4 amendment — see below).audit_artifact_entry.json at bundle_manifest_sha.audit_artifact[].bundle_manifest_sha after verification.Freshness check (§5.2 will run this): at stage transition, orchestrator recomputes the bundle manifest from current file contents (using the same (role, path) set the sidecar declares) and compares the resulting SHA-256 against the entry's bundle_manifest_sha. Mismatch on any path's SHA → manifest mismatch → audit stale → new audit round required.
Why a manifest, not per-file SHA verification: §5.2 stays a fixed-count check list (does not grow with bundle size); reproducibility check is one comparison per audit_artifact entry; humans can read the manifest text directly to debug staleness.
Per consult Q1 + Q2: §3 describes four schemas (entry, JSONL, sidecar, verdict) plus a Schema 9 amendment, but the real object under specification is a lifecycle state machine spread across those four artifact files plus the passport. Local schema rules in §3.1–§3.5 describe each artifact's shape; this subsection lists the cross-artifact invariants in one authoritative place so reviewers can audit top-down. Without this index, an invariant that lives only in one schema's local rule (or only in §4 / §5 prose) is invisible to anyone reading another schema in isolation — the convergence trajectory across rounds 1–9 of this spec demonstrated the cost: each round surfaced new "rules that were never listed together" rather than re-finding the same issue.
The six tables below enumerate every invariant under one of six families. Each row carries: ID (stable handle for citation in lint scripts and review notes), Rule (the constraint), Applies to (which artifact files / passport state the rule covers), Lifecycle states (proposal / persisted / both / wrapper-only / orchestrator-only), Enforcement point (where the rule is checked at runtime), Failure behavior (what happens when it fails). All rules are also implemented in scripts/check_audit_artifact_consistency.py (added in §10 Phase 6.3); the table is the source of truth, the script is the executable mirror.
Constraints that hold inside a single artifact file (no cross-file dependency).
| ID | Rule | Applies to | Lifecycle states | Enforcement point | Failure behavior |
|---|---|---|---|---|---|
| A1 | verdict.status enum agrees with verdict.finding_counts.{p1,p2,p3} per the four PASS / MINOR / MATERIAL / AUDIT_FAILED rules |
entry, verdict file | both | lint at write time; --mode {proposal,persisted} re-check at orchestrator merge |
malformed; rejected at the boundary that runs the validator |
| A2 | verdict.failure_reason required iff verdict.status == "AUDIT_FAILED" |
entry, verdict file | both | lint at write time | malformed; rejected |
| A3 | verdict.round <= verdict.target_rounds |
entry, verdict file | both | lint at write time; wrapper preflight on --round/--target-rounds (§4.2) |
wrapper exits 64 EX_USAGE; lint rejects |
| A4 | acknowledgement allowed only when verdict.status == "MATERIAL" AND entry is in persisted arm; proposal arm forbids acknowledgement entirely (per JSON Schema sketch in §3.2) |
entry | persisted only | --mode persisted schema validation; lint at acknowledgement-write time |
rejected at validation boundary; orchestrator refuses to append the malformed entry |
| A5 | verdict.finding_counts.pN == count(findings[severity == "PN"]) for N in |
verdict file (NOT entry — entry is a counts-only mirror per C1 below) | both | lint at wrapper write time; orchestrator §5.6 Path B re-check before passport merge | wrapper exits non-zero; orchestrator BLOCKs the merge |
| A6 | When verdict.status == "AUDIT_FAILED", findings == [] |
verdict file | both | lint at wrapper write time | rejected |
| A7 | JSONL stream tool-event pairing — true 1:1 with start-before-completion order: for every item.id appearing on a non-agent_message item.started event, the stream contains exactly one item.started and exactly one item.completed event with that item.id (no duplicate starts, no duplicate completions), and the item.completed row index is strictly greater than the matching item.started row index (a tool completion never precedes its start). Equivalently: the multiset of non-agent_message item.started item.id values equals the multiset of non-agent_message item.completed item.id values; each item.id appears at most once in each; and for each id the completed-row-index exceeds the started-row-index. Matching is by item.id only — parallel tool starts may complete in any order (codex starts e.g. item_1, item_2, item_3 then completes them in any permutation), but EACH id is paired exactly once and ordered start-before-completion. Orphan tool completions (a non-agent_message item.completed whose item.id never appeared earlier in an item.started) are rejected. agent_message item.completed events are exempt from the prior-item.started requirement — codex emits assistant messages as single completion events without an opening start, both for the final verdict and for intermediate "thinking aloud" responses between tool calls (per §3.3 canonical tool-using sequence); their item.id values are NOT counted in the multiset above. Stream-level invariant added to fill the L2-4 scope-defer (audit_jsonl.schema.json is per-row only; this is the only stream-shape rule that needs lint coverage in v3.6.7) |
JSONL | both | scripts/check_audit_artifact_consistency.py (Phase 6.3) at orchestrator §5.2 read time |
orchestrator BLOCKs with "audit fake-detected: tool event pairing violation: " |
Constraints that span two or more of {entry, JSONL, sidecar, verdict file, on-disk source files}.
| ID | Rule | Applies to | Lifecycle states | Enforcement point | Failure behavior |
|---|---|---|---|---|---|
| B1 | sidecar.stream.jsonl_thread_id matches the JSONL stream's thread.started event's thread_id field (suspended when companion verdict's status == "AUDIT_FAILED" per §3.4 conditional) |
sidecar, JSONL | both, plus AUDIT_FAILED suspension | §5.2 L3-2 gate check | orchestrator BLOCKs the merge with "audit fake-detected: jsonl_thread_id mismatch" |
| B2 | sidecar.prompt.bundle.primary_deliverables[].sha == entry.deliverable_sha == current_file_SHA256(deliverable_path) |
sidecar, entry, on-disk deliverable file | both | §5.2 L3-3 gate check | orchestrator BLOCKs with "audit stale: deliverable mutated" or "audit fake-detected: sha mismatch" |
| B3 | sidecar.prompt.bundle.bundle_manifest_sha == recomputed_bundle_manifest_sha (recomputed per §3.6 over current SHA-256 of every primary + supporting + template file) |
sidecar, on-disk bundle files | both | §5.2 L3-4 gate check | orchestrator BLOCKs with "audit stale: bundle file changed" or "audit fake-detected: manifest sha mismatch" |
| B4 | sidecar.runner.git_sha resolves to a real commit in the current repo |
sidecar, repo .git/ |
both | §5.2 L3-5 gate check (git cat-file -e <sha>^{commit}) |
orchestrator BLOCKs with "audit fake-detected: fabricated git_sha" |
| B5 | sidecar.timing.ended_at - timing.started_at == duration_seconds within ±1s |
sidecar | both | §5.2 L3-6 gate check | orchestrator BLOCKs with "audit fake-detected: timing arithmetic" |
| B6 | sidecar.process.exit_code == 0 for non-AUDIT_FAILED entries; non-zero exit_code permitted only when companion verdict's status == "AUDIT_FAILED" (per §3.4 rule 6 + F-027 conditional) |
sidecar, verdict | both, plus AUDIT_FAILED conditional | §5.2 L3-7 gate check; AUDIT_FAILED suspension at §5.6 Path B5 short-circuit | orchestrator BLOCKs with "audit attempted but failed" or rejects forged non-AUDIT_FAILED + non-zero-exit pairing (a forger pairing exit_code != 0 with a fabricated PASS / MINOR / MATERIAL verdict) |
| B7 | entry.run_id == sidecar.run_id AND sidecar.run_id is identical to the bare <run_id> basename of every co-located artifact file. The set of co-located files differs by lifecycle: proposal mode = four files (<run_id>.jsonl, <run_id>.meta.json, <run_id>.verdict.yaml, <run_id>.audit_artifact_entry.json); persisted mode = three files (proposal entry file is consumed at §4.9 step 9 — moved to consumed/ and not recorded under artifact_paths, see §3.1 closing note on why pointing at it would create a self-reference loop). One canonical filename convention throughout: bare <run_id> only, no stage / agent / deliverable prefix |
sidecar (canonical source of run_id), entry, the three persisted artifact files plus the proposal entry file when in proposal mode |
both, with lifecycle-asymmetric scope (4 files in proposal mode, 3 in persisted mode) | wrapper at write time uses $run_id for all paths (§4.4); §5.2 L3-1 precondition validates sidecar's own basename against its run_id field; §5.2 L3-8 gating check cross-validates entry.run_id == sidecar.run_id AND the sidecar's run_id against the basenames of the three artifact_paths files (jsonl / sidecar / verdict) for both modes, plus the proposal entry file basename only when running on a Path B unmerged proposal (proposal-mode-only fourth check) — closes the "swap one artifact file for a different run's file" forgery seam |
wrapper aborts with "internal: run_id mismatch"; orchestrator rejects at L3-1 (sidecar self-mismatch) or L3-8 (cross-file basename drift, including entry.run_id mismatch) with "audit fake-detected: artifact filename mismatch" |
| B8 | Persisted ack entries enforce verdict.verified_at == acknowledgement.acknowledged_at (the two timestamps share the same instant by construction per §5.4 strict-monotonic helper) |
entry | persisted only | lint at acknowledgement-write time; §5.6 Path B persisted-mode re-check before passport merge | rejected at validation boundary |
| B9 | When entry.bundle_id is present, it equals sidecar.prompt.bundle.bundle_id |
entry, sidecar | both | lint at wrapper write time | rejected |
| B10 | Persisted ack entry's acknowledgement.finding_ids satisfies all three: (1) non-empty array, (2) every ID exists in the companion verdict file's findings[].id, (3) full coverage — the set equals the set of every current findings[].id (no partial residue acknowledgement; rejected at write time, not at ship-decision time, so §5.3 "MATERIAL + acknowledgement" row never has to handle partial cases) |
entry (acknowledgement block), verdict file (findings[]) |
persisted only | lint at acknowledgement-write time (§5.4); §5.6 Path B persisted-mode re-check before passport merge | rejected at validation boundary; orchestrator surfaces "acknowledgement.finding_ids violates rule N" in the §5.6 BLOCK message |
The artifact-as-contract design (§2.2 Q2 = D) requires fields that appear in multiple files to agree. Each mirror row names the source of truth so disagreements have a deterministic resolution.
| ID | Rule | Source of truth | Mirror in | Enforcement point | Failure behavior |
|---|---|---|---|---|---|
| C1 | entry.verdict.{status, round, target_rounds, finding_counts, failure_reason} mirrors the verdict file's matching fields. On disagreement, verdict file wins (§3.1 line at "Source of truth is verdict.yaml") |
verdict file (<run_id>.verdict.yaml) |
passport entry's inline verdict block |
lint at wrapper write time; orchestrator §5.6 Path A A5 drift check on persisted entries (mirrored fields must match the on-disk verdict file); orchestrator §5.6 Path B B7 drift check on proposal entries before merge (proposal-side mirror must match verdict.yaml exactly — drift treated as Pattern C3 evidence and BLOCKs before passport append) | orchestrator BLOCKs with "audit drift: persisted entry verdict disagrees with verdict file" (Path A) or "Pattern C3: proposal mirror drift from verdict.yaml" (Path B) |
| C2 | entry.bundle_manifest_sha mirrors sidecar.prompt.bundle.bundle_manifest_sha |
sidecar (computed by wrapper) | passport entry | lint at wrapper write time | rejected |
| C3 | Acknowledgement append entry copies the latest persisted entry's (stage, agent, deliverable_path, deliverable_sha, run_id, bundle_manifest_sha, artifact_paths) and the inner verdict shape (verdict.status, verdict.round, verdict.target_rounds, verdict.finding_counts, verdict.failure_reason) byte-for-byte. The new entry's verdict.verified_at and verdict.verified_by are freshly set by orchestrator via the strict-monotonic helper (D3), NOT copied. C1's "verdict file wins" rule still applies — the new entry's mirrored verdict fields point at the same on-disk verdict file as the prior entry; C3 only governs which fields are copied versus refreshed during the append, not the source-of-truth direction |
latest persisted entry (for copied fields); current UTC via D3 (for refreshed fields) | new ack entry | §5.4 ack mechanism (orchestrator-side); lint re-checks copied fields equal prior entry's | rejected at acknowledgement-write time |
| C4 | entry.deliverable_sha == sidecar.prompt.bundle.primary_deliverables[].sha for the matching primary deliverable (this is the entry-side half of B2; B2 is the cross-artifact-plus-disk version) |
sidecar (computed at audit time) | passport entry | lint at wrapper write time | rejected |
The passport is an append-only ledger (§3.1) and Path A selection (§5.6) reads "latest" entries. Two independent ordering mechanisms exist (verified_at for entries, timing.started_at for proposals); each rule below names which mechanism applies and why the alternatives were rejected.
| ID | Rule | Applies to | Mechanism | Enforcement point | Failure behavior |
|---|---|---|---|---|---|
| D1 | Latest persisted entry is selected by max(verdict.verified_at) over all entries matching the current (stage, agent, deliverable_sha) tuple, conditional on D4: D1 applies only when no superseding higher-round proposal exists in <output-dir>. Total order is guaranteed by D3 strict-monotonic. When multiple entries share the same run_id (only reachable via §5.4 acknowledgement append, where ack entry copies the original MATERIAL entry's run_id), D1 selects the latest by verified_at within that run_id group — D3 strict-monotonic guarantees the ack entry post-dates the original by ≥ 1 ms |
passport audit_artifact[] entries |
verified_at (RFC 3339 ms UTC per A1's verdict.verified_at constraint) |
§5.6 Path A entry selection (after A1.5 supersession check); §5.3 ship/block read | tie is impossible by D3; if implementer's max() returns a non-deterministic tie due to bug, lint flags the duplicate verified_at |
| D2 | Path B proposal selection picks latest unmerged proposal by max(sidecar.timing.started_at) over proposals matching the tuple. run_id lex-max is the deterministic tie-breaker only on identical started_at. run_id lex-max alone is NOT chronological — the <ISO-8601-Z>-<4-hex> random suffix (sampled from /dev/urandom per §4.3) does not sort chronologically at sub-second resolution. In supersession mode (§5.6 A1.5 set the flag), B2 additionally filters to proposals whose verdict.round > selected_persisted.verdict.round per D4 |
proposal files in <output-dir> |
sidecar.timing.started_at primary, run_id lex-max tie-breaker; in supersession mode, additional verdict.round > prior filter |
§5.6 Path B2 | orchestrator BLOCKs with "ambiguous proposal selection at started_at; in supersession mode with no candidate surviving the higher-round filter (every candidate was lower-round or invalid at B2 selection time), B3 BLOCKs with "requested higher-round audit artifact is missing" — proposals that pass B2 but fail B6 / B7 / B8a get the proposal-specific BLOCK from P-PB-gate / P-PB-verdict-schema / P-PB-stale-late instead |
| D3 | Every orchestrator-side verified_at write (Path B8d normal merge, another_round re-merge, ship_with_known_residue ack append) goes through _next_verified_at_ms(passport_audit_artifacts) defined in §5.4 — guarantees verified_at strictly post-dates every prior persisted entry's verified_at regardless of clock granularity. Empty-ledger base case returns now_ms |
passport audit_artifact[] entries |
t = max(now_ms, max_prior_verified_at + 1ms), or now_ms when no prior entries exist |
orchestrator at every persisted-entry write | helper enforces invariant by construction; lint additionally verifies the result is > every prior entry as defense-in-depth |
| D4 | Higher-round unmerged proposals supersede lower-round persisted entries for path selection (F-070 closure). When orchestrator scans <output-dir> at §5.6 A1.5 and finds an unmerged proposal for the same (stage, agent, deliverable_sha) tuple whose verdict.round is greater than the A1-selected persisted entry's verdict.round, Path A is preempted and Path B runs with supersession_required = true. The signal is the wrapper artifact itself (round=N+1 written by --round N+1 invocation), not a durable orchestrator state — preserves the artifact-as-contract design (Q2). Covers another_round user choice (§5.4), manual fresh wrapper run, byte-identical abort_stage re-audit, AND higher-round AUDIT_FAILED proposal (which preempts and BLOCKs with failure reason rather than silently falling back to prior MATERIAL) |
passport audit_artifact[] entries + proposal files in <output-dir> |
verdict.round comparison after B1a-style run_id de-dup excludes already-persisted leftovers from the proposal scan |
§5.6 A1.5 (preflight) and Path B B2 (supersession-mode filter) | A1.5 forces Path B with supersession flag; B3 BLOCKs with "requested higher-round audit artifact is missing" if no candidate survives B2's higher-round filter (proposals that pass B2 but fail B6 / B7 / B8a get the proposal-specific BLOCK instead, NOT the supersede-missing diagnostic) |
Constraints on which actor is allowed to perform each mutation. Violations of these rules are the primary attack surface for Pattern C3 (fake-audit forgery via direct passport writes).
| ID | Rule | Actor | Boundary | Enforcement point | Failure behavior |
|---|---|---|---|---|---|
| E1 | Wrapper writes only to its --output-dir; never to the passport file |
wrapper script | filesystem | wrapper-side discipline (canonical wrapper ships in repo per §4.1; deployment-written wrappers risk this rule) | a deployment-written wrapper that writes to passport bypasses E2 — defended by §4.7 (wrapper not LLM-callable) plus deployment hygiene |
| E2 | Orchestrator is the only writer of audit_artifact[] entries into Schema 9 |
orchestrator | code path | orchestrator agent prompt + §10 implementation | lint-detectable as "non-orchestrator writer modified passport" only post-hoc; primary defense is code-path discipline |
| E3 | verdict.verified_at and verdict.verified_by are set ONLY by the orchestrator on one of two write paths: (a) Path B8d normal proposal merge, (b) §5.4 ship_with_known_residue ack append. Both paths use D3's strict-monotonic helper |
orchestrator | code path; schema | wrapper-emitted proposals carrying these fields are rejected at §5.6 Path B4 (--mode proposal schema rejection) |
rejected as Pattern C3 attack surface |
| E4 | A wrapper-emitted proposal file with verdict.verified_at or verdict.verified_by filled is treated as malformed |
wrapper output | schema | §5.6 Path B4 / §4.9 step 4 schema validation in --mode proposal |
rejected; proposal file remains in <output-dir> (not moved to consumed/); user inspects |
| E5 | AUDIT_FAILED entries are never persisted into Schema 9 (proposal-arm only per A1's persisted-arm enum exclusion) | orchestrator | schema | --mode persisted schema validation rejects AUDIT_FAILED verdict.status |
rejected at validation boundary; orchestrator surfaces failure_reason in §5.6 block message but does not append |
| E6 | Append-only ledger semantics: existing audit_artifact[] entries are never mutated. Multiple entries per (stage, agent, deliverable_sha) tuple are allowed and represent multi-round audit history. Acknowledgement is implemented as a NEW append (per C3) not as in-place mutation of the prior entry |
orchestrator | code path | orchestrator agent prompt + §5.4 mechanism | a hand-edited mutation is detectable post-hoc by hash-chain verification (future v3.6.8+); for v3.6.7 this is discipline + code-review |
| E7 | Acknowledgement is a sibling annotation, not a verdict.status mutation. The new ack entry's verdict.status remains MATERIAL regardless of acknowledgement presence |
orchestrator | C3 copy contract | §5.4 mechanism + B8 lint | rejected at write time if implementer mutates verdict.status to a synthetic "MATERIAL_ACKNOWLEDGED" or similar |
| E8 | The proposal entry file (<run_id>.audit_artifact_entry.json) is the LAST file the wrapper writes (after JSONL + sidecar + verdict are all on disk). This guarantees orchestrator never sees a proposal pointing at incomplete artifact paths |
wrapper script | filesystem ordering | wrapper-side atomicity (tmp-file-rename per §4.4) | orchestrator that finds no proposal entry treats the run as not-conducted; no false PASS reachable through partial writes |
| E9 | Wrapper is not invoked by an in-LLM Bash tool call from the same session producing the deliverable (per §4.7). Allowed invokers: human interactive shell, CI workflow step, SubagentStop hook, second-terminal Bash from outside the LLM session. Same-session in-LLM invocation creates Pattern C3 exposure (the LLM that produced the deliverable has motive and means to fabricate audit metadata) — this is the structural attack-surface boundary that the artifact-as-contract design (Q2) was chosen to enforce | deployment + agent prompts | invocation-site discipline | spec-level: §6 partial inversion removes "the orchestrator runs codex audit afterward" sentences from the three downstream agent prompts so the agent has no awareness that a downstream audit exists (no trigger for Pattern C3 fabricated tool-call hallucination); deployment-level: audit runs in CI / SubagentStop hook / human-triggered out-of-band terminal | spec-level: a partial inversion regression (sentence re-added to an agent prompt) is caught by the §6 inversion-sweep lint; deployment-level: a same-session in-LLM call is undetectable from artifact evidence alone — the §1.2 motivated-forgery boundary applies and §9 R2 carries the residual risk discussion |
| ID | Rule | Applies to | Enforcement point | Failure behavior |
|---|---|---|---|---|
| F1 | run_id format: <ISO-8601-Z>-<4-hex> where the date-time component is RFC 3339 second precision and the 4-hex suffix is from /dev/urandom (per §4.3) |
wrapper output, sidecar, entry, all artifact filenames | audit_artifact_entry.schema.json regex ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}-[0-9]{2}-[0-9]{2}Z-[0-9a-f]{4}$ |
rejected as schema violation |
| F2 | All four artifact file basenames use the bare <run_id> stem with extensions .jsonl, .meta.json, .verdict.yaml, .audit_artifact_entry.json. No stage / agent / deliverable prefix |
wrapper output | wrapper at write time uses $run_id directly per §4.4 pseudocode |
wrapper aborts; lint rejects mis-named files |
| F3 | sidecar.run_id is identical to file basename (this is the artifact-side mirror of F2; B7 is the multi-file consistency version) |
sidecar | audit_sidecar.schema.json field constraint |
rejected |
| F4 | All §3.1 / §3.4 / §4.3 / §4.4 / §4.9 spec examples use the bare <run_id> filename pattern. Pre-Round-9 examples that included a <stage>-<agent>- prefix were retired in F-048 closure |
spec text | review at spec edit time | spec drift caught by scripts/check_audit_artifact_consistency.py example-validation harness |
Six families partition the rule space cleanly: A is local (one file), B is cross-file but synchronous-evidence (Layer 3 sees them all together), C is mirror (the same data appearing in multiple places needs an authoritative source), D is temporal ordering (the only rule family where bugs produce non-deterministic behavior rather than rejection), E is who-can-do-what (the attack-surface family), F is naming (the family that bugs in earlier rounds repeatedly missed because filename rules were spread across §3 examples vs §4 contract).
A reviewer auditing the spec end-to-end can read each table once and have full coverage of cross-artifact constraints. Rounds 1–9 of this spec demonstrated the cost of NOT having this index: F-030 (A5 was missing), F-032 (E5 was implicit), F-034 (C3 was undefined), F-039 (D1 had no tie-breaker), F-043 (D3 wasn't named), F-046 (D3's empty-ledger case was undefined), F-047 (D2 had no rule), F-048 (F2/F3/F4 disagreed) were all rules that no single §3.1–§3.5 reader would have known to check. Round 15 added F-070's escalation-deadlock fix (D4 superseding-proposal precedence) — the index made it cheap to add a new D-row and update D1/D2 with the supersession conditional rather than burying the rule in §5.6 prose. Future invariant additions (e.g., when v3.6.8 lands hash-chain verification per E6) get a new row in the appropriate family.
| Schema | Path | Purpose | Touches existing? |
|---|---|---|---|
Schema 9 audit_artifact[] |
shared/handoff_schemas.md |
Append-friendly audit history per deliverable | Adds optional field; no rename, no required change |
audit_artifact_entry.schema.json |
shared/contracts/passport/ |
Per-entry shape for audit_artifact[] |
New file |
audit_jsonl.schema.json |
shared/contracts/audit/ |
Layer 2 — codex JSONL row schema | New file |
audit_sidecar.schema.json |
shared/contracts/audit/ |
Layer 3 — sidecar metadata schema | New file |
audit_verdict.schema.json |
shared/contracts/audit/ |
Verdict file schema | New file |
The shared/contracts/audit/ directory is new in v3.6.7. The README at shared/contracts/README.md (added v3.6.6 per generator-evaluator spec §1) gains one section documenting the audit/ directory's purpose, mirroring the v3.6.6 convention.
The wrapper script is the boundary object between deployment-side audit execution and ARS-side artifact verification. ARS spec defines what the wrapper must produce (§3 schemas); deployment owns how the wrapper runs (codex CLI invocation, network access, retries). This separation is what makes Q2's artifact-as-contract execution model work — the orchestrator never needs to know what wrapper is, where codex lives, or whether the audit happens locally / on a build server / asynchronously.
Path: scripts/run_codex_audit.sh
Shell: Bash 4+ (#!/usr/bin/env bash). The §4.4 pseudocode relies on Bash-only features — indexed arrays (PRIMARY=(...), ${arr[@]}, ${arr[*]}), declare -a, read -ra, and here-strings (<<<) — so POSIX sh will not run this wrapper. Portable across macOS / Linux / WSL where Bash 4+ is available; macOS ships Bash 3.2 by default and users on stock macOS must install a newer Bash via Homebrew (brew install bash) or run on Linux/WSL. Exit code 64 (EX_USAGE) is emitted with a clear message if the wrapper detects BASH_VERSION < 4.
Owner: ARS repo. Maintained as part of the v3.6.7 deliverable; ships with the repo so users do not need to write their own.
Versioned: carries a top-of-file # version: 1.0.0 comment that audit_verdict.schema.json field generator_version must match. Wrapper updates that change verdict shape MUST bump version and update CI fixtures.
Required external commands (wrapper preflights for each at startup; missing one exits 64 EX_USAGE with a "missing dependency:
| Command | Used for | macOS availability | Linux/WSL availability |
|---|---|---|---|
bash (>=4) |
shell | Homebrew (brew install bash); stock 3.2 unsupported |
distro default |
git |
rev-parse --short HEAD, diff --quiet |
Xcode CLT or Homebrew | distro default |
awk |
column extraction in _sha256 helper |
BSD awk preinstalled | GNU awk preinstalled |
jq |
parse JSONL thread.started event's thread_id |
Homebrew (brew install jq) |
distro package |
od, tr |
run_id 4-hex suffix from /dev/urandom |
preinstalled | preinstalled |
uname |
hostname (uname -n) |
preinstalled | preinstalled |
sort |
deterministic ordering for the bundle manifest (LC_ALL=C sort) |
preinstalled (BSD sort) | preinstalled (GNU sort) |
head |
extract first matching JSONL event's thread_id (jq ... | head -1) |
preinstalled | preinstalled |
tee |
split codex --json stdout stream into the JSONL contract file plus the <run_id>.stdout diagnostic file in one pipeline (§4.4 Step 2b). Combined with ${PIPESTATUS[@]} for tee-side error detection so a tee write failure surfaces as wrapper exit 73 rather than silently corrupting the JSONL artifact |
preinstalled | preinstalled |
mv |
atomic tmp-file-rename for the three tmp-rename-bound contract files — sidecar, verdict, proposal entry (§4.4 atomicity guarantee). JSONL is intentionally exempt because codex streams it over the audit's runtime; see §4.4 for the proposal-last guarantee that makes JSONL's mid-write visibility unobservable to orchestrator | preinstalled | preinstalled |
python3 |
fsync inside the §4.4 _atomic_write helper (os.fsync(open(tmp).fileno())); helper falls back to sync(8) when python3 is unavailable. Also assumed present by parse_audit_verdict.py shipped with the wrapper |
preinstalled | distro default (Python 3.x in /usr/bin/python3) |
codex |
actual audit invocation | Homebrew (brew install codex or codex installer) |
codex installer |
SHA-256 helper (one of sha256sum OR shasum) |
content hashing throughout §4.4 | macOS ships shasum only (shasum -a 256); sha256sum requires brew install coreutils |
Linux/WSL ship sha256sum |
The §4.4 pseudocode shows sha256sum for readability; the actual wrapper defines a single _sha256() helper that prefers sha256sum when present and falls back to shasum -a 256 otherwise (see §4.4 helper definition). All sha256sum "$path" | awk '{print $1}' call sites in the pseudocode therefore correspond to _sha256 "$path" in the actual script.
Why a shipped wrapper, not deployment-written: if every deployment writes its own wrapper, sidecar field semantics drift (different hostname conventions, different timing precision, different exit code mapping). Layer 3 cross-file rules then cease to be enforceable across deployments. Shipping the canonical wrapper guarantees sidecar uniformity.
scripts/run_codex_audit.sh \
--stage <stage> # required, integer 1-6
--agent <agent_name> # required, one of three v3.6.7 agents
--deliverable <path> # required, repo-relative path
--supporting <path1,path2,...> # optional, comma-separated paths
--round <N> # required, integer >= 1
--target-rounds <N> # optional, integer >= 1, defaults 3 (matches audit template Section 1 default; orchestrator §5.4 round upper bound is the policy-side cap)
--previous-findings <path> # optional, prior round verdict file (for round > 1)
--output-dir <dir> # optional, defaults audit_artifacts/
--bundle-id <opaque-string> # optional, used to group multi-file Phase 2 audits
--dry-run # optional, validates inputs without invoking codex
Input validation rules (wrapper rejects with exit code 64 = EX_USAGE):
--agent must match the enum from audit_artifact_entry.schema.json §3.2.--deliverable must exist and be readable. Wrapper computes its SHA-256 and pins into sidecar.--supporting paths must exist (audit prompt cannot reference missing files).--round must be <= --target-rounds.--previous-findings is required when --round > 1; absent triggers reject (round 2+ without round-1 findings is per audit template Section 4(a) violation).--bundle-id must be the same value for all files in one multi-file Phase 2 audit (per §4.5 below).For a successful audit run, the wrapper produces exactly four contract files in --output-dir. Three of them (sidecar, verdict, proposal entry) are written atomically via tmp-file-rename (each file fsynced and rename-committed before script returns); the fourth (JSONL) is streamed by codex CLI directly to its final path and may be observed mid-write — see §4.4 atomicity guarantee for why JSONL is exempt and why the proposal-last write order makes JSONL's mid-write visibility never reach orchestrator state:
| File | Schema | Purpose |
|---|---|---|
<run_id>.jsonl |
audit_jsonl.schema.json |
Raw codex CLI output (Layer 2 evidence) |
<run_id>.meta.json |
audit_sidecar.schema.json |
Wrapper-emitted runtime metadata (Layer 3 evidence) |
<run_id>.verdict.yaml |
audit_verdict.schema.json |
Parsed verdict (orchestrator policy input) |
<run_id>.audit_artifact_entry.json |
audit_artifact_entry.schema.json (--mode proposal) |
Proposal entry for orchestrator to merge into Schema 9 audit_artifact[] (§4.9 lifecycle) |
run_id format: <ISO-8601-Z>-<4-hex-suffix> per §3.1, generated by wrapper at script start. The 4-hex suffix is od -An -N2 -tx1 < /dev/urandom | tr -d ' ' to disambiguate sub-second runs.
Plus three non-required diagnostic files (kept for debugging, not part of contract):
<run_id>.stdout — codex CLI stdout<run_id>.stderr — codex CLI stderr (always written, even on success — codex emits warnings here)<run_id>.manifest.txt — the canonical manifest of primary / supporting / template entries hashed into bundle_manifest_sha (§4.4 Step 0b, §3.6 contract). Kept for audit-trail forensics so a future reader can see which files were in scope at audit time without re-deriving from bundle_manifest_sha alone.These are referenced from the sidecar's process.stdout_path / stderr_path (and from §4.4 Step 0b for the manifest) but Layer 3 verification does NOT read them. Their existence is asserted by sidecar but their content is opaque to ARS.
# Pseudocode — actual script implementation lives in scripts/run_codex_audit.sh
# Bash 4+ required (see §4.1): uses indexed arrays, read -ra, here-strings, declare -a.
# Required external commands enumerated in §4.1 dependency table; preflight runs first.
# Portable SHA-256 helper (see §4.1). Prefers GNU sha256sum, falls back to BSD
# shasum -a 256 (the only SHA-256 tool on stock macOS). Every "sha256sum X | awk
# '{print $1}'" pattern in this pseudocode is shorthand for "_sha256 X" in the
# actual script.
_sha256() {
if command -v sha256sum >/dev/null 2>&1; then
sha256sum "$1" | awk '{print $1}'
else
shasum -a 256 "$1" | awk '{print $1}'
fi
}
# Crash-tolerant JSONL thread_id extractor (F-059 closure). Returns empty
# string when the JSONL file is missing, empty (codex killed before opening
# its output file — see Step 2a placeholder), or contains no parseable
# thread.started event with thread_id field. §3.4 conditional already permits
# empty stream.jsonl_thread_id when the companion verdict status is
# AUDIT_FAILED; this helper is the canonical path that emits the empty value
# cleanly without crashing.
#
# codex 0.125+ emits the thread_id only on the opening `thread.started` event
# (per §3.3 event-stream contract). Earlier draft parsed `.session_id` per
# every row — that field does not exist in 0.125+ output and the parse
# returned null on every clean run. The current jq filter selects only events
# whose `type == "thread.started"` and pulls their `thread_id`.
_extract_jsonl_thread_id() {
local path="$1"
if [ ! -s "$path" ]; then
printf ''
return
fi
jq -r 'select(.type == "thread.started") | .thread_id' "$path" 2>/dev/null \
| head -1
}
# Extract semver from `codex --version` output. codex 0.125+ prints
# "codex-cli X.Y.Z" (e.g., "codex-cli 0.128.0"); §3.4 sidecar schema's `codex_cli_version` field is a
# bare semver string with regex constraint `^[0-9]+\.[0-9]+\.[0-9]+$`. Match
# the first dotted-triple in stdout and reject (EX_USAGE) if absent — defends
# against future CLI versions that print multi-line stdout or unexpected
# formats and would otherwise let malformed text leak into the sidecar.
_codex_version() {
local v
v=$(codex --version 2>/dev/null \
| awk 'match($0, /[0-9]+\.[0-9]+\.[0-9]+/) { print substr($0, RSTART, RLENGTH); exit }')
[ -n "$v" ] || exit 64
printf '%s\n' "$v"
}
# Atomic write helper (§4.4 atomicity guarantee). Writes stdin to <path>.tmp,
# fsyncs the tmp file, then renames it over <path>. Used for sidecar, verdict,
# and proposal entry — the three contract files §4.4 guarantees orchestrator
# observes whole-or-not-at-all. JSONL is intentionally NOT routed through this
# helper because codex CLI streams it to its final path over the audit's
# runtime; the proposal-last write order makes JSONL's mid-write visibility
# unobservable to orchestrator. In the §4.4 pseudocode below, every redirection
# of the form `... > "$OUT_DIR/$run_id.{meta.json,verdict.yaml,
# audit_artifact_entry.json}"` is shorthand for `... | _atomic_write
# "$OUT_DIR/$run_id.<filename>"` — the actual script wraps each of those three
# writes in this helper.
_atomic_write() {
local path="$1"
local tmp="${path}.tmp"
cat > "$tmp"
# Best-effort fsync: prefer Python which is already required for parse
# helpers; fall back to `sync` if Python is unavailable for some reason.
if command -v python3 >/dev/null 2>&1; then
python3 -c "import os,sys; f=open(sys.argv[1]); os.fsync(f.fileno()); f.close()" "$tmp"
else
sync
fi
mv -f "$tmp" "$path"
}
# Step 0a — dry-run early exit. `--dry-run` is contracted by §4.2 as input-
# validation only; per §10 Phase 6.1 verification gate it MUST NOT write any
# files under `--output-dir`. Earlier draft had no dry-run branch, so the
# wrapper would walk through run_id generation, manifest write, JSONL
# placeholder pre-touch, codex invocation, sidecar / verdict / proposal
# emission — creating a fake audit artifact at zero codex cost (the exact
# Pattern C3 attack surface §1.2 defends against). All input validation
# (--agent enum, --deliverable readable, --round <= --target-rounds, etc.)
# has already happened upstream in the §4.2 input parser; reaching this
# point means inputs are valid, so dry-run exits 0 with a one-line stdout
# diagnostic and never touches --output-dir.
if [ "$DRY_RUN" = true ]; then
printf '[run_codex_audit] dry-run: inputs valid; no files written\n'
exit 0
fi
run_id=$(generate_run_id)
sha_deliverable=$(_sha256 "$DELIVERABLE")
git_sha=$(git rev-parse --short HEAD)
git_dirty=$(git diff --quiet && echo false || echo true)
# Step 0 — normalize CLI inputs into bash arrays.
# §4.2 specifies --supporting as a comma-separated string; here we split
# into PRIMARY and SUPPORTING arrays for downstream iteration. The PRIMARY
# array always contains exactly one element (the deliverable); SUPPORTING
# may be empty, which is permitted.
PRIMARY=("$DELIVERABLE")
if [ -n "$SUPPORTING_CSV" ]; then
IFS=',' read -ra SUPPORTING <<< "$SUPPORTING_CSV"
else
SUPPORTING=()
fi
# Step 0b — snapshot bundle file SHAs BEFORE rendering the prompt or invoking
# codex (F-060 closure: TOCTOU race). The prompt embeds the at-render-time
# content of every primary + supporting + template file; the manifest written
# into the sidecar must reflect THAT content, not a later state. If a
# supporting file mutates between render time and Step 3a recompute, Step 3a
# detects it and the wrapper writes AUDIT_FAILED with a "bundle file mutated
# during audit" failure_reason — codex saw stale content and the sidecar
# manifest must not silently match the post-mutation state.
audit_template_path="shared/templates/codex_audit_multifile_template.md"
audit_template_sha_pre=$(_sha256 "$audit_template_path")
declare -a primary_shas_pre supporting_shas_pre
for path in "${PRIMARY[@]}"; do
primary_shas_pre+=("$path:$(_sha256 "$path")")
done
for path in "${SUPPORTING[@]}"; do
supporting_shas_pre+=("$path:$(_sha256 "$path")")
done
# Compute the canonical pre-codex bundle manifest (§3.6 deterministic format).
# This value is what gets written to sidecar.prompt.bundle.bundle_manifest_sha
# AND what L3-4 will recompute against current file contents at Path A/B
# verification time.
{
for entry in "${primary_shas_pre[@]}"; do printf 'primary:%s\n' "$entry"; done
for entry in "${supporting_shas_pre[@]}"; do printf 'supporting:%s\n' "$entry"; done
printf 'template:%s:%s\n' "$audit_template_path" "$audit_template_sha_pre"
} | LC_ALL=C sort > "$OUT_DIR/$run_id.manifest.txt"
bundle_manifest_sha=$(_sha256 "$OUT_DIR/$run_id.manifest.txt")
# Step 1 — render audit prompt from template + bundle inputs
# Build a flag-repeated argument list for render_template's --supporting:
# each element of the SUPPORTING array becomes one repeated --supporting
# flag. Naive "$SUPPORTING" or "${SUPPORTING[*]}" would only forward the
# first element (or a single space-joined blob), which would silently drop
# every supporting file beyond the first from the audit prompt.
supporting_args=()
for path in "${SUPPORTING[@]}"; do
supporting_args+=(--supporting "$path")
done
prompt=$(render_template "$audit_template_path" \
--round "$ROUND" \
--target-rounds "$TARGET_ROUNDS" \
--primary "$DELIVERABLE" \
"${supporting_args[@]}" \
--previous-findings "$PREV_FINDINGS")
# Step 2a — pre-touch JSONL placeholder so the file exists even if codex is
# killed by SIGTERM / SIGKILL / OOM before opening its output file. Without this
# the AUDIT_FAILED path (Step 4 else branch) could synthesize a verdict whose
# proposal entry references a nonexistent JSONL, violating the "if proposal
# exists, all three artifacts it references exist" contract (§4.4 atomicity
# guarantee). Empty JSONL is treated by §5.6 Path B5 the same way as a partial
# JSONL — the AUDIT_FAILED short-circuit does not require valid Layer 2 content.
: > "$OUT_DIR/$run_id.jsonl"
# Step 2b — invoke codex CLI with supported flags. codex 0.125.0 dropped the
# pre-0.121 flag set; the current (and only) way to control reasoning effort
# is the `-c model_reasoning_effort=...` config override (TOML-quoted), and
# `--json` replaces the older `--output-format jsonl`. There are no
# `--output`, `--stdout-log`, or `--stderr-log` flags in 0.125+ — codex prints
# the event-stream to stdout, which we split via a real pipeline + tee into
# the JSONL contract file plus the <run_id>.stdout diagnostic file. Stderr
# goes to <run_id>.stderr at shell level.
#
# We use a pipeline with PIPESTATUS rather than process substitution
# `> >(tee ...)` because the latter:
# 1. Leaves tee's exit status outside `$?` (the wrapper sees only codex's
# exit, never tee's — a tee write failure to a full disk would silently
# corrupt the JSONL artifact),
# 2. Race-conditions on draining: orchestrator can Read the JSONL before
# the tee subshell flushes, observing partial JSONL not because codex
# is mid-stream but because tee hasn't written yet,
# 3. Has no clean way to reorder JSONL-write-then-stdout-log.
# The pipeline form below captures both stages via `${PIPESTATUS[@]}`:
# pipe[0] is codex's exit, pipe[1] is tee's. Tee failure (disk full, EIO)
# triggers EX_CANTCREAT (73) per §4.6.
#
# `set +e` is required because §4.1 `set -euo pipefail` would abort the
# wrapper on non-zero codex exit before `${PIPESTATUS[@]}` is captured; we
# capture locally then re-enable. Note: with `pipefail`, $? on a pipeline is
# the rightmost non-zero stage's exit — but we want codex's exit specifically
# (pipe[0]), not the rightmost, so PIPESTATUS is the canonical capture.
#
# This is `codex exec`, not `codex exec resume <thread_id>` — every audit
# run is a fresh thread; the wrapper never resumes a prior thread.
started_at=$(rfc3339_now_ms)
set +e
codex exec \
-m gpt-5.5 \
-c 'model_reasoning_effort="xhigh"' \
--json \
- \
2> "$OUT_DIR/$run_id.stderr" \
<<<"$prompt" \
| tee "$OUT_DIR/$run_id.stdout" > "$OUT_DIR/$run_id.jsonl"
pipe=("${PIPESTATUS[@]}")
exit_code="${pipe[0]}"
tee_exit="${pipe[1]}"
set -e
ended_at=$(rfc3339_now_ms)
# Tee write failure is treated as EX_CANTCREAT (73): the wrapper cannot
# guarantee the JSONL contract file is intact. Cleanup partial artifacts so
# orchestrator never sees a sidecar / verdict / proposal pointing at a
# corrupt JSONL, then exit 73 per §4.6 row.
if [ "$tee_exit" -ne 0 ]; then
rm -f "$OUT_DIR/$run_id.jsonl" "$OUT_DIR/$run_id.stdout" \
"$OUT_DIR/$run_id.stderr" "$OUT_DIR/$run_id.manifest.txt"
printf '[run_codex_audit] tee write failed (exit %s): cannot guarantee JSONL integrity\n' "$tee_exit" >&2
exit 73
fi
# Step 3a — recompute bundle file SHAs and detect intra-audit mutation
# (F-060 closure: TOCTOU race detection). The pre-codex snapshot in Step 0b
# was the canonical bundle state at render-time. If any primary / supporting /
# template file mutated DURING the codex run (user edited in another terminal,
# IDE auto-save, CI build artifact regenerated), this recompute catches it
# and the wrapper falls into the AUDIT_FAILED branch at Step 4 with a
# "bundle file mutated during audit" failure_reason — codex reviewed the
# pre-mutation content and the audit verdict is stale.
declare -a primary_shas supporting_shas
for path in "${PRIMARY[@]}"; do
primary_shas+=("$path:$(_sha256 "$path")")
done
for path in "${SUPPORTING[@]}"; do
supporting_shas+=("$path:$(_sha256 "$path")")
done
audit_template_sha=$(_sha256 "$audit_template_path")
# Compare pre vs post arrays element-wise. Any mismatch indicates intra-audit
# mutation. bundle_mutation_detected=1 forces the AUDIT_FAILED branch in
# Step 4 even if codex itself exited cleanly.
bundle_mutation_detected=0
bundle_mutation_files=()
for i in "${!primary_shas[@]}"; do
if [ "${primary_shas[i]}" != "${primary_shas_pre[i]}" ]; then
bundle_mutation_detected=1
bundle_mutation_files+=("${primary_shas[i]%%:*}")
fi
done
for i in "${!supporting_shas[@]}"; do
if [ "${supporting_shas[i]}" != "${supporting_shas_pre[i]}" ]; then
bundle_mutation_detected=1
bundle_mutation_files+=("${supporting_shas[i]%%:*}")
fi
done
if [ "$audit_template_sha" != "$audit_template_sha_pre" ]; then
bundle_mutation_detected=1
bundle_mutation_files+=("$audit_template_path")
fi
# Step 3b — manifest is fixed at the pre-codex value (computed in Step 0b).
# Sidecar carries the pre-codex manifest because that is what codex actually
# saw. Post-codex SHAs above are diagnostic only (used to set
# bundle_mutation_detected); they do NOT update bundle_manifest_sha. Otherwise
# L3-4 at orchestrator verification time would compare current-state SHAs
# against post-mutation SHAs and silently accept the stale audit.
# Step 3c — write sidecar (Layer 3 evidence)
emit_sidecar \
--run-id "$run_id" \
--codex-version "$(_codex_version)" \
--hostname "$(uname -n)" \
--cwd "$(pwd)" \
--git-sha "$git_sha" \
--git-dirty "$git_dirty" \
--started "$started_at" \
--ended "$ended_at" \
--exit-code "$exit_code" \
--jsonl-thread-id "$(_extract_jsonl_thread_id "$OUT_DIR/$run_id.jsonl")" \
--primary-shas "${primary_shas_pre[*]}" \
--supporting-shas "${supporting_shas_pre[*]}" \
--audit-template-path "$audit_template_path" \
--audit-template-sha "$audit_template_sha_pre" \
--bundle-manifest-sha "$bundle_manifest_sha" \
--bundle-id "$BUNDLE_ID" \
> "$OUT_DIR/$run_id.meta.json"
# Step 4 — produce verdict file. Three triggers force the AUDIT_FAILED branch:
# (a) codex exit_code != 0, (b) parse_audit_verdict.py --probe rejects the
# JSONL, (c) bundle_mutation_detected (Step 3a TOCTOU detection).
# wrapper_exit_code starts as a copy of codex's exit_code; the AUDIT_FAILED
# branch may override it so that wrapper exit status agrees with verdict status
# (§4.6 contract: exit 0 ↔ PASS/MINOR/MATERIAL; non-zero ↔ AUDIT_FAILED).
wrapper_exit_code=$exit_code
if [ "$exit_code" -eq 0 ] \
&& [ "$bundle_mutation_detected" -eq 0 ] \
&& scripts/parse_audit_verdict.py --probe "$OUT_DIR/$run_id.jsonl" >/dev/null 2>&1; then
# Clean codex completion + bundle stable + parseable JSONL agent_message event → real verdict
scripts/parse_audit_verdict.py \
--jsonl "$OUT_DIR/$run_id.jsonl" \
--round "$ROUND" \
--target-rounds "$TARGET_ROUNDS" \
> "$OUT_DIR/$run_id.verdict.yaml"
else
# codex exit != 0 OR bundle mutated mid-audit OR JSONL is partial / missing
# parseable agent_message event → AUDIT_FAILED. Synthesize the failure verdict directly; do
# NOT run parse_audit_verdict.py against partial JSONL (it would either crash
# or hallucinate findings).
if [ "$bundle_mutation_detected" -eq 1 ]; then
failure_reason="bundle file(s) mutated during audit run: ${bundle_mutation_files[*]}"
else
failure_reason=$(synthesize_failure_reason "$exit_code" "$OUT_DIR/$run_id.stderr")
fi
emit_audit_failed_verdict \
--run-id "$run_id" \
--round "$ROUND" \
--target-rounds "$TARGET_ROUNDS" \
--failure-reason "$failure_reason" \
> "$OUT_DIR/$run_id.verdict.yaml"
# If we entered this branch with codex exit 0 (parse-probe failure path OR
# bundle-mutation path), override wrapper_exit_code to 70 (EX_SOFTWARE) so
# the §4.6 contract holds: an AUDIT_FAILED verdict file MUST be paired with
# non-zero wrapper exit. If codex itself already exited non-zero,
# wrapper_exit_code already carries that value and the override is a no-op.
if [ "$wrapper_exit_code" -eq 0 ]; then
wrapper_exit_code=70
fi
fi
# In the AUDIT_FAILED branch, sidecar's stream.jsonl_thread_id may be
# empty (no valid JSONL thread). Schema permits this for AUDIT_FAILED runs;
# orchestrator §5.6 Path B5 short-circuits before reading thread_id.
# Step 5 — emit proposal entry for orchestrator to merge (§4.9 lifecycle).
# Verdict block (status + round + target_rounds + finding_counts + optional
# failure_reason) is mirrored from the verdict.yaml produced in Step 4.
# verified_at and verified_by are NOT set (orchestrator fills them at merge time).
emit_proposal_entry \
--run-id "$run_id" \
--stage "$STAGE" \
--agent "$AGENT" \
--deliverable "$DELIVERABLE" \
--deliverable-sha "$sha_deliverable" \
--bundle-id "$BUNDLE_ID" \
--bundle-manifest-sha "$bundle_manifest_sha" \
--jsonl-path "$OUT_DIR/$run_id.jsonl" \
--sidecar-path "$OUT_DIR/$run_id.meta.json" \
--verdict-path "$OUT_DIR/$run_id.verdict.yaml" \
--verdict-yaml "$OUT_DIR/$run_id.verdict.yaml" \
> "$OUT_DIR/$run_id.audit_artifact_entry.json"
# Step 6 — exit. wrapper_exit_code reflects either codex's own exit code OR
# the parse-probe-failure override set in Step 4 (so wrapper exit status
# always agrees with verdict status per §4.6). Orchestrator additionally
# inspects the proposal's verdict.status to decide ship/block — the two
# signals (process exit code + verdict.status) are independently consumed.
[ "$wrapper_exit_code" -eq 0 ] && exit 0 || exit "$wrapper_exit_code"
Atomicity guarantee:
Three of the four contract files (sidecar, verdict, proposal entry) are written via the tmp-file-rename pattern: wrapper writes to <file>.tmp, fsyncs, then mv <file>.tmp <file>. The fourth file, JSONL, is intentionally exempt: codex 0.125+ --json writes its event stream to stdout (the CLI has no --output flag in 0.125+; see §3.3 + §4.4 Step 2b), and the wrapper splits stdout via a tee | > pipeline into <run_id>.jsonl (the contract artifact) and <run_id>.stdout (diagnostic copy). The pipeline runs over the audit's runtime (potentially minutes), and tmp-rename of a streamed file would defeat live debugging via tail -f. Step 2a additionally pre-touches an empty JSONL placeholder so the file exists at a known path even when codex is killed before emitting any events (F-059 closure). JSONL therefore may be observed mid-write or empty during an audit run, but no orchestrator code path ever consumes JSONL until a proposal entry file references it (see proposal-last guarantee below) — so JSONL's lack of tmp-rename atomicity is not observable to orchestrator. Step 2b's ${PIPESTATUS[@]} capture additionally surfaces tee-side write failures (disk full / EIO during the streaming write) as wrapper exit 73 with cleanup of all partial artifacts, so a corrupted JSONL never reaches the proposal-write step. For AUDIT_FAILED proposals, §5.6 Path B5 short-circuits BLOCK before Layer 2 validation runs (gating against incomplete evidence is meaningless — see §5.6 B5). For PASS / MINOR / MATERIAL proposals, §5.2 L2-1 (file exists) and L2-2 (every event validates against audit_jsonl.schema.json) reject missing or partial JSONL at the gate, so a non-AUDIT_FAILED proposal whose JSONL is partially-written never ships.
If the wrapper crashes between any two writes, only .tmp artifacts (for the three tmp-renamed files), the empty/partial JSONL placeholder, and possibly .stdout/.stderr remain. Orchestrator finds no proposal entry → treats run as not-conducted (per §5 rule). No false PASS reachable through partial writes.
The proposal entry file is the last thing the wrapper writes (after JSONL + sidecar + verdict are all on disk and — for sidecar/verdict/proposal — atomically renamed). This is the load-bearing guarantee that lets JSONL skip tmp-rename: orchestrator never sees a proposal pointing at incomplete artifact paths, so JSONL's mid-write visibility never reaches orchestrator state.
The audit template Section 4 already supports multi-file bundles (Phase 2 chapter deliverable: synthesis + verification + instrument together). Wrapper supports this via:
--deliverable is the bundle's primary file (the one whose audit_artifact[] entry the orchestrator reads).--supporting carries the rest of the bundle (bibliography.json, verification.md, etc.).--bundle-id (CLI flag, kebab-case) populates the schema field bundle_id (snake_case, per §3.2) on the resulting audit_artifact[] entry and the sidecar prompt.bundle.bundle_id field. Same value for all files in the bundle. Orchestrator reads bundle_id to know "these N audit_artifact entries belong to one logical bundle."For Phase 2 chapter ships, three audit_artifact[] entries (one per primary deliverable) all carry the same bundle_id for audit-trail grouping. Each entry's stage-transition gate fires independently when that agent's deliverable is produced (per §5.1 independent-gating semantics) — bundle_id does not collectively gate them.
| Wrapper exit code | Meaning | Files written | Orchestrator response |
|---|---|---|---|
| 0 | Audit completed cleanly | All four contract files (jsonl + sidecar + verdict + proposal entry); proposal carries verdict status PASS / MINOR / MATERIAL | Read verdict, proceed per §5.3 / §5.6 Path B |
64 (EX_USAGE) |
Input validation failed | None | Block, surface error to user; orchestrator never sees a proposal |
70 (EX_SOFTWARE) |
(Wrapper-originated.) Triggered by any of: (a) codex itself exited 70 [EX_SOFTWARE], (b) codex exited 0 but parse_audit_verdict.py --probe rejected the JSONL (no agent_message item.completed event in stream, missing closing turn.completed event, malformed Section 6 verdict text in the final agent_message, etc.), (c) bundle mutation detected — Step 3a's recompute revealed at least one primary / supporting / template file changed during the codex run (TOCTOU race per F-060). In cases (b) and (c) the §4.4 Step 4 wrapper_exit_code override raises codex's exit-0 to 70 so the wrapper's process exit always agrees with the AUDIT_FAILED verdict it just wrote (§4.6 contract: wrapper exit 0 ↔ PASS/MINOR/MATERIAL; wrapper exit non-zero ↔ AUDIT_FAILED) |
jsonl (possibly partial; possibly empty placeholder from Step 2a if codex was killed before opening output — see F-059), sidecar (with process.exit_code carrying codex's actual exit — may be 0 in cases (b)/(c) since sidecar records codex's process exit; wrapper exit is a separate signal), verdict (carrying status AUDIT_FAILED), proposal entry (carrying verdict.status: AUDIT_FAILED, finding_counts: {p1: 0, p2: 0, p3: 0}, verdict.failure_reason: <one-line> — case (c) reason format: "bundle file(s) mutated during audit run: <space-separated path list>") |
§5.6 Path B5 catches AUDIT_FAILED proposal: BLOCK transition without running the eleven gating checks (audit did not complete; gating is meaningless) and without merging into passport |
73 (EX_CANTCREAT) |
Cannot write to --output-dir |
None or partial — wrapper aborts before any contract file is fsync-renamed | Block, surface filesystem error |
75 (EX_TEMPFAIL) |
codex API rate-limited / transient (codex itself exited 75; wrapper preserves the code rather than normalizing to 70 so deployment can detect rate-limiting and apply backoff before retry) | Same artifact set as exit 70: jsonl (possibly partial), sidecar (with process.exit_code: 75), verdict (status AUDIT_FAILED), proposal entry (status AUDIT_FAILED, failure_reason: "codex exit 75: rate-limited" or similar) |
Same as exit 70: §5.6 Path B5 BLOCK without gating |
| Other non-zero (codex's own exit code, preserved) | Codex exited with a code not enumerated above (e.g., 1, 2, 137 from SIGKILL). Wrapper preserves codex's exit code rather than normalizing — preserves diagnostic signal for deployment-side log analysis. The wrapper still produces the AUDIT_FAILED artifact set per the "wrapper writes sidecar + proposal even on codex failure" key invariant below | Same artifact set as exit 70 if wrapper reaches the post-codex step (sidecar process.exit_code carries codex's actual code, verdict status AUDIT_FAILED, proposal carries failure_reason: "codex exit <N>: <synthesize_failure_reason output>") |
Same as exit 70: §5.6 Path B5 BLOCK without gating |
Key invariants:
process.exit_code != 0 paired with a proposal carrying verdict.status: AUDIT_FAILED is the orchestrator's signal that audit was attempted but failed — distinct from "audit was never run" (no proposal at all).AUDIT_FAILED is a fourth verdict status added to the §3.5 verdict schema enum: ["PASS", "MINOR", "MATERIAL", "AUDIT_FAILED"]. It exists for failure-mode signaling only; orchestrator never ships on AUDIT_FAILED and never runs gating against it (rule of §5.6 Path B5: AUDIT_FAILED short-circuits to BLOCK).verdict.failure_reason is required when verdict.status == "AUDIT_FAILED", forbidden otherwise. One-line human-readable reason ("codex exit 70: network timeout after 600s", "JSONL parse error at row 47", etc.).AUDIT_FAILED as a status, but its semantics are "audit aborted" not "audit found no issues". Cross-field rule for AUDIT_FAILED is finding_counts.{p1,p2,p3} == 0 AND failure_reason is set.This contract closes F-019 (round 3): orchestrator can distinguish three states cleanly — (1) no proposal = audit never ran, (2) AUDIT_FAILED proposal = audit ran but failed, (3) PASS/MINOR/MATERIAL proposal = audit completed and produced findings. State (1) and (2) both BLOCK transition; only state (3) reaches gating.
The wrapper is invoked by:
scripts/run_codex_audit.sh ... interactively.github/workflows/audit.yml — to be added in implementation)feedback_subagent_tool_hallucination.md runtime bus pattern)The wrapper is NOT invoked by an in-LLM Bash tool call from the same session producing the deliverable. This is a deliberate constraint, not a technical limitation: an LLM that produces a deliverable AND calls the audit wrapper is structurally susceptible to Pattern C3 (sub-agent fabricated audit metadata, see feedback_subagent_tool_hallucination.md). The artifact-as-contract design (Q2) is the structural fix; this rule is its enforcement.
Spec-level enforcement: the v3.6.7 main spec §6.x agent prompts retain the "DO NOT simulate any audit step" prohibition (already shipped). The Step 6 partial inversion (§6 of this spec) removes the sentence that reveals "the orchestrator runs codex audit afterward" — the agent is no longer told there is a downstream audit at all, removing the trigger for Pattern C3.
audit_verdict.schema.json carries a generator_version field. Orchestrator reads this and applies version-specific verdict parsing rules. v3.6.7 ships wrapper version 1.0.0; future versions may emit additional verdict fields, but core fields (verdict_status, finding_counts, findings[]) are stable across versions per semver.
If orchestrator encounters a generator_version it does not recognize (newer wrapper than orchestrator), it logs a warning but processes the verdict using the latest known schema version. If orchestrator encounters a wrapper version it has been told is broken (deny-list in scripts/check_audit_artifact_consistency.py), it rejects the artifact.
The wrapper produces four contract files (§4.3): three artifact files (jsonl / sidecar / verdict) plus the proposal entry file. Orchestrator is the sole writer of audit_artifact[] into Schema 9. This split prevents concurrency on the passport file (wrapper may run from CI / cron / background hook; passport may be open in an LLM session) and keeps Schema 9 mutation in the same code path that handles other Schema 9 fields (reset_boundary[], literature_corpus[], compliance_history[]).
Lifecycle:
| Step | Actor | Action |
|---|---|---|
| 1 | Wrapper | Produces three artifact files (jsonl / sidecar / verdict, §4.3) under --output-dir. Sidecar and verdict are written via tmp-file-rename (atomic); JSONL is streamed by codex directly to its final path (intentionally non-atomic per §4.4 — codex's runtime is potentially minutes and tmp-rename of a streamed file would defeat live debugging via tail -f). Verdict status is one of PASS / MINOR / MATERIAL / AUDIT_FAILED per §4.6 |
| 2 | Wrapper | Produces the fourth file <run_id>.audit_artifact_entry.json (proposal entry) — this is the LAST file the wrapper writes, ensuring orchestrator never sees a proposal pointing at incomplete artifact paths. verdict.verified_at and verdict.verified_by are UNSET (verification has not happened yet) |
| 3 | Orchestrator | At stage transition, attempts §5.6 Path A first (look up matching persisted entry by (stage, agent, deliverable_sha)). Path A succeeds = A1.5 found no superseding higher-round proposal in <output-dir> (otherwise Path A is preempted and Path B runs with supersession_required = true per D4) + entry valid + 11 gates pass + verdict schema valid + mirror match + A6 late freshness recheck passes. Failure at any sub-step falls through to Path B |
| 4 | Orchestrator (Path B) | Scans <output-dir> for unmerged proposal matching the tuple, picks the one with latest sidecar timing.started_at (chronological; lex-max run_id is the deterministic tie-breaker — see §5.6 B2 for why run_id alone is not chronological). Validates proposal against audit_artifact_entry.schema.json --mode proposal (rejects if verdict.verified_at / verdict.verified_by are pre-filled — Pattern C3 attack surface) |
| 5 | Orchestrator (Path B) | If proposal verdict.status == "AUDIT_FAILED": BLOCK without gating, surface failure_reason. If status is PASS / MINOR / MATERIAL: continue to step 6 |
| 6 | Orchestrator | Runs the eleven gating checks (§5.2) using paths declared in the entry's artifact_paths |
| 7 | Orchestrator | Validates <run_id>.verdict.yaml against audit_verdict.schema.json. On Path A also confirms verdict file's mirrored fields match the persisted entry's mirror (drift = forgery evidence) |
| 8 | Orchestrator (Path B) | If all eleven + verdict schema pass: fills verdict.verified_at via the strict-monotonic helper (§5.4) and verdict.verified_by = "pipeline_orchestrator_agent", re-validates against --mode persisted, appends to passport audit_artifact[] |
| 9 | Orchestrator (Path B) | Moves the proposal file to <output-dir>/consumed/<run_id>.audit_artifact_entry.json so it is not re-merged on a later transition. Files marked AUDIT_FAILED (step 5 short-circuit) are NOT moved — user may inspect |
| 10 | Orchestrator | If any check fails: leaves the proposal file in place (Path B) or falls back to Path B with surfaced reason (Path A failure), surfaces the failed check to user (per §5.6 block message), no passport mutation on the failed entry |
Invariants:
--output-dir.audit_artifact[] entries. This mirrors the existing pattern where reset_boundary[] and literature_corpus[] are also orchestrator-mediated (or user-adapter-mediated for corpus) but never wrapper-mediated.verified_at / verified_by are set ONLY by the orchestrator, on one of two write paths: (a) the normal proposal-merge at lifecycle step 8 (= §5.6 Path B8d), and (b) the §5.4 ship_with_known_residue acknowledgement append, which uses the same write contract (verified_at = current UTC ms, verified_by = "pipeline_orchestrator_agent") and the same strict-monotonic guarantee as Path B8d. Wrappers and any non-orchestrator writer remain forbidden — a wrapper-emitted proposal file with these fields filled is treated as malformed (Pattern C3 attack surface) and rejected at lifecycle step 4 schema validation (= §5.6 Path B4).Why a separate proposal file rather than wrapper writing directly to passport:
--output-dir is bounded by orchestrator gating before reaching passport state.Wrapper proposal file shape (<run_id>.audit_artifact_entry.json):
{
"stage": 2,
"agent": "synthesis_agent",
"deliverable_path": "chapter_4/synthesis.md",
"deliverable_sha": "a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2",
"run_id": "2026-04-30T15-22-04Z-d8f3",
"bundle_id": "phase2-chapter4-2026-04-30",
"bundle_manifest_sha": "9a8b7c6d5e4f3b2a1c0d9e8f7a6b5c4d3e2f1a0b9c8d7e6f5a4b3c2d1e0f9876",
"artifact_paths": {
"jsonl": "audit_artifacts/2026-04-30T15-22-04Z-d8f3.jsonl",
"sidecar": "audit_artifacts/2026-04-30T15-22-04Z-d8f3.meta.json",
"verdict": "audit_artifacts/2026-04-30T15-22-04Z-d8f3.verdict.yaml"
},
"verdict": {
"status": "MINOR",
"round": 2,
"target_rounds": 3,
"finding_counts": {"p1": 0, "p2": 0, "p3": 1}
}
}
This example shows status MINOR because finding_counts.p3 == 1 — a PASS proposal would require all three finding_counts keys to be 0 per the §3.2 cross-field rule, and pairing PASS with any nonzero count is a schema violation that lint catches at lifecycle step 4. verdict.verified_at and verdict.verified_by are intentionally absent. Schema validation at step 4 enforces their absence in proposal files via --mode proposal; their presence is treated as schema error.
Proposal entry shape on AUDIT_FAILED (per §4.6):
{
"stage": 2,
"agent": "synthesis_agent",
"deliverable_path": "chapter_4/synthesis.md",
"deliverable_sha": "a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2",
"run_id": "2026-04-30T15-22-04Z-d8f3",
"bundle_id": "phase2-chapter4-2026-04-30",
"bundle_manifest_sha": "9a8b7c6d5e4f3b2a1c0d9e8f7a6b5c4d3e2f1a0b9c8d7e6f5a4b3c2d1e0f9876",
"artifact_paths": {
"jsonl": "audit_artifacts/2026-04-30T15-22-04Z-d8f3.jsonl",
"sidecar": "audit_artifacts/2026-04-30T15-22-04Z-d8f3.meta.json",
"verdict": "audit_artifacts/2026-04-30T15-22-04Z-d8f3.verdict.yaml"
},
"verdict": {
"status": "AUDIT_FAILED",
"round": 2,
"target_rounds": 3,
"finding_counts": {"p1": 0, "p2": 0, "p3": 0},
"failure_reason": "codex exit 70: network timeout after 600s"
}
}
Both proposal shapes (PASS/MINOR/MATERIAL completed audits AND AUDIT_FAILED aborted audits) write all four contract files; the difference is in verdict.status and verdict.failure_reason. Orchestrator §5.6 Path B5 short-circuits to BLOCK on AUDIT_FAILED without running gating; PASS/MINOR/MATERIAL go through the eleven gates.
This section specifies how pipeline_orchestrator_agent enforces the audit gate at stage transition. The orchestrator never runs codex; it reads the passport, finds the relevant audit_artifact[] entry, runs the verification checks, and decides ship-or-block based on verdict.
The audit gate fires at exactly three transitions, one per v3.6.7 agent. These are the only points in the pipeline where Step 6 hooks intervene; all other transitions retain pre-v3.6.7 behavior unchanged.
| Trigger | Producing agent | Producing skill / mode | Gate fires before | audit_artifact.stage value |
|---|---|---|---|---|
synthesis_agent exit |
synthesis_agent |
deep-research, full mode (Phase 2 synthesis stage) | Stage 1 → Stage 2 transition (deep-research → academic-paper handoff) | 2 (destination stage) |
research_architect_agent survey-designer exit |
research_architect_agent |
academic-paper, full mode (when survey instrument is in scope) | Within-skill internal transition before instrument deliverable enters Material Passport | 2 (destination stage; the survey instrument enters passport at the Stage 2 boundary) |
report_compiler_agent abstract-only exit |
report_compiler_agent |
academic-paper, abstract-only mode | Before Stage 5 (FINALIZE) format conversion | 5 (destination stage) |
Convention — destination stage, not producing stage. The audit_artifact.stage field (and §5.6 Path A's stage tuple match, and the wrapper's --stage flag) carry the destination stage, i.e., the stage the deliverable is about to enter after passing the gate. This matches Phase 6.1's self-test invocation (--stage 2 for the synthesis-agent gate that fires before Stage 1→2 transition) and §7.1's fixture-to-pattern mapping table values. Earlier draft prose that referred to "the just-completed pipeline stage" should be read as "the destination stage that the just-completed deliverable is about to enter" — the fix is in this table; §5.6 A1 prose has been clarified to match.
Why these three only: Q1 fixed scope to v3.6.7-only agents. Other transitions (Stage 1→2 RQ-brief handoff, Stage 2.5 integrity, Stage 3 review, Stage 4 revision, Stage 4.5 final integrity) retain pre-v3.6.7 behavior. The audit gate is purely additive — it does NOT replace any existing checkpoint mechanism (Mode Recommendation, Mid-Entry Material Passport Check, integrity_verification_agent, collaboration_depth_agent observer all continue to fire as today).
Bundle semantics — independent gating, shared audit trail: when multiple v3.6.7 agents produce deliverables in one Phase 2 batch (typical: synthesis_agent + research_architect_agent survey both ship in one chapter run), each agent's deliverable triggers its own audit gate independently at its own stage transition. The bundle_id field is an audit-trail tag that lets users and future maintainers reconstruct "these N audit_artifact[] entries belong to one logical chapter run"; it does NOT define a gating set.
Rationale: ARS pipeline is sequential at stage level (a stage cannot complete with a missing v3.6.7 agent's deliverable — earlier stage-completion checks would already have blocked). A "bundle vacuously passes because it is incomplete" failure mode is structurally impossible: if an agent's deliverable is missing, the stage that produces it is itself incomplete, and the audit gate never fires for the missing entry. Each present entry passes its own eleven gating checks; absent entries are caught upstream by stage-completion logic, not by the audit gate.
This mirrors audit template §4.1 D1 (multi-file > sequential) at the prompt level (one codex audit invocation can review multiple files together for cross-section coherence) but keeps gating at the per-entry level (each entry has its own verdict, its own gate).
For each audit_artifact[] entry that gates a transition, orchestrator runs eleven checks in order. The first failing check stops verification; orchestrator does NOT compose findings across checks. Reasoning: any check failure is sufficient cause to reject the artifact; running more checks just buries the diagnostic.
The eleven checks operationalize three §3 surfaces: the Layer 2 per-row schema (§3.3), the §3.7 family A row A7 stream-shape invariant, and the §3.4 cross-file rules. Each §3.4 cross-file rule maps to exactly one Layer 3 check below; counts agree across §3 and §5 by construction.
Layer 2 checks (codex JSONL evidence — schema-level):
| # | Check | Implementation | Maps to |
|---|---|---|---|
| L2-1 | JSONL file exists at path declared in artifact_paths.jsonl |
Filesystem stat | precondition |
| L2-2 | Every JSONL row validates against audit_jsonl.schema.json |
JSON Schema validator | §3.3 schema |
| L2-3 | JSONL stream contains exactly one thread.started event with a canonical UUID thread_id; the stream opens with thread.started as event #1 and turn.started as event #2 (per §3.3 / Q3 canonical opening) |
Filter events by type == "thread.started"; assert single match; assert it is the first row; assert second row's type == "turn.started"; assert thread_id matches ^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$ (canonical 8-4-4-4-12 layout, not the looser ^[0-9a-f-]{36}$) |
§3.3 schema-level rule |
| L2-4 | JSONL stream contains at least one item.completed event whose item.type == "agent_message"; the last such event's item.text parses against the audit-template Section 6 verdict shape; stream ends with a turn.completed event carrying usage integers |
Filter events by type == "item.completed" + item.type == "agent_message", assert non-empty, take the LAST match (intermediate agent_message events are allowed in tool-using runs — codex thinking aloud between tool calls), hand its item.text to parse_audit_verdict.py --probe; assert final event has type == "turn.completed" with usage.input_tokens > 0 and usage.output_tokens >= 0 |
§3.3 schema-level rule |
| L2-5 | JSONL stream tool-event 1:1 pairing with start-before-completion order per §3.7 A7: for every non-agent_message tool item.id the stream contains exactly one item.started and exactly one item.completed with that id (no duplicate starts, no duplicate completions, no orphan completions), and the completed event appears strictly later in the stream than the matching started event; parallel tool starts may complete in any order; agent_message item.completed events are exempt (codex emits assistant messages as single completion events) |
Invoke scripts/check_audit_artifact_consistency.py --mode jsonl-stream --jsonl <path> (Phase 6.3 contract; exits 0 on pass, non-zero with offending item.id on violation) |
§3.7 A7 |
Layer 3 checks (sidecar evidence — cross-file rules):
| # | Check | Implementation | Maps to §3.4 cross-file rule |
|---|---|---|---|
| L3-1 | Sidecar file exists at artifact_paths.sidecar AND validates against audit_sidecar.schema.json |
Filesystem stat + JSON Schema validator | precondition |
| L3-2 | Sidecar stream.jsonl_thread_id matches the JSONL's single thread.started event's thread_id |
String equality | rule 1 |
| L3-3 | prompt.bundle.primary_deliverables[].sha matches audit_artifact_entry.deliverable_sha AND matches current file SHA-256 |
SHA-256 recompute on disk + equality | rule 2 |
| L3-4 | prompt.bundle.bundle_manifest_sha matches recomputed manifest hash (per §3.6) over current state of all primary + supporting files PLUS the audit template at prompt.audit_template_path |
Manifest reconstruct (3 roles: primary/supporting/template) + SHA-256 + equality | rule 3 |
| L3-5 | Sidecar runner.git_sha resolves to a real commit in current repo |
git cat-file -e <sha>^{commit} |
rule 4 |
| L3-6 | Sidecar timing.ended_at - timing.started_at == duration_seconds within ±1s |
Datetime arithmetic | rule 5 |
| L3-7 | Sidecar process.exit_code == 0 |
Integer equality | rule 6 |
| L3-8 | entry.run_id == sidecar.run_id AND sidecar.run_id equals the bare <run_id> basename of every path in artifact_paths.{jsonl, sidecar, verdict} (always, both lifecycle modes). When running on a Path B unmerged proposal, ADDITIONALLY equals the basename of the proposal entry file (<run_id>.audit_artifact_entry.json) — this fourth check is proposal-mode only because §4.9 step 9 moves the proposal file to consumed/ after merge and §3.1 intentionally does NOT record it under artifact_paths. All co-located artifact files share one bare <run_id> stem per §3.7 family F (F2/F3) |
String parse of basename + equality across all required paths (3 in persisted mode, 4 in proposal mode) | §3.4 rule 7 |
Total: 5 Layer 2 + 8 Layer 3 = 13 numbered slots, of which 11 are gating checks. L2-1 and L3-1 are preconditions (file existence + schema validity); failure of either is reported as "audit not conducted" rather than "audit fake-detected", because either case is structurally indistinguishable from "audit never ran".
The eleven gating checks are L2-2, L2-3, L2-4, L2-5, L3-2, L3-3, L3-4, L3-5, L3-6, L3-7, L3-8.
Composite forgery cost: L2-3 ensures internal JSONL consistency. L2-5 closes the orphan-tool-completion forgery hole (a forger emitting a fabricated command_execution item.completed without a matching item.started would have evaded the per-row schema check). L3-2 cross-bridges JSONL ↔ sidecar. L3-3 + L3-4 pin all bundle file content. L3-5 anchors to repo history. L3-6 ensures temporal arithmetic self-consistency. L3-7 closes the "audit ran but failed" hole. L3-8 closes the "swap one of the four artifact files for a different run's file" hole — without L3-8 a forger could pair a real sidecar from run X with a fabricated verdict from run Y, since per-file schemas would still validate. Cumulative cost makes running real codex the economical path under the §1.2 threat model.
Verdict file is read AFTER all eleven gating checks pass. Orchestrator does NOT read verdict.yaml until Layer 2 + Layer 3 are clean; this prevents a forged verdict from leaking into orchestrator state via partial evidence.
Once all eleven gating checks pass, orchestrator validates verdict.yaml against audit_verdict.schema.json (§3.5). Schema-failed verdict files BLOCK with a diagnostic regardless of how cleanly Layer 2 + Layer 3 verified — schema validation is a separate trust boundary from artifact integrity. After schema validation passes, orchestrator decides per these rules:
Rows are evaluated top-to-bottom; the first row whose Definition matches the latest entry wins. This makes the acknowledgement override explicit rather than leaving an implementer to infer precedence between MATERIAL and MATERIAL + acknowledgement.
| Verdict | Definition | Orchestrator action |
|---|---|---|
| PASS | finding_counts.{p1,p2,p3} all zero |
Proceed to next stage; emit FULL checkpoint with [Audit: PASS at round N] line |
| MINOR | p1 == 0 AND p2 == 0 AND p3 <= 3 |
MANDATORY checkpoint with finding details; user choice required (continue to ship, iterate to dispatch revision, pause to stop) |
| MATERIAL + acknowledgement (higher-precedence override of the MATERIAL row below) | latest entry's verdict.status == "MATERIAL" AND latest entry carries an acknowledgement object whose finding_ids covers EVERY current findings[].id from the companion verdict file |
Proceed to next stage; emit FULL checkpoint with [Audit: MATERIAL at round N, residue acknowledged by user at <acknowledged_at>] line |
| MATERIAL (no acknowledgement, or acknowledgement coverage incomplete) | p1 > 0 OR p2 > 0 OR p3 > 3 AND latest entry either has no acknowledgement object OR its acknowledgement.finding_ids does not cover every current findings[].id |
Block; refuse stage transition; surface findings; require revision + new audit round. Note: the second branch (incomplete coverage) is unreachable under the lint rules at §5.4 (full coverage enforced at acknowledgement-write time, not at ship-decision time); it is named here for defense-in-depth so a hand-edited passport entry that violates the lint rule still fails closed |
MATERIAL handling details:
synthesis_agent) is re-invoked with a "revision based on audit findings" prompt that lists each P1/P2 finding with its dimension and suggested fix.--round N+1 --previous-findings <prior verdict.yaml> to produce the next round audit. Orchestrator picks up the new audit_artifact[] entry on the next transition attempt.MINOR handling details:
p3 <= 3 per §5.3) do not block; they are surfaced to the user as a punchlist for author final pass. P3 findings exceeding the threshold (p3 > 3) trigger MATERIAL per §5.3 and follow the MATERIAL handling path.PASS handling details:
[Audit: PASS at round N] line is appended to the FULL checkpoint's Flagged section, marked as informational. Pre-v3.6.7 Flagged behavior (integrity issues, metric anomalies) is preserved; audit verdict is additive.Decision: default 3 rounds matching audit template Section 1 default target_rounds. Escalate to user after round 3 still MATERIAL.
Rationale: feedback_codex_iterative_spec_review_to_zero.md says "iterate to one zero-finding round" but does not bind the iteration count; feedback_cross_model_review_cascade_inconsistency.md and feedback_codex_review_vs_resume_audit_scope.md both observe round-3 as the typical convergence point under cascade-aware audits. Setting hard upper bound at 3 aligns with empirical convergence observation; a higher cap delays escalation without quality gain in observed runs.
Escalation behavior at round 3 still MATERIAL:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ESCALATION] v3.6.7 audit gate at <stage>
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Audit on <agent>'s deliverable at <deliverable_path> reached round 3
without convergence. Current verdict: MATERIAL with <P1> P1 + <P2> P2 + <P3> P3.
Per ARS feedback feedback_codex_review_vs_resume_audit_scope.md "stop
condition: N rounds OR 'add counter, not change rule' findings".
Findings (round 3):
<finding list>
Options:
1. ship_with_known_residue — Acknowledge findings, ship anyway (orchestrator
appends a new `audit_artifact[]` entry whose `acknowledgement.finding_ids`
carries the IDs the user accepted; not silent).
2. another_round — Override the 3-round cap, dispatch round 4 audit.
3. abort_stage — Return to producing agent, scrap deliverable, restart.
This decision requires explicit user input.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ship_with_known_residue mechanism (preserves append-only ledger): orchestrator does NOT mutate the existing MATERIAL entry. It appends a NEW audit_artifact[] entry that copies the latest persisted entry's (stage, agent, deliverable_path, deliverable_sha, run_id, bundle_manifest_sha, artifact_paths) and the inner verdict shape (verdict.status, verdict.round, verdict.target_rounds, verdict.finding_counts, verdict.failure_reason), then sets verdict.verified_at to the current UTC and verdict.verified_by to "pipeline_orchestrator_agent" (the same fields orchestrator fills at §4.9 step 8 for normal merges — acknowledgement append uses the same write contract, not a copy of the prior verification timestamp), and adds the optional acknowledgement object defined in §3.2 ({finding_ids, acknowledged_at: <same UTC>, acknowledged_by: "user"}). Refreshing verified_at rather than copying it is load-bearing for §3.1's "Orchestrator reads the latest entry by verified_at for verdict" — without the refresh, the acknowledgement entry would tie with the original MATERIAL entry on verified_at and Path A selection would be ambiguous. With the refresh, the acknowledgement entry strictly post-dates every prior entry and Path A deterministically picks it.
Atomicity for ack append: the new audit_artifact[] entry is appended to the passport via the same tmp-write + fsync + atomic rename pattern §5.6 B9 specifies for Path B normal merges (write the new passport content to <passport_path>.tmp, fsync the tmp file, then mv -f over the live passport). This guarantees that a crash between ack-decision and passport-commit leaves the passport in its pre-ack state — the ack is either fully committed or not at all. Combined with §5.6 B1a / B8b idempotent run_id de-duplication, a partially-committed ack append cannot produce a duplicate persisted entry on resume.
Strict-monotonic guarantee (single helper for every orchestrator-side append): ms precision (§3.2 cross-section consistency rule on verdict.verified_at) gives 1 ms resolution, but a same-process append running faster than 1 ms could still tie with the latest persisted entry's verified_at. Every orchestrator-side verified_at write — Path B8d normal merges (§4.9 step 8 / §5.6 B8d), another_round re-merges, and ship_with_known_residue acknowledgement appends — goes through one helper, written here in pseudocode and shipped as scripts/_next_verified_at_ms.py (consumed by both the orchestrator and the lint script):
def _next_verified_at_ms(passport_audit_artifacts: list) -> str:
now_ms = utc_now_ms() # RFC 3339 UTC ms precision
if not passport_audit_artifacts: # empty-ledger base case
return now_ms # no prior entry to outrun
latest_ms = max(entry["verdict"]["verified_at"]
for entry in passport_audit_artifacts)
return max(now_ms, increment_ms(latest_ms, 1)) # 1 ms after latest if clock has not yet advanced
The acknowledgement append additionally writes the helper's return value to acknowledgement.acknowledged_at so the two timestamps share the same instant by construction (§3.7 family B row B8 carries the equality lint rule). Because latest_ms is the max over ALL prior persisted entries (not just those matching the current (stage, agent, deliverable_sha) tuple), the helper guarantees the new entry's verified_at is strictly greater than every prior persisted entry's verified_at regardless of whose tuple wrote them; Path A latest-by-verified_at selection is therefore total-order deterministic at passport scope. The new ack entry's verdict.status remains MATERIAL — acknowledgement is a sibling annotation, not a status mutation (§3.7 family E row E7). This preserves the v3.6.3 reset_boundary[] append-only-ledger pattern (§3.1 closing paragraph) and means a future reader can see both "audit said MATERIAL" and "user acknowledged residue X, Y, Z" as separate facts in the ledger. The three acknowledgement.finding_ids lint rules (non-empty / every ID exists in companion verdict file / full coverage of every current finding) live in §3.7 family B row B10; they are checked at acknowledgement-write time and re-checked at §5.6 Path B persisted-mode validation, and rule 3 (full coverage) is what §5.3's "MATERIAL + acknowledgement" row depends on so partial residue acknowledgement is rejected at write time and orchestrator never handles it at ship-decision time.
The ship_with_known_residue option therefore creates a non-silent paper trail: the passport carries both the original MATERIAL audit verdict and the user's explicit acknowledgement entry, anyone reading it sees both. This matches the v3.6.6 spec philosophy of "transparency over enforcement" for boundary cases the user must own.
Override mechanics: the user's another_round choice MUST be encoded as a new (round, target_rounds) pair where the wrapper is re-invoked with --round N+1 --target-rounds N+1. The schema invariant round <= target_rounds (§3.2 cross-field rule) and the wrapper input rule "round must be <= target-rounds" (§4.2) both hold. Every escalation is a fresh user-owned decision that explicitly raises the cap by 1; this prevents the user from accidentally extending iteration without acknowledging that the original 3-round target was missed. Multiple escalations append additional audit_artifact[] entries with monotonically increasing round and matching target_rounds. There is no path that drifts toward unbounded iteration without per-escalation user consent.
another_round is committed by the wrapper artifact, not by orchestrator state (F-070 closure): the user's another_round choice is durable only after the higher-round wrapper run produces a proposal in <output-dir> carrying verdict.round = N+1 for the same (stage, agent, deliverable_sha) tuple. §5.6 A1.5's superseding-proposal preflight is what makes that proposal preempt the prior round-N MATERIAL persisted entry at the next gate — without A1.5, Path A would succeed on the prior entry and the user-dispatched higher-round proposal would never be merged. If the orchestrator session crashes between the user clicking another_round and the wrapper actually producing the higher-round proposal, the pending intent is intentionally NOT durable: on resume, Path A finds the round-N MATERIAL entry, runs A1.5 (no superseding proposal exists), succeeds at A7 with the prior MATERIAL verdict, and the §5.4 ESCALATION prompt fires again. This is "repeated authorization prompt", not "silent ship of stale verdict" — failing closed in a way the user can re-act on.
Byte-identical abort_stage and manual fresh audits (F-070 generalization): if the user picks abort_stage, the producing agent re-runs and either (a) produces a different deliverable byte-sequence — deliverable_sha changes, Path A finds no matching tuple, Path B runs fresh — or (b) produces the byte-identical deliverable. In case (b), the prior MATERIAL persisted entry still matches the tuple, and unless the wrapper is re-invoked with a higher round, A1.5 finds no superseding proposal and Path A returns the prior MATERIAL verdict. If the user wants the byte-identical re-audit to supersede the prior verdict, the wrapper MUST be invoked with --round N+1 --target-rounds N+1 (same rule as another_round). The same rule applies to any deployment-side wrapper run dispatched outside the §5.4 escalation prompt: a fresh wrapper run that uses --round 1 will not preempt a prior MATERIAL entry; only --round N+1 (where N is the prior persisted entry's round) will trigger A1.5 supersession.
Step 6 audit gate must not collide with pre-v3.6.7 checkpoint mechanisms. Co-occurrence rules:
| Pre-existing mechanism | Audit gate co-occurrence rule |
|---|---|
| FULL checkpoint | Audit verdict appended to Decision Dashboard as informational line; does not change checkpoint type |
| MANDATORY checkpoint (integrity) | Audit gate runs FIRST. If MATERIAL, integrity checkpoint never fires (deliverable is rolled back to producing agent before reaching Stage 2.5/4.5). If PASS, integrity proceeds normally. |
| MANDATORY checkpoint (review decision) | Same: audit gate runs FIRST at the producing agent's exit; review checkpoint runs at Stage 3 (downstream). No collision. |
| collaboration_depth_agent observer | Observer fires on FULL/SLIM checkpoint dispatch as today; orchestrator's audit verification happens BEFORE observer dispatch. Observer does NOT see audit findings (per its v3.5 advisory-not-blocking design). |
| ARS_PASSPORT_RESET reset boundary | Audit_artifact[] entries serialize into passport per Schema 9 append-only ledger semantics, layered on top of the v3.6.3 reset-boundary protocol (academic-pipeline/references/passport_as_reset_boundary.md — the canonical contract for what ARS_PASSPORT_RESET and resume_from_passport mean and how they interact with the reset_boundary[] ledger). On resume_from_passport=<hash>, orchestrator re-reads audit_artifact[] from passport and re-runs the eleven gating verification checks — they MUST still pass even after session reset (note: L3-3 / L3-4 require recomputing SHAs against current file contents, so a deliverable mutated between original session and resume will correctly fail freshness). On resume the orchestrator must also apply §5.6 A1.5 superseding-proposal preflight against <output-dir> before trusting any prior persisted entry — a higher-round wrapper run dispatched between sessions (e.g., the user clicked another_round in session A and the wrapper completed under CI before session B resumes) must preempt the prior entry per D4, otherwise resume would silently use the stale verdict (F-070 generalization across resume). (Resolves L3 from §2.3 — see §9 R4 for full risk discussion.) |
| Mid-Entry Material Passport Check | Existing logic (verification_status / 24h freshness / version_label diff) runs as today. Audit_artifact[] is a separate gate downstream of this entry check. |
Step 6 adds one new subsection (§3.5 Audit Artifact Gate) to academic-pipeline/agents/pipeline_orchestrator_agent.md, inserted between current §3 "Checkpoint Management" and §4 "Transition Management" — audit gate runs at transition time, so structurally it belongs in transition flow.
Prompt-vs-spec split (load-bearing for Phase 6.6 implementers). The fenced ```markdown block below is the complete spec-side reference (~347 lines) describing the orchestrator's runtime gate procedure. It is NOT inserted verbatim into the orchestrator prompt. Phase 6.6 ships a ~50-line decision-policy summary (trigger, ship/block decision rule, hard rules, cross-references, plus the P-PA- / P-PB- failure phase IDs as cross-references back here) into the orchestrator prompt; the procedural detail (Path A → Path B fall-through, A1–A7, B1–B11, A1.5 supersession preflight, B1a/B8b/B8c recovery, F-067 / F-069 / F-070 / F-072 closures, the 24-row Failure State Inventory at the end of this section — 7 P-PA- rows + 17 P-PB- rows) stays in this spec as the implementation contract. The closing prose after the fenced block, the §9.1 R3 risk entry, the §10 Phase 6.6 verification gate, and docs/design/2026-05-05-phase-6.6-scoping-note.md §3 Path 2 all describe the same split and must stay co-consistent. An implementer who reads only the fenced block and copies it into the prompt would blow the +60-line gate by ~580% — read §10 Phase 6.6 deliverable and the closing prose alongside the block before editing the orchestrator prompt.
### 3.5 Audit Artifact Gate (v3.6.7 Step 6)
**Trigger.** At every stage transition where a v3.6.7 downstream agent
(`synthesis_agent` / `research_architect_agent` survey-designer mode /
`report_compiler_agent` abstract-only mode) just completed a deliverable.
**Procedure (Path A → Path B fall-through model).**
The orchestrator first attempts Path A (already-merged persisted entry). Path A's A1.5 step always scans `<output-dir>` for superseding higher-round proposals (per D4); if a superseding proposal exists, Path A is preempted and Path B runs with `supersession_required = true`. Otherwise Path A continues to A2-A7. If Path A succeeds end-to-end, the gate passes by reading the persisted entry's verdict (the wrapper output directory was scanned at A1.5 but no proposal was merged). If Path A finds no matching persisted entry OR finds one but it fails A3-A6 verification (stale SHAs, broken sidecar, etc.) OR is preempted by A1.5 supersession, orchestrator falls through to Path B (look for a fresh proposal that supersedes the failed/preempted entry). This ordering avoids the deadlock where a stale persisted entry blocks the user's freshly-run audit from being merged AND closes the F-070 escalation deadlock where a higher-round user-dispatched proposal would have been silently ignored.
**Path A — already-merged entry (re-verification on resume / re-transition):**
A1. Search passport `audit_artifact[]` for the latest persisted entry matching:
- `stage` = the destination stage the just-completed deliverable is about
to enter (per §5.1 destination-stage convention: 2 for `synthesis_agent`
and `research_architect_agent` survey-designer; 5 for
`report_compiler_agent` abstract-only)
- `agent` = the just-completed v3.6.7 agent
- `deliverable_sha` = SHA-256 of the just-completed deliverable file
- latest by `verified_at` if multiple match (when multiple entries
share the same `run_id` — possible only via §5.4
`ship_with_known_residue` ack append, where the ack entry copies the
original MATERIAL entry's `run_id` — pick the latest by `verified_at`
within that `run_id` group; D3 strict-monotonic guarantees the ack
entry post-dates the original by ≥ 1 ms, so this resolves
deterministically).
A1.5. **Superseding proposal preflight (F-070 closure):** scan
`<output-dir>` for unmerged proposal entries matching the same
`(stage, agent, deliverable_sha)` tuple. After applying B1a-style
`run_id` de-duplication (any proposal whose `run_id` already appears
in passport `audit_artifact[]` for this tuple is a leftover from a
prior session's post-B9 / pre-consume crash; move it to `consumed/`
best-effort and exclude from this scan). If any remaining proposal's
`verdict.round` is greater than the A1-selected persisted entry's
`verdict.round`, GO TO Path B with `supersession_required = true`.
Path A is preempted because the user (or deployment) has dispatched
a higher-round audit that the spec contract requires the orchestrator
to consider before trusting an older verdict — covers `another_round`
user choice (§5.4), manual fresh wrapper run outside the §5.4
prompt, byte-identical `abort_stage` re-audit, AND higher-round
`AUDIT_FAILED` proposal (a higher-round AUDIT_FAILED proposal must
preempt because falling back to the prior MATERIAL would silently
erase the attempted escalation; Path B5 short-circuit then BLOCKs
with the failure reason, which is the correct user-visible signal).
If no superseding proposal exists, continue to A2 with the A1-
selected entry.
A2. If no matching entry from A1 (and A1.5 found no superseding proposal): GO TO Path B (no Path A candidate).
A3. Validate the entry against `audit_artifact_entry.schema.json --mode persisted`
(`verdict.verified_at` and `verdict.verified_by` MUST be present). On schema
failure: GO TO Path B (Path A entry is malformed and a fresh proposal may
supersede it).
A4. Re-run the eleven gating verification checks (§5.2). On resume the L3-3 / L3-4
SHA recompute is what catches deliverable mutation since the original session.
First failure: GO TO Path B (entry is stale; look for a fresh proposal that
supersedes it).
A5. All eleven pass: read `<run_id>.verdict.yaml`, validate against
`audit_verdict.schema.json`. On schema failure: GO TO Path B. On schema pass,
confirm the verdict file's mirrored fields (`verdict_status`, `round`,
`target_rounds`, `finding_counts`) match the persisted entry's mirror — drift
is treated as Pattern C3 evidence (verdict file was forged or corrupted
post-merge): GO TO Path B.
A6. **Late freshness barrier**: immediately before applying ship/block,
re-run L3-3 (deliverable SHA) and L3-4 (bundle manifest SHA) against
current on-disk file contents. The earlier A4 eleven-gate run captured
the bundle state at gate-check time; a deliverable or supporting file
that mutated between A4 and now would otherwise let orchestrator
apply a ship/block decision on stale evidence. First failure here:
GO TO Path B (the entry's audit is stale; look for a fresh proposal
that supersedes it).
A7. All checks pass: apply ship/block per §5.3. **Do NOT append or mutate
the entry** — it is already in passport. Path A succeeded; Path B is
skipped.
**Path B — proposal file (first-time merge OR supersede a failed Path A entry):**
B1. Scan `<output-dir>` for an unmerged `<run_id>.audit_artifact_entry.json`
matching the same `(stage, agent, deliverable_sha)` tuple. **Snapshot the
candidate set** at scan start — record each candidate's `(filename,
file_mtime, sidecar_mtime)` triple. The snapshot is what B2 selects
against; later steps (B6 / B8c) will compare the snapshot's
triples against current filesystem state and BLOCK if they differ
(concurrent wrapper writes during orchestrator scan would otherwise
create a TOCTOU race between selection time and verification time).
Files in `consumed/` subdirectory are excluded — already merged.
B1a. **Idempotent run_id de-duplication**: for every candidate proposal in
the B1 snapshot, check whether passport `audit_artifact[]` already
contains a persisted entry with the same `run_id`. If yes, treat that
proposal as already consumed and split by whether the duplicate
persisted entry matches the current gate tuple `(stage, agent,
deliverable_sha)`:
- **Tuple matches** (the persisted entry IS an audit answer for
this gate): branch by `supersession_required`:
* **`supersession_required = false`** (no higher-round proposal
in `<output-dir>` per A1.5): attempt to move the leftover
proposal to `<output-dir>/consumed/<run_id>.audit_artifact_entry.json`
(best-effort, log on failure but do not block), then **proceed
as if Path A had selected this persisted entry — re-run Path A
steps A3 (schema validation) → A4 (eleven gating checks) → A5
(verdict file schema + mirror match) → A6 (late freshness
barrier) against it before applying ship/block at A7**. This is
what makes the tuple-match branch safe: a prior session committed
the persisted entry, but between that commit and the current
session the deliverable / bundle files / verdict file may have
mutated, gone missing, or been hand-edited. Skipping straight to
B10 (the original Round 13 closure) would let orchestrator
apply a ship/block decision on a stale or untrusted entry.
Re-running A3-A6 in this branch enforces the same trust boundary
as a normal Path A entry.
* **`supersession_required = true`** (A1.5 found a higher-round
proposal that must preempt the prior persisted entry): the dup
is a same-tuple already-persisted lower-or-equal-round entry
(it may be the A1-selected latest entry, or it may be an older
round entry whose leftover proposal was never consumed — e.g.,
passport carries rounds 1, 2, 3 with A1 selecting round 3, while
a round-2 leftover proposal sits in `<output-dir>` because that
prior session never reached B9 consume). In either case, the
dup's `verdict.round <= selected_persisted.verdict.round`, and
the user-dispatched higher-round proposal MUST win. Move the
leftover proposal to `consumed/` (best-effort) and **remove
this dup from the B1 candidate set without re-running A3-A6**
— re-validating and shipping the prior round verdict (whether
round-N or older round-N-k) would silently override the user's
`another_round` choice (F-072 closure). Continue B1a processing
with remaining candidates; B2 supersession-mode filter then
selects the higher-round
proposal and the merge proceeds normally.
If A3-A6 fails (only reachable in the supersession-false branch):
remove ONLY this consumed duplicate from the B1 candidate set and
continue B1a processing with any remaining candidates (those
candidates may include fresh round-N+1 proposals that supersede
the failed entry — see §5.4 `another_round` mechanism). If after
B1a the candidate set is non-empty, proceed normally to B2
selection. If after B1a the candidate set is empty (the only
candidate was the failed-revalidation duplicate, and no fresh
proposal exists), the procedure falls through to B3 BLOCK with
the carried-forward failure reason: `[AUDIT GATE: prior persisted
entry <run_id> failed re-validation: <reason>; please re-audit]`
— same diagnostic shape as a normal Path A → Path B fall-through.
Closes the false-BLOCK race the original Round 13 closure
addressed AND closes the unsafe-trust window F-067 surfaced AND
closes the F-069 multi-candidate cascade where a single failed
re-validation would have prematurely BLOCKed even when other
valid candidates existed AND closes the F-072 supersession
short-circuit where a lower-round leftover dup would have used
the prior verdict instead of yielding to the higher-round
proposal.
- **Tuple does not match** (the persisted entry is for a different
gate, e.g., a stage / agent / deliverable that happens to share
`run_id` — possible only under hand-edit since wrappers generate
`run_id` randomly per `/dev/urandom`): the leftover proposal is
not relevant to this gate. Best-effort move to `consumed/`, then
remove it from the current B1 candidate set and continue B1
processing with the remaining candidates (proceed normally to B2).
Together these closures of the crash-window ensure the next session
cannot re-merge the same proposal (no duplicate persisted entry) AND
cannot false-BLOCK on a proposal that already has a committed audit
answer in the passport.
B2. Selection from the candidate set:
- **Default mode** (no `supersession_required` flag from A1.5): pick
the proposal with the latest sidecar `timing.started_at` (RFC 3339
ms precision per §3.4 — strictly chronological, independent of
`run_id` random suffix). Tie-break on `run_id` lexicographic max
as a deterministic secondary key (the `<ISO-8601-Z>-<4-hex>` random
suffix is not chronologically meaningful at sub-second resolution
because the suffix is sampled from `/dev/urandom` per §4.3 —
lexicographic max on `run_id` alone is therefore not chronological
and is reserved here only for total-order determinism on identical
`started_at`). When Path A fell through on a stale entry, B2 selects
the fresh proposal that supersedes the failed entry; the prior
persisted entry remains in the append-only ledger as historical
record.
- **Supersession mode** (`supersession_required = true` from A1.5):
filter the candidate set to only proposals whose `verdict.round`
is greater than the A1-selected persisted entry's `verdict.round`.
Then apply the same `started_at` primary / `run_id` lex-max
tie-breaker selection within the filtered set. This guarantees the
higher-round audit explicitly dispatched by the user (or deployment)
is what gets merged, not an older same-tuple proposal that happened
to be in `<output-dir>`.
Reading `timing.started_at` requires opening the proposal's companion
sidecar; proposals whose sidecar fails Layer 3 schema validation (B6
below) are rejected at that stage rather than being selected and then
rejected — but if multiple proposals tie on `started_at` AND share an
unreadable sidecar, orchestrator BLOCKs with `[AUDIT GATE: ambiguous
proposal selection at <output-dir>; multiple unverifiable sidecars
share started_at]` rather than guessing.
B3. If no proposal file: behavior depends on whether A1.5 set
`supersession_required`:
- **No supersession**: BLOCK transition. Display
`[AUDIT GATE BLOCKED: no audit artifact for <agent>/<deliverable>]`
with the exact wrapper command the user can run:
`scripts/run_codex_audit.sh --stage <N> --agent <A> --deliverable <P> --round 1`.
If Path A fell through with a specific failure reason, also surface that:
`[AUDIT GATE: prior persisted entry <run_id> failed Path A: <reason>; please re-audit]`.
- **Supersession required but no eligible proposal survives B2's
higher-round filter** (i.e., the candidate set after B2 filtering
is empty — every candidate was either lower-round or invalid at
B2 selection time): BLOCK with `[AUDIT GATE: requested
higher-round audit artifact is missing; expected
verdict.round > <prior_round> for tuple (stage=<N>, agent=<A>,
deliverable_sha=<sha>); please re-audit at round <prior_round + 1>]`.
(Note: a higher-round proposal that DOES pass B2 selection but
then fails B6 eleven-gate or B7 verdict-schema is NOT this case — it
gets the normal P-PB-gate / P-PB-verdict-schema BLOCK with the
proposal-specific failure diagnostic, leaving the proposal in
`<output-dir>` for re-audit.)
Do NOT silently fall back to Path A reading the prior persisted
entry — the user explicitly requested supersession via §5.4
`another_round` (or the deployment dispatched a higher-round
wrapper run), and silently using the prior verdict would erase
that authorization.
B4. Validate the proposal against `audit_artifact_entry.schema.json --mode proposal`.
Reject if `verdict.verified_at` / `verdict.verified_by` are pre-filled (Pattern
C3 attack surface — wrapper must NEVER set these; only orchestrator does).
B5. **AUDIT_FAILED short-circuit**: if proposal carries
`verdict.status == "AUDIT_FAILED"` (per §4.6), BLOCK transition WITHOUT
running the eleven gating checks. The audit was attempted but did not produce
a verdict; gating against incomplete evidence is meaningless. Surface the
`verdict.failure_reason` to the user. Do not move proposal to `consumed/`
(user may want to inspect). User must dispatch a fresh wrapper run.
B6. Run the eleven gating verification checks (§5.2) using the paths declared in
the proposal. First failure stops; surface the failed check and leave the
proposal file in place for re-audit.
B7. If all eleven pass: validate `<run_id>.verdict.yaml` against
`audit_verdict.schema.json`. On schema failure, BLOCK with diagnostic.
Then confirm proposal entry mirrored fields
(`verdict.{status, round, target_rounds, finding_counts, failure_reason}`)
match `verdict.yaml` exactly — drift between proposal mirror and verdict
file is treated as Pattern C3 evidence (wrapper or hand-edit corrupted
one of them) and BLOCKS. The mirror persisted into Schema 9 is sourced
from `verdict.yaml`, not the proposal, so `verdict.yaml` wins on any tie.
B8a. **Late freshness barrier** (mirrors A6): immediately before mutating
proposal → persisted, re-run L3-3 (deliverable SHA) and L3-4 (bundle
manifest SHA) against current on-disk file contents. The B6 eleven-gate
run captured the bundle state at gate-check time; a deliverable or
supporting file that mutated between B6 and now would otherwise let
orchestrator persist a verdict on stale evidence. First failure here:
BLOCK with `[AUDIT GATE: bundle mutated post-gate, before persist;
re-audit required]`, leave the proposal file in place for re-audit.
B8b. **Idempotent re-check** (covers a different window than B1a, despite
the surface similarity): re-confirm passport `audit_artifact[]` does
not already contain a persisted entry with this `run_id`. Defends
against the narrow window where another orchestrator session merged
the same proposal between B1a and B8b. **Unlike B1a, B8b does NOT
re-run A3-A6 on the duplicate persisted entry**, because the current
session has already validated the candidate proposal at B6 (eleven
gating checks against current on-disk file state) + B7 (verdict file
schema + mirror match) + B8a (late freshness recheck of L3-3/L3-4) —
those checks established the same trust boundary A3-A6 would
establish on the duplicate persisted entry, so re-running would
duplicate work. If a duplicate is now present: skip the persist
(treat as already-merged), move proposal to `consumed/`, GO TO B10
reading the pre-existing persisted entry's verdict.
B8c. **Snapshot integrity check**: re-stat the proposal file and its
sidecar; compare `(file_mtime, sidecar_mtime)` against the B1
snapshot. If either mtime changed, BLOCK with `[AUDIT GATE: proposal
or sidecar mutated between scan and persist; re-scan required]` and
return to B1 for a fresh scan.
B8d. Construct the persisted entry **in memory only** by copying the proposal
file's parsed object and adding `verdict.verified_at` (via §5.4's
strict-monotonic helper) and `verdict.verified_by =
"pipeline_orchestrator_agent"`. **MUST NOT rewrite the proposal file on
disk** — the proposal file remains a pure proposal until B9's
consume-move (or B1a/B8b idempotent move on the recovery path). This
guarantee is load-bearing for crash recovery: a crash between B8d and
B9 leaves the proposal file unchanged on disk, so the next session's
B4 `--mode proposal` validation still accepts it for re-merge. If
implementer rewrote the proposal file at B8d (filling verified_at /
verified_by), the next-session B4 would reject it as "wrapper-emitted
proposal with verified_at filled" (Pattern C3 attack surface) and the
audit would be irrecoverably blocked. Re-validate the in-memory object
against `audit_artifact_entry.schema.json --mode persisted`; on
validation failure, BLOCK without proceeding to B9.
B9. Append the persisted entry to passport `audit_artifact[]` via tmp-write
+ fsync + atomic rename (write the new passport content to
`<passport_path>.tmp`, fsync the tmp file, then rename over the live
passport — same atomicity pattern §4.4 requires for wrapper output).
On atomic-rename success, move proposal file to
`<output-dir>/consumed/<run_id>.audit_artifact_entry.json` (best-effort;
failure to move logs a warning but the persisted entry is committed —
B1a's de-duplication catches the leftover proposal on the next session).
Crash window: if orchestrator crashes between passport rename and
proposal move, the next session sees the persisted entry AND the
unmoved proposal; B1a de-duplication then idempotently moves the
proposal to `consumed/` and skips the merge.
B10. Apply ship/block per §5.3:
- PASS → proceed to next stage; append audit line to FULL checkpoint
- MINOR → MANDATORY checkpoint with finding punchlist; await user choice
- MATERIAL → BLOCK; surface findings; await re-audit (with bumped
target_rounds if user selects `another_round`, per §5.4)
B11. After round == target_rounds rounds with MATERIAL: emit ESCALATION block per
§5.4; await user ship_with_known_residue / another_round (bumps cap by 1) /
abort_stage choice.
**Path selection invariants (revised under Path A → Path B fall-through and A1.5 superseding-proposal preflight):**
- Path A is attempted first whenever a matching persisted entry exists, BUT
**Path A success is preempted by A1.5's superseding-proposal preflight**:
if `<output-dir>` carries an unmerged proposal for the same tuple with
`verdict.round > selected_persisted.verdict.round`, A1.5 forces Path B to
run with `supersession_required = true` and Path A is skipped entirely.
This is what closes F-070's `another_round` deadlock and equivalent
user-dispatched higher-round flows (manual fresh wrapper run, byte-
identical `abort_stage` re-audit, higher-round AUDIT_FAILED proposal).
- Outside the supersession case, Path A succeeds only when the entry
validates AND the eleven gating checks pass AND the verdict file is
schema-valid AND mirror fields match.
- Path A failure at any step (A2 / A3 / A4 / A5 / A6) falls through to Path B with
the failure reason carried forward. This closes the deadlock: a stale
persisted entry no longer blocks a fresh proposal from being merged.
- Path B's proposal selection (B2) is by latest sidecar `timing.started_at`
(chronological, ms precision) with `run_id` lex-max as deterministic
tie-breaker — guarantees the latest fresh proposal supersedes any prior
failed entries even when same-second `run_id` random suffixes would have
otherwise sorted incorrectly. In supersession mode (A1.5 set the flag),
B2 additionally filters the candidate set to proposals whose
`verdict.round > selected_persisted.verdict.round`.
- Path A never appends to passport (entry already there). Path B always appends
exactly one persisted entry per successful merge — including when it
supersedes a prior failed Path A entry. The append-only ledger preserves
history of both the failed entry AND the superseding entry.
- Failed Path A entries are NOT deleted from passport (append-only). They
remain as audit history. Future Path A attempts on the same entry will fall
through again until a fresh proposal supersedes via Path B.
**Hard rules.**
- Audit gate cannot be skipped via mode switch. There is no "skip audit"
option in checkpoint command vocabulary.
- Audit gate runs BEFORE collaboration_depth_agent observer dispatch
and BEFORE integrity_verification_agent dispatch — it is the first
transition-time check.
- A `verdict_status: PASS` does NOT imply integrity check is skipped.
Stage 2.5 / 4.5 integrity gates remain mandatory per existing
spec §3 "Hard boundaries" rule 9.
**Failure surfacing.** Any block message uses the standard FULL/MANDATORY
checkpoint visual (━━━ separator) so user attention is preserved.
Block message MUST include:
- Why blocked (which check failed, or which severity finding triggered)
- Where to look (file:line for findings; artifact path for verification failures)
- What to do next (re-audit command, revision dispatch, escalation options)
**Cross-references.**
- v3.6.7 Step 6 spec: docs/design/2026-04-30-ars-v3.6.7-step-6-orchestrator-hooks-spec.md
- Audit template: shared/templates/codex_audit_multifile_template.md
- Schema: shared/contracts/passport/audit_artifact_entry.schema.json
- Wrapper: scripts/run_codex_audit.sh
The fenced block above is the complete spec-side procedure (~347 lines). The orchestrator prompt does NOT receive this verbatim — Phase 6.6 ships a decision-policy summary (~50 lines) that names the trigger, ship/block decision rule, hard rules, cross-references, and lists the P-PA- / P-PB- failure phase IDs as load-bearing references back to this spec section. The detailed Path A → Path B procedure (A1–A7, B1–B11, A1.5 supersession preflight, B1a tuple-match recovery, B8a/B8b/B8c late freshness barriers, F-067 / F-069 / F-070 / F-072 closures) is the implementation contract and remains in this spec. Orchestrator follows the procedure in §5.6 exactly; the prompt's role is to declare the gate, name the decision policy, and reference §5.6 as the procedure source. §9.1 R3 carries the prompt-bloat risk entry; the ~50-line summary addition is within budget for the existing ~580-line orchestrator prompt.
The split (procedure stays in spec; only decision-policy summary enters prompt) reflects the broader orchestrator design pattern: prompt is the LLM-facing decision contract, spec is the implementation contract. Crash-recovery transactional semantics (B1a / B8b / B8c invariants) are reviewer / lint concerns, not LLM-dispatch concerns. See docs/design/2026-05-05-phase-6.6-scoping-note.md §3 Path 2 for the resolution rationale; the scoping note discovered the §5.6 fenced block had grown from ~50 to ~347 lines across rounds (PR #50 → #53 cascade sweeps adding F-070 closure et al.) while §9.1 R3 + §10 Phase 6.6 budget claims inherited the original PR #50 footprint unchanged.
Verification failure state inventory. Path A and Path B above describe what orchestrator does on the happy path and where it falls through; this inventory enumerates what observable state exists after every distinct failure phase. Reviewers can read this table to answer "what has been read, what has been written, what remains in <output-dir>, and what message does the user see after step X fails?" without stitching together §5.2 + §5.6 + §4.9 prose. Each phase is a partial-execution boundary; the inventory is the state machine's failure-side contract.
| Failure phase | Trigger | Passport mutation | Proposal location | User-visible message | Path A/B fall-through |
|---|---|---|---|---|---|
| P-PA-precond | A1 finds no matching persisted entry | none | unchanged | (silent — go to Path B) | A2 → Path B |
| P-PA-schema | A3 entry fails --mode persisted schema validation |
none | unchanged | logged: [AUDIT GATE: stale persisted entry <run_id> failed schema; falling through to Path B] |
A3 → Path B |
| P-PA-gate | A4 eleven-gate failure (any L2-x or L3-x) | none | unchanged | logged: [AUDIT GATE: persisted entry <run_id> failed gate L<N>: <reason>] |
A4 → Path B with reason carried |
| P-PA-verdict-schema | A5 verdict file fails audit_verdict.schema.json |
none | unchanged | logged: [AUDIT GATE: verdict file <path> schema-invalid] |
A5 → Path B |
| P-PA-verdict-mirror | A5 verdict file mirror disagrees with persisted entry mirror | none | unchanged | logged: [AUDIT GATE: persisted entry <run_id> verdict drift from <path>] (Pattern C3 evidence) |
A5 → Path B |
| P-PA-stale-late | A6 late freshness recheck (L3-3 or L3-4) detects mutation | none | unchanged | logged: [AUDIT GATE: persisted entry <run_id> went stale during verification — bundle mutated since A4] |
A6 → Path B |
| P-PA-supersede-preempt | A1.5 superseding-proposal preflight finds an unmerged proposal whose verdict.round > selected_persisted.verdict.round (F-070 closure: typical for another_round user choice; also for manual fresh wrapper run with bumped round, byte-identical abort_stage re-audit, higher-round AUDIT_FAILED proposal) |
none yet — Path A is preempted before A3 | unchanged at this step (Path B will process the proposal next) | (silent — proceed to Path B with supersession_required = true) |
continues at Path B with supersession flag |
| P-PB-empty | B1 finds no proposal (and Path A also fell through OR found none) | none | unchanged | BLOCK: [AUDIT GATE BLOCKED: no audit artifact for <agent>/<deliverable>] + wrapper command |
terminal — user must re-audit |
| P-PB-supersede-missing | A1.5 set supersession_required = true but no eligible proposal survives B2's higher-round filter (every candidate was either lower-round or invalid at B2 selection time). A higher-round proposal that survives B2 but fails B6 / B7 / B8a is covered by P-PB-gate / P-PB-verdict-schema / P-PB-stale-late instead with the proposal-specific failure message |
none | unchanged (the original lower-round proposals stay in <output-dir> for diagnostic) |
BLOCK: [AUDIT GATE: requested higher-round audit artifact is missing; expected verdict.round > <prior_round> for tuple ...; please re-audit at round <prior_round + 1>] |
terminal — user must re-audit at the requested round |
| P-PB-ambig | B2 multiple proposals tie on started_at AND share unverifiable sidecar |
none | unchanged | BLOCK: [AUDIT GATE: ambiguous proposal selection at <output-dir>; multiple unverifiable sidecars share started_at] |
terminal — user must inspect |
| P-PB-proposal-schema | B4 proposal fails --mode proposal schema validation (e.g., wrapper pre-filled verified_at) |
none | proposal stays in <output-dir> (NOT moved to consumed/) for user inspection — Pattern C3 evidence |
BLOCK: [AUDIT GATE: proposal <run_id> rejected: <schema error>] (Pattern C3 attack surface) |
terminal — user must inspect |
| P-PB-audit-failed | B5 proposal carries verdict.status: AUDIT_FAILED |
none | proposal stays in <output-dir> (NOT moved per §4.9 step 9) — user may inspect |
BLOCK: [AUDIT GATE: audit attempted but failed: <verdict.failure_reason>] |
terminal — user must dispatch fresh wrapper run |
| P-PB-gate | B6 eleven-gate failure (any L2-x or L3-x) | none | proposal stays in <output-dir> for re-audit |
BLOCK: [AUDIT GATE: proposal <run_id> failed gate L<N>: <reason>] |
terminal — user must re-audit |
| P-PB-verdict-schema | B7 verdict file fails schema | none | proposal stays in <output-dir> |
BLOCK: [AUDIT GATE: verdict file <path> schema-invalid] |
terminal — user must inspect |
| P-PB-verdict-mirror | B7 proposal mirror disagrees with verdict file | none | proposal stays in <output-dir> (Pattern C3 evidence) |
BLOCK: [AUDIT GATE: Pattern C3: proposal mirror drift from verdict.yaml] |
terminal — user must inspect |
| P-PB-stale-late | B8a late freshness recheck (L3-3 or L3-4) detects mutation | none | proposal stays in <output-dir> for re-audit |
BLOCK: [AUDIT GATE: bundle mutated post-gate, before persist; re-audit required] |
terminal — user must re-audit |
| P-PB-dup-early | B1a early idempotent recheck finds passport already has a candidate proposal's run_id AND the persisted entry's (stage, agent, deliverable_sha) matches the current gate tuple (typical recovery from a prior session's post-B9 / pre-consume crash). Behavior splits by supersession_required from A1.5 |
none in this session (the duplicate was committed by the other session) | proposal moved to consumed/ (best-effort); skip candidate selection. supersession_required = false: re-run Path A steps A3-A6 against the pre-existing persisted entry (F-067 closure: deliverable / bundle / verdict file may have mutated, no unconditional trust); on A3-A6 success: GO TO A7 ship/block; on A3-A6 failure: remove ONLY this consumed duplicate from the B1 candidate set and continue B1a processing with any remaining candidates (F-069 closure). supersession_required = true: do NOT re-run A3-A6 — the dup is a same-tuple lower-or-equal-round entry that the higher-round proposal is meant to preempt (it may be A1-selected or older — see B1a prose for multi-round-history case); remove it from the candidate set and continue B1a so B2 supersession-mode filter can select the higher-round proposal (F-072 closure: re-validating and shipping any prior round verdict here would silently override the user's another_round choice) |
(silent on supersession-true case; silent on supersession-false A3-A6 success or A3-A6 failure with remaining candidates; on supersession-false A3-A6 failure with empty candidate set: B3 BLOCK with [AUDIT GATE: prior persisted entry <run_id> failed re-validation: <reason>; please re-audit]) |
supersession-true: continues B1a/B2 with remaining (B2 supersession-mode filter selects higher-round proposal); supersession-false A3-A6 success: continues at A7; supersession-false A3-A6 failure with remaining candidates: continues B1a/B2; supersession-false A3-A6 failure with empty set: falls through to B3 |
| P-PB-dup-other | B1a finds passport already has a candidate proposal's run_id but the persisted entry is for a DIFFERENT gate tuple (only reachable under hand-edit since wrapper-generated run_id is random per /dev/urandom) |
none in this session | proposal moved to consumed/ (best-effort); proposal removed from candidate set; B1 processing continues with remaining candidates |
(silent — continue to B2 with the remaining proposal candidates) | continues at B2 |
| P-PB-dup-late | B8b late idempotent recheck finds passport already has this run_id (concurrent merge that committed between B1a and B8b) |
none in this session (the duplicate was committed by the other session) | proposal moved to consumed/ (best-effort); skip persist; GO TO B10 reading the pre-existing persisted entry's verdict |
(silent — proceed using the already-persisted entry's verdict) | continues at B10 |
| P-PB-snapshot | B8c snapshot integrity check (proposal or sidecar mtime changed since B1 scan) | none | proposal stays in <output-dir> |
BLOCK: [AUDIT GATE: proposal or sidecar mutated between scan and persist; re-scan required] |
restart at B1 |
| P-PB-persisted-schema | B8d --mode persisted re-validation fails (lint caught a contradiction the proposal-mode validation missed) |
none — atomic-rename did not yet fire | proposal stays in <output-dir> |
BLOCK: [AUDIT GATE: persisted-mode re-validation failed: <reason>] |
terminal — user must inspect |
| P-PB-passport-write | B9 atomic rename of <passport>.tmp over live passport fails (filesystem error) |
none — tmp-write/fsync/atomic rename is a transaction; partial failure leaves passport unchanged | proposal stays in <output-dir> |
BLOCK: [AUDIT GATE: passport write failed: <fs error>] |
terminal — user must check disk |
| P-PB-consume-fail | B9 passport rename succeeded but proposal-move to consumed/ failed |
persisted entry committed | proposal stays in <output-dir> (the unmoved file is the recovery signal — B1a on the next session moves it idempotently) |
(silent — entry is committed, decision proceeds) | continues at B10 |
| P-PB-crash | Orchestrator crashes between any two B-steps after the B1 scan | depends on phase: passport committed iff B9 atomic rename succeeded; otherwise none | depends on phase: in <output-dir> if pre-B9 or post-B9-pre-move; in consumed/ if post-move |
none in the crashed session — recovery happens on the next session via Path A (finds the persisted entry if B9 succeeded) or Path B + B1a (idempotently moves leftover proposals if B9 succeeded but move did not, or re-merges from scratch if neither succeeded) | next session resumes at A1 |
Why this inventory is load-bearing. The Round 12 §4.4 control-flow audit surfaced two §4.4 P1s (F-059 SIGTERM, F-060 TOCTOU) that prior eleven rounds missed because no review walked the wrapper looking for crash / race / partial-state windows. §5.6 has the same complexity at orchestrator scope (multi-step process, side effects across passport + filesystem, can be interrupted by LLM session crash, can race with a second orchestrator session under multi-agent setups). The inventory makes the implicit transaction model explicit so a future review can audit each phase against actual procedure rather than re-deriving "what state should orchestrator be in after step X failed?" each time. New failure phases added in v3.6.8+ get new rows.
To make the additive nature explicit:
continue / pause / adjust / redo / skip / abort). Step 6 introduces audit-gate-specific user inputs that live in their own response namespace and do not enter the checkpoint command vocabulary: iterate (MINOR verdict response, §5.3 MINOR row), and ship_with_known_residue / another_round / abort_stage (round-3 MATERIAL escalation choices, §5.4). These four labels are responses to Step-6 prompts only; the checkpoint command vocabulary stays unchanged. Lint enforces that audit-gate prompts use these four labels exactly and do not redefine the existing six.resume_from_passport). Audit_artifact[] re-verification on resume is part of the existing ledger replay, not a new code path.Style Profile carry-through, Material Passport versioning, or verification_status field semantics.The Step 6 hook is structurally a new transition-time check that runs before existing transition logic. It has its own block conditions but does not alter existing block conditions.
Q3's "Layer 2 + Layer 3 + partial inversion" decision (§2.1) has two halves. Layer 2 + Layer 3 are the orchestrator-side gating checks specified in §3 and §5. Partial inversion is the prompt-side edit that complements them: each downstream agent prompt keeps the explicit prohibition on simulated audit but stops describing the downstream audit pipeline that follows. The asymmetry is the load-bearing part — the agent must still know "do not fake an audit," but must NOT know "an audit is dispatched after my output is read." Without the second clause, sub-agent fake-audit hallucination (Pattern C3, per feedback_subagent_tool_hallucination.md) has no trigger surface.
This section specifies the rule, lists the four sentences to be edited across the three v3.6.7 downstream agent prompts, and defines the inversion sweep checklist that lint enforces in CI.
The partial inversion rule has two clauses, which MUST both apply uniformly across the three v3.6.7 downstream agent prompts:
Clause 1 — Retain the prohibition. Every PATTERN PROTECTION (v3.6.7) block MUST carry the verbatim line:
DO NOT simulate any audit step. DO NOT claim to have run codex/external review. Output metadata must not claim audit-passed state.
This line names the failure mode (Pattern C3 fake-audit hallucination) without revealing that an actual audit pipeline exists downstream. The agent learns "do not invent audit results" as a category of forbidden behavior, not as a hint that its output will be audited.
The exact wording matters. The paired codex/external review phrasing names the forbidden category (so the agent knows which fabrications are off-limits) without naming the orchestrator or saying "my output will be audited later" — the agent learns what not to claim rather than what runs downstream. Earlier drafts that wrote "the orchestrator runs codex audit afterward" leaked the latter and trip-wired Pattern C3; the §6.3 sweep retires every variant of that leak.
Clause 2 — Remove the pipeline disclosure. Every PATTERN PROTECTION (v3.6.7) block MUST NOT carry any sentence that:
a. Names the orchestrator's audit dispatch, OR
b. References codex_audit_multifile_template.md audit dimensions as a downstream consumer ("Cross-model audit follows §3.1..."), OR
c. Names a future audit that will read this agent's output, OR
d. Hints that the agent's output is the input to a verification step (e.g., "downstream audit covers...").
Compliance with Clause 1 is necessary; Clause 2 is what makes inversion partial rather than total. A total inversion (removing Clause 1) would lose the prohibition and let unfettered audit hallucination through; a no-inversion baseline (keeping Clause 2 sentences) trips Pattern C3. The two clauses together define the only point in the design space that defends the boundary without trip-wiring it.
Why this is a v3.6.7-specific rule, not a global one. Per Q1=A, scope is restricted to the three v3.6.7 downstream agents. Other ARS agents (e.g., bibliography_agent, socratic_mentor_agent, field_analyst_agent) may legitimately reference codex / cross-model verification in their prompts because they are not the dispatch-point for the audit gate; their prompts are not Pattern C3 attack surfaces because they do not produce the deliverable that the orchestrator audits. The §6.3 sweep is correspondingly narrow — it lints exactly three files. Widening the sweep is the L2 deferred question (§9 L2); the answer for v3.6.7 is "do not widen" because every additional file in scope is one more file the lint must keep clean against ordinary editorial drift.
The Step 1+2 ship (PR #48, main b4fbffd) added PATTERN PROTECTION (v3.6.7) blocks to three downstream agent prompts. As of main HEAD b76e626, those blocks contain four sentences that violate Clause 2 of §6.1 plus one sentence that already aligns with Clause 1 in only one of the three prompts. Step 6 closes the gap.
| # | File | Line (at HEAD b76e626) |
Existing sentence | §6 action |
|---|---|---|---|---|
| 1 | deep-research/agents/synthesis_agent.md |
164 | "Cross-model audit follows shared/templates/codex_audit_multifile_template.md audit dimensions §3.1, §3.2, §3.3, §3.4 and the bundle-specific Section 4(f) check." |
Remove sentence. It violates Clause 2(b) and Clause 2(d) — it names the audit template the orchestrator dispatches against and frames the agent's output as that template's input. |
| 2 | deep-research/agents/research_architect_agent.md |
190 | "Cross-model audit covers these via dimension §3.5 (instrument quality) of shared/templates/codex_audit_multifile_template.md." |
Remove sentence. Same Clause 2(b) and Clause 2(d) violation. |
| 3 | deep-research/agents/report_compiler_agent.md |
172 | "Cross-model audit covers these via dimension §3.7 (COI adequacy) plus the bundle-specific Section 4(f) check of shared/templates/codex_audit_multifile_template.md." |
Remove sentence. Same Clause 2(b) and Clause 2(d) violation. |
| 4 | deep-research/agents/report_compiler_agent.md |
177 | "DO NOT simulate any audit step. DO NOT claim to have run codex/external review. The orchestrator runs codex audit afterward." | Trim trailing clause. Replace with the canonical Clause 1 line (no trailing "The orchestrator runs codex audit afterward" sentence). The "orchestrator runs codex audit afterward" tail is the most direct Clause 2(a) violation in the inventory. |
| 5 | deep-research/agents/report_compiler_agent.md |
178 | "Output metadata must not claim audit-passed state." | Keep. This sentence is the persistence-side complement of the Clause 1 line and stays. The §6.3 sweep folds it into the canonical Clause 1 line so the same wording appears in all three prompts (the other two prompts add it in §6.2's add step below). |
Two prompts (synthesis_agent.md, research_architect_agent.md) currently have neither the prohibition nor the pipeline-disclosure tail; they have only the pipeline-disclosure sentence (#1, #2 above) without any "DO NOT simulate audit" line. Step 6 must therefore both remove the pipeline-disclosure sentence AND add the canonical Clause 1 line to those two prompts. The third prompt (report_compiler_agent.md) has both — Step 6 removes the disclosure (line 172) and trims the trailing clause (line 177) but keeps the leading prohibition by way of replacing it with the canonical Clause 1 line.
Canonical Clause 1 line (final form to ship in all three prompts):
DO NOT simulate any audit step. DO NOT claim to have run codex/external review. Output metadata must not claim audit-passed state.
The line is one bullet in the PATTERN PROTECTION block, placed last in each block (after the pattern-specific bullets) so the prohibition is the closing instruction the agent reads.
Net change to each prompt:
synthesis_agent.md: −1 sentence (line 164 disclosure) + 1 bullet (canonical Clause 1).research_architect_agent.md: −1 sentence (line 190 disclosure) + 1 bullet (canonical Clause 1).report_compiler_agent.md: −1 sentence (line 172 disclosure), line 177 trimmed and merged with line 178 into the canonical Clause 1 bullet (net −1 sentence, +0 bullets).Total spec-side delta: 3 sentences removed, 2 bullets added, 1 bullet rewritten. The PATTERN PROTECTION block size grows by 0–1 lines per prompt; the "prompt bloat" risk (§9 R3) is bounded by this delta.
The sweep is mechanically enforced by extending the existing scripts/check_v3_6_7_pattern_protection.py lint script (added in Step 1+2 PR #48). Step 6 adds three checks; failure of any check blocks CI on .github/workflows/spec-consistency.yml.
The lint script reads each of the three downstream agent prompts and verifies:
| Check ID | Rule | Implementation | Failure message |
|---|---|---|---|
| INV-1 | The PATTERN PROTECTION (v3.6.7) block contains exactly one bullet whose text matches the canonical Clause 1 line verbatim | Regex match against the literal sentence (whitespace-normalized) | [INV-1] <file>: PATTERN PROTECTION block missing or duplicating the canonical Clause 1 line. Expected exactly one bullet matching: "DO NOT simulate any audit step. DO NOT claim to have run codex/external review. Output metadata must not claim audit-passed state." |
| INV-2 | The PATTERN PROTECTION (v3.6.7) block contains zero sentences matching any Clause 2 violation pattern | Python re patterns compiled with re.IGNORECASE, applied to the block text: (a) r"\bthe orchestrator\b.*\baudit\b", (b) r"\bcross-model audit (?:follows\|covers)\b.*codex_audit_multifile_template", (c) r"\baudit (?:afterwards?\|will be run\|is dispatched)\b", (d) r"\bdownstream audit\b\|\bthis output (?:is\|will be) audited\b". Backslash-pipe (\|) inside Markdown table cells reads as a literal pipe alternation in the underlying regex — the lint script unescapes Markdown table-cell escaping before compiling the patterns |
[INV-2] <file>:<line>: Clause 2 violation: <matched sentence>. Sentence must be removed per docs/design/2026-04-30-ars-v3.6.7-step-6-orchestrator-hooks-spec.md §6.2. |
| INV-3 | No file outside the three v3.6.7 downstream agent prompts gains a Clause 1 line as a side effect of the sweep (defends against well-intentioned over-application that would mark the L2 deferred question (§9 L2) as decided) | Manifest-driven file list in scripts/v3_6_7_inversion_manifest.json (contains exactly three filenames); lint scans only Markdown prompt files under deep-research/agents/ and academic-pipeline/agents/ (excluding docs/**, scripts/**, tests/**, and any other directory — the canonical Clause 1 line legitimately appears in this spec at §6.1, §6.2 and in test fixtures, and scanning those would self-fail) and reports any hit outside the manifest |
[INV-3] <file>: canonical Clause 1 line found outside the v3.6.7 inversion manifest. If this is intentional widening, update scripts/v3_6_7_inversion_manifest.json AND open the L2 question per §9. |
INV-1 + INV-2 are positive-and-negative checks that make the desired state machine-readable: every in-scope prompt must say the prohibition exactly once and must not say anything that names the downstream audit. INV-3 is the scope guard against accidental sweep widening — adding a fourth file to the manifest is a deliberate design choice that opens §9 L2, not a change a casual editor can make by copy-pasting the Clause 1 line into a new prompt.
Manifest format (scripts/v3_6_7_inversion_manifest.json):
{
"scope": "v3.6.7-only",
"rationale_doc": "docs/design/2026-04-30-ars-v3.6.7-step-6-orchestrator-hooks-spec.md#6-partial-inversion-rule-downstream-agent-prompt-edits",
"files": [
"deep-research/agents/synthesis_agent.md",
"deep-research/agents/research_architect_agent.md",
"deep-research/agents/report_compiler_agent.md"
]
}
Adding a fourth entry triggers an L2-resolution requirement: the PR description MUST cite the §9 L2 resolution rationale and update both this spec and the manifest in the same commit. CI does not enforce this PR-description rule; it is a code-review obligation backed by the §9 L2 forward reference. Future v3.6.8+ work that legitimately widens scope (e.g., when bibliography_agent patterns get their own protection layer) will land its own version-tagged manifest rather than retroactively widening v3.6.7's.
Inversion sweep is not the same as audit-mention scrubbing. Lint is narrow: it operates only on the PATTERN PROTECTION (v3.6.7) block. References to codex / audit / cross-model verification in OTHER sections of the same prompt (e.g., the agent's own Output Format section, generic audit-mode invocation instructions outside Step 6 scope) are out of scope. INV-2's regex set is anchored to PATTERN PROTECTION block boundaries via the lint script's section parser (already present from Step 1+2). This is what keeps INV-2 from false-positiving on legitimate audit-mode prompts that the agent files may carry for other purposes.
Step 8 of the v3.6.7 main spec §9 implementation table is "Add evaluation case (synthetic) demonstrating all 17 patterns triggered + protected." Per Q4=C this spec resolves Step 8 with a hybrid fixture set: per-pattern micro-fixtures that exercise each pattern in isolation, plus one chapter-level integration fixture that exercises the whole pipeline end-to-end. Per Q4 the hybrid is the only choice that simultaneously serves unit-level regression detection (granular failure attribution) and integration-level confidence (the runtime really does block when audit catches a finding).
The hybrid choice mirrors the existing 29-mutation lint suite shipped in Step 1+2: that suite ships unit-style mutations against the lint script alone; Step 8 fixtures ship paired BAD/GOOD pipeline traces against the full audit-gate runtime. Together they cover lint behavior + runtime behavior at proportionate granularity.
The v3.6.7 main spec (2026-04-29-ars-v3.6.7-downstream-agent-pattern-protection-spec.md) §3 + §4 inventories 17 numbered pattern IDs (5 narrative-side A1–A5, 5 instrument-side B1–B5, 3 publication-side C1–C3, 4 cross-cutting orchestration lessons D1–D4). Its summary prose now uses the same 17-pattern count, matching the numbered inventory.
Step 8 fixtures track the inventory, not the prose:
Total: 18 fixture artifacts. The fixture count is 18 because it includes 17 micro-fixtures plus 1 integration case; it is not evidence that a hidden 18th pattern exists.
The L-doc-1 prose-retirement follow-up is therefore closed: summary prose says 17 patterns, while the fixture count remains explicitly documented as 17 micro-fixtures plus 1 integration fixture.
Each micro-fixture is a self-contained directory under tests/fixtures/v3_6_7_pattern_eval/<pattern_id>/ carrying a paired BAD-versus-GOOD deliverable plus the synthetic upstream context required to exercise the pattern. The directory schema is fixed across all 17 micro-fixtures.
Micro-fixture directory contract (named "fixture directory contract" rather than "Schema 13.1" to avoid namespace collision with v3.6.6's evaluator_full Schema 13.1 — Step 6 fixtures are §3 / §3.7-anchored and have no relation to v3.6.6's Schema 13 / 13.1):
tests/fixtures/v3_6_7_pattern_eval/<pattern_id>/
├── manifest.json # Schema-validated (see below)
├── upstream_context/ # Synthetic Material Passport snippet + prior-stage outputs
│ ├── passport_snippet.yaml # Minimum passport state to make the agent dispatchable
│ └── prior_artifacts/ # Files referenced from passport (citations, prior chapter, etc.)
├── bad_run/ # The BAD case — agent emits a deliverable triggering the pattern
│ ├── deliverable.<ext> # Synthetic deliverable that exhibits the pattern
│ ├── expected_audit_findings.yaml # Findings codex SHOULD emit when auditing the BAD deliverable
│ └── expected_orchestrator_action.yaml # PASS / MINOR / MATERIAL + which §5.6 phase fires
└── good_run/ # The GOOD case — agent emits a clean deliverable
├── deliverable.<ext> # Pattern-protected deliverable
├── expected_audit_findings.yaml # Empty (PASS) findings list
└── expected_orchestrator_action.yaml # PASS + Path A or B as appropriate
manifest.json schema (one per fixture, validated by scripts/check_pattern_eval_manifest.py added in §10 Phase 6.8):
{
"$id": "pattern_eval_manifest.schema.json",
"type": "object",
"required": ["pattern_id", "agent", "pattern_scope", "stage", "fixture_kind", "upstream_context", "bad_run", "good_run"],
"properties": {
"pattern_id": {
"type": "string",
"enum": ["A1", "A2", "A3", "A4", "A5",
"B1", "B2", "B3", "B4", "B5",
"C1", "C2", "C3",
"D1", "D2", "D3", "D4"]
},
"agent": {
"type": "string",
"comment": "Must match audit_artifact_entry.schema.json §3.2 enum exactly so the harness can feed this value into the wrapper as --agent without §4.2 input validation rejection. cross_cutting D-rows still name a real gate agent here; the cross-cutting nature is recorded in pattern_scope below.",
"enum": ["synthesis_agent", "research_architect_agent", "report_compiler_agent"]
},
"pattern_scope": {
"type": "string",
"enum": ["agent_specific", "cross_cutting"],
"comment": "agent_specific for A1-C3 patterns. cross_cutting for D1-D4 lessons that are orchestration-level rather than per-agent; the agent field still names the gate agent under which the fixture exercises the cross-cutting concern."
},
"stage": {
"type": "integer",
"minimum": 1,
"maximum": 6
},
"fixture_kind": {
"const": "micro"
},
"bad_run": {
"type": "object",
"required": ["deliverable_path", "expected_audit_findings_path",
"expected_orchestrator_action_path"],
"properties": {
"deliverable_path": { "type": "string" },
"expected_audit_findings_path": { "type": "string" },
"expected_orchestrator_action_path": { "type": "string" }
}
},
"good_run": {
"type": "object",
"required": ["deliverable_path", "expected_audit_findings_path",
"expected_orchestrator_action_path"],
"properties": {
"deliverable_path": { "type": "string" },
"expected_audit_findings_path": { "type": "string" },
"expected_orchestrator_action_path": { "type": "string" }
}
},
"upstream_context": {
"type": "object",
"required": ["passport_snippet_path", "prior_artifacts_dir"],
"properties": {
"passport_snippet_path": { "type": "string" },
"prior_artifacts_dir": { "type": "string" }
}
}
}
}
expected_audit_findings.yaml shape (mirrors audit_verdict.schema.json from §3.5 exactly so the same validator can verify both the fixture's expectation and the runtime verdict file; field order, names, and dimension enum match §3.5 verbatim):
run_id: 2026-04-30T15-00-00Z-0a01
verdict_status: MATERIAL
round: 1
target_rounds: 3
finding_counts:
p1: 1
p2: 0
p3: 0
findings:
- id: F-001
severity: P1
dimension: "3.1"
file: bad_run/deliverable.md
line: 42
description: "Cross-section drift on legal-effect characterization of Source X"
suggested_fix: "Pre-list source effect inventory; add explicit cross-section consistency check before output"
generated_at: "2026-04-30T15:00:58.471Z"
generated_by: scripts/run_codex_audit.sh
generator_version: 1.0.0
For GOOD runs findings: [] and verdict_status: PASS.
expected_orchestrator_action.yaml shape:
expected_path: B # Path A or B per §5.6
expected_phase: B10 # Which §5.6 step the gate exits at, or which P-* phase from §5.6 inventory
expected_block_message: "<exact substring expected in the user-visible BLOCK message>"
expected_passport_mutation: "appended" # appended | none
expected_run_id_format: "matches §3.7 family F F1 regex"
This shape lets the test harness (§10 Phase 6.8) compare exact orchestrator behavior against fixture expectations without requiring the harness to run a real codex CLI — the harness reads expected_audit_findings.yaml as the synthesized verdict the codex run would have produced and feeds it into the orchestrator's gate procedure.
Fixture-to-pattern mapping (one row per micro-fixture):
| pattern_id | agent | pattern_scope | stage | What BAD triggers | What GOOD avoids |
|---|---|---|---|---|---|
| A1 | synthesis_agent | agent_specific | 2 | Same source's legal effect drifts across two narrative sections | Pre-listed effect inventory + cross-section consistency check |
| A2 | synthesis_agent | agent_specific | 2 | Pending-verification entry treated as fact | Hedge wrap: "pending verification of X" |
| A3 | synthesis_agent | agent_specific | 2 | Citation mis-anchored to a paper that does not contain the claim | One-line anchor justification per substantive claim |
| A4 | synthesis_agent | agent_specific | 2 | Quoted phrase scope creep (verbatim quote extends beyond the verified phrase boundary) | Verbatim quote bounded; surrounding context paraphrased |
| A5 | synthesis_agent | agent_specific | 2 | Sibling-document fabrication (declarative claim about a chapter not in ground truth) | Conditional language ("if document X argues Y, ...") |
| B1 | research_architect_agent | agent_specific | 2 | Confidentiality / anonymity / pseudonymization terms used interchangeably in consent script | Each term passed through irb_terminology_glossary.md |
| B2 | research_architect_agent | agent_specific | 2 | Pseudo-reverse-coded item (different construct flagged as "reverse-coded") | One-line construct-equivalence justification |
| B3 | research_architect_agent | agent_specific | 2 | Calendar-anchored retrospective item without common event date | Event-anchored phrasing default |
| B4 | research_architect_agent | agent_specific | 2 | Leading-frame item phrasing imports chapter argument vocabulary | Neutral / balanced item phrasing |
| B5 | research_architect_agent | agent_specific | 2 | List-of-options item subsets / over-sets the primary-source list | Primary-source list declared and enumerated fully |
| C1 | report_compiler_agent | agent_specific | 5 | Compression overclaim (abstract drops protected hedge identified upstream) | Protected hedging phrases preserved per protected_hedging_phrases.md |
| C2 | report_compiler_agent | agent_specific | 5 | Reflexivity disclosure with deictic temporal phrase ("during this period") | Explicit year range / past-tense disambiguating verb / "former" prefix |
| C3 | report_compiler_agent | agent_specific | 5 | Output metadata claims audit-passed state (sub-agent fake-audit hallucination) | Canonical Clause 1 line obeyed (no audit-passed metadata) |
| D1 | synthesis_agent | cross_cutting | 2 | Multi-file deliverable audited file-by-file (sequential) misses cross-file inconsistency | Multi-file parallel audit per audit template §4.1 |
| D2 | synthesis_agent | cross_cutting | 2 | Single-round audit declares CLEAN (convergence theatre) | 3+ round iteration; CLEAN only after round-N zero-finding |
| D3 | synthesis_agent | cross_cutting | 2 | PARTIAL closure marked as CLOSED | PARTIAL stays PARTIAL until verdict file shows zero-finding |
| D4 | report_compiler_agent | cross_cutting | 5 | Word count uses regex hyphenated-as-1, busts publisher hard cap | Whitespace-split + 3–5% buffer per word_count_conventions.md |
D-row fixtures use pattern_scope: cross_cutting to record that the pattern is an orchestration lesson rather than a per-agent rule, while agent still names a real v3.6.7 gate agent so the wrapper's --agent input validation (§4.2) accepts the fixture as-is. D1–D3 attach to synthesis_agent (where cross-file Phase 2 audit, multi-round convergence, and PARTIAL→CLOSED behaviour all surface in practice); D4 attaches to report_compiler_agent (where publisher word-count enforcement applies). The harness exercises the cross-cutting concern by simulating the relevant stage's deliverable shape against the named gate agent.
The integration fixture lives at tests/fixtures/v3_6_7_pattern_eval/integration/chapter_level_run/ and exercises the audit-gate pipeline end-to-end on a synthetic chapter that triggers a curated subset of patterns. Its purpose is NOT to exhaustively cover all 17 (the micro-fixtures already do that) but to verify the full audit-gate pipeline — Path A / Path B fall-through, multi-round escalation, ship_with_known_residue, ARS_PASSPORT_RESET interaction — using a realistic deliverable set.
Curated trigger subset (chosen to exercise three structural axes with one fixture):
| Axis | Pattern triggered | Why this pattern |
|---|---|---|
| 3-round MATERIAL convergence (§5.4) | A3 (mis-anchored citation, P1) | Forces round 1 → 2 → 3 escalation; round 3 still MATERIAL after partial fix triggers ship_with_known_residue user prompt |
| Path A vs Path B fall-through (§5.6) | C2 (reflexivity temporal ambiguity, P2) | After round 1 audit finds C2 then user re-runs, fixture exercises both proposal merge (Path B) and persisted re-verification (Path A on resume) |
| Cross-cutting + per-agent stack | D4 (word count) + C1 (compression overclaim) | Stage 5 abstract bundle exercises both publisher hard-cap arithmetic and protected-hedge preservation in one pass |
Fixture artifacts (integration fixture shape — extends the §7.2 micro-fixture directory contract with multi-round + escalation subdirectories):
tests/fixtures/v3_6_7_pattern_eval/integration/chapter_level_run/
├── manifest.json # fixture_kind: "integration"
├── upstream_context/ # Multi-stage synthetic chapter
│ ├── passport_snippet.yaml
│ ├── chapter_body_v1.md # Stage 4 hand-off body
│ ├── chapter_body_v2.md # Post-revision body
│ └── prior_artifacts/ # Citation list, prior-chapter outputs
├── round_1/ # First audit round — round=1, target_rounds=3
│ ├── synthesis_agent/ # Per-agent sub-fixture
│ │ ├── deliverable.md
│ │ ├── expected_audit_findings.yaml
│ │ └── expected_orchestrator_action.yaml
│ ├── research_architect_agent/
│ ├── report_compiler_agent/
│ └── expected_pipeline_state.yaml # What passport / output-dir / checkpoint state should look like after round 1
├── round_2/ # Post-revision audit — round=2, target_rounds=3
├── round_3/ # Round 3 still MATERIAL — escalation prompt
└── escalation/ # User picks ship_with_known_residue in round 3
├── user_response.yaml # Synthetic user-input response
├── expected_passport_state.yaml # acknowledgement entry appended
└── expected_pipeline_outcome.yaml # Stage proceeds, final passport state
Integration manifest is a separate JSON Schema keyed by fixture_kind: "integration". It does NOT extend the §7.2 micro-fixture schema — the two shapes share a fixture_kind discriminator and a pattern_id enum but otherwise have disjoint required fields. The check_pattern_eval_manifest.py lint script (§10 Phase 6.8) branches on fixture_kind and applies the matching schema; this is what keeps the integration manifest clean of micro-fixture-specific fields like pattern_scope, agent, stage, and the BAD/GOOD pair (which the integration fixture decomposes into per-agent / per-round sub-fixtures inside the directory tree above):
{
"$id": "pattern_eval_integration_manifest.schema.json",
"type": "object",
"required": ["fixture_kind", "patterns_triggered", "rounds", "escalation", "rationale_doc"],
"properties": {
"fixture_kind": { "const": "integration" },
"patterns_triggered": {
"type": "array",
"minItems": 1,
"items": {
"type": "string",
"comment": "Same closed enum as the §7.2 micro-fixture pattern_id field — the 17 numbered pattern IDs.",
"enum": ["A1", "A2", "A3", "A4", "A5",
"B1", "B2", "B3", "B4", "B5",
"C1", "C2", "C3",
"D1", "D2", "D3", "D4"]
}
},
"rounds": {
"type": "array",
"minItems": 1,
"items": {
"type": "object",
"required": ["round", "target_rounds", "expected_verdict"],
"properties": {
"round": { "type": "integer", "minimum": 1 },
"target_rounds": { "type": "integer", "minimum": 1 },
"expected_verdict": { "type": "string", "enum": ["PASS", "MINOR", "MATERIAL", "AUDIT_FAILED"] }
}
}
},
"escalation": {
"type": "object",
"required": ["user_choice"],
"properties": {
"user_choice": { "type": "string", "enum": ["ship_with_known_residue", "another_round", "abort_stage"] },
"expected_acknowledgement_finding_ids": { "type": "array", "items": { "type": "string" } }
}
},
"rationale_doc": { "type": "string" }
}
}
Concrete fixture manifest:
{
"fixture_kind": "integration",
"patterns_triggered": ["A3", "C2", "D4", "C1"],
"rounds": [
{ "round": 1, "target_rounds": 3, "expected_verdict": "MATERIAL" },
{ "round": 2, "target_rounds": 3, "expected_verdict": "MATERIAL" },
{ "round": 3, "target_rounds": 3, "expected_verdict": "MATERIAL" }
],
"escalation": {
"user_choice": "ship_with_known_residue",
"expected_acknowledgement_finding_ids": ["F-101", "F-103"]
},
"rationale_doc": "docs/design/2026-04-30-ars-v3.6.7-step-6-orchestrator-hooks-spec.md#73-chapter-level-integration-fixture"
}
The harness runs the integration fixture by:
expected_audit_findings.yaml as synthesized verdicts.expected_pipeline_state.yaml matches the actual orchestrator state.escalation/user_response.yaml as the user's synthetic answer.expected_passport_state.yaml matches actual passport state after escalation.This exercises representative Path A / Path B / Path B-supersession happy-path and escalation code paths in §5.6 within one fixture, without requiring real codex execution. Coverage of failure-side phases (schema rejection, snapshot races, duplicate-recovery branches enumerated in §5.6's verification failure state inventory) is out of scope for the integration fixture; those phases are unit-tested by the lint script's negative cases (Phase 6.3) and by the §10 Phase 6.8 harness's per-phase synthetic injections, not by the integration fixture's chapter-level run.
A pattern is considered "demonstrated triggered + protected" by the Step 8 fixture set when ALL of the following hold:
<pattern_id>/bad_run/ produces an orchestrator decision matching expected_orchestrator_action.yaml. For 15 of 17 micro-fixtures the failure signal is a MATERIAL verdict with a finding in the expected dimension; for C2 the failure signal is a MINOR verdict (one P3 reflexivity-temporal-ambiguity finding — sub-MATERIAL severity by design since C2's deictic temporal phrase is a hedge issue, not a hallucination); for D2 the failure signal is a non-finding convergence-policy assertion ("round 1 PASS but D2 convergence-theatre trigger logged in expected_orchestrator_action.yaml"), since D2 is about whether the round cap mechanism actually iterates rather than emitting a P1 finding. The §7.4 success criterion is generic across all three shapes — the harness asserts the BAD case produces the expected pattern-specific signal (MATERIAL with P1 finding / MINOR with P3 finding / PASS with convergence assertion), not strictly an audit finding.<pattern_id>/good_run/ produces a PASS orchestrator decision and an empty findings: [] list.expected_orchestrator_action.yaml.expected_phase (e.g., B10 for normal MATERIAL, P-PB-gate for eleven-gate failure, B11 for round-3 MATERIAL escalation per §5.4).Audit artifact lifecycle observed. §5.6 persists PASS / MINOR / MATERIAL proposals at B8d/B9 BEFORE applying ship/block at B10, so the lifecycle test splits by failure phase plus duplicate-recovery state, not by ship/block outcome. The harness asserts the following inventory-driven rule:
Fresh non-duplicate proposals that pass B6 / B7 / B8a / B8c / B8d and whose B9 passport write succeeds reach B10 with exactly one new persisted audit_artifact[] entry appended (with verified_at populated by _next_verified_at_ms) before ship/block is applied. This covers all B10-bound paths whether the verdict is PASS, MINOR, or MATERIAL — BAD-case MATERIAL fixtures append an entry then BLOCK at B10 ship/block.
Passport mutation column reads none verifies no new persisted entry is appended at that phase. This covers selection/precondition failures (P-PA-precond, P-PA-supersede-preempt — though the latter continues to Path B), schema/gate/verdict-mirror/late-freshness failures (B4 / P-PB-proposal-schema, B5 / P-PB-audit-failed, B6 / P-PB-gate, B7 / P-PB-verdict-schema, P-PB-verdict-mirror, B8a / P-PB-stale-late, B8c / P-PB-snapshot, B8d / P-PB-persisted-schema), passport-write failures (B9 / P-PB-passport-write), and Path B-only terminal failures (P-PB-empty, P-PB-supersede-missing, P-PB-ambig). Continuation rows (P-PB-dup-other, P-PB-supersede-preempt) only append later if a subsequent selected candidate successfully merges; the harness asserts on the eventual final state, not on the intermediate continuation.expected_orchestrator_action.yaml.expected_phase, looks up the corresponding row in §5.6's inventory, and asserts the passport delta matches the row's Passport mutation column. This way the harness's assertions and §5.6's inventory stay synchronised by construction; adding a new failure phase to §5.6 in v3.6.8+ automatically extends the harness's coverage without code change.Failure of ANY criterion above on ANY of the 17 micro-fixtures or the 1 integration fixture blocks CI on .github/workflows/spec-consistency.yml. CI green requires 18-of-18 fixtures green.
Fixtures are append-only at the v3.6.7 minor version. New patterns documented after v3.6.7 ships (e.g., v3.6.8+ when bibliography_agent patterns get their own protection layer per §6.1's "v3.6.7-specific rule" framing) get fixtures under their own version directory: tests/fixtures/v3_6_8_pattern_eval/.... Existing fixtures are not edited unless the pattern's protection rule itself changes; in that case the change ships under a new spec date and the fixture's manifest carries a superseded_by field pointing to the new fixture.
This mirrors the spec-versioning discipline already in place for docs/design/. It also keeps the §7.4 success criterion stable: "18-of-18 green" refers to v3.6.7's set; v3.6.8+ adds new fixtures with their own count.
Fixture-to-pattern coverage gaps are surfaced explicitly. The manifest.json pattern_id enum is closed: a fixture for an unenumerated pattern fails schema validation. Adding a new pattern to the enum requires updating the upstream main spec §3 / §4 inventory in the same PR, which keeps inventory and fixture set synchronized. The §10 Phase 6.8 lint script enforces this by cross-checking the fixture directory against the inventory's enumerated patterns and erroring on any pattern_id missing a fixture or any fixture missing a pattern_id.
Step 8 fixtures are wired into .github/workflows/spec-consistency.yml as two separate test jobs to make unit-vs-integration failure attribution legible:
| Job | Fixtures | Runtime | Failure surface |
|---|---|---|---|
pattern-eval-unit |
17 micro-fixtures | ~10s (no codex; harness uses synthesized verdicts) | A specific pattern's BAD or GOOD case fails an assertion — narrow blame |
pattern-eval-integration |
1 chapter-level integration fixture | ~30s (multi-round simulation) | Cross-pattern interaction failure or escalation-path failure — broader blame |
Both jobs run on every PR touching any of the following paths (path-filter completeness is load-bearing — a path that produces or consumes Step-6 artifacts but is omitted from the filter would let breaking changes ship without these jobs running):
deep-research/agents/synthesis_agent.mddeep-research/agents/research_architect_agent.mddeep-research/agents/report_compiler_agent.mdacademic-pipeline/agents/pipeline_orchestrator_agent.mdscripts/run_codex_audit.shscripts/check_audit_artifact_consistency.pyscripts/test_check_audit_artifact_consistency.pyscripts/check_v3_6_7_pattern_protection.pyscripts/test_check_v3_6_7_pattern_protection.pyscripts/check_pattern_eval_manifest.pyscripts/test_pattern_eval_runtime.pyscripts/_next_verified_at_ms.pyscripts/test__next_verified_at_ms.pyscripts/parse_audit_verdict.pyscripts/v3_6_7_inversion_manifest.jsonshared/templates/codex_audit_multifile_template.mdshared/contracts/audit/**shared/contracts/passport/audit_artifact_entry.schema.jsonshared/handoff_schemas.mdacademic-pipeline/references/passport_as_reset_boundary.mddocs/PERFORMANCE.mdtests/fixtures/v3_6_7_pattern_eval/**.github/workflows/spec-consistency.ymlThis is the same path-filter discipline .github/workflows/pytest.yml already uses for v3.6.4 corpus adapters; the list is enumerated exhaustively rather than via globs because .github/workflows/spec-consistency.yml itself is in the filter (a typo in the workflow file should re-trigger CI, otherwise the typo could ship).
The 17+1 split mirrors v3.6.7 Step 1+2's 29-mutation lint suite split between unit-style mutation tests (lint behavior) and integration-style fixture tests (the lint actually catches the right thing). Unit-fast / integration-broader is the same separation; unit-and-integration are the same kind of completeness target.
Phase 6.8 CI deployment note (2026-05-05). The "two separate test jobs" framing above is the design-level naming convention. The actual Phase 6.8 ship places pattern-eval-unit and pattern-eval-integration as two consecutive named steps within the existing spec-consistency job, not as two separate top-level jobs. This deviates from the literal "two separate jobs" wording but preserves the spec's intent — failure attribution remains step-name-legible (Run v3.6.7 Step 8 pattern-eval-unit tests vs ... pattern-eval-integration tests), and the attribution surface in CI logs is identical to a job-split. The existing spec-consistency.yml runs unconditionally on push / PR (no paths: filter) — consistent with the 22 sibling lint steps already in the workflow — so the §7.6 path-filter list above is treated as descriptive enumeration of artifacts whose changes SHOULD trigger this job, satisfied by the no-filter (everything triggers it) deployment. A future v3.6.8+ workflow refactor that splits spec-consistency.yml into per-feature workflows with paths: filters would then promote pattern-eval-unit and pattern-eval-integration to top-level jobs at that boundary; until then, named-step deployment is the v3.6.7 ship contract.
v3.6.6's generator-evaluator contract (Schema 13.1 evaluator_full mode, currently in design on branch spec/v3.6.6-generator-evaluator-contract — once merged, the spec lands under docs/design/ per repo convention) and Step 6's audit-artifact gate are orthogonal axes of pipeline quality. Both can coexist on the same deliverable; neither subsumes the other.
| Axis | v3.6.6 generator-evaluator | v3.6.7 Step 6 audit gate |
|---|---|---|
| Where the discipline lives | Inside the writer/evaluator agent pair, same Claude session | Outside the LLM session entirely (codex CLI run by deployment) |
| What it verifies | Writer's draft satisfies an evaluator-derived rubric for in-pipeline self-criticism | Deliverable survives independent cross-model review at xhigh effort |
| When it fires | Per writer iteration, in-line during the writer agent's pass | Once per stage transition, at the boundary between agent and downstream consumer |
| Trust model | Same-session — writer and evaluator share LLM weights and may share blind spots | Cross-model — codex (gpt-5.5) is a different model family from the writer's Claude session |
| Block behavior | Schema 13.1 contract; writer iterates until evaluator passes | Strict Block (§2.2 Approach 1); MATERIAL refuses transition unless §5.4 ack |
| Artifact | evaluator_full envelope inside writer output |
Schema 9 audit_artifact[] entry pointing at four standalone artifact files |
| Threat surface | Genuine self-criticism failure (writer cannot see its own blind spot) | Pattern C3 fake-audit hallucination + cross-model drift detection |
The two layers run sequentially on the same deliverable: a v3.6.7 downstream agent first satisfies its v3.6.6 generator-evaluator contract (in-session), then deployment runs codex audit out-of-session (per Q2=D the orchestrator does not execute codex; it verifies the resulting audit_artifact[] proposal at the next stage transition per §5.6). A passing v3.6.6 contract does NOT bypass the audit gate; a failing v3.6.6 contract never reaches the audit gate because the writer iteration loop blocks first. Either layer alone is insufficient: v3.6.6 catches what same-session reasoning can catch, audit gate catches what only a different-model perspective can catch (per feedback_codex_xhigh_for_drift_audit.md empirical lesson "codex翻案 3 段 Claude 判 'lean' 中 2 段").
Explicit non-goals for Step 6 (clarifying what audit gate does NOT do):
evaluator_full continues to fire per writer iteration regardless of audit-gate presence.bibliography_agent, socratic_mentor_agent, field_analyst_agent, compliance_agent, collaboration_depth_agent, integrity_verification_agent, editorial_synthesizer_agent, literature_strategist_agent, citation_compliance_agent, and every other ARS agent operate under pre-v3.6.7 trust rules. Per Q1=A, scope expansion is a v3.6.8+ concern.evaluator_full's rubric is not part of v3.6.7 scope; if it lands, it will be a deliberate spec change with its own design round.The orthogonality is what justifies running both at once. Merging the two layers into a single contract was rejected at Q4 brainstorm as a category error: in-pipeline self-criticism and cross-model independent review are different epistemic primitives. A single combined "evaluator+audit" contract would force one of two failure modes: (a) the in-pipeline evaluator pretends to be a cross-model auditor (Pattern C3 fake-audit hallucination, the very failure mode this spec defends against), or (b) the cross-model audit is reduced to whatever the evaluator can simulate in-session (eliminating the cross-model trust premise). Keeping the layers orthogonal preserves both primitives without compromise.
The §1–§8 design has accumulated four risks and five open-question entries (L1–L4 deferred from §2.3 plus one docs-only follow-up surfaced in §7.1) that did not have enough surface area to settle within their introducing sections. They are collected here so a single review pass can read all of them together; each carries the load-bearing forward references from earlier sections so the trail is not lost.
R1 — Wrapper script deployment friction. Strict Block (§2.2 Approach 1) means a user without scripts/run_codex_audit.sh installed and a working codex CLI cannot ship a deliverable through any of the three v3.6.7 stage transitions. This is intentional — the audit-as-contract design (Q2=D) requires the audit to actually run somewhere — but it changes the failure mode from "audit is recommended" to "audit is required for ship." Forward reference: §2.3 L4. The risk surface is concrete:
OPENAI_API_KEY (or equivalent xhigh-effort credential) hits the same BLOCK with a different upstream cause (wrapper exits AUDIT_FAILED).Mitigation:
docs/PERFORMANCE.md (added v3.4) gains a Step 6 onboarding subsection covering codex CLI installation, credential setup, and the wrapper's required environment variables (added alongside the wrapper itself in §10 Phase 6.1 — the docs and the script ship together so the BLOCK message's documentation pointer never lands on a missing page).--help output (per §4.2) names every required environment variable and exits with EX_USAGE if any are missing — failure is loud and tells the user what to fix. This is preferred to a soft warning that lets the gate silently degrade.R1 does NOT motivate a "skip audit" command (§5.7 hard rule). The friction is the cost of the artifact-as-contract design; partial mitigation via better diagnostics is acceptable, but escape hatches would re-open Pattern C3 attack surface.
R2 — Motivated forgery boundary. The §1.2 threat model declares Layer 2 + Layer 3 sufficient against inadvertent Pattern C3 over-claim but explicitly NOT against motivated forgery. This is a load-bearing scope decision; misunderstanding it in implementation or future spec work would invite scope creep into cryptographic signing infrastructure that the v3.6.7 timeline does not budget for. Forward reference: §1.2 threat-model statement, §3.7 family E row E9 (deployment-side discipline rule).
The risk: a future maintainer reads the eleven gating checks (§5.2) and assumes they defend against any audit-artifact tampering, then adds new features (e.g., "auto-resume on audit pass") that depend on stronger trust than Layer 2 + Layer 3 actually provide. Such a feature would be safe under the inadvertent-Pattern-C3 threat model but unsafe under the motivated-forgery model that real-world deployment may face.
Concrete gap in 0.125+ stream evidence (load-bearing for §3.3). Codex 0.125+ --json carries no model field and no reasoning_effort field anywhere in the event stream — both are invocation-side properties of the CLI flags (-m gpt-5.5, -c 'model_reasoning_effort="xhigh"') rather than stream-side properties codex echoes back. Layer 2 therefore cannot verify which model actually ran; Layer 3 only records the wrapper-declared invocation in sidecar.prompt. A motivated actor with shell access could invoke codex exec -m gpt-5.6 -c 'model_reasoning_effort="low"' ... and write a sidecar declaring model: gpt-5.5 / reasoning_effort: xhigh; the eleven gates would not detect that mismatch — every JSONL field they cross-check (thread_id, usage integers, the agent_message event's text) is unaffected by which model produced them. This gap stays inside the rejected motivated-forgery scope: it is exactly the case §3.7 family E row E9 (wrapper not LLM-callable from same session) and the §4.7 invocation-site discipline rule are designed to bound, and exactly the case Layer 5 (cryptographic signing on the codex CLI side) would close. The §1.2 closing paragraph already declares "Layer 5 was considered and rejected as over-engineering for the inadvertent-Pattern-C3 threat surface"; this paragraph names the specific gap that decision tolerates.
Mitigation:
docs/PERFORMANCE.md Step 6 onboarding MUST name §1.2's threat boundary so deployment teams know the audit gate is one defense layer among several, not the only defense.R2 is the rejected scope for v3.6.7. Layer 5 (cryptographic non-repudiation) is the natural escalation target if v3.6.8+ deployment scenarios require defending against motivated forgery. v3.6.7 ships with the inadvertent-threat-model boundary explicit so future scope expansion is a deliberate, traceable change.
R3 — Prompt bloat across downstream agents. §6 adds approximately one bullet to each of the three v3.6.7 downstream agent prompts (synthesis_agent.md, research_architect_agent.md, report_compiler_agent.md). §5.6 adds approximately 50 lines to the orchestrator prompt (pipeline_orchestrator_agent.md) as a decision-policy summary; the full Path A → Path B procedure (~347 lines) stays in §5.6 as the implementation contract and is NOT inserted into the prompt verbatim — see §5.6 closing prose for the prompt-vs-spec split rationale and docs/design/2026-05-05-phase-6.6-scoping-note.md §3 Path 2 for how the split was settled after the PR #50 → #53 cascade-expansion was discovered. The cumulative additions are bounded but not free — every line in an agent prompt is context the LLM processes per dispatch, and v3.6.7's PATTERN PROTECTION blocks already grew the three downstream prompts by 6–12 lines each at Step 1+2 ship. Forward reference: v3.6.7 main spec §8.1 risk "prompt bloat across downstream agents."
The risk concretely:
Mitigation:
scripts/check_v3_6_7_pattern_protection.py for §6, the audit-artifact-consistency script for §5.6 references). A future "trim prompt" pass can verify the trimming did not break any check.R3 is monitored, not blocked. v3.6.7 main spec §8.1 already accepted this risk class at Step 1+2 ship; Step 6 inherits the same risk envelope.
R4 — ARS_PASSPORT_RESET interaction with audit artifacts. v3.6.3 introduced ARS_PASSPORT_RESET=1 and resume_from_passport=<hash> (canonical contract: academic-pipeline/references/passport_as_reset_boundary.md). Step 6's audit-artifact gate must remain compatible with reset-then-resume sessions. Forward reference: §2.3 L3, §5.5 last row of co-occurrence table.
The risk: a user runs an audit in session A, the session resets at a FULL checkpoint, the user resumes in session B, and the audit artifact's trust on resume is ambiguous. The settled answer (§5.5 row) is that orchestrator MUST re-run the eleven gating verification checks on resume — passport entries are not trusted "because they were verified once." But the design surface has implications:
another_round in session A and the wrapper completed under CI before session B resumes, the round-N+1 proposal in <output-dir> MUST preempt the prior round-N persisted entry. §5.5's row already requires §5.6 A1.5 superseding-proposal preflight on resume; this closes the F-070 generalization across resume.<output-dir> from a prior session (wrapper exited non-zero, did not produce a clean verdict) is consumed at §4.9 step 9 by NOT being moved to consumed/. On resume, A1.5 may find a higher-round AUDIT_FAILED proposal that preempts a prior MATERIAL persisted entry; per §5.6 D4 closure, this BLOCKs with the failure reason rather than silently falling back. Verified at §5.6 phase P-PA-supersede-preempt + P-PB-audit-failed.Mitigation:
passport_as_reset_boundary.md reference doc gains a paragraph in §10 Phase 6.5 (Schema 9 amendment) cross-linking audit-artifact[] to the reset-boundary contract so future maintainers see both rules together.R4 is a confirmed-and-handled compatibility surface. The risk is documentation drift — if a future v3.6.8+ change to either reset-boundary or audit-artifact contract breaks the cross-link, the resume path becomes ambiguous. Mitigation 3 is the discoverability defense.
These are deferred from §2.3 L1–L4 plus one new docs follow-up surfaced in §7.
L1 — Round upper bound: 3 vs 5. Resolved in §5.4 as "default 3 rounds, escalation prompt allows the user to bump cap by 1 per another_round choice." Forward reference: §2.3 L1 → §5.4. Status: closed in §5.4.
The 3-round default is empirically grounded in feedback_cross_model_review_cascade_inconsistency.md and feedback_codex_review_vs_resume_audit_scope.md observation that round-3 is the typical convergence point. The 5-round alternative would have aligned with feedback_codex_iterative_spec_review_to_zero.md's "iterate to one zero-finding round" guidance but at the cost of delaying user escalation in cases where rounds 4–5 do not improve verdicts. The §5.4 escalation prompt makes the cap user-overridable, which preserves both empirical default and explicit-extension flexibility.
L2 — Partial inversion sweep completeness: v3.6.7-only vs all ARS agent prompts. Resolved in §6.1 as "v3.6.7-only." Forward reference: §2.3 L2 → §6.1, §6.3 INV-3 manifest. Status: closed in §6.1; widening is a v3.6.8+ design decision.
The narrow sweep is justified by the per-agent attack-surface argument (§6.1's "Why this is a v3.6.7-specific rule"): only the three v3.6.7 downstream agents are dispatch-points for the audit gate; other agents (e.g., bibliography_agent) do not produce deliverables that the orchestrator audits, so their codex/cross-model references in prompts are not Pattern C3 attack surfaces. Widening the sweep would add lint-maintenance cost without commensurate threat-surface reduction. The §6.3 INV-3 manifest pattern is what makes future widening a deliberate change.
L3 — ARS_PASSPORT_RESET re-verification model. Resolved in §5.5 as "re-run eleven gating verification checks on resume; do not trust prior verification by passport-presence alone." Forward reference: §2.3 L3 → §5.5, R4 above. Status: closed in §5.5.
The re-verification rule means session B does not inherit session A's audit trust by default. This is intentionally conservative — bundle files may have mutated between sessions, the user may have run a higher-round wrapper between sessions, the session A may have been a different model version. Re-running L3-3 / L3-4 against current on-disk content + L3-2 / L3-7 against the persisted sidecar provides resume-time fresh verification at low cost (the artifact files do not change between sessions; only the recompute is required).
L4 — Wrapper script deployment friction discoverability. Forward reference: §2.3 L4. Status: resolved by R1 above (mitigation 1–3). The R1 mitigation list is the forward-reference target for L4; both are the same design surface viewed from different entry points (L4 = "how visible is the gate?", R1 = "what is the cost of the gate's visibility?"). Treating them as one resolved item rather than two pending items keeps the cross-reference graph compact.
L-doc-1 — "18 patterns" prose retirement (new in §7.1). Closed. Two specs previously carried an off-by-one summary count in eight prose locations:
- v3.6.7 main spec (2026-04-29-ars-v3.6.7-downstream-agent-pattern-protection-spec.md) §1, §2.1, §9 Step 8, and §10.
- This spec at four locations: Scope, Step 8 bullet, §1.2, and §2.1 Q4 row.
Both specs' inventories enumerate 17 numbered IDs (5 A + 5 B + 3 C + 4 D). Step 6 / Step 8 fixtures track the 17-ID inventory per §7.1. Forward reference: §7.1 → here. Status: closed; summary prose has been aligned to the 17-ID inventory.
The retirement updated those eight prose locations to "17 distinct patterns" / "17 micro + 1 integration" without changing any of the 17 numbered IDs. The change is mechanical and not a v3.6.7 spec amendment because the inventory itself was already correct; only the summary count drifted.
If a future review surfaces evidence that the upstream "18" figure was correct and an 18th pattern was dropped from the inventory between drafts, the right response is to amend the main spec inventory (re-add the 18th pattern) rather than the fixture set (which already aligns with the inventory). The fixture set is downstream of the inventory; mismatching them in either direction creates the §7.5 coverage-gap surface that the pattern_id enum closure already defends.
The work in §3–§7 lands in eight phases. Each phase is one logical commit on the implementation branch; phases run sequentially because later phases depend on earlier phase outputs (Phase 6.2 schemas are imported by Phase 6.3 lint script; Phase 6.6 orchestrator prompt cites §3.7 invariant IDs that Phase 6.3 lint enforces; etc.). The phases align with the artifact list in project_ars_v3_6_7_pattern_protection_wip.md memory's implementation roadmap.
Each phase has a defined deliverable, a defined verification gate, and an explicit dependency on prior phases. Phases that can be parallelized are marked. The whole implementation is targeted as a single follow-up PR to land Step 6 + Step 8 together, mirroring the v3.6.7 main spec §9 plan that pairs them under one ship.
scripts/run_codex_audit.shDeliverables: scripts/run_codex_audit.sh AND scripts/parse_audit_verdict.py (the JSONL→verdict parser the wrapper invokes per §4.4 Step 4 to extract the last item.completed.item.text agent_message event's text from the codex 0.125+ event stream — intermediate agent_message events between tool-call item.started / item.completed pairs are skipped — and convert it into the <run_id>.verdict.yaml file per §3.5). Both implement §4 contract end-to-end.
Includes:
- Bash 4+ shebang and dependency check (set -euo pipefail).
- §4.2 input parsing exactly matching the contract: --stage, --agent, --deliverable, --supporting, --round, --target-rounds, --bundle-id, --output-dir, --previous-findings, optional --dry-run. EX_USAGE on missing required flags or on the §4.2 validation rules (--round <= --target-rounds, --previous-findings required when --round > 1, etc.). No --bundle or --audit-template flags — those names appeared in earlier drafts and are not part of §4.2.
- §4.3 output construction: <run_id> generation per F1 regex, four contract files written to <output-dir>/<run_id>.{jsonl,meta.json,verdict.yaml,audit_artifact_entry.json} plus the three non-contract diagnostic files (<run_id>.stdout, <run_id>.stderr, <run_id>.manifest.txt).
- §4.4 atomicity: tmp-file-rename for sidecar / verdict / proposal entry; JSONL streamed by codex CLI directly; proposal entry file is the LAST file written (E8).
- §4.4 helper functions: _atomic_write, _sha256, _extract_jsonl_thread_id, _codex_version (semver extractor — strips the codex-cli prefix from codex --version output so the sidecar's codex_cli_version field stays bare-semver per §3.4), _compute_bundle_manifest_sha per §3.6 contract, _now_rfc3339_ms, _random_4hex from /dev/urandom.
- §4.4 SIGTERM trap that aborts with EX_TEMPFAIL and cleans up partial artifact files (F-059 closure).
- §4.4 TOCTOU detection: snapshot SHA-256 of every primary / supporting / template file at prompt-render time; recompute after codex returns; if any bundle file's SHA changed, emit AUDIT_FAILED with failure_reason: "bundle file(s) mutated during audit run: <space-separated path list>" and let the wrapper exit code rise to 70 per §4.6 (F-060 closure). The TOCTOU path produces the AUDIT_FAILED artifact set per §4.6 — it does NOT exit with EX_USAGE, because mid-run mutation is an audit-failure signal (orchestrator must see the proposal to BLOCK at §5.6 Path B5), not an input-validation error.
- §4.6 failure-mode handling: emit AUDIT_FAILED proposal + sidecar + verdict file for all enumerated non-zero codex exit codes (70 / 75 / other) per §4.6 table; still write the proposal entry last so orchestrator's lifecycle invariant E8 holds.
- §4.5 multi-file Phase 2 audit support via --supporting plus shared --bundle-id for cross-file bundle grouping; manifest hash computed by _compute_bundle_manifest_sha per §3.6.
- §4.7 enforcement: explicit comment block at the top of the script naming this rule (wrapper not LLM-callable from same session producing deliverable); deployment-side discipline reminder.
- docs/PERFORMANCE.md Step 6 onboarding subsection (per R1 mitigation 2): codex CLI installation, credential setup (OPENAI_API_KEY-equivalent), required environment variables, the §1.2 threat-model boundary, and the wrapper's exit-code contract from §4.6.
Verification gate: wrapper passes a self-test invocation (scripts/run_codex_audit.sh --dry-run --stage 2 --agent synthesis_agent --deliverable <test-file> --round 1) that validates inputs and exits 0 without writing any contract artifacts — --dry-run is an input-validation probe, not a synthetic audit, and writing a proposal entry there would create a fake audit artifact at zero codex cost (the exact Pattern C3 attack surface §1.2's threat model defends against). A separate non-dry-run synthetic smoke test (codex CLI mocked or invoked against a tiny fixture deliverable) produces a well-formed proposal entry that validates against the Phase 6.2 schemas in --mode proposal; the AUDIT_FAILED path is exercised by injecting a SHA mutation between Step 0b snapshot and Step 3a recompute. Full integration verification against the §7 fixture set defers to Phase 6.8.
Dependencies: none (Phase 6.1 is the foundation).
Deliverables:
- shared/contracts/passport/audit_artifact_entry.schema.json — per §3.2.
- shared/contracts/audit/audit_jsonl.schema.json — per §3.3 (new directory).
- shared/contracts/audit/audit_sidecar.schema.json — per §3.4 (new directory).
- shared/contracts/audit/audit_verdict.schema.json — per §3.5 (new directory).
Includes:
- JSON Schema draft 2020-12 for each file.
- oneOf lifecycle states (proposal vs persisted) on audit_artifact_entry.schema.json per §3.2 lifecycle-conditional table.
- $ref from Schema 9 amendment (Phase 6.5) to audit_artifact_entry.schema.json.
- shared/contracts/README.md gains one section documenting the new audit/ directory's purpose, mirroring the v3.6.6 convention.
Verification gate: all four schemas parse as valid JSON Schema 2020-12; example payloads in §3.1 / §3.3 / §3.4 / §3.5 validate against their respective schemas (positive examples) and a deliberately-malformed counter-example fails (negative case).
Dependencies: none structurally; can run in parallel with Phase 6.1, but lint script (Phase 6.3) imports these schemas so Phase 6.3 must come after Phase 6.2.
scripts/check_audit_artifact_consistency.pyDeliverable: scripts/check_audit_artifact_consistency.py consuming §3.7 invariants table.
Includes:
- Per-row implementation of §3.7 families A (A1–A7; note A7 is a stream-shape rule against the on-disk JSONL, distinct from A1–A6 which are per-artifact field rules), B (B1–B10), C (C1–C4), D (D1–D4), E (E1–E9 where lint-applicable; E1/E2/E6/E9 are discipline rules with limited lint surface — implement detectable cases like "non-orchestrator writer modified passport" via post-hoc passport-diff analysis), F (F1–F4).
- Three CLI modes:
- --mode proposal (validates a proposal-state entry).
- --mode persisted (validates a persisted-state entry).
- --mode jsonl-stream (validates a JSONL stream against the §3.7 family A row A7 stream-shape invariant). Invoked by orchestrator §5.2 L2-5 as scripts/check_audit_artifact_consistency.py --mode jsonl-stream --jsonl <path>. Exits 0 on stream-shape pass, non-zero on pairing violation with stderr message naming the offending item.id and reason.
- --passport-path <path>, --output-dir <path>, and --jsonl <path> for cross-file / stream rule checks.
- --example-validation-harness mode that walks docs/design/*.md finding §3 example payloads and validating them — closes F4 (spec text drift).
- Test suite at scripts/test_check_audit_artifact_consistency.py with one test per invariant row (positive + negative cases).
Verification gate: lint passes on the Phase 6.2 schemas, on this spec's bundled §3.1 / §3.3 / §3.4 / §3.5 example payloads (via --example-validation-harness), and on the Phase 6.3 positive/negative unit fixtures shipped alongside the lint script. Test suite achieves 100% coverage over §3.7 family rows. Full wrapper-output and orchestrator-prompt validation against the 17 micro-fixtures is Phase 6.8's responsibility, not Phase 6.3's — Phase 6.3's gate must not depend on artifacts produced by Phase 6.6 / Phase 6.8 (which themselves depend on Phase 6.3's lint), or the phase-gate ordering becomes circular.
Dependencies: Phase 6.2 (imports schemas). Phase 6.3 ships before Phase 6.6 / Phase 6.7 / Phase 6.8 so those phases can use the lint as a verification surface.
scripts/_next_verified_at_ms.pyDeliverable: scripts/_next_verified_at_ms.py implementing the §5.4 strict-monotonic helper.
Includes:
- Pure function next_verified_at_ms(passport_audit_artifacts: list) -> str per §5.4 pseudocode.
- CLI mode for orchestrator-side invocation: python scripts/_next_verified_at_ms.py --passport <path> returns the next valid verified_at value as RFC 3339 UTC ms.
- Empty-ledger base case + clock-not-yet-advanced post-condition both covered.
- Unit tests at scripts/test__next_verified_at_ms.py with cases: (a) empty ledger, (b) clock fresh, (c) clock stale (helper bumps by 1 ms), (d) multiple prior entries (max selected, not last).
Verification gate: _next_verified_at_ms() always returns a string strictly greater than every prior verified_at; verified by property-based test.
Dependencies: Phase 6.2 (audit_artifact_entry.schema.json defines the verdict.verified_at shape this helper produces).
shared/handoff_schemas.mdDeliverable: shared/handoff_schemas.md Schema 9 gains the optional audit_artifact[] field per §3.1.
Includes:
- New section under Schema 9 documenting audit_artifact[] shape, semantics, and append-only ledger discipline.
- $ref to audit_artifact_entry.schema.json (Phase 6.2 must land first).
- One worked example showing a persisted entry with MINOR verdict (matching §3.1 example payload).
- Cross-link to academic-pipeline/references/passport_as_reset_boundary.md per R4 mitigation 3 — adds one paragraph to passport_as_reset_boundary.md explaining how audit_artifact[] re-verifies on resume.
Verification gate: scripts/check_passport_reset_contract.py (existing v3.6.3 lint) does not regress; scripts/check_audit_artifact_consistency.py (Phase 6.3) lint passes on the amendment's example payload.
Dependencies: Phase 6.2 (referenced schema must exist).
Deliverable: academic-pipeline/agents/pipeline_orchestrator_agent.md gains §3.5 Audit Artifact Gate subsection per §5.6.
Includes: - Insert a ~50-line decision-policy summary derived from §5.6 between current §3 "Checkpoint Management" and §4 "Transition Management" of the orchestrator prompt. The summary names the trigger and ship/block decision rule, declares hard rules, lists the P-PA- / P-PB- failure phase IDs as cross-references back to spec §5.6, and provides cross-references to the spec / audit template / schema / wrapper. - Path A → Path B fall-through procedural detail (A1, A1.5, A2–A7, B1, B1a, B2–B11) is NOT inserted into the prompt verbatim — that procedure is the implementation contract and stays in spec §5.6. Orchestrator follows the §5.6 procedure exactly; the prompt's role is to declare the gate and reference §5.6 as the procedure source. - Verification failure state inventory cross-reference (the inventory itself stays in this spec; the orchestrator prompt cites it as a load-bearing reference). - Hard rules (audit gate cannot be skipped; runs BEFORE collaboration_depth_agent + integrity_verification_agent; PASS does not skip integrity). - Cross-references to spec, audit template, schema, wrapper.
Verification gate: orchestrator prompt is no more than +60 lines vs pre-Step-6 baseline (R3 budget — the ~50-line decision-policy summary plus 5–10 lines of headroom); the §5.6 inventory's phase IDs (P-PA- / P-PB-) appear at least once each in the prompt as cross-references to spec §5.6, not as inline procedural definitions (the procedural definitions stay in §5.6).
Dependencies: Phase 6.2 (schema names) + Phase 6.3 (lint will check this prompt).
Deliverable: synthesis_agent.md, research_architect_agent.md, report_compiler_agent.md updated per §6.2 sweep table.
Includes:
- Remove the three Clause 2 violation sentences (lines 164 / 190 / 172 at HEAD b76e626).
- Trim line 177 of report_compiler_agent.md to retire the "The orchestrator runs codex audit afterward" tail.
- Add the canonical Clause 1 line as a bullet to all three PATTERN PROTECTION (v3.6.7) blocks (placed last in each block).
- Create scripts/v3_6_7_inversion_manifest.json with the three-file scope per §6.3.
- Extend scripts/check_v3_6_7_pattern_protection.py with INV-1 / INV-2 / INV-3 checks per §6.3.
- Add test cases to scripts/test_check_v3_6_7_pattern_protection.py for each new INV-x check (positive: prompt currently passing; negative: simulated regression that re-introduces a Clause 2 violation).
Verification gate: scripts/check_v3_6_7_pattern_protection.py passes with zero INV violations on all three prompts; test suite green.
Dependencies: none structurally (Phase 6.7 is independent of Phase 6.1–6.6 file outputs); can run in parallel with Phase 6.1–6.4. Recommended: bundle with Phase 6.6 in one PR slice because both touch agent prompts and the lint scripts that check them.
Deliverable:
- 17 micro-fixtures + 1 integration fixture under tests/fixtures/v3_6_7_pattern_eval/ per §7.
- scripts/check_pattern_eval_manifest.py validating each fixture's manifest.json. The script reads fixture_kind from each manifest and branches: fixture_kind: "micro" → validate against the §7.2 micro-fixture schema; fixture_kind: "integration" → validate against the §7.3 integration manifest schema. The two schemas share a pattern_id enum but have disjoint required fields per §7.3, and routing by fixture_kind is what prevents either schema from accidentally rejecting a valid fixture of the other kind.
- scripts/test_pattern_eval_runtime.py test harness — for fixture_kind: "micro" runs the orchestrator gate procedure against each fixture's BAD/GOOD pair; for fixture_kind: "integration" runs the §7.3 round/escalation scenario (the integration fixture has no top-level BAD/GOOD pair — its directory tree is round_1/ / round_2/ / round_3/ / escalation/ per §7.3, with per-agent sub-fixtures inside each round).
- .github/workflows/spec-consistency.yml extended with pattern-eval-unit + pattern-eval-integration jobs per §7.6.
- L-doc-1 follow-up ticket logged in docs/design/2026-04-30-ars-v3.6.7-step-6-orchestrator-hooks-spec.md §9.2 (this spec) — the "18 patterns" → "17 patterns" prose retirement covering eight locations (four in the upstream v3.6.7 main spec and four in this spec) is tracked as an issue or TODO file note, not a v3.6.7 spec amendment.
Includes:
- 17 micro-fixture directories per the §7.2 schema, each with: manifest.json, upstream_context/passport_snippet.yaml, upstream_context/prior_artifacts/, bad_run/{deliverable.<ext>,expected_audit_findings.yaml,expected_orchestrator_action.yaml}, good_run/{...}.
- 1 integration fixture under integration/chapter_level_run/ per §7.3 multi-round structure.
- Path-filtered triggers in spec-consistency.yml matching the file list in §7.6.
- Test harness uses synthesized verdicts (reads expected_audit_findings.yaml as if codex emitted it) — no real codex execution required for CI green.
Verification gate: all 17 micro-fixtures + 1 integration fixture green in pattern-eval-unit + pattern-eval-integration jobs. scripts/check_pattern_eval_manifest.py passes on every fixture. The cross-check between fixture set and pattern inventory (per §7.5) reports 17/17 enumerated patterns covered.
Dependencies: Phase 6.1 (wrapper) + Phase 6.2 (schemas) + Phase 6.3 (lint) + Phase 6.4 (strict-monotonic helper, exercised by integration fixture's escalation/ack append step) + Phase 6.5 (Schema 9 amendment, required for the synthetic passport snippet under each fixture's upstream_context/) + Phase 6.6 (orchestrator prompt) + Phase 6.7 (downstream prompts). Phase 6.8 is the integration phase that exercises everything earlier phases produced.
A successful Step 6 + Step 8 ship requires ALL of the following hold simultaneously after Phase 6.8 lands:
.github/workflows/spec-consistency.yml green: existing checks (spec-consistency, freshness-check, pattern-protection) plus new checks (audit-artifact-consistency, pattern-eval-unit, pattern-eval-integration)..github/workflows/pytest.yml green (regression check on v3.6.4 corpus adapters; Step 6 should not touch these but lint script changes can collide).feedback_codex_review_per_schema_increment.md) on each commit: schema commits (Phase 6.2 / Phase 6.5) and contract commits (Phase 6.6 / Phase 6.7) get incremental codex review; the final integration commit (Phase 6.8) gets a final convergence-to-zero codex review per feedback_codex_iterative_spec_review_to_zero.md.Recommended commit sequence (one PR with multiple commits, squash-merged):
1. Phase 6.2 — schemas (4 new JSON files + README section)
2. Phase 6.4 — helper script + tests
3. Phase 6.1 — wrapper script
4. Phase 6.3 — lint script + tests
5. Phase 6.5 — Schema 9 amendment + reset-boundary cross-link
6. Phase 6.6 — orchestrator prompt update
7. Phase 6.7 — three downstream prompt edits + manifest + lint extension
8. Phase 6.8 — fixtures + CI workflow + L-doc-1 ticket
Phases 6.1, 6.2, 6.4 can run in parallel branch-locally before merging into the PR sequence; the listed order is the dependency-resolved sequence for clean reviewability. The PR description carries the §10 phase table as the review roadmap so reviewers can audit phase-by-phase against this spec.
| Phase | Files added | Files modified | LoC delta (approx) |
|---|---|---|---|
| 6.1 | scripts/run_codex_audit.sh |
— | +800 (Bash incl. helpers + traps) |
| 6.2 | 4 schemas + 1 README section | shared/contracts/README.md |
+400 |
| 6.3 | scripts/check_audit_artifact_consistency.py + test file |
— | +1200 (lint + tests) |
| 6.4 | scripts/_next_verified_at_ms.py + test file |
— | +150 |
| 6.5 | — | shared/handoff_schemas.md, academic-pipeline/references/passport_as_reset_boundary.md |
+80 |
| 6.6 | — | academic-pipeline/agents/pipeline_orchestrator_agent.md |
+60 |
| 6.7 | scripts/v3_6_7_inversion_manifest.json |
3 downstream prompts + scripts/check_v3_6_7_pattern_protection.py + test file |
+50 (mostly lint extension) |
| 6.8 | 18 fixture directories + harness + manifest validator + workflow jobs | .github/workflows/spec-consistency.yml |
+2500 (fixture data dominates) |
Estimated total: ~5300 LoC, of which ~3000 are fixture data + tests. The implementation budget is consistent with the v3.6.7 Step 1+2 ship envelope (~2800 LoC including the 29-mutation suite); Step 6 is roughly 1.9× the size, weighted toward Phase 6.1 (wrapper) and Phase 6.8 (fixtures).
This spec closes Step 6 (orchestrator hooks for automatic per-agent audit + anti-fake-audit guard) and Step 8 (synthetic evaluation case demonstrating the v3.6.7 patterns triggered + protected) of the v3.6.7 main spec (2026-04-29-ars-v3.6.7-downstream-agent-pattern-protection-spec.md) §9 implementation table. After Step 6 + Step 8 ship, v3.6.7 is structurally complete: prompt-level pattern protection (Step 1+2) + version sweep (Step 7) + runtime audit-artifact gate (Step 6) + synthetic evaluation case (Step 8) together deliver the "end-to-end deliverable set passes independent xhigh cross-model audit at 0 P1+P2 finding within three rounds" ship-quality target the v3.6.7 main spec §10 declared.
Step 6 + Step 8 land as one follow-up PR after the §1–§5 design (PR #50, main b76e626) is on main. The PR description carries §10's phase table as the review roadmap. Codex review per feedback_codex_iterative_spec_review_to_zero.md runs to convergence on each commit; the final integration commit (Phase 6.8) is the gate for ship.
The v3.6.7-only scope of §6.2's inversion sweep, §7.1's fixture set, and Step 6's audit gate is preserved; widening to additional agents (e.g., bibliography_agent's five hallucination patterns documented in feedback_ars_bibliography_agent_hallucination_patterns.md) is v3.6.8+ work that will land its own spec round and its own version-tagged manifest.