← Về thư mục
📄 / / academic-research-skills / docs / design / 2026-04-20-v3.4-prisma-trAIce-raise-readcheck-design.md

ARS v3.4 Design Spec — PRISMA-trAIce + RAISE Compliance + Reading-Check

Status: Draft for implementation Date: 2026-04-20 Author: Cheng-I Wu Target versions: v3.4.0 (compliance agent) → v3.5.0 (reading-check) File path: docs/design/2026-04-20-v3.4-prisma-trAIce-raise-readcheck-design.md Supersedes: the three-part roadmap saved in private planning notes (2026-04-19 "下次優化的三項補強")

1. Background

ARS v3.3.6 shipped on 2026-04-15. Three gaps flagged during the NYCU EMBA fact-check exercise:

  1. A. PRISMA-trAIce — 2025 reporting checklist for AI-assisted systematic reviews (Holst et al. 2025, JMIR AI, doi:10.2196/80247). 17 items across 6 sections, 4 priority tiers (Mandatory / Highly Recommended / Recommended / Optional). GitHub canonical: cqh4046/PRISMA-trAIce.
  2. B. RAISE — Cochrane / Campbell / JBI / EPPI-Centre / CEE joint framework for Responsible AI in Evidence Synthesis (Thomas et al. 2025, NIHR ESG Working Group). Role-based ecosystem, not an item checklist. 8 roles × 2–3 responsibilities, organised around four principles: human oversight, transparency, reproducibility, fit-for-purpose. Source: eppi.ioe.ac.uk/CMS/Portals/0/RAISE ESG BPWG.pdf (accessed 2026-04-20).
  3. C. Reading-check probe — Socratic-mentor nudge to detect whether the user has actually read the AI-cited material.

v3.3 anti-sycophancy and S2 verification already cover ~80% of EMBA speaker warning points. v3.4 closes the reporting-framework gap.

2. Positioning Decisions

2.1 Why a single compliance agent (not two)

PRISMA-trAIce and RAISE share the same evidence base (the manuscript + material passport) and the same checkpoint trigger surface (Stage 2.5 / 4.5). Merging reduces token cost by ~50% and keeps compliance logic in one place.

2.2 Why mode-aware scope

2.3 Why non-SR warn-only

RAISE's formal scope is evidence synthesis. When ARS runs RAISE principles on academic-paper full (primary research), that is principle extension — it should not produce hard blocks that suggest false authority. Warn-only + honest disclaimer is the honest design.

2.4 Why C ships separately

Reading-check has zero coupling with the compliance agent. Same bundle = larger blast radius for rollback. Separate minor bump keeps each PR's regression surface small.

3. Architecture

3.1 File layout

shared/
├── agents/
│   └── compliance_agent.md              ← NEW: shared agent, single source
├── prisma_trAIce_protocol.md            ← NEW: 17 items embedded verbatim
├── raise_framework.md                   ← NEW: principles + 8-role matrix + scope disclaimer
├── compliance_checkpoint_protocol.md    ← NEW: 2.5/4.5 gate behaviour spec
└── handoff_schemas.md                   ← CHANGED: add Schema 12 compliance_report

academic-pipeline/SKILL.md               ← CHANGED: Stage 2.5/4.5 payload extended
deep-research/SKILL.md                   ← CHANGED: systematic-review mode hooks compliance
academic-paper/SKILL.md                  ← CHANGED: full mode adds pre-finalize RAISE principles (warn-only)

docs/PERFORMANCE.md + .zh-TW.md          ← CHANGED: new "Long-running session management" section
README.md + .zh-TW.md                    ← CHANGED: v3.4 key additions
CHANGELOG.md                             ← CHANGED: 3.4.0 entry

v3.5.0 adds:

deep-research/SKILL.md                   ← CHANGED: 8th mode = reading-check
deep-research/agents/
├── reading_check_agent.md               ← NEW
└── socratic_mentor_agent.md             ← CHANGED: optional probe layer
deep-research/references/
└── reading_check_protocol.md            ← NEW

3.2 Coupling diagram

              ┌─────────────────────────┐
              │  academic-pipeline      │
              │  (orchestrator)         │
              └────────────┬────────────┘
                           │
      ┌────────────────────┼────────────────────┐
      │                    │                    │
      ▼                    ▼                    ▼
┌───────────┐      ┌──────────────┐      ┌──────────────┐
│Stage 2.5  │      │Stage 4.5     │      │other stages  │
│Integrity  │      │Integrity     │      │(unchanged)   │
│Gate       │      │Gate          │      └──────────────┘
│+compliance│      │+compliance   │
└─────┬─────┘      └──────┬───────┘
      │                   │
      └─────────┬─────────┘
                ▼
      ┌─────────────────────┐
      │ compliance_agent    │  ← shared/, mode-aware
      └──────────┬──────────┘
                 │
    ┌────────────┴────────────┐
    ▼                         ▼
[SR mode]                [non-SR mode]
PRISMA-trAIce 17 +       RAISE 4 principles
RAISE full               only
(tier-based block)       (warn-only)

3.3 Design principles

  1. Single compliance agent. Not two agents. compliance_mode parameter dispatches payload.
  2. No new stage. 2.5 / 4.5 are existing Integrity Gates. Compliance is a payload extension, not a structural change. Fail-loop and resume reuse existing machinery.
  3. Scope honesty. raise_framework.md opens with a disclaimer: RAISE's official scope is evidence synthesis; non-SR use is principle extension, not RAISE compliance.
  4. Reading-check decoupled. v3.5.0 has zero dependency on compliance_agent; ships and rolls back independently.

4. Components

4.1 shared/agents/compliance_agent.md

Metadata

name: compliance_agent
version: 1.0.0
data_access_level: verified_only
task_type: open-ended
owner_skill: shared
invoked_by: [academic-pipeline, deep-research, academic-paper]

Input contract

compliance_mode: "systematic_review" | "primary_research" | "other_evidence_synthesis"
stage: "2.5" | "4.5"
materials:
  manuscript_draft: <path or inline>
  methodology_blueprint: <path>
  bibliography: <path>
  material_passport: <Schema 9 payload>
user_metadata:
  intended_venue: <string or null>
  ai_tools_used: [<list of {name, version, stage, purpose}>]

Output contract — full schema defined in §4.5 Schema 12.

Scope guardrails

Self-checks (compliance agent's own failure modes)

Check What On fail
CA-1 Am I quoting PRISMA-trAIce from memory or from protocol? Force re-read shared/prisma_trAIce_protocol.md
CA-2 Are block decisions anchored to Mandatory tier, not tier-promoted? Cross-ref protocol tier table
CA-3 Did all 17 items pass? (Sycophancy risk) Force second pass on 3 Mandatory items; require explicit evidence citation
CA-4 Do RAISE principle passes have evidence citations? Downgrade no-evidence passes to warn with [weak evidence] tag

4.2 shared/prisma_trAIce_protocol.md

Full verbatim snapshot of the 17-item checklist from GitHub cqh4046/PRISMA-trAIce. Each item contains:

  1. Original description (quoted from upstream)
  2. ARS check procedure — what material fields to consult, what evidence counts as pass
  3. Tier behaviour — Mandatory=block, Highly Recommended=warn, Recommended/Optional=info

Stage-assignment table:

Stage Items
2.5 M1, M2, M3, M4, M5, M6, M7, M8, M9, M10
4.5 T1, A1, I1, R1, R2, D1, D2

Header metadata (required):

snapshot_date: "2025-12-10"
upstream_source: "https://github.com/cqh4046/PRISMA-trAIce"
upstream_version_commit: "<commit hash at snapshot time>"
update_policy: "Manual sync. CI freshness check runs weekly; drift emits warning, not merge block."

Citation: Holst D, et al. Transparent Reporting of AI in Systematic Literature Reviews: Development of the PRISMA-trAIce Checklist. JMIR AI. 2025. doi:10.2196/80247

4.3 shared/raise_framework.md

Structure

  1. Scope disclaimer (must be first, non-removable):

    RAISE's official scope is evidence synthesis (systematic review, meta-analysis, scoping review, rapid review, evidence mapping). When ARS applies RAISE principles to non-SR work (academic-paper full, primary research, etc.), this is a principle extension. ARS does not claim official RAISE compliance for non-SR outputs.

  2. Four principles — human oversight, transparency, reproducibility, fit-for-purpose. Per-principle: definition, ARS check procedure, pass/warn/fail criteria.

  3. Full 8-role matrix (SR mode only) — 8 roles × 2–3 responsibilities each. Default ARS-user role assumption: Evidence Synthesists. ARS itself also occupies AI Development Team + Methodologist roles; this dual-role declaration is stated in the doc.

  4. Citation: Thomas J, Flemyng E, Noel-Storr A, et al. Responsible Use of AI in Evidence Synthesis (RAISE). NIHR ESG Best Practice Working Group in AI/Automation, EPPI-Centre, UCL Social Research Institute. 17 July 2025. https://eppi.ioe.ac.uk/CMS/Portals/0/RAISE%20ESG%20BPWG.pdf

4.4 shared/compliance_checkpoint_protocol.md

Dual-gate behaviour

Aspect Stage 2.5 Stage 4.5
PRISMA-trAIce scope M1–M10 T1, A1, I1, R1, R2, D1, D2
RAISE scope 4 principles (focus: oversight, fit-for-purpose) 4 principles (focus: transparency, reproducibility) + 8-role matrix (SR only)
Block condition SR mode AND any Mandatory PRISMA-trAIce item = FAIL Same
Warn condition Highly Recommended fail OR RAISE principle warn Same
Info condition Recommended / Optional fail Same
Non-SR mode RAISE principles only, warn-only (never block) Same
Checkpoint display FULL checkpoint + compliance section FULL checkpoint + compliance section

User action UX

Fail loop reuses pipeline_state_machine.md §Integrity Check FAIL Loop: after 3 unresolved rounds, user picks [manual handling / remove AI usage / continue with "partially non-compliant" warning]. See §6.3.

4.5 shared/handoff_schemas.md — Schema 12

compliance_report:
  mode: "systematic_review" | "primary_research" | "other_evidence_synthesis"
  stage: "2.5" | "4.5"
  generated_at: <ISO-8601>
  prisma_trAIce:        # null when mode != systematic_review
    items_total: 17
    by_tier:
      mandatory:
        total: 9
        pass: <int>
        fail: [<item_id>, ...]
        gaps: [{item_id, reason, evidence_path}, ...]
      highly_recommended: {...}
      recommended: {...}
      optional: {...}
    block_decision: "block" | "warn" | "pass"
  raise:
    mode: "full" | "principles_only"
    principles:
      human_oversight: "pass" | "warn" | "fail"
      transparency: ...
      reproducibility: ...
      fit_for_purpose: ...
    principle_evidence:
      human_oversight: [<path into manuscript / methodology>, ...]
      ...
    roles:                # only when raise.mode == "full"
      evidence_synthesists: [<responsibility fulfilment notes>, ...]
      ai_development_teams: [...]
      methodologists: [...]
      publishers: [...]
      users: [...]
      trainers: [...]
      organisations: [...]
      funders: [...]
    block_decision: "block" | "warn" | "pass"
  overall_decision: "block" | "warn" | "pass"
  user_action_required: <boolean>
  evidence: [<path into manuscript / methodology>, ...]
  user_override:        # only present when user overrode a block
    decision: true
    timestamp: <ISO-8601>
    rationale: <string, min 100 chars on 3rd-round override>
    scope: [<item_ids or principle_names>]
  upstream_sync_status: "current" | "stale"   # from freshness check

Material Passport extension (Schema 9):

compliance_history: [<compliance_report>, ...]   # append-only

4.6 SKILL.md changes

academic-pipeline/SKILL.md - Stage 2.5 section: add "compliance payload" bullet listing what compliance_agent contributes - Stage 4.5 section: same + note AI Self-Reflection Report includes compliance summary - Checkpoint dashboard example: add compliance row

deep-research/SKILL.md - systematic-review mode flow: add compliance entry at Stage 2.5 and 4.5 positions - Mode selection guide: note PRISMA-trAIce coverage for SR mode

academic-paper/SKILL.md - full mode: add pre-finalize RAISE principles check (warn-only, mode=primary_research) - disclosure mode: unchanged; stated as complementary (venue policy ≠ RAISE)

4.7 docs/PERFORMANCE.md — new section

Long-running session management

5. Data Flow

5.1 SR pipeline with dual-gate compliance (primary scenario)

Stage 1 Scoping → Stage 2 Investigation
        │
        ▼  Material Passport v1
        │
Stage 2.5 INTEGRITY GATE
  1) integrity_verification_agent (existing)
  2) failure_mode_checklist (v3.2)
  3) compliance_agent (NEW)
       mode=systematic_review, stage=2.5
       → PRISMA-trAIce M1–M10 check
       → RAISE principles (oversight, fit-for-purpose)
       → compliance_report_v1
  overall decision: block / warn / pass
    block → FAIL loop (max 3 rounds)
    warn/pass → FULL checkpoint
  user confirms → append to compliance_history
        │
        ▼
Stage 3/4/4' Write + Revise (compliance_agent not invoked)
        │
        ▼  Material Passport v2
        │
Stage 4.5 INTEGRITY GATE
  1) integrity_verification_agent
  2) failure_mode_checklist
  3) compliance_agent
       mode=systematic_review, stage=4.5
       → PRISMA-trAIce T1/A1/I1/R1/R2/D1/D2
       → RAISE principles + 8-role matrix
       → compliance_report_v2
  same block/warn/pass logic
  user confirms → append to compliance_history
        │
        ▼
Stage 5 Review → Stage 6 Finalize
                  AI Self-Reflection Report
                    + compliance summary section
                      - pass/warn/fail counts across history
                      - full compliance_report JSON in appendix

5.2 Primary research (non-SR)

academic-paper full mode
  Stage A Plan → Stage B Draft → Stage C Revision
        │
        ▼  Material Passport (primary research variant)
        │
  pre-finalize compliance check (NEW)
    compliance_agent
      mode=primary_research
      → RAISE 4 principles only (principles_only payload)
      → PRISMA-trAIce section null
      → block_decision always ≤ warn
        │
        ▼
  Finalize (warns do not block; listed in disclosure)

5.3 Other evidence synthesis (scoping / rapid review, etc.)

compliance_mode = "other_evidence_synthesis"
  → RAISE full suite (4 principles + 8-role matrix)
  → PRISMA-trAIce runs in "adaptation" mode: items marked as informational, not Mandatory
  → block condition relaxed to warn (flagged "not formal SR")

5.4 Mode × skill × stage matrix

Skill / entry Stage 2.5 Stage 4.5 compliance_mode
academic-pipeline, SR route full full systematic_review
academic-pipeline, primary research route n/a pre-finalize hook primary_research
deep-research systematic-review (standalone) full terminal-hook systematic_review
deep-research full / lit-review n/a pre-output hook other_evidence_synthesis
academic-paper full (standalone) n/a pre-finalize hook primary_research
academic-paper-reviewer (any mode) n/a n/a not invoked

5.5 Resume flow

session ends → Material Passport + compliance_history persisted to local file
new session starts → user pastes passport YAML
  orchestrator reads compliance_history[]
  locates last completed stage
  resumes from next stage
  already-run compliance is NOT re-run (passport already has compliance_report_v1)

Append-only invariant: a Stage 2.5 FAIL → backfill → retry pass produces compliance_history = [v1_fail, v1_retry_pass, ...]. Full audit trail preserved; AI Self-Reflection Report can cite correction trajectory. This implements RAISE's transparency principle on the pipeline itself.

6. Error Handling

6.1 Error taxonomy

# Error Trigger Handling
E1 Missing material compliance_agent needs M4 (Input Data) but passport.bibliography is empty Apply Anti-Leakage Protocol: mark [MATERIAL GAP: M4 — bibliography not provided], no hallucination. Item auto-FAILs, tier dictates block/warn
E2 Mode mismatch academic-paper full passes compliance_mode=systematic_review by mistake compliance_agent entry-point validates mode vs skill context; on mismatch returns {decision: abort, reason: "mode/context mismatch"}, orchestrator re-evaluates mode
E3 Block loop > 3 rounds Stage 2.5 FAIL → backfill → FAIL → … exceeds 3 Existing pipeline_state_machine §Integrity Check FAIL Loop: list unresolvable items; user picks [manual handling / remove AI usage / continue with "partially non-compliant" warning]
E4 User override block User chooses "acknowledge limitation" after a block Allowed with audit: compliance_report.user_override.decision=true; AI Self-Reflection Report must record override; disclosure statement auto-adds line "PRISMA-trAIce Mandatory item X not met; authors acknowledge limitation"
E5 Schema validation fail compliance_report does not match Schema 12 Pipeline halts; orchestrator reports internal error; NOT appended to compliance_history (avoid polluting audit trail)
E6 Upstream reference drift prisma_trAIce_protocol.md snapshot_date diverges from GitHub canonical No pipeline block; compliance_agent adds upstream_sync_status: "stale" to report; weekly CI check_prisma_trAIce_freshness.py emits warning annotation

6.2 E4 user override design

Override is not "skip" — it is "leave a trace." 3-round friction ladder:

Round Behaviour
1st block User sees warning, must acknowledge; rationale string optional
2nd FAIL User must provide rationale (any length); system notes "second override attempt"
3rd FAIL User must provide rationale ≥ 100 chars; only then released

Generated disclosure_addendum is auto-injected into final manuscript's AI disclosure section. Non-removable. This is the concrete form of the no detection evasion iron rule.

disclosure_addendum: |
  The authors acknowledge the following compliance limitations:
  - PRISMA-trAIce M4 (Input Data) not fully reported; rationale: <user text>
  - RAISE principle: transparency — partial; rationale: <user text>

6.3 Boundary behaviour across skills

When compliance_agent is invoked outside the pipeline (e.g., academic-paper full standalone):

6.4 Resume-time error cases

Case Behaviour
Loaded passport predates Schema 12 Auto-migrate: append empty compliance_history=[], warn user
compliance_history shows unresolved Stage 2.5 block Do not auto-retry; display dashboard, let user choose next step
Upstream PRISMA-trAIce updated during pause E6 path; do not interrupt resume

6.5 What does NOT change

7. Testing Strategy

7.1 Layer 1 — Unit tests

Test file Coverage
scripts/test_compliance_agent.py Input contract, mode dispatch, tier-based decision
scripts/test_compliance_report_schema.py Schema 12 validator + positive/negative fixtures
scripts/check_compliance_report.py CI validator (patterned after check_repro_lock.py, check_benchmark_report.py)

Key unit cases: - mode=systematic_review, stage=2.5 → output contains M1–M10 results only, no T1/R1 etc. - mode=primary_research → prisma_trAIce field is null, raise.mode="principles_only", block_decision ≤ warn - Mode mismatch → agent refuses (E2) - All 17 items pass with empty evidence → CA-3 self-check triggers - user_override with non-empty scope → disclosure_addendum must be non-empty - Tier misclassification (HR item marked Mandatory) → Schema validation fails

7.2 Layer 2 — Integration tests

scripts/test_pipeline_with_compliance.py:

Scenario Verification
SR pipeline at Stage 2.5, Mandatory M4 FAIL Orchestrator shows block; user picks "backfill" → pipeline returns to Stage 2
Same, user overrides 3 times 3rd round demands ≥ 100 char rationale; post-pass compliance_report.user_override.rationale non-empty
Primary research path, Stage 2.5 skipped Compliance re-hooked at pre-finalize; block_decision always ≤ warn
Resume: passport has compliance_history[0]=v1_pass Stage 2.5 not re-run; advance to Stage 3
Append-only: FAIL → retry pass compliance_history length +2, not overwritten

7.3 Layer 3 — Fixture-based compliance tests

examples/compliance/ (fictional manuscripts only, no real data):

Fixture Expected result
fixture_sr_full_compliant.md Full SR report with all AI usage disclosed → all pass
fixture_sr_missing_M4.md Missing Input Data description → block at 2.5, Mandatory FAIL
fixture_primary_raise_weak.md Primary research, weak human-oversight description → warn (principles_only), no block

Serves double duty: regression tests + user reference templates.

7.4 CI wire-up

Extend .github/workflows/spec-consistency.yml:

- name: Validate compliance schema
  run: python scripts/check_compliance_report.py examples/compliance/*.yaml

- name: Test compliance agent
  run: python -m scripts.test_compliance_agent

- name: Test pipeline-compliance integration
  run: python -m scripts.test_pipeline_with_compliance

- name: Check PRISMA-trAIce snapshot freshness
  run: python scripts/check_prisma_trAIce_freshness.py
  # Non-blocking; emits warning annotation (E6 behaviour)

7.5 Anti-regression coverage

Existing feature v3.4 verification
v3.2 Failure Mode Checklist Stage 2.5/4.5 runs failure-mode + compliance simultaneously, separate reports, no cross-contamination
v3.3 Score Trajectory After compliance block → retry pass, trajectory correctly records delta
Material Passport Schema 9 (repro_lock) Adding Schema 12 does not break Schema 9; passport migration smoke test
check_spec_consistency.py Add SKILL.md version ↔ compliance_agent.md version consistency check

7.6 Calibration (release gate)

compliance_agent is itself subject to calibration — reuses v3.2 academic-paper-reviewer calibration framework:

7.7 v3.5.0 reading-check tests (independent)

scripts/test_reading_check.py: - reading-check sub-mode invoke → generates 3–5 questions - Question types cover argument / method / limitation (not fact recall) - Soft-nudge variant: socratic_mentor triggers ≥ 1 probe between rounds 7–15 - Opt-out flag: user says "goal-oriented" / "skip probe" → entire session probes off

8. Release Plan

8.1 v3.4.0 — Compliance Agent

8.2 v3.5.0 — Reading-check

8.3 PR ordering

  1. Open and merge feat/v3.4.0-compliance-agent
  2. After v3.4.0 stable, open feat/v3.5.0-reading-check
  3. Each PR carries its own CHANGELOG entry and calibration note

9. Open questions / deferred

10. References