← Về thư mục
📄 / / academic-research-skills / docs / design / 2026-04-22-ars-v3.7.3-reading-check-probe-design.md

ARS v3.5.1 Reading-Check Probe — Design Spec

Status (2026-05-04 retrospective): SHIPPED in v3.5.1, 2026-04-22. All §1–§4 components landed; §5 tests landed as scripts/test_reading_probe_lint.py covering 7 of 8 specified tests. §5.1 test 8 (test_version_bump_sweep) is OBSOLETE — superseded by PR #33 which generalised check_version_consistency.py to dynamic CHANGELOG-driven version comparison, replacing the per-version pinned-fixture pattern this spec assumed. Pinned fixtures for v3.5.1 strings are no longer the right shape; the dynamic guard already covers the regression class. Filename retains v3.7.3 because the roadmap slot was originally v3.7.3 before being reshaped into v3.5.1; spec body uses v3.5.1 throughout.

Retained as historical design reference. Implementation evidence: .claude/CLAUDE.md ## v3.5.1 Key Additions, deep-research/agents/socratic_mentor_agent.md ## Optional Reading Probe Layer (v3.5.1 ...), scripts/test_reading_probe_lint.py.

Status: DRAFT — pending implementation Date: 2026-04-22 Target version: 3.5.0 → 3.5.1 (patch) Roadmap slot: v3.7.3 "Did the user actually read the source?" probe — parked from v3.4 design, reshaped in 2026-04-21 ROADMAP §3.7.3 Author: Cheng-I Wu Design precedent: docs/design/2026-04-20-v3.4-prisma-trAIce-raise-readcheck-design.md §7.7 (tests) and §8.2 (release plan). The earlier v3.5.0 proposal bundled a standalone reading_check_agent + reading-check sub-mode. The 2026-04-21 roadmap reshape drops both and keeps only the Socratic Mentor soft-nudge. This spec is authoritative for v3.5.1.

1. Scope & non-goals

1.1 What this ships

1.2 What this explicitly does NOT do

1.3 Out-of-scope alternatives considered and rejected

Alternative Rejected because
Standalone reading_check_agent (original v3.5.0 design) Roadmap §3.7.3 collapses to Mentor-only. Adding an agent violates "minimum surface" principle.
Probe fires in exploratory mode Exploratory sessions assume the user has not yet done the homework; the probe premise doesn't hold.
Probe fires on every user-cited paper Violates "do not extend pipeline" constraint; one probe is enough for honesty signal.
Probe result stored in Material Passport reading_probe_history[] Requires Schema 9 change and Socratic-only sessions don't produce Passport. Inline storage is lighter.
User-initiated probe ("probe me on X") Removes the detection value. Honesty probes have to be AI-initiated.

2. Architecture

2.1 Surface

One primary file: deep-research/agents/socratic_mentor_agent.md. The Mentor agent is defined by prompt, so "architecture" here means what the agent prompt instructs the model to do. No runtime code executes the probe logic — Claude interprets the prompt.

Three supporting files get aligned text updates: deep-research/references/socratic_mode_protocol.md, deep-research/SKILL.md, academic-pipeline/references/process_summary_protocol.md.

2.2 Probe trigger state machine

           ┌──────────────┐
           │ session start│
           └──────┬───────┘
                  │
                  ▼
         probe_fired = false
         candidate_paper = null
                  │
                  ▼
     ┌────────────────────────────┐
     │ on each user turn:         │
     │ 1. classify intent         │
     │ 2. scan for paper citation │
     └────────────┬───────────────┘
                  │
           ┌──────┴──────┐
           │ paper cited?│
           └──┬────────┬─┘
        no   │        │ yes
             │        ▼
             │  candidate_paper = first cited
             │  (never overwritten this session)
             │        │
             └──┬─────┘
                ▼
    ┌──────────────────────────────────┐
    │ at Layer 2 → Layer 3 transition: │
    │ probe_gate =                     │
    │   env ARS_SOCRATIC_READING_PROBE │
    │   AND intent == goal_oriented    │
    │   AND candidate_paper != null    │
    │   AND probe_fired == false       │
    └──────────────────┬───────────────┘
                       │
              probe_gate?
                 │yes   │no
                 ▼      ▼
         fire probe   continue
         probe_fired = true   Layer 3
                 │
                 ▼
         user response:
         paraphrase | decline | other
                 │
                 ▼
         log [READING-PROBE: ...]
                 │
                 ▼
         Mentor acknowledges briefly,
         continues Layer 3

2.3 Env gate semantics

2.4 Data flow across stages

Socratic session (deep-research, mode=socratic)
  ├─ Mentor emits dialogue turns, possibly with
  │  [READING-PROBE: paper=..., outcome=..., turn=N,
  │                 paraphrase_quote="..."]  ← tag inline
  │
  └─ Session end → Research Plan Summary
                    ├─ existing sections
                    └─ "Reading Probe Outcomes" subsection
                       (populated only if probe fired OR env was set
                        and probe did not fire with reason)

If user continues into academic-paper / pipeline:
  academic-pipeline Stage 6 (report_compiler_agent)
    └─ AI Self-Reflection Report
         └─ "Reading Probe Outcomes" section
            (transcribes Research Plan Summary subsection verbatim,
             with a line noting the probe is a user-honesty observation,
             not an AI-verified claim)

If user stops after socratic mode:
  Probe data persists in Research Plan Summary only.
  No data loss — the downstream Self-Reflection Report simply
  never runs.

2.5 Components changed

File Change type Approx size
deep-research/agents/socratic_mentor_agent.md NEW section "Optional Reading Probe Layer (v3.5.1)" +60–90 lines
deep-research/references/socratic_mode_protocol.md NEW subsection "Reading Probe (opt-in, goal-oriented only)" +15–25 lines
deep-research/SKILL.md Socratic Mode section gains env-var note; INSIGHT Compilation Output template gains "Reading Probe Outcomes" subsection slot +10–15 lines
academic-pipeline/references/process_summary_protocol.md Self-Reflection Report spec gains "Reading Probe Outcomes (if present)" item +8–12 lines
scripts/test_reading_probe_lint.py NEW — probe prompt-lint test suite ~120 lines
CHANGELOG.md v3.5.1 entry +15–25 lines
README.md + README.zh-TW.md v3.5.1 key-addition paragraph +5 lines each
.claude/CLAUDE.md Suite version 3.5.0 → 3.5.1, Skills Overview row adjustment if any 1–2 lines
deep-research/SKILL.md frontmatter version 2.9.0 → 2.9.1 (patch, because SKILL.md text changes and mentor behaviour gains optional layer) 1 line
deep-research/references/changelog.md corresponding skill-level changelog entry +10 lines

Not changed: shared/handoff_schemas.md, any orchestrator agent, any other deep-research agent, academic-pipeline/SKILL.md, any compliance / rubric / scoring file.

3. Prompt specification for the probe layer

This is the literal content that goes into socratic_mentor_agent.md — not full prose, but the key invariants the prompt must encode.

3.1 Section header

## Optional Reading Probe Layer (v3.5.1 — Internal, Never Mention "Probe" to Users)

3.2 Activation clause

This layer activates only when ALL of the following hold:
- Environment variable ARS_SOCRATIC_READING_PROBE is set to "1"
- Current intent classification (from Intent Detection Layer) is goal-oriented
- The user has, in a prior turn of THIS session, cited a specific paper
  (named authors + year minimum, e.g., "Smith 2024" or full reference)
- The Layer 2 → Layer 3 transition is imminent
- The probe has not yet fired in this session

If ANY of these is false, this layer is dormant. Do not mention the probe.
Do not prepare for the probe. Do not hint that a probe exists.

3.3 Candidate paper tracking

While this session is active, silently track the FIRST concrete paper
citation the user produces. Store internally as candidate_paper. Once set,
never overwrite. If the user cites multiple papers later, they do not
replace the candidate. Rationale: one probe, one paper, fair detection.

3.4 Probe wording template

When all activation conditions hold, at the Layer 2 → Layer 3 transition,
ask ONE question in this form:

  "You mentioned [candidate_paper] earlier. Before we move into evidence
   strategy — could you tell me, in your own words, one specific passage
   from that paper that's shaping your thinking? Feel free to paraphrase
   a paragraph or an argument. Or skip this if you'd rather keep moving."

Do NOT:
- Frame it as a test, check, or verification
- Imply that the user must answer
- Use evaluative language ("make sure", "prove", "demonstrate")
- Preface with "I want to check if..."
- Follow up with a second probe question in the same session

3.5 Response handling

The user's response maps to one of three outcomes:

OUTCOME = paraphrase
  User offers any content that references the paper (even if vague,
  even if arguably wrong — Mentor does NOT judge accuracy).
  Action: Acknowledge in ≤ 15 words. Do not praise, do not evaluate,
  do not grade. Example: "Got it — noted. Let's move into evidence."
  Log: [READING-PROBE: paper="<candidate>", outcome=paraphrase,
                      turn=<N>, paraphrase_quote="<user text, trimmed to
                      first 280 chars>"]

OUTCOME = decline
  User's response is a clear skip/pass signal: "skip", "pass",
  "let's move on", "不用了", "跳過", or similar, AND contains no
  content referencing the paper. If the user's response mixes a skip
  signal WITH paper content (e.g., "skip, but briefly — the paper
  argues X"), classify as OUTCOME=paraphrase and log the paper-content
  portion only.
  Action: Acknowledge briefly. Example: "No problem — moving on."
  Decline is NOT penalised. It does NOT count toward Persistent-Agreement,
  Conflict-Avoidance, or Premature-Convergence indicators. It does NOT
  shift convergence signals. It does NOT affect intent classification.
  Log: [READING-PROBE: paper="<candidate>", outcome=decline, turn=<N>]

OUTCOME = other
  User answers something off-topic or asks a clarifying question back.
  Action: Answer their question, then proceed to Layer 3 without
  re-asking the probe. Probe is fired once per session regardless
  of what the user said.
  Log: [READING-PROBE: paper="<candidate>", outcome=other, turn=<N>,
                      user_response="<first 280 chars>"]

Regardless of outcome, set probe_fired = true and NEVER probe again
this session.

3.6 Banned phrases in probe and acknowledgement

The probe question and the acknowledgement MUST NOT contain:
- "correct", "right", "wrong", "good answer", "well said"
- "make sure", "verify", "prove"
- Any praise of the user's paraphrase content
- Any judgement of the user's decline

(Note: "check" is not banned because it appears in non-evaluative
senses elsewhere in this section — e.g., "activation conditions".)

Rationale: evaluative language turns the probe into a sycophancy hook
(user answers well → Mentor praises → user feels graded). Probe is
observation, not grading.

3.7 Research Plan Summary subsection

When the Mentor compiles the Research Plan Summary at session end, if
ARS_SOCRATIC_READING_PROBE was set at any point during the session,
include this subsection immediately before "Complete INSIGHT List":

  ### Reading Probe Outcomes

  Probe status: <fired | not_fired_no_citation | not_fired_exploratory_mode>
  <If fired:>
  - Paper: <candidate_paper>
  - Outcome: <paraphrase | decline | other>
  - Turn: <N>
  - User text (verbatim, if paraphrase or other): <quote>

  Note to reader: This section records whether the user chose to
  paraphrase a paper they cited. The Mentor did NOT verify factual
  accuracy of any paraphrase. Interpret at your own discretion.

If ARS_SOCRATIC_READING_PROBE was NOT set, omit this subsection entirely.

4. Companion file updates

4.1 deep-research/references/socratic_mode_protocol.md

Add a subsection after "Dialogue Management Rules":

## Reading Probe (opt-in, goal-oriented only)

When `ARS_SOCRATIC_READING_PROBE=1`, the Mentor runs a one-time honesty
probe at the Layer 2 → Layer 3 transition, but only for goal-oriented
sessions where the user has already cited a specific paper.

The probe asks the user to paraphrase one passage from that paper. The
user may decline; the decline is logged without penalty. The probe is
not a gate — it records user self-report only. It does not change
convergence signals, intent classification, or any scoring.

Default is OFF. Exploratory sessions never probe. See
`agents/socratic_mentor_agent.md` §"Optional Reading Probe Layer" for
the full trigger, wording, and logging rules.

4.2 deep-research/SKILL.md

In the "Socratic Mode" section, append:

### Opt-in Reading Probe (v3.5.1)

Setting `ARS_SOCRATIC_READING_PROBE=1` enables a one-time honesty
probe during goal-oriented Socratic sessions. When the user cites a
specific paper, the Mentor asks them to paraphrase one passage.
Decline is logged without penalty. Default OFF. See
`agents/socratic_mentor_agent.md` §"Optional Reading Probe Layer".

In the INSIGHT Compilation Output template (in the agent's own file, not SKILL.md — SKILL.md only references the template location), add the "Reading Probe Outcomes" subsection slot so the layout is documented in one canonical place.

4.3 academic-pipeline/references/process_summary_protocol.md

In the Self-Reflection Report structure list, add:

- **Reading Probe Outcomes (if present)** — transcribes the
  "Reading Probe Outcomes" subsection from the Research Plan Summary
  verbatim, with a one-line note that the AI did not verify paraphrase
  accuracy. If the Research Plan Summary has no such subsection,
  this item is omitted entirely (no "not applicable" noise).

5. Tests

TDD entry: scripts/test_reading_probe_lint.py. Style matches existing test_check_compliance_report.py and _skill_lint.py — file-content lints, not runtime agent tests.

5.1 Test list (failing-first order)

  1. test_mentor_file_has_probe_section Assert socratic_mentor_agent.md contains heading "Optional Reading Probe Layer" and required subsections (activation, candidate tracking, wording, response handling, banned phrases, summary subsection).

  2. test_env_var_name_consistent_across_repo Grep for ARS_SOCRATIC_READING_PROBE across agent, protocol, SKILL, README. All occurrences use the exact same string. No typos, no case drift, no alternate names.

  3. test_probe_gated_by_goal_oriented Assert Mentor file contains explicit text stating the probe activates only in goal-oriented intent, and there is NO instruction elsewhere that activates it in exploratory mode.

  4. test_decline_is_zero_penalty Assert Mentor file's probe section contains the "no penalty" clause and explicitly excludes decline from Persistent-Agreement / Conflict-Avoidance / Premature-Convergence / convergence signals / intent classification.

  5. test_probe_tag_format Assert the [READING-PROBE: ...] tag regex is defined consistently in the agent file and referenced identically in process_summary_protocol.md pickup rules.

  6. test_no_probe_in_scoring_files Grep all scoring / rubric / convergence files (collaboration_depth_rubric.md, compliance_report.schema.json, convergence-signal blocks in socratic_mentor_agent.md) for reading_probe or READING-PROBE. Expected: zero matches. Prevents accidental regression where the probe becomes a gate.

  7. test_banned_praise_phrases Assert the probe section's banned-phrases list contains the exact strings: "correct", "right", "wrong", "good answer", "well said", "make sure", "verify", "prove". The word "check" is intentionally excluded from the banned list because it has non-evaluative uses in the surrounding section (e.g., "activation conditions hold").

  8. test_version_bump_sweep — OBSOLETE (2026-05-04) Spec at draft time assumed extending a pinned-fixture in check_version_consistency.py. PR #33 (2026-04-22) generalised that script to dynamic CHANGELOG-driven version comparison, removing the per-version pinned-fixture pattern. Adding v3.5.1 pinned strings would re-introduce the technical debt the dynamic redesign retired. Skip; the dynamic guard already covers this regression class.

5.2 Test execution

Part of the existing suite invocation (pytest scripts/). No new CI wiring needed — check_version_consistency.py guard already triggers on PR.

5.3 What tests do NOT cover

6. Release plan

6.1 Branch & PR

6.2 Version bump

The 7 bump sites in this repo (grep targets for check_version_consistency.py):

  1. CHANGELOG.md — new ## [3.5.1] heading + entry
  2. README.md + README.zh-TW.md — "Version Info" / "版本資訊" section
  3. .claude/CLAUDE.md — Skills Overview table version column and "Version Info" footer
  4. deep-research/SKILL.md — frontmatter version: field
  5. deep-research/references/changelog.md — skill-level entry
  6. docs/ARCHITECTURE.md — any mermaid/table row citing the suite or deep-research version
  7. Any script docstring or spec that pins a version string (lint script catches these)

Not every change touches all seven; the consistency check fails the PR if any site drifts.

6.3 Pre-push public-repo checks

This is a public repo. Before git push:

6.4 CHANGELOG entry shape

## [3.5.1] — 2026-04-22

### Added
- **Opt-in Socratic reading-check probe.** When
  `ARS_SOCRATIC_READING_PROBE=1` is set, the Socratic Mentor fires a
  one-time honesty probe during goal-oriented sessions where the user
  has cited a specific paper. The probe asks the user to paraphrase
  one passage. Decline is logged without penalty. Outcome is recorded
  in the Research Plan Summary and flows into the Stage 6 AI
  Self-Reflection Report when the pipeline continues. Default OFF.
  Roadmap slot: v3.7.3. See
  `deep-research/agents/socratic_mentor_agent.md` §"Optional Reading
  Probe Layer".

### Changed
- `deep-research/SKILL.md`, `deep-research/references/socratic_mode_protocol.md`,
  `academic-pipeline/references/process_summary_protocol.md` — aligned
  text updates for the new probe section. No behaviour change when the
  env var is unset.

### Version
- Suite: 3.5.0 → 3.5.1 (patch; opt-in, default OFF, no breaking change)
- `deep-research` skill: 2.9.0 → 2.9.1

6.5 ROADMAP update (memory-sync)

After PR merges, update the private roadmap §3.7.3: mark shipped, cite v3.5.1, note that the original "sub-mode + soft-nudge" dual-track was collapsed to Mentor-only.

6.6 Post-merge verification

7. Risks & mitigations

Risk Likelihood Impact Mitigation
Env var set but intent detector misclassifies user as exploratory → probe never fires Medium Low Mentor documents the gate; user can read Self-Reflection Report and see probe_status=not_fired_exploratory_mode. Troubleshootable.
User games the probe by citing a paper they did not read, then offering plausible paraphrase High Low Feature is not a gate; accuracy is for human reader to judge. "Gaming succeeds but yields no benefit" is acceptable.
Probe becomes de-facto sycophancy hook (user answers well → Mentor praises) Medium High Banned-phrases list in prompt; test_banned_praise_phrases lint; post-merge manual calibration.
Future v3.6.1 Material Passport reset-boundary work wants probe data cross-stage High Low Inline storage now; when Passport schema evolves in v3.6.1, migrate then. Not now.
Probe tag format drifts between Mentor emission and Stage 6 pickup Medium Medium test_probe_tag_format lints both sides from one regex source of truth.
User runs Socratic-only session (no pipeline), expects probe data somewhere formal Low Low Research Plan Summary is the formal output — probe subsection is right there. Acceptable.
Affiliated-institution name or internal topic accidentally used as probe example in prompt / CHANGELOG Low High Pre-push PII diff review (§6.3).

8. Success criteria

  1. All 8 lint tests (§5.1) pass.
  2. With ARS_SOCRATIC_READING_PROBE unset, deep-research/agents/socratic_mentor_agent.md behaviour is byte-identical to v3.5.0 for all sections outside the new probe layer. Verified by diff.
  3. scripts/check_version_consistency.py passes across all 7 bump sites.
  4. PR diff has zero PII / internal-URL hits on manual review.
  5. CHANGELOG, README (EN + zh-TW), .claude/CLAUDE.md, ROADMAP memory-sync entry all synchronized.
  6. Single PR, reviewable in one sitting (< 400 lines net addition).
  7. /simplify review run before marking PR ready.
  8. Manual calibration: one goal-oriented probe session + one exploratory session, both transcripts committed to PR description or linked calibration log.

9. Decision log

Decision Rationale
Probe target is user-cited papers, not Phase 2 bibliography_agent output Mentor lifecycle ends before Phase 2. Extending Mentor across phases violates "minimum surface" and contradicts mode boundaries. User-cited papers are the natural artifact Mentor has access to.
Inline storage in Research Plan Summary, not Material Passport Schema change is out of scope for a patch. Socratic-only sessions don't produce Passport anyway. Self-Reflection Report already reads Mentor output.
Goal-oriented only Exploratory sessions assume the user has not done homework; probe premise fails. Cleaner than trying to gate probe on "only if user cites" because both conditions end up equivalent in practice, and goal-oriented-only better matches the roadmap intent.
Single probe per session Roadmap language: "paraphrase ONE specific paragraph from ONE paper". Multiple probes extend pipeline and violate "non-gate" stance (repeated probes feel like interrogation).
Mentor does not verify paraphrase accuracy Content-check requires the actual paper in context. Out of scope. Feature value comes from the presence/absence/quality of response being visible to later reviewers, not from AI judgement.
Env var name ARS_SOCRATIC_READING_PROBE Matches ARS_* prefix convention (ARS_CROSS_MODEL). Explicit about scope (socratic) and purpose (reading).
Version 3.5.1 patch, not 3.6.0 minor Opt-in, default OFF, zero breaking change. Patch semver is correct. v3.6 slot reserved for long-running-agent harness themes (§3.6.1-3.6.4).
Roadmap label "v3.7.3" kept in memory-sync notes, actual release tag is v3.5.1 Roadmap slot names are organisational markers, not release tags. Decoupling preserves both: roadmap stays readable, semver stays correct.
PR not bundled with D (sprint contract) Sprint contract is large (1-2 days) and touches orchestrator. Bundling would create a PR too large to review, and rollback granularity would suffer.

10. References