← Về thư mục
📄 / / profiles / r-and-d / raw_evidence / OpenAI4S / openai4s / storage / README.md

Storage repositories

中文说明

The domain repositories behind Store live here. Store owns the single SQLite connection, the schema and migrations, the query guard, the re-entrant lock, the cached facade generation, and the compatibility API; every repository in this package is handed that connection and that lock, and none of them opens a database of its own.

Where this fits

The outer loop writes its canonical action ledger and its execution attempts here. Web and CLI projections go through the same Store facade to persist frames, messages, Cells, Artifacts, permissions, plans, delegation, kernel-generation identities, checkpoints, and recovery events. Host services read narrow repository projections for data, policy, session control, Skills, connectors, and progress.

A SQLite transaction can make a defined set of rows atomic. It cannot stretch across SQLite, workspace files, content-addressed blobs, a running Python/R namespace, remote compute, and WebSocket events all at once. That is why the repositories keep three things apart: append-only history, mutable materialized projections, and the best-effort binding to files that live outside the database.

Files

File Responsibility
__init__.py Re-exports the repository classes that the Store composition facade wires up.
actions.py The canonical action ledger. Groups and events are immutable once written, and a reducer replays them to rebuild provider history, tool batches, and Cell observations. An execution attempt is allocated before the work starts; each lifecycle milestone is then filled exactly once, so a finished attempt can never be rewritten.
activation.py Activates one checkpoint branch in a single transaction: selected branch, session capabilities, conversation permission rules, visible Artifact heads, checkpoint state, and the selected Python environment all move together. A crash cannot publish a branch id whose surrounding policy and data still describe another branch.
agents.py Named specialist Agent profiles, with their skill and connector overrides stored as JSON.
annotations.py Figure-review pins for one session/Artifact context: normalized coordinates, body, ordinal, status transitions, deletion. Allocating the ordinal and inserting the row stay in one critical section, so two concurrent pins cannot land on the same number. It also owns the reservation that admits a pin into exactly one message. reserve is a single frame-scoped UPDATE off open, so of two racing requests exactly one claims a given row and the loser sees it absent rather than silently shares it; release and mark_sent settle only their own reservation_id, and only within their own frame, because a reservation id travels in a response and is therefore a value a caller holds. Every predicate carries its expectation instead of checking first and writing after — Store's re-entrant lock is per instance and the daemon runs more than one instance against one file, so holding it was never a claim about the database. annotation_admissions is the durable record of what became of each reservation, and a snapshot that captured a pin mid-flight is settled back to open with no holder on restore, since the request that held it does not survive the gap.
artifact_observations.py Durable per-producer capture observations, separate from version identity. When a Stage 1 capture reuses an equal-checksum head, this row records the new producing Cell, frame, environment/source and input-version ids without rewriting the version's original provenance. Cursors and deltas are root/project scoped; the read used for a turn has no unscoped fallback. Observations remain local-only in Stage 1 and are not yet part of Session package/share/export.
artifacts.py Everything the system knows about a produced file. An Artifact is a stable identity; each changed write appends a version that records where the bytes are, the environment snapshot they were made under, and the Cell that produced them. browse_artifacts pages one project by (created_at, artifact_id) with escaped filename LIKE and team visibility applied before LIMIT. The Stage 1 opt-in may instead reuse an identical current head while writing an observation, so version means bytes and never merely “captured again”. Lineage edges connect an output version back to the input versions it was derived from, which is what lets the UI answer "what produced this figure, and what fed it" without reading the workspace. A version derived from a public database also carries its retrieval envelope — where the data came from and when — stored as canonical JSON so two versions of the same retrieval compare equal as text rather than by key order; the projection that decides how much of it a client sees is the gateway's. Rows written before that column existed keep NULL rather than a backfilled retrieval this installation never made, for the same reason the environment migration did not invent one. The environment snapshot's content address includes the interpreter, the environment name and the kernel generation: without the generation, a kernel restarted into an unchanged environment produced an identical id, and every artifact from the second lifetime pointed at a row still naming the first. Restore records, priority, and the latest head are kept here as well.
auto_mode.py Stage 2's SQLite source of truth for Auto Run selections, runs, canonical events, result audits and findings, repair ownership/action-group bindings, and permission assessments. Every fact is bound to root, branch, turn, and execution; writes use immediate transactions and request digests for restart-safe idempotency. Checkpoint cursors project logical branch history without deleting the physical audit trail. Imported history is identity-remapped, quarantined, inert, and never imports an approval grant. The repository only persists and projects facts in Stage 2; it does not invoke a Reviewer, Repair Agent, or Permission Guardian. Version 29 adds additive auto_mode_budget_state / auto_mode_budget_reservations (and an append-only budget event log): one BEGIN IMMEDIATE reserve per admission, no optimistic refund after an action may have started, and old runs without a budget row project legacy=true.
branch_projection.py Rebuilds the logical, branch-aware history from immutable checkpoint cursors plus the rows written after the current head. Physical append-only history is never deleted to make a branch read correctly.
capabilities.py Durable capability enablement. Precedence is the same for every kind of capability (session over project over global; an absent row means enabled), a materialized table answers the fast policy check, and each change also appends to an event table. Bootstrap manifests live here too.
checkpoint_state.py The session-domain state that has to travel with a branch: plans, reviewer activity/settings/annotations, and project memories, captured as canonical JSON under a SHA-256 integrity digest. Imported state is validated and quarantined rather than trusted, identities are remapped, and only the verified scope is restored.
compute_jobs.py Remote work outlives the process that submitted it — an SSH job keeps running under nohup, a BYOC sandbox keeps billing — so two tables answer two different questions: compute_jobs for where a job is now and the handles needed to reach it again, compute_job_events for how it got there, append-only and monotonically sequenced. A status alone cannot tell "we never submitted" from "we submitted and lost the response before recording the handle", and those demand opposite actions on restart. The idempotency key is written before the submit is attempted, so a crash anywhere in the submit path still leaves a row to find, and reconciliation looks the job up by that key rather than assuming — the difference between recovering a job and paying for it twice. owner_key is why a restart does not hand one session's live jobs to another.
connectors.py What an MCP server is configured to be. The command, its arguments, and its environment go in as JSON and are decoded back out on read, next to the enabled flag and the display name. Starting one of these servers is the MCP client's job, not this table's.
datapro_index.py Lossless local indexing for successful DataPro searches. It stores the complete canonical redacted result envelope, verifies source and indexed leaf manifests match, projects top-level and nested dataset items without collapsing repeated occurrences, and provides literal scoped search with no field whitelist or silent truncation.
delegation.py The bounded sub-Agent tree, made durable so it still reads correctly after a restart. Child slots are reserved against the session spawn cap inside one immediate transaction and released when the run ends, so a fanout cannot quietly overrun its budget. Child lifecycle, results, and steering messages are stored alongside it. When a request identity is present, reserve is idempotent against delegation_requests and charges the spawn budget only for a newly inserted request.
delegation_attempts.py Durable request identity separate from attempt identity. delegation_requests is unique on (root_frame_id, parent_action_group_id, native_call_id); delegation_attempts is unique on (request_id, attempt_no). Same digest reuses the original child; a digest mismatch is a 409 conflict. Restore never creates an attempt — only an explicit continue increments attempt_no. The constructor is passive; DDL is numbered migration 30.
deletion.py Deletes every SQLite aggregate owned by one session or one project in a single transaction. Most of the compatibility schema has no foreign keys (DataPro entries are the narrow exception), so each owned table is named explicitly. It returns the filesystem paths that are now cleanup candidates and unlinks nothing itself.
delivery.py Recoverable completion publication. It commits the final assistant message and its canonical, hash-bound Artifact manifest in one immediate transaction under a root/branch-scoped idempotency key. A conflicting reuse fails closed; queryable committed rows prove the durable message-and-manifest fact, while the idempotent published transition records the best-effort emit marker together with message metadata. Stage 1 recovers the message through REST reopen and does not consume this set to drive automatic socket re-emission; ordinary bounded live-turn WS sequence replay is separate. Session-package import binds a verified remapped manifest to an existing safe pending message in one transaction. Corrupt message/manifest hashes or relations are rejected on read rather than projected as delivered.
session_imports.py Creates the safe placeholder project, root frame, and Session-package mutation quarantine in one immediate transaction. Package-authored display metadata is applied only after this boundary commits, so an interrupted import cannot leave an unquarantined partial Session.
like.py The one substring LIKE pattern the repositories share: \, % and _ become literals and an embedded NUL becomes a private-use stand-in, because SQLite reads the pattern as a C string and stops at the NUL — three byte-identical private copies meant the NUL fix in the DataPro index never reached project or filename search, so lab\x00zzz matched every lab.
frames.py The session spine: projects, the frame hierarchy and the scope a frame resolves to, the messages a user sees, activity steps, token counters, and frame search. The Cell execution log lives here too, and each logged Cell carries a visibility and a replay policy — that is how a protocol-only Cell can stay in the audit record while the read-only Notebook keeps it out of view. Message reads page from either end: oldest-first by default, and newest-first from a before_seq keyset cursor for the direction a long conversation actually needs — an ascending limit returned the oldest page, so a 640-message session opened on messages 0–299 with the newest 340 absent. A cursor rather than an offset because ordering newest-first is what makes an offset skew on every arriving message; seq is already monotonic per root frame, so it needs no tiebreaker. A Cell capture step ("Saving ...") written before 0.3.0 labelled every Cell with the Python kernel, R included; a row that never recorded its language is read through the Cell that produced its artifact versions, on the step read every consumer shares, and the stored row is left as written.
kernels.py Durable UUID identities for Python and R kernel generations, plus manifests, owner and process metadata, ordinals, activity, and terminal state. These rows describe a process lifecycle; they never claim that a live namespace was serialized.
model_capability_receipts.py Exact receipts from an explicit model-profile probe. Bound to profile_id + revision + endpoint SHA-256 + model + wire + probe_version. Three-state fields (true / false / unknown) never store timeout, auth failure, or 5xx as false. Only a current revision's positive evidence is adopted as a get_model_capabilities() overlay.
memories.py Project-scoped long-term memory, and a deliberately small table. Add, correct, delete, and the block projection callers read instead of grouping the rows themselves. resolve returns the effective set and what inheritance did to produce it — a project's own memories first, then the global ones no block of its own overrides — because a merged list alone cannot tell an overridden memory from one that was never saved, and order is priority here: the budget truncates from the end, so the project's own survive a full context. The cross-project view is something a caller has to ask for by name. Expiry withholds a memory from injection and never deletes it, so the per-scope cap counts live rows — counting stored rows meant an expired scope refused every new write over memories the pane already showed as omitted — with a separate, larger ceiling on what is stored so the first rule cannot let a scope grow without bound.
metadata.py Five small repositories grouped in one module: project notes, folders, managed endpoint metadata, compaction archives, and the Host-call audit log. Credential reads are derivable and are never duplicated into the log; secret-bearing RPCs stay auditable by method name, but their raw arguments do not cross the persistence boundary.
migrations.py Versioned, transactional schema migrations. There used to be no version marker at all: every open re-probed each table and issued ALTER TABLE ADD COLUMN under a bare except OperationalError: pass, which made a genuinely failed ALTER indistinguishable from the benign duplicate-column of a re-run — so the process continued against a schema missing a column it believed it had. Version 1 is defined as "the legacy pass ran to completion" — it is idempotent by predicate, adding only absent columns and guarding every backfill with a WHERE that selects only the rows still needing it, so re-running it on any database converges to the same shape; from 2 on, steps are numbered, applied exactly once, and recorded with a checksum. Version 24 installs capture observations and completion deliveries in the same migration transaction. Version 32 adds the reversible Artifact browse index (project_id, created_at DESC, artifact_id DESC), and only that step creates it: Store commits its base schema script before migrating, so an object added there lands in an older database ahead of the pre-upgrade copy and survives a rolled-back upgrade. The whole set rides one explicit BEGIN, and that is the load-bearing part: pysqlite opens a transaction before DML but not before DDL, so a bare ALTER runs in autocommit and survives the ROLLBACK meant to undo it. The invariant bought is that a database is fully at N or fully at N-1, which is what makes an interrupted upgrade recoverable by running again. PRAGMA integrity_check gates the attempt, and a pre-upgrade copy is taken through SQLite's own backup API — kept on failure as the operator's escape hatch, removed once the upgrade commits.
permissions.py Resolves scoped allow/ask/deny rules, seeds the local defaults, persists approval requests and events, and expires pending decisions that pass their deadline. Versioned security replacements can revoke an unsafe legacy default without restoring deleted or stricter rules; v4 changes the old silent skills_edit allow to ask because Skill documents and sidecars feed future execution. A restart-continuation grant is narrowly bound and consumed atomically, once.
plans.py Structured plans for a frame, with per-step status and notes. The status set is enforced in Python rather than as a SQL CHECK: plans is created with CREATE TABLE IF NOT EXISTS, so a constraint added today would bind new databases and silently not existing ones — a guard protecting whoever needs it least. Status moves by compare-and-set, and rows a dead process left executing are reconciled at startup beside the kernel generations: paused rather than failed, because the steps that completed did complete. get_by_frame answers with the newest plan that has not been discarded, which is why one row stuck executing used to shadow every later draft for that session.
recovery.py The recovery journal. Every attempt and every repair appends an ordered entry, so a failed or partial restore stays inspectable after the daemon restarts.
settings.py One key/value table, with two structured views built on top of it. Model profiles are stored as a JSON list, and mutate_model_profiles reads, edits, and writes them back under the Store lock so a concurrent edit cannot lose a profile. Message feedback is keyed by frame.
skills.py Content-addressed Skill packages: immutable blobs, files, and manifests; an installation pointer that only moves under optimistic concurrency; and an append-only history of activation and deactivation whose metadata can prove the trusted authoring boundary for a retained version. Package validation and materialization belong to skills_loader/versions.py, not here.
shares.py The shares table for web sharing: one durable row per share holding its lifecycle status (publishing/ready/failed/revoked), current snapshot id, bundle hash, and optional expiry. A partial unique index enforces at most one active share per session; the filesystem publish and lease GC live in server/share_service.py.
snapshots.py Two halves. WorkspaceCAS is a stdlib content-addressed store for workspace bytes, with restore previews, conflict detection, materialize/diff_trees for child-private scratch (never deletes Artifact versions), and release of trees plus the blobs nothing else shares; it is told which trees to keep. SessionSnapshotRepository holds the session branch and checkpoint envelopes, forks, and operation journals, and it is the half that discovers the retained trees, by querying the checkpoint rows. Neither half reads or writes the researcher's Git repository.
governance.py Team-mode governance (M2): project_members, invites, usage_ledger, quotas. Invites store only sha256 digests and redeem through a single UPDATE whose WHERE carries the liveness predicate, so two racing redemptions cannot both win. The usage ledger is append-only and aggregates in SQL; quota verdicts are computed here beside the ledger they read, over sliding day/week/month windows, and come back as a raised QuotaExceeded (code QUOTA_EXCEEDED) rather than rows a caller could misread. All four tables sit on the host.query denylist.
leases.py A session's hold on a resource and the two clocks that end it, plus the session↔workload binding that makes a cluster kernel findable from the chat session in both directions. The load-bearing column is last_active_at, and it is load-bearing in one direction only: a worker being alive is not a user being present, so nothing in the transport, the watchdog, or the reclaimer's own probe writes it — only a user executing something does. Getting that backwards yields a lease that renews itself forever, which is the same as having no lease at all while looking like diligence.
user_keys.py Per-user LLM credentials, by reference (M4-1). The secret goes through the same SecretBroker as every other credential here, so this table copied off the machine names slots it cannot open. One row per (user, provider), because a lab with its own Anthropic account and the group's OpenAI key is the ordinary arrangement, and a single-key design would force that user to choose between paying for everything and paying for nothing. Absence of a row is the fallback — which is what makes the feature additive rather than a migration (INV-1).
workloads.py Durable workloads and allocations, converted to and from the orchestration dataclasses so the reconciler never sees a row and this module never sees a backend. The partial unique index on allocations(workload_id) is the whole of INV-3 — the Python check beside it exists for a readable error, but the failure it prevents happens between a check and an insert: a tick that dies right there, restarts, and submits a second job holding a second GPU. submission_token is UNIQUE for the same class of reason: INV-8's "does anything carry this token" only means something if a token could never be minted twice.
team.py Team-mode identity (OPENAI4S_TEAM_MODE): the users, auth_sessions, and team_audit_log tables. Passwords are PBKDF2-HMAC-SHA256 with a per-user salt and the iteration count recorded on the row, so a future cost bump re-hashes lazily instead of invalidating accounts; login cookies store only the sha256 of the token; disabling a user or resetting a password revokes every live session. All three tables sit in the host.query denylist — agent SQL never sees credential material.

Durable model

Consistency and security boundaries