← Về thư mục
📄 / / profiles / r-and-d / raw_evidence / OpenAI4S / tests / README.md

Tests

中文说明

The offline correctness gate for OpenAI4S. uv run pytest runs every module here against deterministic fakes, and it has to stay green: the provider-neutral agent engine, the Host services, the persistent Python/R kernel protocol, the repositories, the security boundaries, the tools, the Skills, and the Web composition. Reusable scenarios and scored evaluations are a separate layer and live in ../harness/.

Offline contract

Support and smoke files

File Responsibility
test_windows_launcher.ps1 Native PowerShell 5.1 launcher contracts for user distribution selection, preserved data ownership and explicit preparation identity. Runs in Windows CI without entering WSL.
test_wsl_parity.py Offline regressions for WSL adapters, DPAPI transport, relocated scientific commands, default UI assets and failed-install preservation.
conftest.py Sets up import paths, gives every test its own data directory, configures a fake LLM provider and key, cleans up the Store afterwards, and holds the shared pytest fixtures. It also pins the posture the suite is supposed to measure rather than inherit from whoever is running it: OPENAI4S_UNATTENDED_APPROVAL=deny, OPENAI4S_SECRET_STORE=plaintext (so no test writes to the developer's login keychain), OPENAI4S_NOTEBOOK_REPL=0, the share and MCP-deadline variables cleared out of any git-ignored .env, and the telemetry endpoint aimed at https://127.0.0.1:1/ — not hypothetical hygiene, that one: a benchmark case granted consent to itself and really POSTed a fresh install id to the live endpoint from every developer machine and CI runner. Finally it installs the response-shape recorder when OPENAI4S_CAPTURE_SCHEMAS is set, and pauses it around every stubbed_backend test.
_ports.py Race-free port allocation for the tests that bind a real gateway ThreadingHTTPServer. The probe-then-rebind _free_port pattern loses its port to a concurrent xdist worker between the probe's close() and the server's bind() — the Frozen-shapes CI job failed exactly that way through test_team_auth_routes — so bound_gateway_server binds port 0 once, keeps the socket, and lets the caller attach the real handler class only after make_handler has snapshotted the true cfg.port for the Host allowlist. Tests that put a port into a Config without ever binding it keep their local _free_port: an unbound number has no race to lose.
_envs.py Real interpreter prefixes for the environment-binding tests. A conda-shaped fixture used to be one bare bin/python symlink to sys.executable, which stopped self-reporting the fixture's own prefix: a copied python-build-standalone launcher loses its runtime tree on Linux 3.10, and a bare symlink resolves to the base interpreter on newer builds. real_python_prefix builds a pip-free venv — pyvenv.cfg plus a link — so the kernel's sys.executable names the selected env on every interpreter in the CI matrix, and the incantation lives once rather than in each module that builds an env.
browser_auth.mjs The credential every browser harness presents; all five that use the loopback-token flow import authenticate from here. The daemon requires an access token by default, including on loopback, so a harness that navigates to / with nothing would fail every check on a 401 and report it as a product failure. It normally reads the token from the same file the CLI does; a security-sensitive harness may instead pass an already validated, fd-captured token so authentication does not reopen its environment/default-file lookup. Either path logs in the way a user does — open the printed ?token= URL once and let the 303 set the cookie — so the bootstrap and cookie hand-off are exercised rather than routed around. It also exports minimalChildEnvironment (a pinned daemon environment that the developer's shell cannot drive) and boundedLogCollector (the tail of a child's stream, so a failure can quote the daemon) for harnesses that own a disposable daemon.
browser_p1_controls.mjs The P1-A/P1-B controls, in a real browser. The three existing browser_*.mjs files scored zero keyword hits on all eight — before_seq, newest_first, chip, profile, attachment, delegation, steer, memory — so the group rested on one manual walkthrough taken 43 commits earlier. Paging is checked by calling the client's own fetch helpers against the live route (the defect is which query string goes on the wire, which no DOM assertion can see); the rest call the real render functions into a real DOM. Fixtures are created by the test, never assumed from the developer's database. Also covers the truthful delegate step card (D9: completed/blocked/partial/max_turns/cancelled/error shapes, no raw-JSON default), the live delegation_child_event merge (D8), and the Executed-code surface (D10: child-frame navigator, failed cell rendered, execution-history labeling distinct from Artifacts, and the sources.zip entry in the download menu).
browser_sandbox_preview.mjs Real Preact Viewer acceptance on a disposable daemon, in both loopback directions: script and sibling-image rendering, parent isolation, cross-session and app-origin grant replay refusals, exact versions, inert fallback, API CSP refusal with both origins authenticated, and real Ketcher edit/save. Its daemon runs under the pinned minimal environment and its log tail is attached to the SUMMARY on failure; the Ketcher phase waits for the wrapper's deferred reload to land and refuses an (unchanged) save.
browser_smoke.mjs Drives a real browser against the running Gateway UI and its streamed interaction paths. You invoke it separately from local pytest; the normal PR CI workflow runs it automatically.
browser_stage0_acceptance.mjs Machine-readable Stage 0 baseline against an explicitly disposable daemon. Before reading a token it refuses port 8760, non-loopback or credential-bearing URLs, the default data directory, an OPENAI4S_TOKEN override, and any data-dir daemon state whose host, port, or pidfile does not match. Cross-origin HTTP, API, and WebSocket requests are blocked and counted; every execute-refusal probe is preceded by a frame kernel state of repl_enabled=false; and the Artifact link comes from the production completion projector plus production UI renderer. The summary carries only one-way hashes for its project/frame/Artifact identities. finally deletes that exact project, proves it is absent from the production project-list response, and requires the Artifact readback to return 404. It truthfully reports zero Agent requests while marking provider-call count as unobservable at this browser boundary.
browser_stage1_trusted_delivery.mjs One-command local browser acceptance for Stage 1 trusted delivery. It owns a random-loopback disposable daemon and metadata-only standard-readiness fixture, uses the production completion and stored-message renderers, and checks three passes over 100 exact-version links for HTTP 200 and SHA-256. It also proves same-head dedup retains the capture observation, delegated code/native producers reopen with their real child frame and no fabricated Notebook link, an old link stays immutable after a new head, zero Agent/LLM/external-network work occurs, and cleanup removes precisely its fixture.
browser_team_mode.mjs Team mode as a person meets it, which is the half no socket-level test can see: the anonymous redirect to /login, the identity chip, the admin governance panel rendering live users/usage/quotas/invites/audit, a member refused that panel and not even shown its button, a guest invite redeeming into an account whose entire world is /replay, and the cross-user refusals (a non-participant's project, artifact bytes through /preview/). The only gate that needs its own daemon: OPENAI4S_TEAM_MODE=1 plus two seeded accounts, whose passwords arrive in OPENAI4S_TEAM_ADMIN_PW / OPENAI4S_TEAM_MEMBER_PW because a password must not ride argv.
scientific_renderers_smoke.cjs A dependency-light Node runner that checks the UMD scientific artifact parsers in the Web UI against their contract. Run by hand with node tests/scientific_renderers_smoke.cjs. The existing browser-smoke CI job runs it as a step (not a new job) before Playwright or the daemon.
webui-contract.md Machine inventory of the three E2E compatibility surfaces the frontend rewrite must keep: bare window globals referenced from page.evaluate, S field read/write (including nested writes), and DOM selectors. Generated by scripts/extract_webui_contract.mjs; do not edit by hand.

None of the browser_*.mjs files runs under uv run pytest; install the integrity-locked driver with npm ci --ignore-scripts and ./node_modules/.bin/playwright install <engine> on Node 20 or newer — the package's own engines floor is >=18 for the published Skill installer, but the pinned Playwright devDependency refuses older runtimes and npm ci only warns. Most need a separately started daemon. Stage 1 is the deliberate exception: node tests/browser_stage1_trusted_delivery.mjs owns its random-port disposable daemon and local fixture. The Stage 0 pack is intentionally stricter than the other externally managed browser probes: start a throwaway daemon with its own temporary OPENAI4S_DATA_DIR, OPENAI4S_NOTEBOOK_REPL=0, and a non-8760 loopback port, then run env -u OPENAI4S_TOKEN OPENAI4S_BROWSER_DISPOSABLE=1 OPENAI4S_BROWSER_URL=http://127.0.0.1:<disposable-port>/ OPENAI4S_DATA_DIR=<same-temp-dir> node tests/browser_stage0_acceptance.mjs; consume only its single SUMMARY line. The daemon must still own matching daemon.json, openai4s.pid, and owner-only access-token files in that directory. Its redaction/schema/binding/production-projection self-check is OPENAI4S_STAGE0_SELF_TEST=1 node tests/browser_stage0_acceptance.mjs and does not load Playwright or contact a daemon. CI runs the workbench walk, the admission-fault case and the P1 controls under Chromium — depth, not breadth — and browser_matrix.mjs in all three engines. The same browser-smoke job also runs scientific_renderers_smoke.cjs as a Node-only step before the daemon.

Test modules

Every test_*.py file is listed below, with the reason the module exists rather than an inventory of its cases. Most modules also carry extra regression cases around failure, restart, redaction, and concurrency that the line does not name.

File Why you would open this file
test_action_ledger_repository.py Four assertions about what the Ledger storage refuses to do: overwrite a group or an event, split an atomic tool group, or rewrite a terminal execution attempt. The column migration is additive.
test_action_ledger_runtime.py The other half of the Ledger: what the running loop actually writes into it. Arguments are redacted before they land, a branch inherits only its checkpointed parent prefix, an interrupted native group is closed atomically, and the history rebuilds after the store is reopened.
test_action_routing_eval.py Less an assertion suite than a scorer. It replays offline routing fixtures through the Tool/Code/Finalize router and emits a failure and confusion report for a human to review.
test_action_timeline_service.py Open this when the Timeline shows a user something it should not. The projection is public, so the tests are mostly about what it withholds — raw payloads, secrets, provider IDs — and about what it must not withhold, such as an earlier tool failure.
test_attempt_terminal_state.py Open this when an execution attempt's terminal state disagrees with what the Cell did. Both the CLI and Web writers share one classifier: a returned result is failed, interrupted or completed however often its traceback says timeout, and only a watchdog TimeoutError is timed_out.
test_attention_view.py Cross-session "needs attention" aggregation. A read-only GET unions running/queued executions, pending approvals, recoverable failures, view-only/blocked sessions, and live/unknown remote compute into a fixed card shape; team visibility is applied before aggregation, sort, and limit; the server returns no arbitrary URL. Six fixtures produce one card each, cross-user leakage is zero, the cursor is bound to caller scope, a frozen 50-Session page is p95 < 200ms, and GET does not spawn a kernel, call a provider, retry, approve, or harvest.
test_actions.py Reply parsing, shared by both outer loops. The precedence rules are pinned here: a native call beats a code fence, an unclosed fence is never executable, and only the first Cell in document order runs.
test_admet_genetic.py The bundled ADMET genetic Skill: that it is discoverable, that its helpers aggregate deterministically, and that the dashboard it emits escapes script and HTML delimiters.
test_agent.py The broadest module in the suite — the offline outer loop end to end. Code-as-Action cycles, an R Cell that soft-fails into an observation when R is absent, token estimation, compaction that keeps a Cell and its observation in one atomic segment, and the delegation caps. A change that breaks the loop usually breaks here first.
test_agent_control.py Native tool batches under stress. Every call in a batch reaches a result even when one invocation fails or the turn is cancelled mid-batch. Independent read-only calls may run in parallel, but a mutating call is a barrier for everything after it.
test_agent_engine.py AgentEngine on its own, driven through fake ports. The last test is the point of the module: the engine must not import runtime infrastructure. The rest pin routing precedence, replay-safe history grouping, and the exact moments at which cancellation wins.
test_agent_hybrid.py Two tests on the hybrid Agent facade: a native call beats code and its canonical history survives into the next turn, and a reused agent clears the previous submission before a new task.
test_agent_profile_repository.py Named agent profiles in SQLite, mostly the awkward parts — legacy serialization of falsy values, JSON decoding edges in listings, and the read-then-write gap an upsert has to survive.
test_agent_progress_circuit.py Generic Agent no-progress circuit: same-action / malformed / similar-error / long-text thresholds, different args and reasoning inserts that must not false-trip, and restart/compaction rebuild from the Action Ledger rather than a live RunState. A trip is no_progress, never completion.
test_agent_runtime.py The local adapters between the pure engine and real infrastructure. Two rules dominate: a native parse error or a limit must never dispatch, and it must never drop a tool result either. Compaction expands its tail to keep an assistant tool group atomic, and a circuit breaker stops repeated low-yield compaction.
test_compaction_summary_robustness.py Lane A of the long-context fix. The summary call gets a real output budget (max(8192, cfg.llm.max_tokens), env-overridable) and no temperature; an empty or truncated summary raises CompactionSummaryError instead of archiving the middle behind a placeholder the model would then believe; the summarizer input drops wire_state and caps tool arguments.
test_compaction_cli_budget.py Lane B. The CLI now hands CompactionPolicy the same provider-derived context budget and tool-schema accounting the Web loop already had, so a one-shot openai4s run compacts against the model's real window rather than the config default.
test_compaction_failure_breaker.py Lane C. Two consecutive compaction failures open a circuit that reopens once the context has grown 1.5×; a success clears the streak; and the real input_tokens from the previous reply calibrates the estimate (clamped to [0.5, 8]) so an under-estimate can still trigger.
test_compaction_chunked_summary.py Lane D. A long middle is summarized as bounded rolling chunks — each chat() input under 0.35× the budget, the previous handoff carried forward and excluded from the transcript JSON — the retained tail is chosen by tokens, never splitting an atomic pair, and CJK characters estimate as one token each.
test_compaction_externalize_retrieval.py Lane E. An oversized output is copied to a workspace-relative .openai4s/context/<sha256>.json the kernel can json.load, the marker names that path and nothing absolute, and the digest-only filename cannot be steered by content.
test_compaction_ledger.py Lane F. Compaction is a kind="compaction" Action Ledger group with assistant_message=None, so a pre-compaction reducer skips it and replays the full history while the new reducer keeps the head user, drops the covered groups, and inserts the handoff; the coverage bound is the last group whose messages all lie in the compacted middle; a daemon rebuild via _seed_messages restores the compacted view.
test_compaction_wave2_integration.py The seam between Lanes D and E that neither lane's own tree could cover: compact() must thread workspace into externalize_large_outputs, and the policy must thread it into compact().
test_compaction_wave3_integration.py The seam between Lanes D and F: the note the ledger restores must be byte-identical to the one compact() built in memory (shared COMPACTION_NOTE_PREFIX), and restoring an already-framed note must not double the prefix.
test_compaction_wave4_review.py What the Wave 4 review found across the integrated lanes. The durable ledger record is written only for a compaction the policy adopts; coverage is keyed by group id in the branch's spliced sequence (ordinals restart at 0 on a fork) and any misalignment covers nothing rather than the tail; a compaction group without its event is skipped, not applied as an empty note; the summary max_tokens is clamped to the model's declared cap and a truncated chunk (length/max_tokens/MAX_TOKENS/incomplete, one retry) is never padded; a zero context budget means unknown; a poisoned workspace .openai4s/context costs the kernel copy, not the compaction; the Artifact path also writes that copy; and a reopened breaker gets its full failure budget back.
test_analysis_skills.py The bundled analysis Skills are executed here, not just listed. Their data audit catches group leakage, the AUC is tie-aware, and the bootstrap is deterministic.
test_annotation_repository.py Image annotations. Three tests, and the one that matters is ordinal allocation under concurrent pins — it has to be atomic.
test_artifact_control_tools.py Artifact restore is the dangerous operation, so most of this module is about refusing it: a corrupt snapshot, an untrusted ID, a workspace that drifted since the version was cut. Restore stays behind approval and rolls back a failed store write.
test_token_mint_atomicity.py Whether two daemons starting together agree on the access token. The mint read the file, saw nothing, and finished with an unconditional os.replace — which never fails on a name already taken, so every racer won: each overwrote the file, and the re-read meant to adopt the winner's value sat outside any exclusion. Eight processes released at one barrier produced seven different tokens with only the last write on disk, so the surviving daemons authorised cookies against a value the CLI and the printed ?token= URL could no longer read. The winner is now decided by os.link onto the final name — the one publish that can only succeed once — and the content is complete before that name exists, so a loser's re-read cannot land on a half-written file the way an O_EXCL create on the final path itself would allow. The file and its directory are both fsynced, because a published name that is not directory-fsynced can vanish on a crash while its bytes survive, and 0o600 is on the inode before the first byte of the secret rather than chmod-ed after it. The deciding test spawns real processes rather than threads: the gap is between two syscalls, which the GIL hands off anyway, so a threaded version would have passed against the broken code.
test_first_run_access.py Whether a first-run user can get into their own daemon. The access token became required by default — correct, this daemon executes code — and nothing that hands a URL to a human was updated: _url() returned the bare origin and fed the browser auto-open, status, url, and the macOS .app. Every recovery path was closed too: /static/app.js is behind the same gate so the SPA cannot load and cannot offer a way in, the 401 body was JSON, and the one working URL goes to stderr which the .app redirects into a log file. A .dmg user had no in-product route to a working session. Now the printed URL carries the token (and machine callers that append paths opt out, or they build ...?token=Xhealth), and an unauthorised browser navigation gets an HTML page naming the command to run while scripts still get parseable JSON. The page embeds no credential and fetches nothing, since every asset it could reference is behind the same gate.
test_token_query_bootstrap.py How the access token is allowed to travel, driven through the real Handler. The ?token= bootstrap was written as a subtraction — anything that is not /api/v1/* or /static/* — and described in its own comment as "paths that serve the SPA shell". /preview/<id> is neither: it answers with artifact bytes, so a link carrying a token there set a durable HttpOnly cookie for the whole daemon and then served the file to whoever held the link, which is the exact leak the restriction existed to prevent. The rule is now an allowlist of the root page, the only URL the product ever prints. Separately, a token in the query string of a mutation was merely not used to authenticate: a request that also carried a valid cookie succeeded with the credential still in its URL, so the caller had no way to discover they were shipping a secret — it is refused outright now. The rest pins what must not break: the 303's Location carries no credential, the cookie is HttpOnly; SameSite=Strict; Path=/, /health stays open, and Authorization: Bearer still authenticates.
test_frame_cap_units.py Two caps that measured different things, and a refusal one tool skipped. MAX_OUTPUT caps stdout and stderr at 1,000,000 characters; _MAX_FRAME_BYTES capped the frame at 8,000,000 bytes with a comment claiming it sat above the largest legitimate frame — true only for ASCII. Measured at the documented caps: ascii 2,000,059 bytes (ok), CJK and control 12,000,059 (dropped), emoji 24,000,059. A cell obeying every documented limit lost stderr, the exception text, error_lineno, guards and usage; only stdout survived, and only via the manager's chunk backfill. The cap is now derived from MAX_OUTPUT so the two cannot drift apart again — and the derivation's first constant was wrong (6, for \uXXXX) because Python counts an astral character as one while JSON emits a twelve-byte surrogate pair; a test here caught that. Separately, secret_path_key is the hard refusal that stops a tool overwriting a credential file, write_file and edit_file declared it and web_download — the only other tool with writes_files — did not, so write_file(".env") was refused while web_download(url, ".env") was not. Asserted over every file-writing tool, so a fourth cannot be added without it.
test_message_paging_client.py Which end of a long conversation the workbench opens on. newest_first, before_seq, next_before_seq and has_earlier were built through the store, the repository and the route; app.js contained neither newest_first nor before_seq, and all four message fetches sent ?from=0&limit=300\|500 — the OLDEST page. A 640-message session opened on messages 0–299 and the work you came back for was off the end, while gateway.py carried a comment describing "the client asks for the newest page" — describing a client nobody had written. All four call sites now go through one helper, which also sorts the descending rows back into reading order (rendering them raw would show the conversation backwards, a worse bug that looks like a server fault). Records honestly that has_earlier/next_before_seq are still unread, so a session longer than one page still has unreachable older messages — it now hides the old ones rather than the recent ones.
test_model_binding_recovery.py Whether a session pinned to a model configuration can ever be sent to again. D2's 409 model_revision_unavailable — "choose one to continue" — was correct and unanswerable: both writers of model_profile_id sit past the raise, PATCH /frames/{id} allowlists name and task_summary, forking inherits the pin, ids are random so re-creating the profile does not match, and app.js had zero references to the code. Deleting a model profile permanently bricked every session bound to it — history and artifacts readable, never sendable again. Two triggers, only one involving a delete click; the other is a profile that exists whose bound revision does not. Now delete releases what pointed at it, and POST /frames/{id}/model-binding answers the rest. Deliberately its own route rather than a flag on send: the client sends model on every message, so treating that as consent would re-pin silently on every turn — the drift D2 removed. The 409 itself is unchanged and a test fails if it is softened.
test_retrieval_provenance.py A retrieved record says when it was fetched and hashes what upstream returned, so a rerun that quietly changed is distinguishable from one that did not — and that envelope reaches the artifact derived from it.
test_artifact_env_provenance.py An artifact records the environment that produced it: an R cell's artifact stops carrying a Python package list, a selected conda env is named rather than the daemon's, and an unreadable interpreter reports absence instead of borrowing this process's packages.
test_response_schemas.py The HTTP surface has a shape that can be checked. Response shapes are captured from real responses rather than written by hand, so the frozen file cannot describe a response the code does not produce; the tests pin the shape algebra (an optional field must not stay required, an observed null makes a field nullable rather than optional) and the distinction the gate exists to make — a removed guarantee is BREAKING, a new field is additive.
test_response_capture_assembly.py The capture survives being split across xdist workers. Workers atomically publish during pytest_sessionfinish, before xdist reports success, and every share names the run plus expected worker count. Assembly rejects missing, corrupt, stale, or mixed-run evidence before writing; a complete split must equal the single-process schema even for nullable numeric and optional-field observations.
test_server_errors_module.py The HTTP failure type is importable without importing the gateway. A prerequisite for decomposing Handler._api, not tidying: GatewayError sat ~5,800 lines below gateway's own imports, so a route module importing it from there failed the daemon at boot. Pins the no-cycle property in a fresh interpreter and the re-exported names ten test modules already use.
test_followup_admission.py What a session accepts while a turn is already running. A message had no text limit at all — the only bound was _MAX_JSON_BODY_BYTES, the session archive cap of 128 MiB standing in as a chat-message cap — and the message is persisted and replayed, so an 8 MiB paste is eight times the whole context window on its own and compaction cannot rescue it, because summarising it means sending it. A session pinned to a deleted model revision was refused with 409 inside the worker thread, so over HTTP it arrived as 200 with {"status": "failed"}, the same soft-dictionary shape six other routes were fixed for. A full queue reached the client as 500 internal_error, telling a retrying client to loop against a queue that cannot accept anything until the user acts. Everything here drives the real handler: each defect lived in the gap between a function that behaved correctly and the route that reported it.
test_gateway_kernel_routes.py The twelve kernel routes, pinned before they are moved out of Handler._api. Four had no frozen shape and no route-level test at all; four more had only their 403 "REPL disabled" envelope pinned, so swapping the restart and stop bodies passed pytest, the response-shape gate, and the browser smoke run together. Also records two asymmetries as-is rather than tidying them: ten of the twelve answer 200 for an unknown frame id, and kernel/install is deliberately ungated by notebook_repl. Marked stubbed_backend where a service is replaced, so the fabricated responses never reach the frozen contract.
test_telemetry_schema.py Telemetry may say counts and enumerations, and this is what stops it saying more. Attacks values, not keys — {"error_type": "ValueError"} and {"error_type": "FileNotFoundError: /home/y/unpublished/cohort.csv"} pass the same key check. The leak shapes are taken from what this repo really produces (an exception class defined inside an agent-authored cell, a skill name from SKILL.md frontmatter, an environment directory name, a provider error sentence), crossed against every declared field. Also asserts the package never imports the credential redactor, which exempts absolute paths on purpose.
test_egress_surface.py Every module that can open an outbound connection is declared, with a reason. Written as a prerequisite for telemetry but not about it: before this, nothing recorded which modules can reach the network, and "not a single packet leaves the machine" is a claim about the whole tree that cannot be checked one file at a time. Seven modules today. A new one fails with its file and line, so adding a line to the table is a decision while adding urlopen to a module is a Tuesday. Listening sockets are deliberately out of scope — a different risk with a different answer.
test_telemetry_consent.py Consent is what telemetry may not work without. Pins the two properties whose reasonable-looking alternatives fail quietly: revoke destroys the install id along with the permission (and leaves no tombstone, since a record that a grant once existed is itself a fact outliving its withdrawal), and an environment variable can veto but never grant. Writing it found a real defect — a vetoed grant() still wrote the row, so a scripted grant landed silently and would take effect the moment the variable went away.
test_telemetry_off_by_default.py The proof that with no consent nothing leaves the machine. The claim is absolute so the test is too: every outbound primitive — getaddrinfo included, because a DNS lookup is itself the signal consent is asked for — raises rather than being absorbed by a mock that returns a plausible 200. Covers import time, daemon construction, revocation, the environment veto, and each of the sender's refusals. Writing it caught a vacuous test of my own: asserting _NoRedirects.redirect_request returns None proved nothing about whether send installs the handler, and the mutation that dropped it passed.
test_telemetry_emit.py emit sits on hot paths (daemon start, end of every turn), so its first duty is to do no harm: a store that raises, a consent read that throws, a send that crashes — none may reach the caller, and the send never runs on the caller's thread. Also pins the stop-reason mapping against the engine's real six values (completed/done/stopped/cancelled/max_turns/failed); the first call site mapped finalized, which the engine never produces, and dropped three it does.
test_telemetry_transmission.py Revocation as a boundary rather than a request to stop soon: a barrier steps a sender to the exact point between the consent read and the socket open, then revokes from another thread, and the invariant is one comparison — nothing reached the opener after revoke returned. Also holds a collector open and counts what 200 events cost the daemon in threads, queue depth and counted drops.
test_telemetry_consent_route.py The Customize toggle at the HTTP boundary. A GET reports {enabled, env_locked}, a PUT grants or revokes; the route records and never sends. Proves the wiring carries the consent module's guarantees through the gateway: a fresh install reads disabled, granting mints an id, revoking destroys it, re-granting is a new identity, and an environment veto is surfaced so the UI does not present a toggle that silently does nothing.
test_telemetry_collector.py The receiving end, exercised on an ephemeral loopback port (never a public bind) with the same wire.seal the client uses. The property a relay would not have: the server does not trust the sender, so a hand-built envelope smuggling a path into error_type has that field dropped on arrival — "counts and enumerations only" holds even against a client that ignores it. Plus the boring refusals: Host allowlist (not Origin==Host, per the relay's DNS-rebinding history), body cap on Content-Length before the read, rate limit, and an aggregate that keeps counts rather than a per-install log.
test_artifact_step_environment.py The "Saving ..." step card names the runtime that wrote the files. Its writer used the Python kernel's label for every Cell, so an R Cell's card was stored and shown as "environment python" while the artifact's env snapshot said r. Drives the real kernel/execute and steps routes, with only the R worker stubbed. Rows 0.2.x stored carry no language, and are read through the Cell that produced their artifact versions. A row that recorded its language is read as written.
test_artifact_manager.py Workspace Artifact capture, versioning, and cell promotion. A lot of it is anti-duplication: a provisional version must coalesce rather than become a second version, and under Stage 1 identical head bytes must reuse the version while retaining the new producing-Cell observation. An R capture must not run the Python figure probe. Promotion rejects a symlinked output path and embeds figures only as safe data URLs.
test_artifact_mutation_service.py Interactive edit, rename, upload, and delete from the UI. The event shapes are pinned exactly because the frontend reads them, and any path that tries to escape the workspace fails closed.
test_artifact_repository.py The artifact, version, environment, lineage, and capture-observation repositories underneath the manager. The transactional claim is the one to check: if lineage/observation persistence fails, the whole record_cell transaction rolls back, the version with it. It also proves that same bytes across different Cells produce one version plus two scoped observations without rewriting the original producer.
test_code_evidence.py Completion evidence for a source-code deliverable. The reconciler only ever asked whether a claimed file exists, and the model's own "all tests passed" was the only thing standing in for a test run. These pin the four fields for reusable_pipeline/codebase_change and every way each one fails — a deleted file, a path outside the evidence roots, a mismatched sha256, a file never captured as an artifact, an entry point that does not compile, a forged cell id, a real cell from another session, an error-status cell, an ok cell with no stdout, and an ok cell whose recorded stdout says FAILED. One test matters more than the rest: an entry point that writes a file at import time must be compiled and never run, so the beacon it would drop must not exist. analysis_run is asserted unchanged in both directions — no fields required, and bogus fields not validated either — and a broken evidence store is asserted to REFUSE rather than degrade to an accept.
test_completion_delivery.py Stage 1's exact-version delivery boundary: canonical percent-encoded URLs, immutable snapshot/scope/size/checksum verification, atomic message+manifest persistence, idempotency conflicts, committed-ledger reads and idempotent publication marks, and fail-closed corruption/fault cases.
test_artifact_ref_chip.py An @file reference as a durable record rather than a token re-parsed on every send. Two refusals were reproducible before it: the same sibling-session file could be referenced exactly once (materialisation refuses when the live filename is taken, and the first successful reference is what took it, so the second turn came back materialise_failed and the model got nothing), and two artifacts sharing a name were mutually exclusive for the same reason — in a form whose entire premise is that a name is not an identity. Covers the reuse of an existing copy, the version-scoped landing name, the six fields (artifact_id/version_id/sha256/display_name/source_session/materialized_target), the rule that a reference the budget dropped is not recorded as sent, persistence through the real send path, the conversation route over real HTTP, and the branch projection. Also locks in the one thing that was already right: a chip typed and then deleted leaves no Artifact and no lineage edge, because the text is the only source of truth and materialisation happens at send.
test_memory_scope_and_budgets.py The UI wrote memories to "default" while injection read the session's project, so a memory a user created was silently invisible to the sessions it was written for — a feature that quietly did not work. Global vs project scope with inheritance, per-item size and per-scope count enforced before the insert, and a scoped delete so an id-only delete cannot cross a project boundary.
test_queued_followups.py The FIFO queue, per-ticket identity, the 413/409/429 admission refusals and sibling-safe cancel were all real on the server and reachable over HTTP — and the composer was simply disabled during a run, with no queue UI at all.
test_model_protocol_menu.py profiles_payload already served protocols: list(PROFILE_PROTOCOLS) and the client ignored it, hardcoding the dropdown — so gemini and openai_responses, both dispatchable and both accepted by POST/PATCH, were unreachable from the UI.
test_session_and_message_paging.py The session-list keyset cursor was real server-side and had no production consumer anywhere in the repo: loadSessions was hard-capped at limit=100, so a project with more than 100 sessions simply could not reach the rest. Walked over 260 sessions and 640 active-branch messages asserting no gap, no duplicate, and an export carrying the first and last message. The load-more control firing and the scroll anchor holding are browser-only and are named as such rather than faked with a grep against app.js.
test_project_paging.py Home project directory paging. A 1000-row fixture still answers in pages of 100 (never more than 100); the (last_active_at, project_id) keyset covers a same-timestamp tie and empty last_active_at sorts last; a page is at most two SELECTs (JOIN aggregate + exact filtered total); escaped LIKE treats \\/%/_ as literals on name/description, ASCII case-insensitive and non-ASCII exact; q over 128 code points is 400; a filter or principal change, or a malformed cursor, returns 400 invalid_cursor rather than silently restarting at page one; team visibility is applied before LIMIT so hidden rows cannot occupy slots; unparameterized GET stays a full dump and offset is executed in the compatibility window.
test_annotation_version_binding.py An image attachment's bytes were not bound to an immutable version: annotations recorded no version_id/checksum, and resolve_artifact_path returns the artifact's latest version — so overwriting the figure between pinning and sending silently changed what the model received. MIME is now decided by magic bytes, not the filename.
test_artifact_refs.py What @file in a message actually sends. The old resolver read the artifact's live path, so the same reference meant different bytes once a later cell overwrote the file; an unresolvable name was dropped in silence, leaving the user asking about a file the model never received; and every artifact was decoded as UTF-8 with errors="replace", so a .npz arrived as a wall of U+FFFD that reads as corrupted text rather than as "this is not text". Covers the pinned form, the one-release legacy spelling, cross-session materialisation at send, and the budgets. Also covers the budget the other two did not add up to: MAX_REFS bounds how many and MAX_REF_BYTES bounds each, but eight at the per-file cap is 1,600,000 characters — about 1.5× a 262,144-token window, and 8× the cap on the message a person types. References now share one budget, and what it cuts is named rather than dropped.
test_artifact_index.py The project Artifact index route. A 500-row fixture still answers in pages of 50 (never more than 100); the (created_at, artifact_id) keyset covers a same-millisecond tie; escaped LIKE treats %/_ as literals and searches filename only; a filter change returns 400 invalid_cursor; team visibility is applied before LIMIT; same-name artifacts across sessions/versions stay distinct; the legacy array route stays a bare array. The browse index is additive and rolls back with a failed migration step.
test_artifact_scope.py Who owns an Artifact when a child frame produced it. Scope is inherited from the root project, an explicit root that conflicts with the known producer is rejected, and a migration repairs sessions written before the rule existed.
test_artifact_scope_closure.py Three scope holes on complete production chains, each reachable from one line of agent code. artifacts, artifact_versions, lineage_edges and frames were on no denylist, so one SELECT returned every project's filenames, checksums and absolute paths — and the filter that did exist matched the SQL text, which a bound parameter is invisible to. _scoped_artifact raised PermissionError for foreign and KeyError for absent while _scoped_version twelve lines below collapsed both and said why. view_image deferred its scope check to a place that did not perform it, and save_artifact(input_version_ids=…) wrote cross-project lineage edges with no validation at all.
test_artifact_workbench.py Stage 9 Artifact workbench Go/No-Go: full-dataset CSV sort/filter, unchanged text edit does not create a version, benzene molfile save/reopen has six carbons, PDF/HTML locators reach the next-turn block, and /ketcher serves real Ketcher 3.7.0 assets only when the flag is on. Also pins optional version_id on /table (never falls back to latest) and flag-off 403 for /table/profile and /table/export.csv.
test_table_profile.py Table workbench GA backend: shared query-parser snapshot, histogram boundary, approximate unique, export chunk and 32 MiB total 413, malicious Parquet metadata, missing optional engine stays unavailable, profile stats stay out of SQLite, and the resource-acceptance manifest blocks default-on.
test_upload_scope_resolution.py POST /uploads was impossible for any session outside the default project. upload read payload.get("project_id") or "default", so a request naming a frame_id and no project asserted "default" on the caller's behalf — and artifact_write_scope, correctly, refuses a stated project that disagrees with the producer frame. The refusal was a bare ValueError nothing caught, so it reached the dispatcher's catch-all as 500 internal_error: a client that named the wrong project was told the server had broken. The suite now also pins the exact writer transaction: pinned live and immutable parents, held pending/final inodes, journal identity, rollback/recovery at every crash point, and same-byte or attacker-byte final-component swaps.
test_artifact_ref_chip.py The reference existed only as a text token re-parsed on every send, so no reader could recover which version the model actually read — and for a cross-session copy that is not the version the user named. Six fields are now persisted. It also caught a regression this batch introduced: refuse-on-name-collision in materialise_artifact meant the second turn referencing the same sibling file failed, because the first reference had taken the filename.
test_backend_import_contract.py A source scan rather than a behavior test. While implementations move into packages, callers may only import the declared facade surface, and no new dependency on an undeclared legacy internal is allowed to appear.
test_background_cleanup.py Background-kernel lifetime and trusted-capture admission: shutdown interrupts then kills workers, failed starts and cleanup release their leases, malformed or corrupted coordinator state fails closed, external mutation stays owner-reentrant but writer-exclusive, and buffered output remains bounded. Its malformed-state matrix uses stable xdist IDs so every CI worker collects the same suite.
test_bash_authorization.py host.bash is kernel-local, so the Host never executes it — it can only authorize it. This module is the whole argument that the token is enough. It is bound to command, cwd, generation, and challenge, it is single-use and expiring, and the worker rejects a tampered one before any subprocess is spawned.
test_biosecurity_web_parity.py Whether the biosecurity screener runs on the surface people actually use. OPENAI4S_BIOSECURITY is on by default and config.py documents it as splicing the accountability prompt and running the diO trajectory screener. The CLI's pre-exec gate did both; the Web daemon's _safety_refusal called classify_code and returned — so a cell uv run openai4s run refused was executed by ./start.sh. That is the worse half to lose: the prompt asks the model to behave, the screener checks whether it did. The screener judges a trajectory, so the port had to widen to pass the session; _gather_trajectory is imported from the CLI loop rather than reimplemented, because two definitions of "the trajectory" would be two safety policies wearing one name.
test_bioprobench_skill.py The BioProBench Skill's two silent scorers. Ground truth encoded as 0/1 or "true"/"false" has to score exactly like a JSON bool, an unrecognized label has to reach failed_rate instead of being scored as a confident miss, and a bare correct PQA answer like 0.3 must not be eaten as its own confidence.
test_capability_state.py Durable Skill and Specialist enablement. The failure it prevents is a Skill that stays disabled in the prompt but not in search, or a loader still holding a repository from a closed Store.
test_catalyst_sar_screening.py The catalyst SAR Skill: its documented pipeline and hard lock, its vendored CONTCAR catalog, and the POSCAR parser's edges — a negative scale factor, a truncated file, a structured name that has to be sanitized before it becomes a filename.
test_cell_cancelled_during_bootstrap.py A Notebook Cell whose fresh kernel was still bootstrapping when daemon shutdown cancelled it. Nothing can interrupt a bootstrap no lease is bound to yet, and nothing looked at the cancellation once it returned, so on a real daemon the user's code started during shutdown and ran until the watchdog's next poll. Driven through a real SessionRunner, a real Python kernel and runner.close(), with the bootstrap held host-side: the Cell's code never runs, its attempt and execution-log row end interrupted, and the Notebook event says so. Each signal the check reads refuses the Cell on its own: the session's stop marker with no admitted ticket, and a ticket that Stop or shutdown cancelled before mirroring it onto that marker, for that ticket's session only.
test_cell_dependencies.py Static dependency analysis for Python and R Cells, deliberately allowed to say it does not know. A dynamic namespace write or a parse failure marks the Cell uncertain instead of guessing, and staleness propagates transitively from there.
test_cell_execution_service.py The Web Cell service, where the transaction ordering lives. An attempt is allocated before the Cell is prepared and finalized even when the worker throws. An R protocol exception shuts down only the lease that was executing, never its replacement. A protocol-only host.submit_output is still audited, just not streamed as a Notebook cell.
test_cell_watchdog.py Timeout recovery for a Cell that will not stop. The subtle case is the one to read: while a Cell is paused waiting for a permission approval, the timeout budget freezes, or a human thinking for a minute would kill the kernel.
test_checkpoint_state_snapshots.py Plan, review, and memory state bound to an immutable checkpoint. Two behaviors are load-bearing: a legacy checkpoint is treated as partial rather than as an empty state that would wipe live data, and corrupt state fails closed instead of restoring half of itself.
test_checkpoint_binding_survival.py The capture read annotations with SELECT * and the restore wrote an explicit column list without version_id/checksum, so every restore unbound every pin and put back the resolve-to-latest behaviour migration 12 exists to prevent — on a path whose promise is that it puts the session back the way it was. Memory updated_at was lost the same way. Asserted on the columns in the database after a real restore, not on the snapshot dict: the snapshot was already right, which is why the loss was invisible.
test_cli_daemon_api_root.py Whether the CLI's daemon calls reach a route at all. openai4s share built URLs from /api/...; the daemon serves the API only under /api/v1 and answers anything else with a deliberate "the API is versioned" refusal, so all nine share subcommands 404'd — the feature had never reached a route in any released form, including the openai4s share import <url> line the generated share page tells a recipient to run. The prefix now comes from contract.API_ROOT, the constant the gateway routes on, and a path carrying its own /api/ is refused loudly rather than producing another 404 nobody reads as a defect. Asserted as a property over every call site, because nine were wrong the same way.
test_cli_catalog_posture.py The headless CLI's model-facing tool projection is posture-aware. define_dynamic_tool and the other approval-required dynamic-tool lifecycle tools were declared in every CLI run although, with no channel and the shipped unattended deny (or under the Guardian), every call was refused -- models reached for them first and lost a turn. The rule is general, not a dynamic-tool list: every approval-required tool no approval path could allow is left out. Under the default posture that is the four dynamic-tool lifecycle tools, web_download and rollback_skill_version from the first turn, plus restore_artifact_version, exec_background, read_mcp_resource, get_mcp_prompt, request_network_access, stage_model_asset, register_remote_capability and compute_submit once their groups activate. Every one of them was already refused there. search_capabilities reads the same predicate: it used to report them under visible_tools, a live model then called exec_background and lost the turn to the refusal, so now visible_tools and each matched group's tools name only what the provider tools= list offers, and the rest are named under unavailable_tools with reason approval_unreachable (a key the Web root, which sets no predicate, never gets). Pins what hides them (default posture, the Guardian even with the unattended env allow, a deny for every target) and what keeps them (a standing allow rule, OPENAI4S_UNATTENDED_APPROVAL=allow, an attached approval channel as on the Web path), and that a hidden tool called anyway is still refused and audited by the gate.
test_cli_code_mode_test_runner.py Explicit code modes in the headless CLI. reusable_pipeline/codebase_change completion needs a Host-authorized host.bash receipt for each cited test command, which a run with no approval channel and the shipped deny posture could never obtain -- every run looped to max_turns guessing cell ids it was never shown. Pins the exact-command --allow-test-command opt-in (glob-escaped, conversation-scoped, validated before any rule exists), the refusal before the first model call when nothing could authorize the runner (including under the Guardian), the [cell id: …] Observation line that exists only while the contract is armed (Web and CLI), and the mode fragment that names host.bash and the keyword-argument shape.
test_cli_contract.py The CLI surface people script against: entry points, subcommands, options, and help text. status reports the local data directory without claiming the daemon is healthy.
test_cli_auto_mode.py openai4s run --auto: the autonomous preset it sets (and the operator choice it will not overrule), and the post-run reviewer adapter that gives a one-shot run a real terminal. --auto is not a blanket grant -- it never sets OPENAI4S_UNATTENDED_APPROVAL. The reviewer packet carries what the run recorded -- its cells' code and output, its native tool ledger -- because a correct run reviewed from cells: [] came back with the task's own instructions flagged as unbacked claims; and a review with no execution record, or with executed cells missing from it, is an omission that never reports verified.
test_cli_stop_slow_daemon.py openai4s stop against a daemon that outlives the shutdown grace period. The old code discarded the poll's outcome: it cleared the pidfile and printed "daemon stopped" unconditionally, orphaning a live daemon on a bound port that status then called "not running". Driven against a real SIGTERM-ignoring child: on timeout the pidfile survives and the exit code is non-zero, --force escalates to SIGKILL, and a serve whose bind fails prints one clear error instead of a traceback after a success banner. The reverse case too: a daemon that exits a little after the 5s SIGTERM grace is a clean stop, because stop keeps polling up to --timeout (default 30s) behind a shutting down… line instead of returning 2.
test_compute_nvidia.py The NVIDIA BYOC provider with every docker call intercepted by a fake subprocess layer, so no Docker, GPU, or network is needed. The security half is the two-stage secret scrub: a provider's top-level code must not be able to read credential-shaped or known-prefix environment variables when it is imported.
test_config.py Layered configuration, and mostly one bug class. A placeholder API key copied out of a template must never be treated as a real key, whether it arrives from the environment or is passed explicitly, and it must not shadow a real per-provider key.
test_container_deployment.py The Dockerfile, compose.yaml and Kubernetes manifests, tied to the code that gives their values meaning: the probe path to the gateway's own unauthenticated set, the port to Config's default, the injected API-key variable resolved through LLMConfig rather than trusted by name. Configuration cannot fail on its own — a renamed variable or a moved route just stops working later, for someone else. Two properties are asserted because a plausible edit reverses them: a single replica (the store is SQLite) and a published port bound to loopback. It also covers the singleton across a container restart, where bare pid liveness cannot tell a stale pidfile from a reused pid and a killed container therefore refused to start again.
test_connector_repository.py MCP connector rows: JSON normalization, ordering, enable and disable, and the Store facade that feeds them to the Host MCP service. Also pins secret-preserving edits: unmentioned env names survive, explicit replacements update their brokered value, and explicit removals retire the stored secret.
test_context_policy_web.py Two tests. A large context output becomes a deduplicated Artifact version instead of a second copy, and the compaction payload is linked back into session history instead of being dropped.
test_data_background_tools.py The data and background tool classes. Two policy lines are asserted rather than assumed: query stays strictly read-only with no approval path that could widen it, and submitting background work is gated while interrupting it stays available.
test_datapro_index.py The completeness boundary for successful DataPro responses: every recursively returned field is searchable without a schema whitelist, large batches are not truncated, duplicate occurrences retain their JSON-pointer identity, retries are idempotent, and a mid-batch SQLite failure rolls back the batch and all earlier entries. It also pins strict integer-code admission, command-palette discovery, and Artifact/session/project lifecycle cleanup.
test_doubao_search.py The offline security and routing boundary for primary Doubao Search: Agent Plan credentials are resolved from SecretBroker for every request and survive rotation without persistence, the fixed endpoint/header/body contract is exact, only a proven result list becomes source: doubao, auth and quota failures cannot be disguised by fallback engines, redirects and oversized or malformed replies fail closed, and reflected credentials are removed before any result or error escapes. Every upstream here is fake, so the whole module carries stubbed_backend; real upstream success belongs to the opt-in browser gate.
test_dangerous_tool_declaration.py What a tool's dangerous declaration actually did. Ten control tools declare it and two tests assert it; nothing else in the repository read it — not the dispatcher, not the broker, not the audit record, not the approval card. The consequence was in the prompt: every card looked alike and its remember-scope defaulted to "conversation" with a pre-filled pattern, so one Allow click on restore_artifact_version granted artifact restore for the rest of the session exactly as it would for a file read. Now the declaration reaches the broker, persists in the durable request payload, and the card reads it — a risk badge and a default grant covering only this call. The wiring is driven through a real dispatcher call rather than handed to the broker, because a test that supplies the flag itself passes even when the call site never does — which was the defect.
test_delegated_cell_record.py Delegated child Cells become durable execution_log rows keyed under the child's own delegate frame (origin="delegate"), failed and host-side-crashed cells included, stamped with the child kernel's real generation id — driven end to end through the real DelegationRunner and a real kernel, with only the LLM scripted. Rows survive a Store reopen, nested children key under the nested frame, the parent Notebook stays clean by construction, and lineage cell_recorded becomes true through the unchanged view. Also the ComposedCellHooks contract: the recorder runs even when the stage-1 capture hook raises, and native calls reach only the capture half.
test_delegation_env_inheritance.py Delegated children inherit the parent session's selected environment at all three kernel construction sites (foreground, background factory, R pin) and through nesting; child kernels register durable generations that artifact environment provenance resolves instead of the "assumed" daemon fallback; host.env.use without a switch callback is an error, and a child's switch really retargets the next cell's interpreter.
test_delegation_idempotency.py Durable request identity separate from attempt identity. Ten concurrent identical requests create one request and one child; two native_call_ids in the same action group create two requests; a digest mismatch is 409; terminal reuse makes zero provider calls; restore never auto-continues — only an explicit continue increments attempt_no. Caps stay 48/1000/4.
test_delegation_persistence.py Delegation across a daemon restart. A child that was live when the process died has to be stopped and its lease fenced, not silently adopted by the new generation. Steering, budgets, cascade cancellation, and cleanup on deletion are here too.
test_delegation_policy.py Four tests, one claim: a child's capability and permission policy is enforced, not decorative. An invalid policy fails before it consumes budget, and a nested child can neither widen what its parent allowed nor relax a deny.
test_delegation_control.py Stopping and steering a sub-agent from outside the agent. _stop_subtree and send_message were already correct and already tested — and unreachable: a user watching a runaway child could only stop the whole session. What the routes add is status mapping, and three answers have to stay apart: 404 no such child in the durable record, 409 the record has it but nothing live can act on it, 200 done. The 409 is ordinary — a daemon restart marks every pending/running child stopped with stop_reason='daemon_restart', so any page opened beforehand holds ids for children that are gone; 404 would say that work the user watched run never existed, and 200 would claim a stop that stopped nothing. send_message's {"ok": False} becomes a 409 rather than riding out as success. Includes success-path tests, without which every case here asserts a refusal and a handler wired to the wrong method still passes.
test_delegation_runtime.py Running sub-agents. Budget is shared across the whole tree and reserved atomically between concurrent runners, depth four is an unconditional leaf, and a model reply that arrives after a stop can neither execute nor submit.
test_delegation_null_semantics.py Explicit null == absent == default, identically through both delegation doors. The SDK codec always dropped top-level Nones; the native delegate_task now strips them before schema validation instead of refusing the provider's literal null, and a None nested in a fan-out item inherits instead of clobbering (or crashing the turn-budget parser).
test_delegation_step_projection.py The delegate step card tells the truth. The old projection flattened every child result to a 2000-char JSON string under a hardcoded green "done"; now the structured bounded projection carries task_status/turns/environment/limitations/artifacts, the summary word is the task_status itself, and the status vocabulary gains warning: completed→done, partial/blocked/stopped/max_turns→warning, failed/malformed/transport→error, fan-out worst child wins — proven both on the pure projection and end-to-end through the real dispatcher's on_step.
test_delegation_task_status.py The delegation completion contract. stop_reason says how the child's engine terminated; task_status (completed/partial/blocked/failed) says whether the TASK is done — derived once in the envelope build, where the child's declaration is input and machine checks (require_artifacts against the store) can only downgrade it. max_turns is a failed lifecycle, every terminal persists its stop_reason, and the bounded retries option re-runs a not-done child with its previous limitations appended — the only automatic retry anywhere.
test_delegation_workspace.py Feature-flagged private scratch (default off). Two children writing the same filename do not overwrite; each output version's producer frame is the child that wrote it; the parent workspace is unchanged until explicit materialize; materialize/rollback never deletes published Artifact versions.
test_dynamic_tool_scopes.py Dynamic Tools resolved at session, project, and global scope, in that priority order. Re-promoting identical content reuses the version but still audits the operation; a cross-project activation and a tampered scoped manifest both fail closed.
test_dynamic_tools.py Agent-authored tools are the most direct way to get new code onto the machine, so this is a containment module. Each one runs in its own one-shot worker, behind a source gate, with no Host secret in its environment, and refuses to run at all unless the OS sandbox is enforced. Schema, TTL, permission, and timeout checks apply on top.
test_e2e.py Two tests, but the widest stack in the suite: a real Skill loader, a real kernel subprocess, and a real HostDispatcher, with only the LLM scripted. The second one checks what the agent actually observes when the kernel raises.
test_image_attachment_budget.py What one turn may attach as images. Every pinned figure was attached at full size with no count, per-item or total limit, so eight pins on a large raster sent about 10 MiB to the provider and eighty sent ten times that. All three budgets are asserted against fixed values rather than against the constants under test — the first version of two of these defined their expectation in terms of the very limit they were checking, so removing the budget broke nothing and mutation testing is what exposed it.
test_host_network_capability.py The Host's outbound capability and the guards that make it one. Three bundled skills used raw urllib because host.web_fetch could not express a HEAD probe, a contactable User-Agent or a binary download — and a request made that way is subject to neither the egress allowlist nor the SSRF guard, so the gap in the API was why part of the product's own traffic went around its own fence. Closing it granted three powers, so every test here asserts a refusal or a limit: the byte ceiling is enforced while reading, downloads stream to a daemon-private stage and publish through a verified parent descriptor instead of collecting checkpoint-sized bodies in daemon memory, a parent-directory swap cannot redirect publication, a probe is guarded like any other request, and a download that escapes the workspace fails before it contacts anything. WSL Fake-IP compatibility is narrower than the private-fetch override: only catalogued hostnames may use 198.18.0.0/15, while unknown/exact-host children, IP literals, metadata, and other private ranges still fail closed. It drives the real WorkspaceFileService rather than a double, because a double could be the stricter of the two and hide an open capability.
test_egress.py The outbound domain allowlist. Read the first tests carefully: the mode defaults to off, and off fails open. This is a control an operator turns on, not a property the system has by default. Once on, lookalike hosts are rejected, a runtime grant can widen it through the broker, and a denial leaves the fence closed.
test_environment_readiness.py Stage 1's local-only standard preflight across the exact Python/R direct manifests: complete gaps, path-free DTOs, no network/mutation/interpreter start, persistent UI projections, and both explicit managed repair commands. Control-only/finalize turns remain routable without a kernel; a Code Cell fails closed before pending Cell state or execution, while approved/resumed scientific plans are checked before their CAS transition. Unknown inventory fails closed instead of becoming ready.
test_environments.py Conda environment discovery, run against fake conda directories so the result never depends on what the developer happens to have installed. An R-only environment is discovered but never offered as runnable Python — while env_list, the model's view, reports it runnable on the R channel its ``r cells actually use and recommends it for the R packages it has, so the tool can no longer tell the model R is unavailable in the environmentenv_use` just accepted.
test_env_setup_failure_projection.py A failed host.env.create must not render as "ready". preinstall.install reports failure as a structured ok: false result rather than the single-key soft-fail, and the activity-step projection only understood the latter — so the card said "ready" over an install that never happened. The projection now reads the declared flag, carries the real reason (e.g. "No module named pip") onto the card, and ends the step as an error, while the structured result the model sees is unchanged.
test_preinstall_uv_fallback.py On-demand installs in a pip-less uv virtualenv — including the daemon's own .venv that setup.sh builds with uv. _pip_install probes for the pip module and falls back to uv pip install --python <sys.executable>; with neither tool present it fails naming both remedies. Only tool selection falls back: a failed install under a present pip is reported as-is rather than retried with uv.
test_execution_coordinator.py The FIFO coordinator by itself. One writer owns a session at a time, separate sessions never block each other, and a cancel or interrupt has to name the exact ticket and owner — a ticket from another coordinator cannot release anything.
test_execution_principal.py Team-mode execution identity is explicit and fail-closed: an absent principal is refused, members cannot browse/search/open another member's private cells, owners and admins keep their intended reach, and identity context is restored between executions.
test_execution_sources.py The executed-code view and its sources.zip export against the real Store and real gateway routes: the JSON projection walks root + nested delegate frames with counts, source hashes, generation/environment identity and artifact-version links but never inlines code; the zip carries the actual executed source files (failed cells included and marked), per-frame # %% session files, a new manifest and bilingual persistent-kernel READMEs — byte-deterministic across repeat exports and a Store reopen, and free of prompts/stdout/credentials. Also: legacy root-only sessions export cleanly and the root Notebook projection is untouched.
test_execution_view_service.py The DTOs the Execution log and the Notebook are built from. The retry projection is the fiddly part: consecutive failures collapse into one row without dropping any, and the collapse must not cross a runtime or non-agent boundary.
test_frame_repository.py Projects, frames, messages, steps, and cell logs. The execution log is append-only and orders equal timestamps by state revision; deleting a project cascades in a single commit.
test_auth_exit_matrix.py P0-1's exit matrix, over a real socket. Three legs the row claimed and nothing tested: an unauthenticated WebSocket upgrade (every /api/v1/ws test presented a credential, on a synthetic handler that could not tell a refusal from a live socket that accepts cancel_execution), an unauthenticated REST matrix that was one route (now swept from contract.http_routes(), so a new route arrives on its own), and a cookie replayed against the handler that replaced its issuer, where the old evidence asserted only token-file stability. It also drives a daemon started with the retired OPENAI4S_REQUIRE_TOKEN=0, on loopback and off it, and asserts the gate over the wire: the loopback opt-out was granted for exactly one minor release (D1) and is ignored from 0.3.0.
test_gateway.py The largest module here, and the one with the widest blast radius. It reaches from the hand-rolled WebSocket — frame encoding, unmasking, the resume buffer and its byte budget — up through HTTP body framing across keepalive requests, cross-origin refusal for both API writes and WS upgrades, the artifact and environment routes, the generic connector edit route and its masked env projection, and the placeholder-key filtering that has to survive persistence.
test_gateway_engine.py The AgentEngine-backed Web runner at its composition boundary, with the kernel kept offline. Streamed deltas hide fences and land before the tool and terminal events, and a conversational JSON fence must not cut off the public prose that follows it.
test_gateway_host_allowlist.py The DNS-rebinding Host allowlist (GHSA-fm3g-2c7x-8qj8), asserted on the wire rather than from a synthetic handler: a real ThreadingHTTPServer on a real port, spoken raw HTTP/1.1 at. It pins what a monkeypatched _json cannot — that the refusal is a 403 status line, that the Host check precedes the token gate (a forged Host with no credential is refused as a Host, not as a 401), that it covers the SPA shell, /static/ and the token-exempt /health and not just /api/ writes, and that the WS upgrade is refused instead of switching protocols. The positive controls are half the point: every loopback Host a browser really sends, plus the absent-Host CLI client, must still be served.
test_gateway_kernel_lifecycle.py Python and R slots under a supervisor, and the races are the content. A stop intent must not be overtaken by a new start, bootstrap has to run outside the supervisor lock, and an R failure must not touch the Python worker. Team composition covers foreground/background/delegated/R/recovery workers before first use: whole daemon data, personal data-root namespaces, exact current Skill and Artifact-input exceptions, and symlink/overlap refusal.
test_gateway_lazy_runtime.py Laziness is a load-bearing property here, not an optimization. A plan turn, a native tool turn, and a tool-only structured finalize all complete without a kernel process ever existing; the first Cell or REPL use is what spawns one.
test_gateway_session_domain_routes.py The session-domain HTTP routes over one shared composition. Worth knowing before you read the UI code: fork_from_cell deliberately fails closed because it is not supported yet, and the variable-inspector route never starts a worker.
test_gateway_session_lifecycle.py Durable generation and attempt identity across the daemon's own lifecycle: TTL sweeps, startup reconciliation of a generation left behind by a dead daemon, and the admission check that stops new work entering a project being deleted.
test_fsprobe.py The security/fsprobe.py contract, stated so it fails on the interpreter that motivated it: lstat_is_symlink / lstat_kind answer absence and links plainly and raise PermissionError for a path under a mode-000 directory — where CPython 3.14's Path.is_symlink() answers False and turns every except OSError: refuse guard into dead code.
test_gateway_static_cache.py Weak ETag, 304, gzip negotiation, and immutable Cache-Control for /static/ and the SPA shell. Asserted on the wire: a matching If-None-Match is an empty 304 that still carries the security profile, gzip is offered only for text types over 1KB with Vary and an identity fallback, immutable is only for fingerprint names, files over 8MB stream, and a symlink out of the web UI tree is 403 after realpath.
test_gateway_webui_next.py SPA-shell switch after the F-23 flip. Unset serves dist/index.html at /, /index.html, and /projects/{pid}/frames/{fid}; OPENAI4S_WEBUI=legacy is the escape hatch that serves webui/index.html. /static/ resolution is unchanged. Also pins the package-data dist globs, the _WHEEL_REQUIRED sentinel, and the favicon 10 fps floor.
test_global_research_views.py Two read models that span sessions. The project Timeline merges them without carrying raw payloads, and the lineage view connects Artifact versions to the cells that produced them.
test_governance.py Repository governance expressed as a test. Every workflow action is commit-pinned and its version comment is checked in PR CI, the source secret scan covers the working tree, and Dependabot tracks all five ecosystems with only schema-level entry keys and one entry per ecosystem and directory.
test_harness_characterization.py A characterization golden, so it deliberately records current behavior, known bugs included and labelled as such. Regenerating the golden has to be an explicit act, which is what keeps a silent drift from passing.
test_harness_contract.py The harness itself: the scenario schema, the scripted provider, the fault schedule, and a normalizer that preserves event order instead of sorting it. A declared fault that never fires fails the scenario.
test_auto_budget.py Atomic Auto Mode budget admission against a real SQLite repository: last-slot races, identical admission_id collapse, no optimistic refund, restart-stable remaining, same-action/no-progress circuits, unverifiable tokens fail closed, and one authority per public field.
test_auto_mode_faults.py Stage 2 crash, rollback, expiry, multi-connection, and side-effect-ledger recovery contracts. It proves that committed facts replay idempotently, partially committed phases do not escape, repair action groups are bound before use, and a lost WebSocket never becomes the source of truth.
test_auto_mode_portability_regressions.py Adversarial package/share portability coverage: per-run identity remapping, start-cursor-stable owner ordering, direct/shared graph-validation parity, constant event-query count, and multi-hop source provenance that remains inert.
test_auto_mode_recovery_regressions.py Post-commit proof-tamper, replay, finding-owner, event-order, terminal-integrity, history-prefix, fork, and revert recovery regressions.
test_auto_mode_routes.py Focused adapter contracts for the exact frame-scoped Auto Mode GET/PATCH/audit routes: method/path ownership, query bounds, error mapping, canonical live-event names, and the deliberate absence of an HTTP transition route. The stubbed service is marked so its synthetic shapes never enter response-schema capture.
test_auto_mode_service.py Selection precedence and compatibility mapping across quarantine, frame, project, deployment, legacy boolean, and built-in defaults. Explicit deployment unset stays distinct from off, and Stage 2 remains runtime-inert.
test_auto_mode_storage.py The durable Auto Mode state machine, event ordering, idempotency, Verified proof integrity, finding lifecycle, repair ownership, permission scope, branch projection, migration, quarantine, and import identity remapping. Corrupt hashes and owner/event disagreement fail closed.
test_auto_mode_surface_regressions.py Real Gateway, team-scoped REST, public-field redaction, disabled/default behavior, WebSocket-hint loss, daemon reopen, package, share, and quarantine regressions proving SQLite remains authoritative and corrupted Verified proof is never published.
test_auto_mode_terminal_contract.py The five Stage 0 non-Guardian terminal contracts: exact independent golden payloads and canonical digests, closed reason-specific conditions, fail-closed cross-routing, no Reviewer/Guardian audit masquerade, and bounded recovery truth.
test_scientific_reviewer.py Stage 3 V2 schema: incomplete evidence cannot pass, material findings force issues, and the Reviewer sees only the injected snapshot.
test_evidence_adapters.py PDF/image/structure/table adapters return complete coverage or an honest omission; a filename is never enough.
test_evidence_snapshot.py Immutable snapshot completeness, plan/cell/adapter refs, and digest stability.
test_review_scratch.py Isolated scratch refuses formal-workspace writes, network, submit_output, and inherited secrets.
test_scientific_review_service.py Shadow orchestrator: independence fail-closed, same-model review_only, deterministic claim checks, forged refs, plan-included shadow steps, and flag-off inertness.
test_scientific_review_golden.py Stage 3 Go/No-Go pack: ≥100 golden/injected cases through the shipped evaluator, high/medium recall, clean-task false positives, and zero forged evidence refs.
test_completion_gate.py Stage 4 promotion order: candidate before terminal, issues stay unverified, flag-off is inert, and reopen reads the durable message/setting stamp.
test_review_finding_identity.py A finding's identity is its content within one review, not its content: two sessions reaching the identical conclusion both persist and both reach their real terminal, the id the caller quotes is the one stored, a storage failure never escapes as a bare exception, and a collision no longer strands the branch in reviewing.
test_stage4_atomic_promotion.py Stage 4's final transaction: one exact Candidate message, its optional committed Stage 1 delivery, and the Auto Run terminal/event promote together or all roll back. It covers scope/content CAS faults, post-update event faults, exact replay, and rejection of omitted or changed promotion identities.
test_stage4_candidate_promotion.py Stage 4 candidate → review → promotion: the answer is withheld until a verdict exists, the candidate is durable before the reviewer is called, a repaired candidate is handed back for delivery, Verified requires the delivered bytes to be the reviewed bytes, and a delivery that fails after the review retracts the promotion.
test_auto_repair.py Stage 5 planted residual-row and missing-value cases, version reuse on identical bytes, no self-certify, and review_only never repairs.
test_guardian_shadow.py Stage 6 exact-action hash fail-closed, no standing allow, shadow does not execute, and flag-off is inert.
test_guardian_enforce.py Stage 7 unattended allow_once vs dangerous deny, and flag-off leaves the legacy path.
test_host_completion_service.py host.submit_output, the only completion signal a Python Cell has. A schema failure is soft and must not replace a previous completion, and the past-tense rule on completion bullets accepts CJK phrasing as well as English.
test_host_contract.py The worker-to-Host wire: the injected facade, the single-key soft error, the camelCase codec. The last test is the useful one — every host.* method the SDK can call must have a dispatch route, so a new capability cannot ship half-wired.
test_host_accelerators.py Local GPU discovery stays distinct from SSH registration, container-runtime availability, and model-backend readiness.
test_model_assets.py A user-supplied local checkpoint is imported under an exact-path approval, confined to the session workspace, streamed through SHA-256 verification, and remains staged rather than self-admitting.
test_model_admission.py Connector-neutral checkpoint admission binds a parsed canary to exact model bytes, backend revision and execution route, rejects digest drift, and resets on process restart.
test_host_credentials_service.py Session-local credential references. The secret is never returned, a lease is bound to one action and expires, rotating a credential revokes the leases outstanding against it, and replay excludes every credential method and value.
test_host_data_service.py host.query and the data lookups around it. Read-only SQL is enforced before the Store is touched, the lineage graph is bounded, and an Artifact marker carrying an untrusted ID is rejected.
test_host_delegation_service.py The Host side of host.delegate. Profile injection must not mutate the caller's own argument, and the delegate and steering sources are resolved at call time rather than captured at construction.
test_host_endpoint_service.py Managed endpoints through create, status, request, and close. Re-pointing an existing local registration at a new port needs approval, the free-port scan closes every socket it opens, and the projection carries no secrets.
test_host_llm_service.py In-kernel host.llm. A batch preserves order and bounds the concurrency it was asked for, and a failed item stays a hard failure rather than being softened into an empty reply.
test_host_mcp_service.py MCP on the Host: connector lookup by ID then exact name, tool discovery and call routing, resource and prompt reads, and the soft-error contract each of them uses. A disabled connector is refused at every entry point, not only in the listing.
test_host_progress_service.py Session todos and progress against an approved plan. The plan mutation is committed before the event sink runs, and a failing sink is best-effort — it cannot undo a step that already ticked.
test_host_remote_capability_service.py Remote capability probes. Validation soft-fails before SSH is ever started, a transport failure never writes the registry, and the registry is updated only after a probe actually succeeds.
test_host_remote_science_service.py Remote folding and mutation scoring, where the no-fabrication policy is the whole point. An incomplete or unparseable remote result produces an exact diagnostic, never a plausible-looking number. Provenance is parsed on a best-effort basis and drained by identity.
test_host_skill_service.py The Host Skill service: path confinement, the read-only bundled root, the sidecar gate, and a catalog that refreshes after a publish or delete. A user Skill cannot claim the declared name of a bundled one.
test_host_workspace_service.py Host file access, confined to the workspace. Parent, absolute, secret-directory and symlink escapes are refused; external hardlinks and final-component swaps cannot cross the boundary; and read, glob, grep, list, write and edit use pinned descriptors so a parent-directory swap cannot redirect later I/O.
test_jupyter_adapter.py The optional Jupyter path — and because it is optional, half the module is about its absence. Describing and exporting a KernelSpec has to work with no ipykernel installed, and the bridge reports a missing or broken dependency rather than crashing. The bridge itself is executed against the real hardened Python worker.
test_kernel.py The real Python worker, driven over its actual protocol. Namespace persistence across Cells, stdout and stderr attribution, error_lineno, usage accounting, a mid-cell Host RPC that has to come back, the wire cap and the bounded desync recovery, SIGINT versus a user-raised KeyboardInterrupt, and the locks that could deadlock. The interrupt half is now pinned window by window, because the CI failure that prompted it was in the test's own premise as much as in the worker: a signal that arrives before user code is owed rather than dropped, an idle worker survives one, a protocol write finishes before one is raised, an abandoned host RPC does not poison the next, and a chunk stamped with another cell's id is not this cell's output. Three later seams joined the list: the interrupt is aimed at the worker's main thread via a tgkill helper, because a helper thread pool (OpenBLAS's included) can consume a process-directed SIGINT while the main thread sleeps on; children start with SIGINT/SIGQUIT back at their defaults even when the spawning process ignores them, since a backgrounded daemon inherits SIG_IGN through every exec; and the worker runs in its own session, so killing it also ends what the cell started — while an arming failure is announced rather than returned as silence. The variable inspector lists only the user's bindings: a Web session's skill import gate binds 32 _o4s_*/_OpenAI4S* helpers into the Cell namespace, driven here through the gateway's own bootstrap call. Re-run this after any change to the kernel protocol.
test_kernel_font_cache.py The matplotlib font list a sandboxed kernel starts with, and who built it. An enforced sandbox gave every kernel an empty private MPLCONFIGDIR, so each new kernel's first plot was a full font scan; the cure must not become a channel between kernels. Pinned: a new enforced kernel is seeded with the host-built list, and a list a kernel rewrote in its own temp never reaches the next kernel or the host cache; builder output that is not a font list (a path in the name, a version that disagrees with it or is neither matplotlib's string nor its pre-3.11 integer, the wrong class, not JSON, oversize, an error) is never stored, while a matplotlib 3.10 list (fontlist-v390.json, "_version": 390) is stored byte for byte and seeded; a builder report that cannot pin the matplotlib install it came from (no source, a relative or vanished path, a file that changed during the build) is never stored, and upgrading that install rebuilds the list instead of seeding the old one; a newly installed font, a cache inside the kernel's own workspace, a symlinked cache file, and an unconfined kernel all mean no seeding; builds start only where enabled, once at a time, and a failed one is not retried on every spawn; the fixed builder program returns what the font manager wrote rather than a pre-existing shared cache, and never imports a json.py or matplotlib/ from the directory it was started in; a build started from such a directory through the real confined probe stores nothing. Stopping the daemon through run_server's real teardown stops a build in flight -- the builder, the scan child it started in a process group of its own (as system_profiler does), its scratch directory and the probe workspace are all gone when run_server returns, without waiting out the scan -- and a build that had not spawned its builder when shutdown began never starts one. End to end on a host with a really enforced sandbox, a kernel reports the seeded (deliberately empty) list instead of scanning; the version it seeds is read in both spellings (__version__ = 390 and '3.11.0'), and it skips as "matplotlib is not installed" only when it is not.
test_kernel_generation_storage.py Where kernel generations and attempts are written down. Startup reconciliation may only abandon generations left by an older daemon, never a live one, and the generation audit is not reachable from host.query.
test_kernel_generation_supervisor.py A persistent UUID identity laid over the supervisor's integer counter. The integer stays as the ABA guard; the UUID is what a lease and the status projection bind to, so a replacement worker can be fenced.
test_kernel_oversized_frame.py What happens when a cell's response is too big to send. _write_frame rightly refuses a frame over _MAX_FRAME_BYTES — the host would have to materialise it whole, and a truncated JSON line desynchronises the reader — but it replaced the frame with a log, so a dropped response left the host waiting on a cell id that never arrived. Measured before the fix: raise ValueError("x" * 12_000_000) had not returned after 90 seconds. To a user that is a hang, not a refusal. Two independent halves: error is now capped like the stdout and stderr beside it, and a dropped response still answers its id. Each prevents the hang on its own, and each is separately falsified.
test_kernel_recovery.py Rebuilding a dead kernel. Recovery runs in an isolated candidate and is published only after hydration, replay, and validation all pass. A step with an external side effect is not replayed at all, and a partial recovery keeps the old generation rather than claiming the new one.
test_kernel_sandbox.py How the Seatbelt and bubblewrap commands are built, and how the mode behaves. Single-user auto degrades visibly, while a team read policy never does. Coverage includes whole-data and multi-root masks, exact runtime/Skill/personal exceptions, system-temp leftovers, bounded streaming inventories, pre-existing hardlinks and symlink aliases, private PID/mount order on Linux, namespace-init-to-command pidfd adoption with ambiguous process trees refused, and mandatory complete self-tests.
test_user_llm_keys.py A user's own LLM credential (M4-1). The claim that matters is negative: Alice's key is used for Alice's session and never for Bob's, because a per-user credential that leaks across sessions is worse than no feature at all — the billing looks right while the trust boundary is gone. Also: absence falls back to the group key unchanged (INV-1), the override is per provider, no route reads a value or a reference back, disabling an account takes its keys with it, and a configured-but-unreadable key refuses the turn rather than quietly charging the group.
test_compute_session_routes.py The run-location routes over real HTTP against a real daemon, because the status code is half of what they promise and a direct call cannot observe it. The claims are the boring-sounding ones: a daemon with no worker listener answers "this session runs locally" rather than erroring, since that is what every install does; an unconfigured profile is refused rather than guessed at; and another user's session answers 404, not 403 (INV-13).
test_cluster_session_e2e.py The whole chain offline: a fake sbatch that really runs the job it was handed, and above it the production path — real broker shelling out, real backend, real reconciler, real Store, real gateway, real credential, real worker.py in its own process, real frame protocol over a real socket. A stubbed launch would assert that the code does what it was written to do; the questions here are the ones only a subprocess can answer — whether the credential the daemon wrote admits the worker it was written for, whether a variable survives to the next turn, whether an interrupt reaches a process with no local pid, and whether a lapsed lease actually takes the resource back while the kernel is demonstrably alive.
test_query_scoping_team_mode.py host.query must not read a colleague's prompts. The denylist covered identity and governance tables and the artifact family sat behind scoped views, but messages, execution_log and frames were readable — in a single-user install that is the user's own work and is documented as allowed; in team mode it is every colleague's prompts, the model's replies and the code they ran. Both halves are asserted, because the closure is conditional: team mode refuses the direct read and answers through the scoped view, and team mode off reads exactly what it always read (INV-1).
test_execution_principal.py host.frames must not read a colleague's cells. The team-mode scoping lived in the SQLite authorizer, which only host.query goes through; host.frames reads the same rows — frame names, cell code, stdout — through ordinary Store methods, so a member's agent could browse every session, regex-search every cell and open any colleague's frame by id. What is asserted is the principal, not a parameter: absent is refused (team mode with no principal raises rather than widening), None is never an operator (the single-user daemon and the loopback CLI carry explicit principals), and the filtering happens in SQL before the code and stdout are read. The owner-still-sees-their-own case is asserted too, because a predicate that refuses everything passes every "the stranger cannot" assertion.
test_cluster_session_production_wiring.py The daemon's own wiring for cluster sessions, not a rehearsal of it. Every earlier cluster test builds the Kernel or the manager itself, which is why attach_worker having no production caller stayed invisible through a green CI. These go through SessionRunner and build nothing the daemon would build for itself — verified by removing each wire and watching exactly its test fail.
test_orchestration_distributed.py A distributed task runs inside the resource a workload already holds (INV-4), and a multi-node session is ready only when every rank has registered (M4-3). Both are cases where a wrong system would look plausible rather than broken: srun without --jobid allocates, which turns one interactive session into two jobs that are both billed and one of which nobody watches; and a run started while other nodes are still being placed fails inside the user's computation, where it reads as their bug. The gang tests use real sockets, because the defect they were written for — every rank after the first silently replacing the last — is invisible to a double that stores one registration.
test_orchestration_session.py Persistent sessions on a resource plane, against a real Store and the real Reconciler — the claims are about what is durable after a step, which an in-memory double cannot answer. The tests that carry weight are the negative ones: three of INV-5's four conditions holding is not ready; a session whose kernel is perfectly alive and heartbeating still expires, because a worker being alive is not a user being present; a lost node produces a new epoch and an announcement rather than a silent reconnect (INV-6/11); and a session being cancelled that also loses its node is not "recovered" into resubmitting the work the user asked to stop.
test_worker_tcp_transport.py A real worker subprocess, over a real socket, driven by the real Kernel — the only arrangement that can support the claim, since a fake socket exercises the parts that were never in doubt. It asserts what the plan pins rather than what is convenient: the credential works once, an old epoch is refused (INV-7), a forgery and an expiry and a replay all come back as the same word, the secret rides a 0600 file and never the environment (INV-9), and a mid-cell host_call round-trips — because the host-call transaction discipline is the part a transport change is most likely to quietly break.
test_kernel_supervisor.py Worker lifecycle with no assumptions about the protocol underneath. The recurring theme is exactness: a stale lease cannot interrupt, kill, restart, or abandon a new worker, and a dead replacement is rejected without destroying the healthy current one.
test_lazy_kernel.py Four tests on the CLI's one-shot kernel ownership. A context that never runs code never creates a worker, and a failed bootstrap does not publish a broken one.
test_llm_anthropic_streaming.py The Anthropic SSE path, and the one property that makes it safe to add: streaming and blocking must normalize the same reply. wire_state is replayed verbatim as the next turn's assistant content, so a block the reconstruction does not recognize — redacted_thinking, a server-tool block — has to survive intact rather than collapse into an empty text block, which would fail the request after the one that produced it. Also pins the fallback boundary in both directions: a stream that dies before its first event retries blocking without stream in the body, and one that dies after a delta is raised rather than replayed.
test_llm_anthropic_tool_calls.py Anthropic Messages tool calls through an encode/decode round trip. Parallel results have to come back in one adjacent user message, which is exactly where a naive history rebuild goes wrong.
test_llm_capabilities.py The provider capability catalogue and token accounting. Usage normalization tolerates a missing or nonsense counter, and cost is computed only from explicitly configured prices, never from a guess.
test_llm_gemini_tool_calls.py Gemini generateContent, and its awkward corner. When the wire supplies no call ID a stable local one is synthesized, but it must never be replayed back to Gemini, and opaque part metadata — signatures included — has to be replayed on the original parts.
test_llm_openai_tool_calls.py OpenAI Chat, including the streaming path. Interleaved tool calls aggregate losslessly across SSE deltas, a partial call with no terminal event is rejected rather than half-executed, and an error event is not mistaken for an empty success.
test_llm_providers.py The stdlib transport with the network patched out, so wire selection, config resolution, URLs, auth headers, and image translation can be asserted exactly. A loopback endpoint may run without an API key, but it does not inherit the vendor's tool support just because it speaks the same wire.
test_llm_responses_tool_calls.py The OpenAI Responses wire: output items preserved and replayed before the function result, incomplete or unterminated tool streams rejected, and a canonical call history rebuilt without any leftover wire state.
test_local_model_discovery.py Discovering a local model endpoint, which is an SSRF surface. Candidate URLs must be literal loopback, the opener refuses redirects, and an unreachable, invalid, or oversized probe stays inert.
test_marker_policy.py Two tests protecting the offline contract. Every opt-in marker — external, network, live-LLM, GPU, SSH, Docker, browser, lab — is deselected by default, and an edit to the default deselection expression cannot quietly re-enable a live test.
test_mcp_client.py MCP offline: JSON-RPC framing, the stdio lifecycle, request correlation, timeouts. The boundary that matters is the child environment, rebuilt from a strict allowlist so connecting a connector cannot hand the daemon's ambient secrets to a subprocess.
test_mcp_control_tools.py The MCP native tools. Listing a catalog and reading content are different policies, prompt arguments are revalidated as strings, and content coming back from a connector is screened for injection before the agent sees it.
test_mcp_lifecycle.py MCP lifecycle and bounds, driven against real subprocess connectors so the claims are pids and processes rather than calls. A failed handshake leaves no process and no reader threads; each fault class evicts that exact cached instance and the next call reconnects on a new pid, while a JSON-RPC error does not; the in-flight and abandoned-id queues are capped; an 8 MB stderr line is truncated while it is read; and termination reaches a wrapped server's real worker without hanging. Also covers the connect path's locking: _connect spawns a child and runs the initialize handshake, and it used to do that inside the manager's global lock — the one every connector's get, _evict, disconnect and shutdown need — so one server hanging until its timeout stalled every other connector in the process, including ones already connected and merely being looked up. Asserted on wall-clock ordering rather than on the lock object, because what a caller experiences is the wait; a second case pins that two callers for the same id still produce one child.
test_mcp_tools_positional_server.py mcp_tools driven through dispatcher.__call__ with the kernel SDK's positional-string server — the shape that crashed with 'str' object has no attribute 'get' once the method grew a control tool, while host.mcp.list() and host.mcp.call() kept working. Pins the dual-shape execute/resource_keys contract, the dict spec, the locally answered volcengine-datapro descriptor, and the soft error for an unknown server.
test_memory_repository.py Long-term memory: filters preserved across the Store boundary, the legacy default category, and a project delete that cascades its memories.
test_metadata_repositories.py The small repositories — notes, folders, dynamic endpoints, the compaction archive. The host-call log is the one with teeth: it scrubs, skips, and truncates before it commits.
test_methodology_skills.py Three tests over the methodology-only bundled Skills: discovered read-only, retrievable, and one of them exercised inside the agent loop.
test_mineral_spectra_analysis.py The mineral spectra Skill, and the design it protects. The pipeline is blind: the report is produced without the ground truth, and evaluation against truth is an explicit step after the loop, never folded into it. The pure helpers import without the science stack installed.
test_model_profile_probe_reachable.py Whether a user can find out that their model configuration is wrong. readiness (local-only, on every listed profile) and POST /model-profiles/{id}/probe were both implemented, tested and served — with no call site. The Models tab derived its own status from has_api_key, a binary carrying no reason and unable to express unsupported: a profile naming a protocol this build cannot dispatch has a key, so it read as configured and failed only when a turn used it. Nothing could reach the probe at all, so "is my endpoint reachable" was answered by sending a message. The probe stays behind a click and stays POST — a render-time call or a GET would spend the user's own provider quota, which is exactly what the local/remote split exists to prevent, and a mutation moving it to render time fails here. Records honestly that needs_model is unreachable for the five protocols on offer, since each has a default model.
test_model_profile_readiness.py Whether a model profile is usable, and who decides to find out. Readiness is derived from local state so listing profiles costs no network; reachability needs a request, and a request is something the user asks for — a card that quietly probed on render would be the implicit outbound call P0-1 removed, aimed at the user's own paid quota. Also pins that every dispatchable provider is either selectable or declared withheld: gemini and openai_responses were neither.
test_model_revision_binding.py D2: a session names the model configuration it ran under, not the one its profile happens to describe today. Covers what a revision is keyed on — (provider, base_url, model), deliberately not the name and cannot be the key, since the credential reference is derived from the profile id — plus the send-only binding, both 409 rebind paths, and the v9→v10 upgrade of a populated database, which must leave historical rows unbound rather than defaulted.
test_memory_budget.py How much remembered context enters a system prompt. The injection bounded the count and nothing else, so 50 memories of a pasted protocol added ~600,000 characters (~150k tokens against a 262k window) to every turn — a count cannot bound this, because length is the thing that varies. Each budget test asserts the other two cannot fire first: an earlier version of the total-budget case appended an index suffix that pushed every item over the per-item cap, so it was silently re-testing the wrong limit. Also covers that an over-long memory is skipped rather than clipped, that order is preserved so the oldest is what goes, and that the model is told when something was withheld. The second half of the same gap is the Context panel: it reported Text: N tokens for the system prompt and the conversation together, so a 150k-token standing context read as "your conversation is long" and sent the user to compaction — the one remedy that never touches a system prompt. The projection now separates system_prompt, keeps total unchanged, and reports what the budgets omitted as aggregate counts rather than re-rendering memory text that was deliberately withheld.
test_memory_edit_and_expiry_quota.py Retention meant one thing when reading and another when writing: expiry withheld a memory from injection and deliberately never deleted it, but admission counted every row, so a scope of two hundred expired memories refused every new write about entries the pane already lists as omitted. Admission now counts live rows, with a separate stored ceiling and its own code because the remedies differ. The second half is the operation that never existed — a memory could be written and deleted, never corrected — and its updated_at, without which a correction expires on the clock of the instruction it replaced.
test_message_pagination.py Which end of a long conversation you see. Opening a 640-message session returned messages 0–299 — the oldest page, with the newest 340 absent — because the only order was seq ASC with a limit and no direction. Covers latest-first paging, the before_seq keyset cursor (an offset would shift on every arriving message), that a page stays stable while a message arrives, and that the old oldest-first response is unchanged when the flag is absent.
test_model_catalog.py Provider and model catalogues you can extend. A custom provider is validated before it can route, a hyphenated provider name still yields a shell-safe environment prefix, and the built-in presets stay immutable while user presets are added around them.
test_native_tools.py Four tests on the native tool declarations. Names must fit the intersection of four providers' limits, every call returns an independent schema copy, and neither shell nor completion is ever declared as a native tool.
test_notebook_export.py Three tests on export. Python and R become separate read-only notebooks, the bundle is deterministic with manifest checksums that match, and an unknown notebook language is rejected.
test_notebook_lineage.py Stage 8 official live Notebook and host-side cross-language lineage. The flag opens the REPL without the old developer switch; R CSV → Python JSON writes a version edge and two env snapshots; Agent and user REPL share one kernel generation.
test_model_capability_receipts.py Exact model-capability receipts from an explicit probe. true only from a schema-valid native tool call or a fully terminated stream; timeout/auth/5xx stay unknown; a new profile revision adopts zero old receipts; restart reloads the exact row; the key never enters the receipt.
test_onboarding.py First-run provider setup, where the write has to be all or nothing. An invalid API key cannot leave the model settings half-written, switching provider drops the previous provider's key instead of reusing its defaults, and no response ever carries the secret back.
test_web_onboarding.py Web first-run routes. GET /onboarding is redacted and contacts nobody (Test 前 outbound=0); POST /onboarding/complete is instance-global and admin-only in team mode; the key hits GET/diagnostics zero times.
test_orchestration_backend_opacity.py INV-2, written before the first backend existed — a leak guard written afterwards codifies whatever leaked and calls it the rule. Holds that the orchestration core never names a scheduler in its source or its import graph, and that importing the contract loads no implementation. Writing it paid for itself twice before any backend code was: the source scan caught scheduler names in the core's own docstrings (the check is literal on purpose — allow "explanatory" mentions and every leak arrives as one), and the falsification case caught a hole in the guard itself, since \bslurm\b does not match slurm_backend — _ is a word character — so import paths are matched as substrings, not words.
test_orchestration_routes.py BatchJob end to end over real HTTP with the reconciler actually running, because the DoD's chain — submit → PENDING → RUNNING → terminal → logs — crosses four components and driving them separately leaves the seams untested. Local jobs run real processes; the cluster case runs the same routes against the fake scheduler, which is the point of the abstraction. Also pins what must not appear: a response carries allocation_id and never a backend job id, queue name or pid (INV-2); a command string is refused rather than split; someone else's job is 404 everywhere; an admin cancelling another user's job is recorded as ADMIN_CANCELLED. OPENAI4S_RECONCILE_INTERVAL shortens the daemon's 5s cadence here — the file took 75 seconds of waiting for ticks before that, and a suite slow enough to skip is the expensive kind.
test_orchestration_reconciler.py The state machine with no database and no daemon — an in-memory store that asserts INV-3 the way the real partial index does, so a test can never pass with two live allocations. Pins the invariants (an Unknown submission is never retried blindly; repeated unknowns never accumulate submissions; recovery is a new epoch rather than a rewrite) and the two things reconcilers get wrong in practice: a BACKEND_UNAVAILABLE observation must move nothing, and the cancel barrier must be walkable twice. That second one caught a real defect — RELEASING was not counted as an active allocation, so the barrier could not find its own allocation on the next pass and marked a workload cancelled while its job was still running. The local-backend half runs real processes: completion, nonzero exit, cancel-kills-the-tree, the concurrency refusal, token reconciliation, and a check that the daemon's environment (which holds API keys) is not inherited.
test_orchestration_slurm.py The Slurm backend against a fake cluster: real sbatch/squeue/sacct/scancel programs generated into tmp_path and put on PATH, with a programmable state sequence (deliberately not in fixtures/, which is byte-exact captured data — these are programs). Covers the cases where being wrong is expensive: a lost submission becomes Unknown and the retry after it reconciles by token into Existing rather than putting a second job on the cluster (INV-8); a job in neither the queue nor accounting is LOST, never COMPLETED; an unreachable scheduler leaves the phase alone; argv is constructed so a hostile profile name is an argument rather than a command, and a credential-shaped environment variable is refused (INV-9). Two findings came from writing it: swallowing "no scheduler on PATH" as "no such job" declared every running job lost, and a fake that ignored --format let a parser read the wrong column and pass. A last test asserts the fake is really on PATH — otherwise the whole file could pass against a machine with no scheduler and prove nothing.
test_orchestration_skills.py The runtime-orchestration Skills driven against a real worker and a real dispatcher rather than fakes: the kernel-side host facade, Skill customization, self-awareness queries, managed endpoints, compute and environment setup, and the audit trail they leave.
test_permission_repository.py Permission rules and permission requests in storage. Two things to know: a request is atomically bound to its Action Ledger group and rolls back if that group is unknown, and a restart-scoped one-time grant is consumed atomically so it cannot be replayed.
test_permissions.py The permission gate itself. Scope precedence and pattern specificity are here, but so is the part with no override: reading a secret file or a secret environment variable is denied even when a conversation rule says allow, and the headless broker fails closed unless an operator explicitly allowed the action.
test_plan.py Plan extraction and the plan lifecycle. A model reply that gives prose instead of JSON still yields a plan through the prose fallback, and approving one runs it as a normal turn.
test_plan_repository.py Three tests that PlanRepository behaves identically through the Store facade, malformed JSON included: it falls back rather than raising.
test_plan_resume_claim.py Resuming a paused plan, raced by two real concurrent POSTs against a live ThreadingHTTPServer. The transition was a read in the route and an unconditional write in the job it spawned, so both requests were accepted and both turns executed the same steps — the turn is counted here, not inferred from the status codes.
test_plan_service.py The plan service boundary. Finalizing reuses the draft row and its Artifact instead of creating a second one, and it survives a failed Artifact write rather than losing the plan.
test_ports_helper.py The _ports.py contract, stated so it can fail: the allocation is a single bind of port 0 — asserted by recording the bind sequence, because "the port is held after the call returns" cannot tell a clean bind-0 from a probe that closed and rebound (a rebinder also holds the port by then), and mutation-checking proved exactly that. The second half: the handler attached after the port is known is the one that answers.
test_protein_mutation_enhancement_skill.py The protein mutation Skill's pure helpers. Enumeration, ranking, and the selection rounds are deterministic, and the error paths are checked explicitly: a mutation whose wild-type residue does not match, an out-of-range position, and a variant with no position all raise rather than quietly being scored.
test_protein_design_mcp.py The atomic protein-design MCP connector, driven against real stdio and fake backend subprocesses. It pins its bundled connector-directory entry, the exact nine-tool catalog, RFdiffusion's deterministic + design_startnum seed contract and PDB/TRB pair, ProteinMPNN's official chain/fixed-position flags plus independent motif/target/map checks, blind ColabFold's network namespace and frozen parameters, terminal failure records, path confinement, and the corrected Rosetta unsatisfied-H-bond field. The whole module is stubbed_backend, so fabricated scientific shapes never enter the captured API schemas.
test_prov_record_confinement.py Which files a cell may register as artifacts of its session. provenance_record resolved its path with Path(path).expanduser() and checked only that the file existed, so any absolute path on the host was registerable — verified before the fix: a private key written outside the workspace came back with a version id, listed and downloadable through every surface that shows artifacts. The confinement was not missing, it was unused: self._resolve_path is injected into this service and its siblings call it. The same function also read the whole file into daemon memory to checksum it while every other artifact path streams.
test_provenance_identity.py Whether a tagged object is still the object that was tagged. The side table mapped id(obj) -> tags, and an id is not an identity: CPython reuses a freed address immediately, so an unrelated object inherited a freed one's lineage — reproduced on the first allocation. That is a fabricated provenance edge in the subsystem whose whole claim is that a result is reconstructible, and the table is the primary path, not a corner: list and dict both land there, which is what json.loads returns. Two of these tests exist because mutation testing broke the first version — waiting for a real address collision cannot falsify the fix (pinning prevents the collision, so the test fell through to a weaker assertion), and sizing a loop from MAX_SIDE_TAGS makes a raised cap hang instead of fail.
test_provenance_paths.py Filesystem identity inside the worker, which is where provenance quietly breaks. A Cell that changes its working directory between opening a file and writing it must still produce the same canonical path, and the last test proves that against a real kernel.
test_public_api_contract.py The public import surface, pinned so the backend behind it can keep moving: package version, constructor parameter names, run_task's calling convention, and the Host and server facades. It pins what a caller can see and nothing more.
test_public_exception_projector.py One exception carrying a credential, an absolute path and a shell command, raised on every public surface — HTTP dispatcher, WebSocket chunk, async job result, plan job, REPL job, connector call and remote compute — asserting that none of the three reaches the response while a stable code and this daemon's own request_id do. public_failure is an envelope and never rewrote error, so an except Exception that wrote str(e) had the raw text shipped with a tidy code bolted on. The connector call is here twice over: it answered 200 with {"error": …}, and the web client only rejects on a non-2xx, so a connector that never ran was reported as one that did.
test_r_kernel.py Includes the variable inspector's contract, rewritten. substitute() does not substitute bindings from .GlobalEnv — the only environment the inspector reads — so every ordinary variable came back as an opaque symbol with no type, length or preview, and the previous test wrote that outcome down as the contract. Safety and uselessness had been conflated: reading an ordinary binding runs nothing, only a promise has anything to force. The inspector now probes with substitute first and reads for real when the probe says nothing. Stated rather than hidden: on current R a delayedAssign binding is forced when inspected, and a promise whose body raises degrades to the opaque answer instead of failing the whole list — which is how this surfaced, one stop() turning every variable into "failed closed". Invoking an active binding remains a hard no. Also covers live output, which the R half did not have: the host has always accepted stdout_chunk frames and the R worker never emitted one, so a long R cell showed nothing until it finished. Streaming used to happen only between top-level expressions, and this row used to state the limit that followed: a chatty for loop arrived in one piece, because mid-expression streaming needs C-level work in a single-threaded interpreter. That stopped being true when the host began draining the cell's fifo — the drain sees bytes as R's connection buffer empties, which happens inside an expression too. The worker-side flush remains, for the expression that ends without filling that buffer. The interrupt path has its own regression pair now: a shell backgrounds a daemon with SIGINT set to SIG_IGN, that disposition survives every exec down to Rscript, and R — unlike the Python worker, which installs its handler unconditionally — declines to install its interrupt handler over an inherited ignore, so the R worker was born uninterruptible; asserted as interrupted=True under a SIG_IGN spawner against both the protocol fake and real R. And rm(list = ls(all.names = TRUE)) is a user's idiom, not a kernel kill: the worker kept its loop, helpers, connections and lineage controller in .GlobalEnv, so that cell deleted the worker and the session with it; the real-R tests assert the same pid, working file-read tracers and an inspector listing exactly the user's bindings afterwards, and a source tripwire pins the private-environment shape for CI jobs that have no Rscript.
test_recovery_recipe.py Three tests on the recovery recipe. It compiles a dependency-closed Python or R namespace, leaves external and unresolved state as manual steps for a human instead of replaying them, and refuses to run at all unless the recorded source hash still matches.
test_skill_catalog_readiness_route.py What the Web Skills catalogue tells a user about whether a Skill can run here. The loader computed requirements and ready/needs_setup/unknown; the Web projection rebuilt each row without them, so the route never carried them and Customize could not draw them — a GPU-only Skill looked exactly like one that runs anywhere, and only the agent-facing host surface saw the difference. Drives the real route, keeps readiness beside enabled instead of folded into it, and pins the property that makes a catalogue safe to render: the guarded request starts no subprocess and opens no socket.
test_skill_network_contract.py B-02 Skill network manifest and execution admission. Catalog traversal starts no socket and no subprocess; every bundled Skill has an explicit mode; raw_required is fail-closed even with the raw-network compat switch; host_only undeclared/unapproved domains and non-compliant redirects are refused; Cell and shell sinks have zero bypass; a loaded manifest never changes egress grants; third-party collection Skills stay legacy/unknown and do not gain authorization from the new field.
test_skill_requirements.py What a Skill needs before it can run. Fourteen bundled Skills have declared requirements: [gpu] since they were written and nothing read it, so a GPU-only Skill was indistinguishable from one that runs anywhere until execution time. Covers the three states (ready/needs_setup/unknown — an unknowable requirement is never guessed in either direction), that browsing the catalogue reaches no network, and that readiness does not spawn a subprocess per Skill.
test_resource_allowlist.py Tri-state Skill allowlists, and why None and [] must not agree despite both being falsy — if not allowed: turns "deny everything" into "allow everything", failing open for exactly the configuration a user chose to lock something down. Also covers the four surfaces the exit criterion names: an unlisted skill is absent from the catalogue, from search, from load/get, and from read (the one that hands back file contents), with the same answer as a skill that does not exist.
test_retrieval_source.py What a client may see of where retrieved data came from. The artifact_versions.source envelope has recorded the request URL, query and response hashes since retrieval provenance was added, and was selected by nothing — so a figure built on a live API fetch looked exactly like one computed from thin air. Sending it raw is not an option either: it is free-form JSON written by unaudited retrieval code, and scientific APIs routinely take the key in the query string. Covers the allowlist, the length cap, and redaction in the query, the path and the userinfo.
test_release_gates.py The gates that run before a release. The secret scanner reports a hit without echoing the value, and the archive verifier rejects a third-party core dependency, a missing collection marker, a manifest hash/size mismatch, or an unmanifested collection payload. Installed-wheel and desktop-bundle checks also prove the pinned collection cannot make its own floor optional, while the directory-docs gate keeps the collection boundary pair in scope.
test_remote_capability_probe.py Command injection over SSH, closed one input at a time. A shell string is rejected before SSH starts, an ambiguous or ill-typed spec never reaches it, and even a rejected probe still projects an activity step so the attempt stays visible.
test_remote_compute_control_tools.py The remote compute tools. Submitting a job needs approval; cancelling and closing one does not, because cleanup has to stay reachable even while new work is being refused.
test_renderer_capability_honesty.py What the renderer catalog is allowed to claim. A capability nothing implements — sort, filter, compare_versions — has to be absent rather than documented, and a columnar binary with no parser is declared download-only instead of promising a table the viewer then fails to draw. Flag-on catalogs may name sort/filter/profile/export, and parquet only when the optional engine is actually importable.
test_renderer_registry.py Three tests: renderer selection is deterministic, duplicate renderer IDs are rejected, and the public descriptor keeps version and provenance bound to the renderer that produced it.
test_retrosynthesis_evidence_regressions.py Evidence retrieval for retrosynthesis routes. Every test here exists to stop a candidate from passing itself off as verified: a candidate-only set of sources cannot push coverage past 30, an explicitly empty record wrapper never becomes pseudo-evidence, and the verified flag stays false until something actually verifies it. A search or fetch that fails is isolated rather than taking the sources that did come back down with it.
test_retrosynthesis_evaluator.py Installed public/private dataset hash verification, strict mapping fixture types, and directly runnable GT entrypoints.
test_retrosynthesis_generated_programs.py Generated Scenario CLI contracts: production chemistry canonicalization, unique mapping identifiers, and public fixture compatibility.
test_retrosynthesis_generation.py Generator process isolation, secret-free diagnostics, and preservation of per-Scenario generation provenance.
test_retrosynthesis_planning.py The retrosynthesis Skills, including discovery/model identity for six independently testable tasks, the six-Scenario document contract, and the executable class-unknown single-step protocol: hidden reaction fields fail closed, unordered precursor sets compare correctly, invalid/duplicate beams remain visible, and private metrics aggregate by target. Two rendering hazards get most of the remaining attention: LLM annotations must degrade with a warning when the model returns something unparseable, and graph payloads must escape molecule names received from outside.
test_retrosynthesis_scenario_protocols.py Executable protocol regressions for Scenarios 2–6 plus the foreign-environment model boundary: AND/OR stock closure, AiZynthFinder route conversion, atom-map secrecy and symmetry, forward stereochemistry diagnostics, complete Parrot beam tuples, un-clipped worst-group yield evaluation, pinned environment plans, artifact snapshots, and manifest-bound worker handshakes.
test_retrosynthesis_scoring_regressions.py Route scoring and rendering, which is where the fiddly bugs live. A rerank must not leave annotations pointing at the routes they used to describe. The string "false" in a stock flag never counts as available. Normalized decision weights sum to exactly one hundred, and a solved route with no reactions is not penalized for evidence that could never have applied to it.
test_revert_projection.py Three tests on Revert and Undo history: it is append-only and survives a restart, sibling branches never mix their messages, and legacy rows are backfilled to a canonical root branch.
test_review.py The Reviewer end to end through the gateway. Its verdict is constrained: an invalid schema is reported as unavailable rather than coerced into a pass, and a provider error is nonfatal — a failed review must not fail the turn it was reviewing.
test_review_service.py The same Reviewer as a service, outside the gateway, and here the concurrency is the substance. A cancelled provider call still finishes on its own thread while blocking duplicates, and a failed thread start cleans up the reservation it had already made.
test_science_connectors.py Every scientific database connector against offline HTTP fakes. They share one record schema, they never build an arbitrary URL out of user input, and an upstream that returns a 204 or a schema it never promised produces a bounded connector error instead of a crash.
test_stage10_connectors.py Stage 10 ClinVar/PubMed/ClinicalTrials: flag-off catalog stays the original seven, fixtures return accession/URL/time and a versioned Artifact, and empty/429/schema-drift answers are honest. Artifact publication uses a pinned parent directory and an inode-verified private staging file, so workspace-path swaps cannot redirect it and a hardlink substitution fails closed without publishing or leaving an alias behind.
test_stage10_live_canaries.py Live canaries against public ClinVar, PubMed, and ClinicalTrials.gov. Marked network/external; not in the default offline suite.
test_stage11_remote_compute.py Stage 11 durable remote compute: restart only reconciles, cancel hits the exact receipt, unknown/timeout are not success, and harvest Artifacts name receipt, input versions, and checksums.
test_stage12_ga.py Stage 12 GA kill switch stays default-off and the evidence table names every stage.
test_connector_manifests.py A connector manifest must describe the connector, not decorate it. Proves, against the connectors' own offline fixtures, that every required path is present AND load-bearing — delete it and the adapter stops returning the record. Without the second half a manifest drifts toward listing every field anyone might like, and the canary built on it alarms on things that do not matter.
test_connector_canary.py The canary's one job is to tell an outage apart from a contract change. Drives the classifier with an injected fetch — no network — through a healthy 200, a 200 whose required field was renamed (drift), a timeout, an HTML 200, and a transient failure that retries then succeeds. Getting the distinction backwards in either direction is the failure mode: alarm on an outage and it gets muted, stay quiet on drift and it was pointless.
test_platform_support.py Windows is unsupported, and unsupported has to mean refused. The load-bearing test is not "does it raise" but "does it raise before it spawns" — a refusal after Popen would satisfy a naive check and none of the intent. Mutation-checked: removing the guard, adding Windows to the supported set, and downgrading the refusal to a warning (the original defect) each fail.
test_science_control_tools.py Three tests over the two scientific search surfaces, native and in-kernel: two flat schema-checked registry entries, a catalog that runs with no network, and a Host wire that encodes only top-level fields.
test_sdk_compute.py The worker-side host.compute namespace: providers, instances, transfer, attach, cleanup. Path handling is the part with a security edge — local paths are relativized against an allowlist and an absolute path is rejected outright.
test_security.py The defense-in-depth layers with the LLM mocked. Read the fast paths and the fail-open cases as carefully as the blocks: routine code skips classification entirely, and the injection screener fails open when no key is configured. What does not fail open is the query denylist, which keeps host.query away from settings, connectors, and secret tables even through a split-identifier bypass.
test_server_agent_run.py The Web adapters around the pure engine, and most of this is about what the user must not see. A fence stays hidden across delta boundaries, a code draft is transient and discardable, and a completion-only Cell has its draft cleared before it runs. Legacy fenced Tool calls still execute and come back as a user observation.
test_server_completions.py What the user is shown at the end of a turn. The completion message is rendered from the real Artifact/capture delta and the actual output, never from hidden reasoning; trusted links come from the server helper and exact version id, while a code-only turn gets an honest status about what ran instead of an implied success.
test_stage1_browser_fixture.py Builds the browser acceptance's local metadata readiness fixture and uses the production Store, Artifact manager, and completion-delivery verifier to seed 100 atomic deliveries plus same-head/head-change and delegated-producer evidence. Focused tests keep the seed path deterministic, scoped, and free of LLM/network calls.
test_server_execution_coordinator.py Admission and cancellation on the Web side. Cancellation without an exact execution and owner identity fails closed; it must never fall back to cancelling whatever happens to be running. A queued REPL cancel cannot touch the active Agent.
test_session_sidebar_scope.py Whether a project's sessions are visible when you open that project. loadSessions fetched /frames?limit=100 globally and filtered by project in the browser, so a project whose sessions sat outside that page appeared empty — and openProject reads empty as a reason to call newSession(), presenting a blank session to a user whose work was intact in SQLite. Measured: 120 sessions in one project push a three-session project entirely out of the global page; 0 of 3 visible before, 3 of 3 after, confirmed again in a real browser. The server has supported project_id, cursors and next_cursor the whole time — this was one unused query parameter.
test_session_and_message_paging.py Reaching the part of a project, and of a conversation, that is off the page. Walks the session-list keyset cursor across 260 sessions and the message cursor back through 640, asserting no gap, no duplicate, and that a session or message arriving mid-walk does not shift a later page. Pins the two page-size defects the walk exposed: ?limit=banana reached the browser as a 500 internal_error while the same route already answered 400 for before_seq=banana, and ?limit=-5 answered 200 with an empty page and has_earlier: false — the exact shape of "you have reached the beginning". Exercises the session package export end to end: first message, last message, and a forked branch's own message, with every message_id still attached to its own row.
test_session_branching.py Four tests on checkpoint, fork, revert, and undo. Revert preview reports every state dimension without writing anything, and an external edit to the workspace blocks the revert and is recorded instead of being overwritten.
test_session_control_tools.py The current-session and capability tools. Capability search is always visible and activates the session group, approval inputs are redacted through the dispatcher, and a mutation fails closed when the domain it needs is not filesystem-aware.
test_session_deletion.py Deleting a session or a project without deleting someone else's data. The content-addressed store is shared, so its GC waits for a checkpoint's references to be published; snapshot cleanup never follows a symlink out of the tree; and a feedback delete escapes LIKE metacharacters.
test_session_domain_service.py One composition behind snapshots, checkpoints, cursor forks, branches, Timeline, export, renderer, and recovery, all reached through the Store facade. A failed cursor checkpoint is audited without claiming a fork happened.
test_session_package.py Session export and import, which means a full untrusted-archive surface. Traversal, symlinks, compression-ratio failures, duplicate or dangling identities, and embedded secrets are all rejected; anything resembling a replay hook is quarantined until a confirmed fresh restart; and an import that faults rolls back the database, the workspace, the environment, and the CAS together.
test_session_recovery.py The idle sweeper. Every blocker has to be clear before it releases a session, and an in-progress recovery is one of them — the sweeper cannot pull a kernel out from under it. TTL parsing, persisted activity, and startup reconciliation are here too.
test_session_snapshots.py Workspace snapshots in the content-addressed store. Snapshotting excludes secrets, symlinks, and oversized files; restoring refuses to run over an externally changed workspace and leaves untracked files alone; branch heads move under a compare-and-swap.
test_share_expiry.py Share expiry / auto-revoke: expires_at is recorded, the sweeper revokes lapsed shares and keeps live ones, no-expiry shares are never swept, update keeps or clears the expiry, restore revokes anything that lapsed while the daemon was down, and duration parsing.
test_sandbox_credential_denies.py Which credentials an enforced kernel sandbox actually keeps from a cell — measured against a real sandbox-exec, not inferred from the profile text. The deny list carried a prefix rule on openai4s.db and the daemon's access token is a sibling, so the DB was blocked and the token was read; that token gates the whole HTTP API. And a cell could still run /usr/bin/security and reach the macOS keychain, where OPENAI4S_SECRET_STORE puts the LLM API key by default. Denying the keychain files is the tempting wrong fix — securityd opens them on the caller's behalf — and the tests fail on it. TLS is asserted to still work under the same rules, because that was the trade this fix could have got wrong.
test_share_gateway.py The gateway wiring: sharing is off with no tunnel by default, enabling while unconfigured reports the missing settings, create/revoke works without a tunnel, session deletion revokes shares, and the storage cascade drops share rows.
test_share_import.py URL-import SSRF policy (HTTPS-only off loopback, no credentials, private-address and non-HTTP rejection, bundle-path normalization) and the import-side static injection annotation of untrusted messages.
test_share_projection.py The flattened share snapshot: it round-trips through import_bytes, carries no dangling references, keeps memories/permissions/capabilities empty, flattens revert-to-sibling history, redacts secret-shaped text, fails closed on residual known-secret bytes, and matches the viewer document.
test_share_protocol.py The tunnel wire protocol: data-frame header round-trip and flags, size limits, unknown category, control-frame JSON round-trip, unknown/oversized/ malformed control frames, and the request-header allowlist.
test_share_relay_tunnel.py End-to-end over a loopback relay + tunnel: visitor fetch, Range, HEAD, unknown/bad-host 404, method 405, a poisoned response header becoming 502, different-principal conflict, and XFF ignored without a trusted proxy.
test_share_router.py The read-only router: the viewer shell CSP and security headers, the static whitelist, view/meta/artifact/bundle, inline vs attachment, Range, HEAD, method 405, a uniform 404 for unknown and revoked, and an AST guard that it imports no execution surfaces.
test_share_service.py Two-phase publish and lifecycle: ready snapshots, one active share per frame, bundle/view sharing one projection identity, update replacing the snapshot, a reader lease blocking GC of an old snapshot, revoke, revoke-for-session, and crash recovery.
test_share_viewer.py The static viewer assets load and the router serves the real shell/JS/CSS; the CLI share subcommands parse; and relay gen-token prints a token.
test_session_title_service.py Background session titles. It runs on another thread, so the tests are mostly races: it must not overwrite a title the user just typed, and it has to resolve a replaced Store if one arrived while the background work was starting.
test_setup_environments.py The bundled Python and R environment manifests, checked without conda anywhere near them. Two promises carry the module: an existing environment is never mutated unless you ask for an update, and an update targets the prefix it discovered without pruning, so packages you installed yourself survive it. It also pins that the Python manifest never pip-installs the NumPy-1-ABI RDKit, and that --only and --profile refuse to be combined.
test_single_cell_rna_analysis_skill.py Discovery, sidecar compilation and retrieval for the curated single-cell workflow; structural preflight failures for missing/fully confounded Harmony metadata and mixed references; raw-count and donor-replication gates; a generated deterministic paired-donor AnnData run that preserves the counts layer, produces Artifacts and resumes byte-stably; plus an opt-in network smoke over Pertpy Kang 2018.
test_session_tool_catalog.py Two tests on session tool composition. The Host gates the Dynamic Tool lifecycle before the proxy executes in that same catalog, and progressive disclosure keeps the core group while activating only the relevant ones.
test_settings_repository.py Settings, model profiles, and feedback. Model profile mutations are serialized by the shared lock, and permission seeding runs through this same repository, which is why the two are tested together.
test_skill_customization_service.py User Skills authored in the Web Customize panel. The path handling is the interesting part: deleting foo must not take foo-bar with it, and neither the directory nor the document may be a symlink pointing out of the user root. Bundled Skills stay read-only and win a name collision.
test_skill_recipes_are_callable.py Every bare name a bundled recipe tells the agent to call resolves — as a builtin, a binding in the recipe, a kernel.py export, or one of the two names the kernel injects. Two phantom APIs shipped before this gate existed.
test_skill_frontmatter_edit.py What a Web-Customize save does to the rest of a skill's frontmatter. It rebuilt the block from name/description/origin, so license and category (23 of 34 bundled skills), a nested metadata block (17), requirements (13) and fold_cue (1) were deleted by anyone fixing a typo. requirements is what makes it a correctness bug: readiness is computed from it, so a skill that lost [gpu] stops reporting needs_setup and starts claiming it can run anywhere. Both tempting wrong fixes are falsified — parse-and-re-emit (the parser flattens lists and ignores nested blocks by design) and a plain line filter (which mistakes the name: inside a nested metadata block for the top-level one). Covers the block-scalar continuation, idempotence across repeated saves, an unknown future field, and the real service end to end.
test_skill_product_surface.py Versioned personal and project Skills as a product surface: the control tools, the SDK calls, and the HTTP history and rollback routes. A rollback is scoped to the current project and audited, and a cross-project version scope is refused. Security coverage drives the real dispatcher: skills_edit asks by default, and even an explicitly allowed call from a team member cannot replace a project peer's SKILL.md or kernel.py. The real authenticated HTTP route keeps member rollback for a Web-authored recipe, while a retained legacy Host recipe or executable sidecar cannot reactivate until an admin authorizes it.
test_skill_reference_reading.py Reading one file inside a Skill directory. admet_genetic's recipe tells the agent to read references/data_contracts.md before running the pipeline, and until now the only spelling was host.skills.read(...) from inside a Python cell — so an agent in the tool plane, and a delegated child that never runs a cell most of all, structurally could not follow that instruction, while its natural fallback (read_text_file, workspace-confined) failed with a path error indistinguishable from "the file is not there". These drive a real fixture Skill through both doors: the new read_skill_file native tool with no kernel at all, and a delegated child with a scripted model and a REAL persistent kernel reading both references in one cell and returning both markers. Also the loader traversal guard that had no test (the only escapes skill dir assertion in the suite was against the edit path, a different function): .., an absolute path, and a symlink out of the directory. A child scoped to another Skill gets "no such skill", deliberately indistinguishable from absent; a child without the skills capability never sees the tool at all.
test_skill_sidecar_recovery.py Two tests, run against a real worker. Only sidecars that actually loaded are frozen into the generation manifest, and a tampered sidecar record marks the generation unrecoverable rather than replaying it.
test_skill_versions.py Content-addressed Skill install, upgrade, publish, rollback, and delete. Two guards are worth knowing: a bundled Skill is never an install, publish, or rollback target, and a manifest that lies about its sidecar bytes is rejected before activation.
test_skill_write_authorization.py Two defects found while tracing CodeQL alerts that were themselves noise. _permits gated load/get/read and none of edit/publish/delete, so a delegated child restricted to skill_names=["a"] could not read skill b and could overwrite, publish or delete it — the worse half, since the parent goes on to execute the recipe a restricted child rewrote. Bundled skills hid it: they are read-only for an unrelated reason, so the first probe came back all-refused and looked fine; the gap only shows on user-authored skills. edit is also the create path, so authoring a name outside the allowlist is refused too. Separately, JobManager.submit called mkdir(parents=True) before its realpath/commonpath check, so a cwd that was then correctly refused had already created directories wherever it pointed — the refusal was real and the side effect happened first — and an uncreatable target raised out of submit instead of returning the error dict every other refusal there returns.
test_import_bioskills.py The vendoring importer, against a two-skill fixture instead of a 561-file refresh: clean/pinned checkout enforcement, reading committed Git blobs rather than dirty checkout bytes, the skill count, frontmatter conversion and command-text normalisation, examples/ → scripts/ and usage-guide.md → references/, POSIX paths in every manifest section, and the SHA-256 verifier — asserted by breaking it. The failure path is the point: a duplicate declared name must leave no partial tree behind, because the emptiness guard would then refuse to overwrite it.
test_skills.py The Skill loader. A surprising share of it is YAML frontmatter parsing — folded scalars, literal blocks, chomping indicators, inline comments — because a stray > surviving into a summary is how progressive disclosure starts printing garbage. The import hints in every bundled Skill are also checked as real Python.
test_specialists_builtin.py Built-in specialists as one source of truth. The gateway catalog advertised SCIENTIST/EXPLORE/GENERAL/PLAN/REVIEWER as full specialists while the runtime resolver knew only REMOTE_GPU_PROVISIONER's prompt, so delegating to a catalog name produced a generic child and the catalog's unrestricted: False was decoration. Every catalog entry must now resolve to a runtime profile with a real persona (and vice versa), the gateway roster must derive from openai4s/specialists.py byte-identically, restricted builtins must derive a fail-closed read-only policy, a stored row with the same name overrides its builtin, and unrestricted builtins inject persona only — injecting unrestricted: True would turn a restricted parent's delegation to GENERAL into a policy-widening error.
test_specialist_delegation_roundtrip.py Delegating to a specialist that was actually saved. agents.unrestricted is INTEGER NOT NULL, so SQLite returns 0/1; the repository decoded skill_names and connectors and left this column alone; and child_execution_policy refuses anything that is not exactly a bool. Every stored specialist failed to delegate — restricted and unrestricted alike. The strict check is right and did not change: "false" is truthy and must not pass for True. The existing suite stayed green because it builds profiles in memory with a genuine Python False, so the int never appears — these tests write and read back.
test_specialist_allowlist_armed.py Whether the specialist Skill allowlist is ever switched on. set_allowed_skills narrows all four read paths and test_resource_allowlist.py covers them thoroughly — and it had one definition and six call sites, every one of them in that test file. No production code called it, so _allowed_skills stayed None (permit everything) for the life of every real dispatcher: a specialist restricted to one Skill got all 34 bundled ones and could skills_read the body of any (measured: 33 leaked, 4972 chars returned). The tests passed because they armed the allowlist themselves — the failure mode per-commit review structurally cannot see. Now armed at set_child_execution_policy, the choke point every child passes. The second half matters as much: _normalize_item merged a child's spec over its parent's with dict.update, so a nested child could replace the list it inherited — a lock with a bypass — and inheritance now narrows. Records that the connector half has no mechanism to arm at all. One test here was itself vacuous first (search("") returns 0 rows, so all(...) was trivially true) and was rewritten against a query that matches.
test_specialist_connector_allowlist.py The other half of B.4, and it was never built. connectors was stored on the specialist, inherited through delegation, narrowed in _normalize_item and echoed back by the API — with no field on ChildExecutionPolicy to carry it into the runtime and no setter to enforce it, so there was nothing to arm. Measured through the real dispatcher with connectors=["allowed"]: mcp_list returned both connectors on the host, and mcp_tools("denied") returned the denied server's tool list and started its process. The gate now sits in MCPService.connector() — the one lookup all six MCP RPC methods share, and the one the launch config is built from, so a connector this specialist may not reach has no command to be spawned with. A denial is reported as "not found" so refusals cannot be used to enumerate. The nested hole is closed on both halves: _normalize_item narrows against the delegate() call's own kwargs, but only _apply_parent_execution_ceiling sees the parent child's spec, and it narrowed capabilities and permissions while leaving both resource allowlists alone — a grandchild could widen. 14 of the 19 tests fail without the change; the other 5 are the guards that it did not quietly become deny-by-default.
test_stop_kernel_drains_queue.py Whether Stop can stop a session that has work waiting. stop_kernel cancelled the running execution and then submitted its own lifecycle ticket to the same FIFO, so anything already queued ran first — and a turn admitted after stop_requested is set blocks on stop_finished, which is what Stop sets when it completes. Stop waits for the queue; the queue waits for Stop. Measured: no return after 40s with three items queued behind a turn that cancelled correctly, against 0.1s with nothing queued — which is why this went unnoticed, since the existing tests stopped idle sessions. cancel_queued already existed and is deliberately exact so a queued cancellation never disturbs a sibling; the lifecycle drain was the missing caller, not a missing mechanism.
test_store.py Schema, migrations, and the row shapes callers depend on today. The artifact coalescing rules are the ones that bite: a provisional version has to merge rather than duplicate, and a lineage failure rolls the version back with it.
test_structured_finalize.py finalize_response is owned by the Engine, not the tool registry, and it completes a run only when it arrives alone. A mixed batch containing it is not completion, and an invalid one becomes a canonical error result the model can see and retry.
test_submit_output_reconciliation.py The sibling of the finalize ledger reconciliation, for the in-cell completion path: host.submit_output may not name files under produce-shaped output keys that no artifact, no executed cell's recorded writes, and no disk root backs, and a summary may not restate a metric with a number that contradicts the same submission's metrics. Rounding, percent forms, input-shaped keys and free prose pass; refusals stay the {"error": msg} soft failure so the model can write the file or align the summary and resubmit, and a broken evidence provider degrades to the legacy accept rather than blocking the loop's only completion signal.
test_task_modes.py Task modes and the two doors that select one. The system had exactly one shape of guidance — analysis — so "build me a reusable pipeline" was answered with figures and a report while the implementation stayed inside the kernel namespace. These pin the classifier's conservative contract (a target signal AND an action signal, word-bounded, zh+en, or it stays analysis_run: a false code mode would demand source/entry-point/test evidence a plain analysis turn cannot produce), the explicit override that always wins with a loud error on a typo rather than a silent fall-through, the fragment registry, and both wiring seams — the Web per-turn user-message append (durable message row untouched, exactly the _EXPLORE_PROTOCOL pattern) with a 400 on an invalid body value, and the CLI Agent.run seam that had no per-turn augmentation at all.
test_team_isolation_hardening.py Regressions for the isolation holes an adversarial review found on this branch, grouped by the shape of the mistake. A guard wired to one call site of several: artifact bytes were checked by path inside _api, so /preview/ (dispatched earlier) and version-/filename-addressed serves walked past — the check now lives at the byte chokepoint and fails closed when an artifact's session cannot be resolved. Project visibility is read-only: a same-project member may read another owner's Artifact/share but cannot edit, restore, rename, delete, annotate, upload into, or revoke it. A resource family with no guard at all: /projects/* let any member read, rename and irreversibly DELETE any project, and the list leaked every team's names; participation is now membership or owning a session there, so creating a project does not lock you out of it. Plus: a WS identity captured once outlived its own revocation; a substring denylist pre-check made SELECT … FROM active_users fail once users was denied (INV-1); a LIKE prefix let DELETE /team/invites/% revoke every live invite; a lost username race burnt a single-use invite; the reviewer reached the provider through its own port, around the LLM quota gate; and the login-bucket trim tested a value it could never reach, so the dict grew without bound.
test_team_governance_routes.py The M2 acceptance matrix over a real socket. /team/* is admin-only with one declared gate; user management routes audit; same-project members see each other's project sessions while private stays the owner's and admin's, but every frame-scoped write (turns, permissions, plans, annotations, Auto Mode, kernels, branches, recovery and deletion) remains owner/admin-only while Revert preview stays readable. Every admin view of a private session (HTTP and WS subscribe alike) writes an admin_read_private row, per view; the visibility toggle is owner-only over HTTP; an invite redeems into a replay-only guest whose whole API surface is auth + replay, a username collision does not burn the single-use token; a session quota answers 429 QUOTA_EXCEEDED on the second creation and the ledger reports it; the runtime ledger attributes LLM tokens to the session owner and stays inert without an ownership row; the ChatModel quota gate runs before the provider is ever reached.
test_team_governance.py Governance storage (M2) and the D4 visibility it enables. Membership upserts and role validation; invites are single-use (the racing-redemption UPDATE), expiring, digest-only at rest and in listings; the ledger aggregates and drops user-less rows rather than attributing them to nobody; quota verdicts bind at the limit, project scope binds every member, other scopes stay free. Visibility: members see fellow members' project sessions but not private or no-project ones, a guest membership never widens access, the toggle is owner-only, and the browse filter enumerates accordingly.
test_team_files.py The team file area (M1-8). FileArea's policy directly: containment survives .., the prefix-string sibling trap (/x/datasets-sibling vs /x/datasets), and a symlink pointing out of a root; upload names must be plain filenames. The routes over a real socket: list/download/upload round-trip with the temp file gone and a 409-then-overwrite; traversal is 404 over HTTP; an oversized Content-Length is 413 before any byte is read; guests get 403 everywhere here; with no roots configured the bare listing answers 200 {roots: [], configured: false} (in team and single-user mode, and configured roots still list with team mode off) while an addressed listing, download or upload keeps the stable no_data_roots 404.
test_team_ws_isolation.py WebSocket event isolation (M1-7), with a real masked-frame WS client against a real WSHub — a stub hub would prove nothing about replay or per-connection filtering. B subscribing to A's session gets view_denied with the same "session not found" sentence as HTTP, refused before hub.subscribe, so the replay buffer, pending approval prompts, and the queue snapshot sit behind one check; a broadcast into A's session never reaches B's connection while B's own arrives; an unknown id cannot be pre-subscribed (a guessed id parked on a future session would stream it from its first event); the WS cancel — the one mutating inbound — answers like an unknown session; admin subscribes to anything.
test_team_session_ownership.py The cross-user authorization matrix (INV-13), over a real socket with two members and an admin. Creation records the owner in the same lock as the frame insert; enumeration, search, and the project timeline are filtered in their source queries; every frame-addressed read or write of someone else's session answers 404 — not 403, because which sessions exist is itself the protected fact — while the owner's same request gets past the guard; a session with no ownership row (pre-team history, demo seeds, CLI runs) is admin-only; deletion takes the ownership row with it.
test_team_auth_routes.py The team guard and login routes over a real socket (the auth-exit-matrix reasoning: a _route call has never written a status line). What must hold with team mode on: every API answers 401 and every browser GET 303s to /login until a login cookie exists; ?token= no longer buys a browser cookie; the login cookie is HttpOnly/Lax and becomes Secure behind an explicit HTTPS proxy origin; logout and disable kill it; the sixth bad login in a minute is 429 — the right password included; the WS upgrade needs the same login; the loopback header token keeps the CLI's service identity only when no trusted proxy topology has erased peer provenance. Proxy-shaped token requests using either accepted header stay unauthenticated while an admin cookie still works. With team mode off, the routes answer their stable disabled shapes (what the contract capture freezes).
test_team_cli.py openai4s user (add/list/disable/reset-password), offline against the real store. The properties that matter: a password never rides argv — it arrives on stdin or is generated and printed exactly once, and the printed value really is the account's password; duplicate users and unknown users exit 2 with the reason on stderr; every management action lands in the team audit log.
test_team_repository.py Team-mode identity storage (storage/team.py). The contracts that must not drift: the production PBKDF2 cost stays pinned at 600k with the per-row iteration count making a future bump lazy rather than account-invalidating; plaintext passwords and raw login tokens never reach a table (only salted hashes and sha256 digests); disabling a user or resetting a password revokes every live session and re-enabling does not resurrect old cookies; and host.query refuses all four identity tables, so agent SQL cannot read credential material.
test_tool_schema.py The pure-stdlib validator for the JSON Schema subset the Host accepts. Nested paths and constraint violations are reported together rather than one at a time, and provider strict mode requires recursively closed required objects.
test_tools.py The control-tool classes and the legacy fenced-tool parser. The parser tests exist to stop something executing by accident: a tool fence nested inside a Python cell, inside a longer fence, or inside a tilde fence is not a tool call. execute is required never to raise, however hostile the arguments are.
test_unbounded_session_memory.py Two daemon accumulators nothing emptied. host.app.render() appended to an uncapped list with no payload limit: 2000 renders of 50 KB HTML — a tile per loop iteration, which is what the API is for — held 100 MB for the session's life, and nothing outside the cell can read a tile, so none of it was ever displayed. Now bounded to the most recent 200, with oversized payloads refused rather than clipped and evictions reported. Separately, a SessionState left _sessions only on explicit close, so the idle sweeper stopped a cold session's kernels and left ~1.1 MB of provider history resident for a kernel that no longer existed. The history is dropped on release and rebuilt from the store by _seed_messages — the same state a daemon restart produces — while the model override, plan/explore flags and pinned env, which live nowhere else, are kept. Measurements use distinct payloads: identical ones are one interned string and make the leak look 100x smaller.
test_variable_inspector_service.py Three tests keeping the variable inspector narrow. It never enters Cell state, it checks language and lease before it reads anything, and a protocol failure surfaces as a generic message rather than kernel internals.
test_webtools.py HTML-to-Markdown and the search backends, on offline fixtures. The first three tests are a regression net around one bug: an arXiv abstract page whose abstract, authors, and title were all being dropped by the converter.
test_webtools_redirect_guard.py Whether the SSRF and egress checks survive a redirect. _http_get documents that it follows redirects manually so every hop is guarded — true only on the optional requests branch. The fallback called urllib.request.urlopen, which follows redirects inside the stdlib, and that is the branch a zero-dependency install always takes. Measured before the fix against a local 302: two URLs fetched, one guarded. Deliberately not marked network — the server is an in-process loopback socket, and addopts deselects network, so the marker would have removed three security tests from every run while leaving them looking present.
test_webui_static_contract.py The static frontend read as source, with no browser and no gateway. It catches what nothing else can — a referenced asset that is missing, a duplicate DOM ID, an icon name with no SVG behind it — and it pins safety claims including the connector editor reading masked env names rather than secret values, promoted Markdown allowing only safe raster data images, and the Timeline being an allowlisted projection. It does not prove the UI works; browser_smoke.mjs is what drives that.
test_ws_frames.py The hardened RFC 6455 codec: the length ladder, client-mask round-trip, mask-direction enforcement, truncated/RSV/fragmented/unknown-opcode/oversized-control/non-canonical-length rejection, the payload cap, strict-mode UTF-8, and the RFC accept example.
test_workbench_state_service.py The Context and Security panels. The security projection is written to under-claim on purpose: it never says a sandbox is active before the worker has started, and when Python and R disagree it reports the weaker of the two claims.
test_worker_runtime_alias.py Six tests that openai4s_worker_runtime is nothing but a re-export: the same __all__, identical objects under both names, no shadow submodules, and no entry point of its own.
test_workspace_streaming_budgets.py The allocation budgets under the workspace tools and save_artifact. Each case allocates a sparse file that would be fatal to read whole and watches tracemalloc across the call, because a test that only checks the returned window passes just as happily against the implementation that read the whole file first. Measured before the fix, in the test process: 768 MiB to return two lines of a 256 MiB file, 192 MiB to grep a 64 MiB one, 64 MiB to change seven characters in a 32 MiB file, and 64 MiB to register a 64 MiB output.
test_volcengine_connector.py The official Ark CLI boundary and its Gateway adapter: child-environment secret stripping, exact masked-key selection, missing-Key and opaque multi-Key decisions, public identity/quota projection, single-flight login cancellation, and the proof that configuration stores a broker reference while no route response contains the raw Ark key or cloud Key ID. Live Volcengine access is never required.

Subdirectories

Directory Responsibility
fixtures/ Byte-sensitive captured HTML, fake interpreter helpers, and renderer samples. The subtree is deliberately left out of directory README generation and automatic formatting.

Choosing the right place

Put a focused regression assertion here. Put reusable scripted scenarios, fake providers, normalized trajectories, scored evaluations, and reviewed goldens in the ../harness/ layer. Default tests in both layers stay offline unless their entry point is explicitly marked and invoked as an opt-in smoke.

Trust Foundation additions

Module What it pins
test_compute_trust_boundary.py The manager never reports success it did not observe. The fault matrix — no network, probe failure, a killed remote process, a partial transfer, a hostile archive, a wedged helper, a cancel that never lands — must resolve to failed, incomplete, or unknown, never done. Includes the real-shell class that caught & binding looser than &&.
test_compute_alias_registration.py An agent could name any ssh destination and the daemon would dial it. _safe_alias checks that an alias cannot be read as an ssh option — a question about argv parsing — and that was the only check, so provider="ssh:<anything well-formed>" reached ssh <anything> and was resolved by whatever a Host * stanza or a DNS search domain supplies. On host.compute(...) that string comes from the model. The refusal sits at that boundary rather than in _split, which is also on the CLI and Compute-panel paths where the alias is something a person typed and demanding prior registration would refuse names the product itself offers. Fault-injected: subprocess.run/Popen fail the test if called at all, because "refused" and "refused after dialling" are different outcomes and only one is the claim.
test_harvest_remote_provenance.py A harvested artifact said it came from this machine. ArtifactManager.capture takes drain_remote_provenance and every cell-path caller passes it; the compute-refresh harvest — the one path whose files genuinely came from another host — did not. Two costs: the artifact was stamped with the local environment and carried no record of the host, engine or remote directory; and because the drain never ran there, a buffered remote entry stayed put and landed on whatever cell wrote a file next, which is the cell-3-fold-becomes-cell-7-figure bug capture's own comment describes as fixed. Split deliberately: three cases drive capture directly (what it does with a drain) and two drive refresh_compute_task and _remote_provenance_drain (that the route supplies one at all) — the first set passes either way, which is why the second exists.
test_write_file_atomicity.py write_file could destroy the old bytes without producing the new ones. Both write and edit now stage through a verified parent directory descriptor and publish with an atomic descriptor-relative rename, so a full disk, interruption, or parent swap cannot replace the intended file with partial bytes or redirect the write. Fault injection lands at the write and rename boundaries rather than replacing the tool; the tests also pin that no stage outlives a failure and an overwrite preserves the target's permission bits.
test_workspace_scan_deadline.py The walk was bounded in entries, which is not the same as bounded. MAX_SCAN_ENTRIES's own comment names the reason it exists — "the walk is unbounded work, not just unbounded memory" — and then bounds the work in syscalls. How long those take is the filesystem's property, so on a network mount a scan well under a hundred thousand entries outlives any timeout the caller set, and the caller cannot intervene because the whole walk is inside one tool call. grep, glob and list_dir now carry a wall-clock budget reported through the same scan_truncated flag, so a partial answer never looks exhaustive whichever budget ran out. Fault-injected by making the clock advance per entry rather than by shrinking the budget — a zeroed budget would pass against a walker that checked the clock once and never again.
test_compute_states.py One job vocabulary, enforced on write: terminal states cannot be re-opened, unknown stays live so nothing bills unnoticed, and the migration folds done/incomplete/closed without losing what they meant.
test_compute_manifest.py A job is checked against what it promised: every harvested file gets a size and a sha256, a declared output that never arrived is reported, and featured_files is the declared subset rather than everything.
test_compute_durability.py A remote job outlives the daemon. The row is written before the submit, the receipt survives a restart, and reconcile reports without resubmitting.
test_secret_broker.py Credentials behind an opaque reference; auto fails closed rather than degrading; env injection is read-only.
test_secret_canary.py Canaries asserted on the secret's bytes, not on field names — a denylist fails open the moment someone adds a field.
test_schema_migrations.py A database is fully at version N or fully at N-1. A failing step rolls back the whole set, and DDL really is transactional.
test_data_dir_permissions.py The credential store is not readable by another local account, including its WAL sidecars.
test_sandbox_grants.py Signed preview capabilities bind a nonempty frame, primary version, exact spend origin, sole minting ancestor and deadline; tests refuse tampering, expiry, foreign secrets, malformed authorities, same-origin grants and empty scopes.
test_security_headers.py The app shell and ordinary Artifact routes retain their strict policies; executable grant responses name only the minting ancestor, and JavaScript string compilation is permitted only for the exact pinned Ketcher editor document. Artifact bytes are named inline after their last path component, with controls, format characters and quotes removed.
test_local_jobs.py Cancelling a local background job really stops it: the whole process group dies, not just the shell, and a cancel that could not stop the job says so instead of reporting success.
test_compute_job_races.py The ledger decides the terminal state and the caller is told what it decided: a cancel or close that loses the compare-and-swap reports the state that actually landed, a persistence failure is never swallowed, and a two-thread race leaves memory, SQLite and the answer agreeing. Also runs the real harvest script under sh to prove a residency: remote output is never archived.
test_byoc_confinement_scope.py What the boundary hands back. The allowance named the helper package's parent — the repository root in a source install — so a real Seatbelt profile is built here and a real process tries to read pyproject.toml, list the tree and list .git through it. Also: available() must establish a boundary rather than ask which, its verdict may be cached but not across a changed backend, and doctor must describe the same posture the runtime does.
test_evidence_walkthrough_provenance.py One artifact assembled from four independent retrievals needs four envelopes. Attaching the first one left three accessions inside a file whose whole purpose is to preserve their evidence, with no request URL, retrieval time or response hash — provenanced-looking, which is worse than bare. Round-trips the aggregate through the real repository and traces every accession back out.
test_env_kernel_binding.py An environment transaction no kernel can see is not a transaction: after apply, a real cell runs in a real worker and sys.executable must be the current generation's interpreter; after rollback, it must have moved. Also pins that the pointer never moves onto an interpreter that was not executed, and that a spec edited between plan and apply is refused.
test_llm_transport_typed.py A 429 carries its status and Retry-After and is retried within a bounded budget; a stream that committed output never is.
test_llm_transport_wiring.py The typed transport is reachable from a real chat(): errors name their provider and Stop interrupts a backoff, while four-argument injected transports keep working.
test_llm_system_placement.py Only leading system messages become the provider's initial system field, so compaction cannot invade the cache prefix.
test_startup_no_implicit_install.py serve diagnoses the environment and never mutates it.
test_startup_no_implicit_side_effects.py A boot on a fresh data dir opens no socket, spawns no process, runs no cell and persists no session — guarded at socket.connect and subprocess.Popen, with OPENAI4S_SEED_DEMO=1 as the positive control.
test_unstarted_worker_admission.py A worker whose thread never started must not hold the session forever, and the abort has to survive its own cleanup failing: the ticket is released, the job is terminalised rather than merely forgotten, the plan row goes back to the status its own route can claim, and none of it depends on rendering an exception that may refuse to render.
test_update_contributors.py The Community Contributors generator keeps publicly recognized non-commit contributors after the commit-count-ordered API users, de-duplicates them case-insensitively, refuses a recognized login that is also excluded, and still fails closed on an empty API result instead of replacing the wall with only the maintained recognition list. A transient avatar failure preserves that current contributor's committed PNG while departed identities are pruned, and reports zero refreshed rather than a healthy count. The expectations name the recognized login literally, because an expectation re-derived from the constant under test passes just as happily when that constant is empty. Two link-integrity regressions are pinned: a login whose casing drifted from the committed filename keeps its file and falls back to the remote avatar instead of emitting a local <img src> for a file the prune deleted, and the unauthenticated github.com/<login>.png fallback never carries the repository token across its cross-host redirect.
test_diagnostic_archive_boundary.py Nothing unknown leaves the machine in a shareable diagnostics ZIP: the archive is deny-by-default, so a structured line survives only as validated metadata and an unstructured one only as a count, a class and a fingerprint.
test_diagnostics.py The support bundle is safe to paste into a public issue, including a token sitting mid-sentence in a log line.
test_diagnostics_routes.py Web Diagnostics routes: unauthenticated 401, member 403 (same code/size/filename, never 404 or 429), admin 200; page-load GET adds no network, subprocess, or Store/config/bundle write; canaries are absent from the zip; temp files return to baseline on complete, disconnect, and error.
test_evidence_verification.py An exported package verifies with no daemon, and four tamper shapes are caught — including a payload rewritten together with its recorded hash.
test_observability.py Correlation IDs and shape-based redaction of structured log fields.
test_doctor.py openai4s doctor works without a daemon, tells a degradation from a refusal, exits with the verdict, and never prints a credential value. It never upgrades the database it diagnoses: an older schema is a pending upgrade (warn), a kept pre-upgrade backup beside it is an upgrade that did not complete (fail), and no check reports ok over a Store that failed to open.
test_endpoint_probe_guard.py Where a managed-endpoint readiness probe may reach. register takes spec["url"] and only defaults to loopback when none was given, so the probe target is agent-supplied — and probe_ready called urlopen on it unguarded. A cell could register an endpoint at 169.254.169.254, or any port on any reachable host, call probe, and read existence off the boolean. An oracle rather than exfiltration, but a port scanner is a real capability. The fix could easily have been wrong in the other direction: a managed local endpoint is loopback, so guarding everything would refuse the normal case. The exemption is the one address this daemon allocated — not "allow 127.0.0.1", which is precisely the local scan, and the tests fail on that widening.
test_env_provenance_confidence.py Whether the environment panel's claim matches what the row can support. env_snapshots carries generation_confidence (verified when the snapshot's address includes its generation, so no later kernel can share the row) and provenance (the fallback path's own words). The migration that added the confidence column says a reader needing certainty filters on the label — and no reader did: 0 occurrences in app.js. The panel drew one distinction, captured vs live, so every captured snapshot rendered as "Recorded from the kernel environment at the time this artifact was produced", including rows the store had marked as unable to support that. The display-layer version of what capture_environment was fixed for: provenance that is wrong rather than absent. Both halves are checked — the payload carries the fields, and the shipped branch is extracted and run under node — because either alone passes while the panel stays wrong.
test_error_envelope.py One success/error envelope with stable codes and a request id.
test_compute_task_centre.py Looking at remote work without paying for it. A remote job outlives the turn, the kernel and the daemon, but the durable record was only reachable from inside a cell — a person whose GPU job had run for two hours had nowhere to look. The page cannot poll because the probe is the harvest: result() contacts the remote, and contacting it pulls files back and closes the job, so a self-refreshing page would harvest into a session nobody was watching. The test for that reads the module's imports via ast, not its text — the first version grepped for "ComputeManager" and failed on the docstring that names it to explain why it is unused. Also covers cross-session non-enumerability (not listed, not counted, not reported as hidden), survival across a restart, that unknown is never rendered as failure (the job may be running and billing), and that pids, sandbox handles and cluster paths stay out of the projection.
test_compute_owner_and_harvest.py Two compute defects. The idempotency namespace was installation-wide while every other view of compute_jobs is per-owner, so one session's key blocked every other session's and the duplicate refusal handed back the other session's job_id and status — the leak shape for_owner's own docstring already names. The index is now per-owner with COALESCE(owner_key,''), because SQLite treats NULLs as distinct and NULL is exactly the CLI rows. And compute_result declared no writes_files, which is the only thing the Web wrapper gates on, so a native harvest of N files produced zero Artifact versions while the in-kernel path produced them correctly.
test_materialisation_atomicity.py Both write paths ran checks that could still refuse after they had already destroyed bytes. Materialisation hardlinked the borrowing session's writable live file to the source session's immutable snapshot, so one ordinary write rewrote another project member's frozen provenance and left its checksum describing bytes that no longer existed — measured, not reasoned. It also trusted is_file() followed by copyfile(), so a same-length snapshot swap could inherit the trusted row's checksum for attacker bytes. Now the shared held-FD verifier runs before any mutation, both destination files are private durable copies, and every refusal leaves zero row/file.
test_public_exception_projector.py public_failure is an envelope: it decorates the message a payload already carries and never rewrites it, which is right for author-written refusals and left one hole it cannot close — an except Exception putting str(e) in the payload got the same tidy treatment, so a credential, an absolute path and a shell command shipped to the client with a code bolted on. The fix is a projector at the exception boundary; a filter in the envelope would have to guess which messages are safe and would guess wrong in the direction that destroys the good ones.
test_token_mint_atomicity.py The first-run mint was check-then-act followed by an unconditional os.replace, so concurrent daemons each overwrote the token file and the loser's re-read sat outside any exclusion — the survivor could authorise a token no longer on disk. Raced with real processes, not threads: threads share the GIL and hide exactly this.
test_plan_resume_claim.py paused → executing was a read-then-write, not a compare-and-swap. Measured on the unpatched tree: two concurrent POST /plan/resume gave CODES [202, 202] and TURNS 2 — the same steps ran twice. The claim is now the UPDATE's own rowcount, not a re-read.
test_mcp_lifecycle.py Every MCP fault class left the child running. An initialize failure escaped the constructor, so the object was unreachable from the cache and each "Test" click leaked a process and two reader threads. No fault evicted anything, so a server that closed stdout poisoned its connector id permanently — five calls, one pid. _abandoned grew by one entry per unanswered request and was pruned only when the server chose to answer. stderr used an unbounded readline, so a 20 MB newline-free line was retained whole and copied again before anyone truncated it. And close() hung forever when a grandchild held the stdout write end.
test_check_css_tokens.py F-21: var(--x) references in style.css minus --x: declarations is empty; the checker fails a hole and passes the shipped sheet; :root and dark both declare --text-100/--text-300/--surface-0/--warn; the dark comma leak is gone; markdown tables wrap in .md-table-wrap; ≤900px touch targets are ≥40px; eight dead rules are absent.
test_ci_gate_independence.py Every declared gate, including the Linux private-PID interrupt smoke, must fail independently. Four once lived as steps inside another job — check_directory_readmes after pre-commit, and the harness tier plus both response captures after pytest. A step only runs when every step before it passed, so all four were unreachable whenever one unrelated thing was red, and a run that said “tests failed” was indistinguishable from one where the goldens had also drifted. Asserted against the parsed workflow graph, because a step that did not execute cannot be observed from a working copy.
test_skills_installer_contract.py The half of the npm installer's contract that lives on this side of the language boundary. tools/skills-installer/ is gated by its own Node self-test, which cannot see skills/ growing a directory with no SKILL.md, a bin path renamed out from under package.json, or a files glob that stopped matching — the last of which publishes a command that runs, finds nothing bundled, and falls back to downloading 100 MB of source tarball per install. Asserted in pure Python on purpose: a test that shelled out to node would report success by skipping wherever Node is absent. Every openai4s-skills@X.Y.Z install pin in tracked docs must name a release listed in PUBLISHED_NPM_VERSIONS, and package.json may be ahead of that list only by one release step (the next patch, minor or major version after the newest published one), since npm publish reads it from there.
test_channel_counter_contract.py Seven buffers drop input to stay inside a budget and two reported nothing. host.bash cut stdout at 30,000 characters, returned the tail, and recorded chars: 30000 beside a sha256 of that tail as the command's stdout digest — indistinguishable from a command that printed exactly 30,000, which len(stdout) cannot tell apart either. One definition (execution/budget.py), asserted as an absence: no producer may spell a counter key itself.
test_producer_output_budgets.py stdout was producer-bounded; stderr and the traceback were accumulated in full and capped at response-build time, so the bound was on what was returned rather than what was allocated. Measured on a patched worker: stderr peak 452 MB → 3.3 MB, traceback 122 MB → 4.3 MB. The R error string had no cap at all.
test_workspace_streaming_budgets.py save_artifact read a whole artifact into the daemon just to hash it, so registering a large scientific output OOMed the daemon; read_file and edit_file allocated O(file) with no byte budget; list/glob sorted the whole tree before truncating. Now streamed with real budgets and structured counters.
test_local_job_envelope.py A local job ran with the daemon's environment verbatim — no env= on the Popen — so every provider API key the daemon holds reached every job, while build_kernel_environment (called from the kernel manager, the dynamic tools and preinstall) sat unused, and the architecture doc claimed child environments are rebuilt rather than copied. It also used bash -lc, a login shell that sources a dotfile the daemon never saw; counted no active jobs before spawning a thread and a process; and confined cwd with normpath, which is lexical, so a symlink inside the jobs root escaped it. Asserted on what the child can actually read, not on which function was called.
test_local_job_lifecycle.py The other half of a local job's life: its end. There was no deadline at all — the ceiling on concurrency could not bound duration, so eight jobs could hold the machine forever — and no close(), so daemon shutdown left process groups running, reparented, with their stdout pipes shut and no record they existed. The output cap was applied after the allocation it existed to prevent, because the read loop was readline(): a gigabyte on one line was materialised whole before anything could drop it. The chunk sizes the read loop actually asks for are recorded and asserted, because a test on the surviving buffer cannot tell this fix from the bug. Also: the timeout and abandoned states, which may not borrow failed's or cancelled's name; the receipt that lets a restarted daemon name a job it was holding; and that neither str(e) nor an OSError's path reaches the job log or the submit refusal.
test_model_profile_identity.py Three ways a pinned session still dispatched somewhere its pin did not name. _pinned_llm_config preferred st.model — the request's bare model string, which the browser sent on every message — so provider, endpoint and credential came from the pin while the model name came from the header selector: a configuration that exists in no profile. It also returned None for a revoked key and for any exception, after which the caller used whichever profile is active now: recorded as A, executed as B. And delete hard-removed the row plus the revisions history depends on, then NULLed the pin on every frame, so the next send silently re-pinned.
test_contract_inventory.py Every external route and event is covered by the contract inventory.
test_frame_pagination.py Keyset pagination over the session list.
test_ws_resume_cursor.py A monotonic event sequence and a resume cursor a reconnecting client can use.
browser_admission_fault.mjs The one admission property Python cannot show: the page surviving the loss of its own answer. The POST reaches the daemon and succeeds; only the response is destroyed, so the turn really was accepted. A reload then has to reconcile from an id the client stored before dispatch — without resending the turn and without reopening a pin the running turn is already carrying.
browser_matrix.mjs The workbench driven in three engines rather than the friendly one. A renderer that works in Chromium and silently fails in WebKit is a broken product for the half of the users on it, and only a real engine says which half.
test_benchmark_workflows.py The versioned workflow benchmark actually runs: every manifest in ../workflows/ loads, every step name resolves, and a declared outcome is compared against the observed one in both directions — a case expecting failure fails on a clean run. It also freezes the Stage 0 acceptance manifest by version plus canonical digest, rejects caller-supplied packs with removed probes or weakened nested expectations before execution, checks exact claim/execution/assertion shapes, proves failed Cell/Reviewer/duplicate attempts remain in their denominators, makes an observed Reviewer workspace write fail the frozen contract, keeps offline contract metrics structurally separate from null-at-zero live observations, honors the production global network deny, and verifies the real loopback Ketcher probe leaks no ephemeral credential through CLI streams.
test_benchmark_bringup.py The tool bring-up contract's verifier, unit-tested: the self-vouching seal, weights digests and sizes, canary parse and downstream consumption proofs, admission, budget and path confinement — with the tamper shapes ordered by subtlety, up to the full forgery only the evaluator-held reference digests catch.
test_byoc_confinement.py The boundary tried from inside. The home directory is unreadable, the stage directory is the only writable place, the interpreter still starts — and the keychain is unreachable, which the filesystem invariant does not imply, because security find-generic-password asks securityd rather than reading the file. Uses a scratch keychain with a planted canary, never the developer's own.
test_compute_harvest_safety.py Where the remote's bytes are allowed to land, and what happens when they misbehave. A symlinked harvest root or per-job directory is refused, a hostile archive leaves no staging tree behind, a terminal job re-polled returns its recorded evidence instead of re-harvesting, and the capped transfer aborts on transferred bytes rather than on the size the remote claimed.
test_compute_lifecycle.py close() is not a synonym for "write cancelled": it must actually signal the remote, must not claim a cancel it could not deliver, and must terminate a sandbox it only knows from a job row. Also pins that recovery is scoped to the owning session, so a restart cannot hand one session's jobs — and their harvested outputs — to another.
test_compute_output_caps.py What the daemon is allowed to allocate for a remote it does not control. Driven with real subprocesses, because a cap asserted against a BytesIO proves the arithmetic and nothing about pipes, threads or deadlock: flooded stdout, flooded stderr, both flooding at once, and a hung command that must raise TimeoutExpired rather than hang.
test_env_generations.py An environment change as a transaction — plan touches nothing, a failed apply leaves the current environment intact, rollback is a pointer move — and the id that names a generation is confined to its own environment, so rollback python ../../r/generations/env-R cannot point Python at an R prefix and label its provenance Python. Stage 1 additionally proves --repair builds a fresh generation despite an unchanged spec and cannot move the pointer until the real Python/R interpreter and complete standard package list verify.
test_preinstall_probe_confinement.py The interpreter being probed is foreign: a hostile .pth or sitecustomize runs before the probe's own code. So the probe runs confined, with the environment scrubbed, and under enforce it refuses rather than degrading. It also runs in its own empty directory rather than the daemon's launch directory, which python -c would put first on sys.path. A probe that times out is ended together with everything it started, including a child that left its process group; a probe stopped through its stop event ends at once rather than waiting on the pipes that child holds, kills it, and removes its workspace; and one stopped before it starts runs nothing.
test_release_pipeline.py The release, tested without cutting one. The GitHub flip is last and refuses until PyPI has the version; a staging run does not rebuild; a disk image is signed on evidence rather than on a configured variable; and the draft must carry exactly the distributions PyPI carries, in both directions.
test_release_rehearsal.py The non-publish rehearsal contract: publish=true cannot masquerade as a rehearsal, macos_asset=omit cannot carry a DMG, a skipped 3.14 check is not attestation, a missing digest is refused, and Linux ci_attestation/release_reexecution cannot collapse into a flat passed+unproven status.
test_progress_document.py docs/next-version-progress.md is read by section number — §6, §12, §13, §15 all appear inside it — and it had two ## 15 sections with ## 16 between them, so one reference had two possible referents and a reader following it landed on the wrong one. Its last section also stopped 67 non-merge commits behind the tree, which does not read as incomplete but as “nothing has happened since”. Numbering is now 1..N, unique, ascending, every §N resolves, and the file must name the commit plan-crosswalk.json was audited at — so the two documents cannot describe different trees. It also holds the last crosswalk distribution this file states to the crosswalk's real counts, because §8's "no open row" outlived a reopened P0 row.
test_query_authorizer_lifecycle.py Connection.set_authorizer(None) removes the authorizer from Python 3.11 onwards and does not on 3.10 — the project's declared floor — where it leaves the C trampoline installed with no callable behind it and SQLite reads that as SQLITE_DENY. The Store holds one connection, so Store.query's finally left it deny-all for the rest of the process: after a single agent SQL statement an ordinary new_frame raised not authorized. Nineteen CI tests reported this accurately and were read as a py3.10 quirk. Note the falsification is only visible on 3.10 — on 3.11+ the old code is correct.
test_plan_crosswalk.py The plan's 56 original proposals must each have exactly one recorded outcome, so the crosswalk is data and this is the check: 56 unique (source, original_id) keys each appearing once, all seven source reports contributing eight, and the status vocabulary used honestly — closed must name a test file that exists, implemented_unverified must name the run it is missing. Prose could gain a duplicate, lose a row, or say Completed about something no call chain reaches, and nothing would notice. An open P0 row must be named in docs/v03-decisions.md, since it means D12 is not met and that has to be a recorded waiver rather than an unnoticed state.
test_release_gate_manifest.py The quality receipt must describe the canonical gate manifest exactly, driven through the production consumer rather than the manifest helpers. Each case is one mutation of a good receipt — a missing gate, a duplicate padding the count back up, an unknown gate, a substituted or merely extended command, an older schema version, a different manifest digest, a check run from another commit or one that was skipped. The old consumer accepted all of them, and test_release_pipeline.py's own fixture was that forgery. It also pins the independent hosted-Linux bubblewrap interrupt job: real Python/R packages, enforced sandbox, explicit raw-network exception, Ubuntu's capability-stripping bwrap AppArmor profile without a global userns relaxation, an exact non-optional command, and its check run as required release evidence.
test_release_docs.py Release and install documentation must describe the machinery that exists. Each check reads a fact from the workflow files or the version declarations and a claim from the prose, and fails when they disagree. The first cases pin trusted publication to the draft-first dispatch: docs/release-validation.md and .github/CONTRIBUTING.md once told maintainers to publish the GitHub Release by hand, which made the release public before release.yml ran and left only pypi_only, which attaches nothing. Version examples must name this tree's version or the X.Y.Z placeholder. The upgrade cases hold docs/upgrading.md and its Chinese half to the 27 to 32 schema move, a backup before the first start, no supported downgrade, and a backup of the container's /data volume (the directory is read from the Dockerfile); the claim that the migration deletes its own pre-upgrade copy is read from migrations.py, so keeping that copy fails here until both halves change. The future-schema paragraph must say that 0.2.0 predates the refusal. The platform cases read release.yml, publish-image.yml and ci.yml: while the DMG is omitted by default no install doc may put macos-arm64.dmg near a releases/latest link and platforms.md may not claim a notarized image ships; while the Windows zip is staged on every publish no sentence in the README halves' install sections may pair Windows with deferral wording, so a paraphrase is caught too; the Windows known limitations must disclose what the final "Fix verification — 2026-09-07" section of the parity audit leaves unverified (a Windows reboot, Conda provisioning, real provider sign-in, the full browser smoke that did not pass, WSL service timeouts), keyed on the audit's own wording, and may not repeat the superseded "WSL or Windows restart" list; the published ghcr image must be named; and the workflow README pair may not call the full Linux boundary smoke manual.
test_response_contract_coverage.py Every external route has a response contract, and the contract is the one the route actually produces — a coverage file that drifts from the server describes a product nobody is shipping.
test_response_contract_downloads.py The routes whose success is bytes, and the one refusal that carried no envelope. The notebook export, the Session package and an artifact download were each frozen as kinds: ["json"], statuses: [404]: the parameterless sweep had nothing to ask for, and the four unimplemented verbs supplied that 404, so the coverage gate counted a download endpoint as covered while nothing a client depends on was written down anywhere. PATCH\|POST\|PUT /annotations/<id> answered 404 with {"annotation": null}, the one refusal outside the PublicFailure envelope, so the UI's api() — which builds its error from j.error — reported a failure that said nothing.