← Về thư mục
📄 / / profiles / r-and-d / raw_evidence / OpenAI4S / openai4s / kernel / README.md

Kernel runtime

中文说明

The persistent Python and R kernels live here, and with them the system's scientific execution plane. The outer agent loop hands this package at most one complete Cell at a time; the worker starts on first use, speaks a language-neutral JSON-lines protocol, and keeps its namespace alive across Cells. The inner synchronous Host RPC loop is Python-only.

Where this fits

  1. agent/engine.py routes a Cell action without depending on any concrete kernel.
  2. CLI and Web composition create a lazy Kernel or a supervised Python/R slot.
  3. The manager sends one execute frame. A Python worker may answer with host_call; the manager dispatches it and writes back the matching host_response before it goes on waiting for the Cell's final response. Compatibility acknowledgements are not the normal completion path.
  4. The worker returns captured output, error and interrupt details, guard reports, and resource usage. Namespace inspection is a separate bounded request, not a synthetic Cell. A Cell result becomes another outer-loop observation; only Python host.submit_output(...) can complete a task from inside a Cell.

For each worker, the manager must be the only party that reads frames. The worker's protocol-write lock serializes frames on the wire, and its Host-call transaction lock allows only one synchronous RPC in flight at a time. The Web execution coordinator and KernelSupervisor coordinate writers and lifecycle around that protocol; neither proxies it.

Files

File Responsibility
__init__.py Public exports for Kernel, KernelBusyError, KernelLease, KernelSupervisor, and InterruptDelivery.
background.py The home of host.exec_background. A Cell that will run for a long time — a training run, a long simulation — gets a worker process of its own, so it never blocks the foreground kernel or the agent's turn. exec_peek reads its accumulated stdout without waiting; exec_interrupt sends one idempotent SIGINT, and when the stop reached nobody the report says so (interrupt_undelivered) instead of leaving a running status to be read as "still unwinding". Such a job cannot see the foreground namespace, and nothing about it is written to disk. What it accumulates is bounded on both axes: the peek buffer keeps a head of MAX_PEEK_CHARS and marks the cut instead of growing for the life of the job, and sixteen jobs may run at once — each owns a subprocess, so that cap counts processes, and the slot is claimed before the spawn rather than registered after it, since a check that runs afterwards is one that concurrent launches all clear together and every one of them spawns.
errors.py The kernel's exception types, in a module that imports nothing, preserving the distinction between a transient interrupt failure and a remote kernel that has no signal path at all. KernelInterruptUnavailable has to be catchable by supervisor, and manager reaches supervisor through the watchdog — so defining it in manager closed an import cycle that only failed on the orders where manager happened to be initialised first. manager re-exports both names, so existing imports keep working.
environment.py Decides what a kernel is allowed to inherit. The child environment is built from a small explicit allowlist rather than copied from os.environ, so provider keys, cloud tokens, agent sockets, and loader-injection variables stop at the process boundary. Anything a Cell then launches, host.bash included, inherits that same filtered mapping. PATH is led by the selected conda prefix's bin, or, with none selected, by the Python worker's own interpreter directory (unresolved, so a venv stays a venv), so python in host.bash is the Cell's interpreter even when the daemon or CLI was started by absolute path from a venv nobody activated. An R worker keeps the host PATH.
environments.py Environment selection: how a task moves to an interpreter that already has what it needs instead of installing packages every time. Prebuilt conda environments are discovered under OPENAI4S_ENV_ROOTS or the usual install roots, probed for bin/python or bin/Rscript, and cached with their package sets. The daemon's own interpreter is always offered as a synthetic base environment, so a selection can never leave a session with no Python kernel.
font_cache.py The matplotlib font list a kernel in an enforced sandbox starts with. That sandbox hands every kernel an empty private MPLCONFIGDIR, so each new kernel's first plot rebuilt the list from scratch — 8-48 seconds on macOS, where matplotlib shells out to system_profiler. A cache kernels could write, or a list copied back out of a finished kernel, would be one Cell poisoning what another session's kernel parses, so the list is only ever built by the host: the kernel's own interpreter, confined through preinstall.run_confined_probe, running a fixed program that writes into a fresh empty cache directory. That probe runs in its own empty working directory, and the program drops the working-directory entry python -c puts first on sys.path before any other import, so a json.py or matplotlib/ in the daemon's launch directory (which a CLI kernel's workspace can be) cannot decide what it prints. The host validates what it prints, stores it under <data_dir>/cache/matplotlib-fonts/<interpreter key>/, and the manager copies it into each new sandbox's private temp through symlink-refusing directory descriptors before the worker starts. A cache inside the kernel's own workspace is refused, and the list is reused only while a fingerprint of the font directories still matches, so a newly installed font reaches the next kernel, and only while the font_manager.py it was built by keeps the size and mtime the builder reported: matplotlib names its list after its own version, so an upgraded matplotlib triggers a rebuild instead of a month of per-kernel scans behind a cache that still looked valid. Only the serving daemon (run_server) starts the background build; a CLI run or a test only reads an existing cache. The daemon also stops a build still running when it shuts down (shutdown_background_builds): a build not yet spawned never starts; a running builder's process group gets SIGTERM (the builder then removes its scratch directory) and SIGKILL after a short grace, and every process it started is killed, including the helper macOS system_profiler re-spawns in a process group of its own; and the build thread is joined for a few seconds so it removes its probe workspace. The scan is never waited out; openai4s stop used to report success while it ran on, re-parented to PID 1. Every miss degrades to the old in-kernel scan, never to a kernel that fails to start.
guards.py Looks for state that leaks from one Cell into the next: pyplot figures a Cell opened and never closed, plus a few process-global registries pinned before the Cell and diffed after. These are cheap probes, not confinement. A probe whose optional library is missing does nothing, and OPENAI4S_GUARDS_OFF=1 turns the whole bundle off. The figure probe reads pyplot only when a Cell already imported it: importing it here made every new kernel's first Cell, x = 1 included, pay matplotlib's font scan.
lazy.py Keeps a turn that only calls tools or only finalizes from starting an interpreter it never needs. One owner, one start, thread-safe. The candidate worker is published early enough that a cancellation can still reach it, and a failed bootstrap is detached and shut down rather than reused.
manager.py The host side of one worker. It spawns the subprocess, wraps the command in the OS sandbox, and is the only party that ever reads that worker's JSON-lines frames. One execute frame goes out; what comes back may be a streamed output chunk, the final response, or a host_call that has to be answered with a host_response before the blocked Cell can go on. Interrupts and restart generations are driven from here: interrupt() returns an InterruptDelivery (falsy when nothing was delivered) naming which path was taken and why a failure failed, and on Linux the SIGINT is aimed at the worker's main thread via tgkill(2) — a process-directed signal may be handed to any thread, and an OpenBLAS pool thread that consumes it leaves a sleeping main thread uninterrupted. A streamed stdout chunk is attributed only to the cell whose id it carries; a stale chunk from a previous cell's logging handler or background thread is counted, not concatenated. The worker's stderr is drained by a thread of its own and bounded in bytes at the read — os.read on the descriptor underneath the text decoder, since the pipe is opened in text mode for the protocol frames — and it reports what it saw, kept and dropped, so the tail printed when a worker dies is not read as the whole of the failure.
preinstall.py Package management for the kernel, held at arm's length: it is support tooling, and the stdlib core never takes a hard import dependency on it. Startup only ever reports what the scientific baseline is missing; applying that plan is a separate act a human or an explicit API asks for. serve used to apply it itself, on a background thread, resolving ~23 unpinned names against PyPI and installing them with --break-system-packages — so launching the daemon mutated the user's interpreter, two cold starts a week apart disagreed, and an offline one failed where nobody was watching. A newly installed package only reaches a fresh process, so restarting the kernel afterwards is the caller's job. An interpreter with no pip module at all — uv-built virtualenvs ship without one — falls back to a uv binary on PATH pointed at that same interpreter; only tool selection falls back, and a failed install under a present pip is reported as-is rather than retried. It also owns the confined probe every foreign interpreter runs under, which on a timeout ends the probe and everything it started -- its process group and any descendant that left it, as system_profiler's scan helper does (a leader-only kill left that work running) -- and takes a stop event another thread can set to end it early without waiting on a pipe such a helper still holds, and freeze_for, which reads a package list from one named interpreter and returns nothing rather than falling back to this process's own — an artifact's environment provenance is attributed from that.
protocol.py The shared byte ceiling for one JSON-lines frame. The worker producer and the remote socket receiver import the same value, so a response admitted at one end cannot be rejected as oversized at the other.
provenance.py Installs object-level lineage instrumentation inside the Python worker. Objects read through a supported reader carry the source version_id of the artifact they came from, and a later write reports the input versions they accumulated. It sees what it can see, not everything. The side table that holds tags for objects which cannot carry an attribute stores the object next to its id: an id is not an identity, CPython reuses a freed address immediately, and an unrelated object was inheriting the lineage of whatever used to live there. That table is bounded and evicts oldest-first, which loses a tag rather than inventing one. matplotlib is never imported to be instrumented: a sys.meta_path hook wraps Figure.savefig the moment a Cell imports matplotlib.figure, so a Cell that imports, plots and saves in one go still reports its edge, and a Cell that never plots never pays for the font scan.
r_kernel.py Resolves a real Rscript and builds the file-descriptor-safe command that runs the R sibling through the common manager. Python is never silently substituted.
r_worker.R The persistent R worker, holding the same execute/response contract as Python: captured output, interruption, the failing line and call, resource usage. A Cell's two streams are sunk to the host's fifos with blocking = TRUE — R's fifo() defaults to non-blocking and silently drops whatever will not fit the pipe buffer — so the host's reader paces the Cell and its output arrives live instead of all at the end; the response sets sink_capture to say the host, not the worker, is holding it. Output is still flushed between top-level expressions, for the expression that ends without filling that buffer. Inbound frames are parsed with jsonlite, but outbound JSON is escaped by hand, so an R install without jsonlite still reports a clean structured error instead of dying. It is an analysis channel: no host object, no mid-Cell RPC, no way to complete a task from inside a Cell. Nothing of the worker's own is a .GlobalEnv binding: the whole file runs inside one private environment (parented to .GlobalEnv, so a user's own S3 print/conditionMessage methods still dispatch) and the quit()/q() guards sit in an attached openai4s:guards environment, so rm(list = ls(all.names = TRUE)) in a Cell clears the user's variables and nothing else, and the Variable Inspector lists every .GlobalEnv binding unfiltered.
readiness.py The Stage 1 local-only readiness projection for the standard pair. It parses the shipped direct Python/R dependency lists, compares them with discovered local package metadata, and returns a path-free ready/needs_setup/needs_repair/unavailable DTO plus explicit managed repair commands. The check starts no interpreter, imports no science package, contacts no network, and mutates nothing; an unreadable inventory fails closed rather than pretending every package is missing or installed.
recovery.py Builds a replacement kernel from a content-addressed, canonical bootstrap recipe plus conservatively classified replay steps, and validates it before anyone else sees it. The candidate is published only once validation passes, and recovery reports partial when state cannot be safely reconstructed.
env_generations.py An environment change as a transaction: plan touches nothing, apply builds a new generation and only then moves the current pointer, a failed apply leaves the previous environment exactly as it was, and rollback is a pointer move to a generation still on disk. The generation is built at its final prefix — Conda bakes absolute paths in, so a renamed staging directory is a broken environment — and what is made atomic is its visibility instead. The id that names one is confined to its own environment, because it is joined onto a path and read back by every later kernel spawn. Stage 1's explicit --repair forces a fresh generation even when the spec hash is unchanged; Python/R standard generations must start their real interpreter and contain every direct manifest package before the pointer may move.
sink_drain.py Bounds a worker's captured output from the host end, for a worker that cannot bound it from its own. R is single threaded and no callback fires inside a top-level expression, so a cap living in the R worker could only act between expressions — one expression printing 300 MB still wrote all of it to a tempfile that is a tmpfs on much of Linux. Here the cell's two streams go to fifos this module drains: it keeps the first megabyte, discards the rest as it arrives, and reports exactly how many bytes it saw and dropped, so a capped result cannot be confused with output that was silently lost. It reports the fact too — stdout_truncated / stderr_truncated — rather than leaving a consumer to derive it: output ending exactly at the cap was never cut, and a cell whose own text happens to contain the marker's wording is evidence of nothing. The retained head is handed on incrementally as it lands, which is what makes an R cell's output live.
supervisor.py Owns the durable Python and R session slots, and stops there. A caller receives a lease naming the exact generation it acted on, and interrupt, restart, and watchdog replacement only fire while that lease still matches the live slot — so a late caller cannot kill the kernel that already replaced the one it meant. It never reads a protocol frame.
transport.py How a Kernel reaches its worker: PipeTransport for the local child (moved rather than rewritten — the Popen call, the stderr drain thread and the shutdown sequence each record a failure already paid for once: a filled stderr pipe deadlocking a cell, a daemon thread parked in a buffered read turning a clean exit into SIGABRT, a restart leaking a zombie), and OutboundTcpTransport for a worker that dialled in from a compute node. The local child is spawned into its own session with SIGINT/SIGQUIT reset to SIG_DFL between fork and exec — an inherited SIG_IGN survives every exec and R honours it, so a backgrounded daemon's R kernels were born uninterruptible — and kill() gained the group ladder, so stopping a worker also stops what its cell spawned instead of leaving grandchildren running. The manager's protocol discipline is untouched by this file — one frame reader, id-routed host_response, the host-call transaction lock — because only how bytes get to the other end differs. Interrupt is the one operation that does not survive the move: there is no local pid across a cluster, so the remote transport takes an explicit hook and reports honestly when it has none, rather than returning success for a signal nobody sent.
worker.py The persistent Python worker, and the file that has to get the fiddly parts right. The protocol file descriptors are moved off stdout, so a stray print from Cell code lands on stderr instead of corrupting the wire. host is injected into the namespace and held to one Host-call transaction at a time. Cell source is registered with linecache, so a traceback points at the line the researcher actually wrote. SIGINT handling, guards, the audit hook, provenance, and the bounded inspection and usage replies are all armed here. One delivered SIGINT ends exactly one cell and never the worker: a signal that arrives where the handler cannot safely raise — before the cell's first bytecode, or inside a protocol write — is recorded as pending and the handler stays armed, so the stop is deferred rather than dropped, and it is raised the moment the write completes. Outbound frames are bounded at the producer, in three layers that each catch what the one before cannot: a write() streams in 64k chunks, a Cell's captured stream stops retaining at MAX_OUTPUT and says so with exactly one marker, and a frame still over the byte ceiling is replaced rather than truncated — a cut JSON line is not a frame, and the host's single reader would desynchronise on it. A dropped response keeps its type and id and carries the reason, because a frame the host is blocked waiting for that simply never arrives is a hang, not a refusal.

Security and failure boundaries