← Về thư mục
📄 / / profiles / r-and-d / raw_evidence / OpenAI4S / openai4s / orchestration / README.md

openai4s/orchestration/

The cluster control plane: what work was asked for, what resource was granted for one attempt at it, and the boundary any resource plane must present. Deliberately outside server/, beside execution/ — the CLI path needs these values too, and a control plane that can only be reached through a web handler is one that cannot be tested without one.

The package's defining property is what it does not contain. INV-2 (Backend Opacity) says the orchestration core's source and import graph never mention a scheduler: no slurm, no partition, no qos, no sbatch. Those words live only in a backend subpackage and in cluster.toml. This is not a style preference — it is checked, so a scheduler's vocabulary cannot leak into the modules that decide policy, and a second backend does not require re-reading the first one's assumptions.

Two naming rules, fixed by the plan's §2 so two vocabularies cannot drift into one confusing one:

file what it is
bootstrap.py The credential a worker placed by a scheduler presents to prove it is the worker this daemon asked for: an HMAC over (allocation_id, epoch, rank, expires_at, nonce), signed with a per-daemon 0600 secret published through os.link so two daemons cannot each mint their own. Every field earns its place — without epoch a credential from a previous incarnation still works, which is exactly what INV-7 forbids; without nonce a captured one replays. It travels as a file, and the scheduler is told only its path, which is why the broker refuses credential-shaped environment names: a job's environment is readable by anyone who can run scontrol show job. 0600 excludes other Unix identities; it does not isolate untrusted sibling allocations sharing the same uid.
worker_gateway.py Where those workers dial in. Off unless OPENAI4S_WORKER_LISTEN says otherwise — a listener on by default is an attack surface on every laptop that will never run a cluster job. The worker is the client because a compute node is usually reachable from nothing while the daemon usually is, and the credential is verified and burned before a single protocol byte is exchanged: this socket carries host_call traffic, so a listener that served first and checked later would be a remote execution surface for the duration of "later". Refusals say only "refused" — expired, replayed and forged are the same word, because the difference is an oracle. The rendezvous is keyed by (allocation, epoch), so a straggler from a previous epoch cannot satisfy a wait for the current one.
reclaimer.py Leases that lapse, so an idle session stops holding a GPU. It states an intent — request_stop with a reason — and stops there; the reconciler's barrier is the single place a resource is actually given back, and a second teardown path would drift from the first on exactly the failure where it matters. The subtlety it exists to get right: a worker being alive is not a user being present. A session whose process is healthy, whose socket is connected and whose heartbeats arrive on time is still idle if nobody has run anything in it, and it is still holding what somebody else is queued for. So this loop reads last_active_at and never writes it. Which clock ran out picks the reason code, because "come back later" is the wrong thing to tell somebody whose session cannot come back.
session.py A chat session whose kernel lives on a granted resource. Readiness is a conjunction of four conditions (INV-5) and names the one that is missing, because the resource plane saying "running" means a process started somewhere — not that it reached us, not that its files are where the user's data is, not that it can evaluate 1+1. The other half is AttemptPreparer: the durable spec is what an operator reads back, and the spec that is submitted is derived from it once per attempt so a credential bound to (allocation, epoch) can exist at all. That derivation is why no signature is ever written to spec_json, and why a recovery is a genuinely new attempt rather than a replay of the lost one's identity. Both admission and attempt preparation first require the backend's verified per-allocation session-isolation capability and fail before filesystem or durable writes when it is absent; BATCH never uses this credential or capability.
local/ The default backend: this machine. Every install has it, which is why INV-8's reconciliation is implemented rather than stubbed there — otherwise the invariant would be untested outside a cluster nobody has in CI. It does not claim per-allocation session isolation, so it remains available for BATCH but not interactive remote SESSION placement.
reconciler.py The loop: one periodic pass comparing desired against observed, at most one step per workload per tick, every step written to be safe to repeat — because a tick that dies half way through is the only failure mode a daemon actually has. An Unknown submission is never retried blindly; the next tick asks find_by_token first (INV-8). A BACKEND_UNAVAILABLE observation moves nothing, so a scheduler restart is not a wave of dead workloads. The cancel barrier is one method so the plan's fixed order — fence → cancel tasks → drain → release → observe terminal → mark terminal — is a thing you can read rather than an ordering that emerges from where calls happen to sit, and it is re-entrant, because a barrier that cannot be walked twice strands a workload the first time a backend lags. Optional backend recovery facts are acknowledged only after reloading a terminal allocation plus a terminal parent or later recovery epoch from the durable store; constructor and end-of-tick sweeps close both sides of the commit/crash window.
ports.py — TaskRunner Optional and separate from AllocationBackend, because not every resource plane has the concept and a Protocol nobody can implement is one that gets implemented badly. A caller asks isinstance(backend, TaskRunner) and says plainly when the answer is no, rather than falling back to submitting a second allocation — which is exactly what INV-4 forbids.
slurm/ The Slurm backend — the one directory allowed to name a scheduler, which is what makes the rule above checkable rather than aspirational. The leak guard skips it by name, so a scheduler word anywhere else is a defect. It does not claim per-allocation session isolation, so it remains available for BATCH but interactive remote SESSION placement fails closed.
__init__.py Re-exports the contract and nothing else. Importing this package must not drag in an implementation — that is one of the runtime claims the leak guard makes, and it is why a backend is imported by composition code rather than from here.
models.py The vocabulary: Workload (kind ∈ SESSION/BATCH), Allocation (one attempt, one epoch), ResourceProfile (what a scientist asks for, in units they state), Phase, and the Reason codes from the plan's appendix C. Two shapes carry invariants that are easy to lose otherwise: ExternalHandle keeps a backend's own id wrapped so INV-2 survives contact with a dozen call sites, and SubmissionToken is minted before a submission is attempted — which is the whole of INV-8, because "did my submission land?" has to be a question about something the backend was told to record. Phase.is_terminal and Phase.is_active_allocation are the single readable copies of the sets the schema's partial unique index enforces.
ports.py The AllocationBackend Protocol — submit / observe / cancel / find_by_token / diagnostics — and SubmitResult as four cases rather than a boolean. Created, Existing (a submission carrying this token was already there, which is what makes a retry safe), Rejected (an answer: the workload can fail cleanly) and Unknown (not a retry-me: the caller must reconcile by token first, because a blind retry is how one submission becomes two jobs holding two GPUs). Unknown carries the token for exactly that reason — making the caller fish it out of its own state is how the step gets skipped. Optional capabilities stay separate: TerminalAllocationAcknowledger is for durable recovery facts, while SessionIsolationProvider may assert interactive placement only when a verified per-allocation OS identity, container, or mount namespace prevents sibling workspace and credential access. Missing, false, or failing isolation checks refuse SESSION; shared uid plus 0700/0600 is insufficient.