`openai4s/orchestration/slurm/`

The Slurm backend — the only place in the tree allowed to name a scheduler. INV-2 is a claim about the orchestration core, not a claim that no code anywhere may know how jobs are submitted; something has to. Concentrating that knowledge in one directory is what makes the rule checkable: the leak guard skips this path by name, so anything scheduler-shaped outside it is a defect rather than a judgement call.

file	what it is
`__init__.py`	Re-exports the backend, the broker and the cluster config. Imported by composition code; the orchestration core never reaches it.
`backend.py`	The `AllocationBackend` implementation, and the state table that turns `TIMEOUT` into `(FAILED, TIME_LIMIT_EXCEEDED)`. Three behaviours are load-bearing rather than incidental. A lost submission is `Unknown`, never a failure — and `submit` begins by asking whether the token is already out there, so the retry after an `Unknown` returns `Existing` instead of putting a second job on the cluster (INV-8). That lookup matches on JobName as well as Comment, because `sacct` persists Comment only where a site enables `AccountingStoreFlags=job_comment` while JobName is an ordinary accounting field; and when the lookup itself cannot be answered, `submit` returns `Unknown` rather than pressing on — absence of an answer is not an answer of absence. A job in neither the queue nor accounting is `LOST`, never `COMPLETED`: claiming success for work we cannot find is the one answer that silently loses science. And an unreachable scheduler leaves the phase where it was — a cluster outage must not terminate everyone's jobs. A multi-node `SESSION` is launched as one `srun` task per node, each worker resolving its rank from `SLURM_PROCID`, so gang readiness has a worker on every node while no credential value enters the job environment. `run_task` runs steps inside the allocation the workload already holds and refuses when there is no live handle — the tempting fallback, submitting a fresh allocation, is exactly what INV-4 forbids.
`broker.py`	The subprocess boundary: every `sbatch`/`squeue`/`sacct`/`scancel` call and all the parsing, so INV-9 is checkable in one place. argv is a list with `shell=False`, so a profile named `gpu; rm -rf /` is an argument rather than a command; job names and comments are additionally restricted to a character class, because those two fields carry our identifiers into scheduler output and back out through a parser. A credential-shaped environment variable is refused outright — a secret travels as the path to an 0600 file. Environment values are validated as well as names: `--export` is a comma-separated list with no escaping, so a value containing a comma, newline, or NUL — or reading as `ALL`/`NONE`/`NIL` — is refused before `{"X": "1,ALL"}` can propagate the daemon's entire environment, credentials included, into the job. `StepSpec` holds in-allocation `srun` steps to the same discipline, and a long-lived step's output is drained under a byte cap instead of the control-command timeout, so the transport does not kill the step it is supposed to observe. `SlurmCommandError` distinguishes `timed_out` from `unreachable` from a plain refusal; writing the tests proved why, since swallowing "no scheduler on PATH" as "no such job" declared every running job lost.
`profiles.py`	`cluster.toml`: the one file that maps a profile name a scientist says (`gpu-interactive`) onto the partition and QoS an administrator controls (D5). `ClusterProfile.public()` is the admin-readable view and deliberately omits both — a queue name in a JSON body is the same leak by a slower route. Parsed with `tomllib` where it exists and with a deliberately tiny fallback where it does not: `requires-python` is 3.10, CI runs 3.10, and a tomllib-only reader would have made the whole cluster feature dead on the floor version. The fallback understands tables, strings, integers and booleans, and refuses anything else with a line number rather than guessing — failing to read is recoverable, misreading schedules onto the wrong queue. It is kept in strict parity with `tomllib` for what it does accept: quoted dotted keys stay one key, basic-string escapes and non-TOML integer spellings are refused rather than reinterpreted, because both readers are production paths and a divergence is a profile silently landing on the wrong queue with `configured: true`. Unknown keys are refused by name instead of ignored, and a profile may declare `nodes` for multi-node gangs.