The engine, under the hood.

Four binaries on one bus, contracts before code, and a single rule that never bends: the event log is the runtime, and nothing executes outside it.

The pipeline

Four binaries, one bus, two governed doors.

The problem: agents today run as loose processes — their access, their steps, and their failures live in different places, and reconstructing “what happened?” afterwards is archaeology. The answer: one pipeline, one bus, one door — every transition an event on the log.

aioctl / console — the operator starts a run or approves a held step, words on the log.
aio-server orchestrates definitions and dispatch; aio-reflex scores every gated step over aio.reflex.gate — request/reply, fails closed.
NATS JetStream — every event lands here before the next thing happens: the log is the runtime.
aio-worker consumes task queues, ensures the model, mints the run key; one aio-sidecar per job: engine-owned prompt, settings, MCP allowlist.
agentgateway — the only door: LLM calls on JIT per-run keys, tools deny-by-default, everything attributed.

Nothing crosses the gateway that the registry didn't declare. A request enters, the gate decides, the worker spawns one sidecar, and every step lands on the bus before the next one happens.

a governed code loop

dev-review-loop runs its 8 steps; the gate holds review_2 at risk 0.81; a human approves with words on the log; the verdict lands machine-readable — VERDICT: PASS.

an ad-hoc A2A send

A peer sends aio.a2a.send.claude — it becomes a single-turn run behind the same gate: input-required on hold, rejected on block, never a silent direct call.

no AI at all

echo-demo exercises the whole pipeline deterministically, gateway included. Governance is not a model feature; it’s the plumbing.

The logic

Event-sourced, replayable, deterministic by test.

The engine’s state is a fold over an append-only log. A run is not a process you monitor — it is a legal sequence of decisions that anyone can re-verify:

Every transition is an event first: run.started, step.dispatched, gate.decided, step.held, step.completed, trigger.fired, task.created — the log entry lands before the next thing happens.
aioctl replay re-executes the fold and checks the run against its definition hash: determinism is a tested property, not a promise.
Projections — the run index, the approvals queue, usage rollups — are cache. Lose one and aioctl reindex rebuilds it from the log.
Exactly-once behaviors ride explicit intent/fired event pairs; crash windows are re-driven by a recovery sweep, never guessed.

This is what makes the governance claim falsifiable: an audit is a query, not an archaeology project — and a disputed decision is replayable in seconds.

Contracts first

Schema before code, drift fails CI.

Every wire payload — envelopes, run events, gate requests and decisions, task definitions, agent cards, the gateway config — is pinned to a published JSON schema in contracts/schemas/, and the contract tests enforce the discipline:

a new event or payload type lands with its schema in the same change;
every tracked manifest, workflow, provider and task validates against its schema and the strict loader — unknown keys are loud errors, not warnings;
cross-references are swept: workflow→agent, manifest→provider→model, task→workflow, declared capacity ↔ parallelism;
no secrets, no operator paths — tracked files carry neither, and a guard test says so;
the CLI and the control API are two faces of one OpenAPI contract — --format=table|json|ndjson and exit codes are pinned the same way.

Two services never agree “by convention” on what a message means. They agree by schema, or the build is red.

The reflex gate

What reflex is — exactly.

Reflex is a small, local, non-generative decision model that answers typed questions. Not a chatbot, not an LLM jury: a 421M-parameter model (Laya, CPU-served) that maps a structured question to a structured answer it literally cannot leave.

The problem: the bigger the model, the worse the babysitter — you cannot gate a fleet on human attention alone, and you should not gate it on something that can talk its way past a rule. The answer: a tiny non-generative model that answers only typed questions — and fails closed.

Typed question in — route · verdict · goal · risk · escalate, plus run context. Never free text.
Laya scores it — 421M parameters, local, non-generative; every request scored, warmed before serving.
Threshold rails decide — risk ≥ t-hold parks, ≥ t-block refuses, verdict FAIL judges; gate down fails closed.
go · hold · block out — hold carries a reason and a risk score; block fails the run loudly.
Every decision recorded — gate.decided on the log, later paired with the outcome as fine-tuning data.

Calibration is published, not promised: 94.9% on verdict questions, 90.0% on escalate, 75.6% on goal — measured on held-out cases and re-measurable on your own decision log. The same judgment sits on the tool path: tool.decide scores every tool call pre-flight, and allow/hold precision is measured per tool.

a verdict question

A review step ends VERDICT: PASS. The gate answers the verdict question and a risk score; the rails let the run continue — and record the decision for later judgment.

an escalate question

prospecting-crew’s outreach step scores escalate 0.90 → the step is held for a human. Approve or reject; the decision — your words — is on the log.

gate down

Reflex unreachable at dispatch? Every gated step fails closed to hold. The run parks and tells you why — it never proceeds ungoverned.

Just-in-time access

Credentials that exist for exactly one run.

Provider keys are platform property — scoped, rotated, auditable. So the worker mints them per run instead of handing out durable ones: minted when the run starts, revoked when it ends — an agent never holds a permanent key.

The problem: a durable API key is a standing privilege — whoever holds it, holds it forever, and the audit usually happens after the damage. The answer below: credentials that exist for exactly one run.

Dispatch — the task is admitted; the worker mints an llm.apiKey over the gateway admin API.
Env-only hand-off — the sidecar receives AIO_GATEWAY_TOKEN in its process env: never the envelope, the log, or a file.
Token-only sidecar ⇄ gateway ⇄ provider — calls are scoped and attributed; no provider key exists in the agent's world.
Revoked at terminal — and a boot sweep revokes crash leftovers; engine-managed tokens only.

Verify before trust: the serving box is checked at dispatch (/v1/models — listed, loaded, context length fits) before any agent spawns; self-hosted models are loaded on demand with the full contract and loud, named timeouts.

a provider-backed run

The worker mints an llm.apiKey over the gateway admin API and hands the sidecar AIO_GATEWAY_TOKEN in its process env. Calls flow gateway → provider; the terminal revokes the key.

a Claude Code session

Subscription credentials are inherited, never minted — no gateway key exists at all for the LLM plane. The tool plane still transits the gateway, attributed per run.

a crash mid-run

Worker dies before the revoke? The boot sweep revokes leftover engine-managed tokens at startup. Nothing durable leaks by accident.

Provider-agnostic, capacity-contracted. The provider registry treats frontier APIs and your own GPU box as first-class citizens — each declaration carries context_length, per-model parallelism, and optional single_model switching exclusivity. The worker enforces residency: a task past its slots waits on the gate, never head-of-line blocks the queue, and a single_model box is never asked to swap models under a running agent.

Honest scope: wired today — Claude, Codex, Kimi, local models (Qwen, Gemma), any HTTP LLM. Battle-tested in CI today: Claude Code (with subscription-session credentials) plus provider-backed manifests. Nothing gets the “proven” label until its smoke suite passes.

Proven, not reinvented

The gateway is someone else's good work.

AiOverload did not build a proxy. The agentgateway is a proven, tiny, secure, stable open-source product (Apache-2.0) maintained by its own team — and this engine composes it rather than recreating it:

the registry feeds it at boot, the worker mints per-run keys over its admin API, the sidecars speak to it token-only — AiOverload owns the policy, the gateway owns the proxy;
it doubles as a general model gateway: point it at your own models and providers, and the same door serves the rest of your stack;
and it is core, not decoration. Remove the gateway and the JIT credential boundary, the per-run tool attribution, and the rendered JSON access logs go with it — a real part of the observability and the governance goes dark with one deletion.

License-clean by construction: a separately deployed container, integrated wire-only — no vendored code, no copyleft entanglement, in either direction.

The sidecar

One ephemeral process per job — the hard boundary.

The sidecar is the harness boundary the engine owns. For every cli task it materializes the reality the manifest only declares:

the engine-owned system prompt (base prompt + the manifest’s mandatory persona text) — the agent never brings its own instructions;
the settings file: provider spec and the per-run gateway token, read from env at the last responsible moment;
the per-run MCP allowlist — the only tool servers that exist on the wire;
argv-only execution inside an env allowlist, one process group, with a scratch directory that is SIGKILL-proof cleaned up.

Around it, the execution-isolation contract:

sandbox levels — restricted (read-only world) vs workspace-write (the worktree, nothing else), enforced by the runtime and the env allowlist, not by prompt begging;
liveness — heartbeat acks per sidecar; three missed beats and the process group is killed with a named sidecar_wedged error (the §7 process-group kill rules are verified, never assumed);
zero progress, named death — a cli stream silent past progress_timeout (default 10m) is killed mid-run with a named error instead of burning its whole budget;
one result line — exactly one structured result on stdout, or the runtime synthesizes a malfunction error. There is no third outcome.

Tools, denied by default

MCP: nothing on the wire the registry didn't declare.

Agents do not configure their own tools. A manifest names its mcp_servers; the sidecar renders exactly that set into the per-run configuration; the gateway enforces the same set at the wire. A manifest that declares none gets {"mcpServers": {}} — an empty world, not an error, because default-deny is the steady state, not a punishment.

The registry (registry/mcp/) is the catalog: each server is a typed, schema-pinned entity with contract tests — a five-piece set, like every other registry entity.
All tool traffic transits the gateway by default — with per-run attribution headers, rendered JSON access logs, and reflex tool.decide pre-call vetoes at the wire.
The engine feeds the gateway its model catalog at boot — file-rendered models and runtime resources conflict, so the registry stays the single source of truth and the app feeds the gateway, never the other way.
Knowledge reads ride the same discipline: scoped tools, refusals named (knowledge_scopes_not_declared, knowledge_scope_refused), never silent empty results.

Security

Stated as facts, not posture.

Secrets move through env — only

Read at the last responsible moment, never logged, never on envelopes. Tracked files carry neither secrets nor operator paths, and a CI guard test proves it on every change.

Per-run credentials

Minted just-in-time, revoked at terminus, swept at boot. A key that outlives its run is a bug, not an inconvenience — see the lifecycle above.

Wire-level deny-by-default

MCP allowlists rendered per run and enforced at the gateway. Undeclared servers do not exist, declared ones are attributed per run.

Least-privilege accounts

Role-based NATS accounts — every grant pinned by auth-matrix contract tests. TLS on the API and the bus, optional bearer token at the edge.

Sandboxed execution

restricted vs workspace-write sandbox levels, argv-only execution, verified process-group kills, and a progress timeout that kills silent agents with a named error.

Scanned continuously

Secret scanning (gitleaks) and vulnerability scanning (govulncheck) ride the hermetic CI ladder on every PR — the same ladder that gates every merge.

The threat model is written down — not implied: docs/threat-model.md in the repository, published with the source at release. Known limits stay named there, the same way they stay named on this site.