Run each durable session's turns on the agent that created it

- `createFxAgent()` takes a `name`, which defaults to `"default"`, and `agent.name` returns it. A name that isn't 1 to 64 letters, digits, `.`, `_` or `-` throws.
- A session's turns run only on agents with the name of the agent that created it, so one app can run several durable agents with different models, instructions and tools.
- A prompt sent through another agent, such as `coding.session(supportSessionId).prompt()`, runs on the session's own agent, and the sending agent's `onEvent` gets a `session.forwarded` event.
- A process with no agent of a turn's name refuses the delivery with an `FX_NO_AGENT` error that names it, and the queue delivers it again.
- `wakeHandler` from `libfx` returns one queue route for every agent in the process. `agent.wakeHandler()` also routes by name.
- Agents created without a durability share one for each sessions location, or one on Vercel, so they can reach each other's sessions.
- Sessions stored before this change belong to the `default` agent.

Closes #1246.
This commit is contained in:
Pranit committed 2026-10-13 01:57:35 -04:00
1 parent 4e9f14916a
commit 871e93d845
22 files changed
+1073 -133

No files matched your search

+2
View File
@@ -379,6 +379,8 @@ jobs:
bun sdk/tests/model/drive.mjs cutoff --walks 20 --seed 5
bun sdk/tests/model/drive.mjs leased --walks 40 --seed 12
bun sdk/tests/model/drive.mjs queue --walks 40 --seed 4
bun sdk/tests/model/drive-agents.mjs send --walks 60 --seed 11
bun sdk/tests/model/drive-agents.mjs consume --walks 60 --seed 13
- name: Test libfx on a World
run: |
+1 -1
View File
@@ -58,7 +58,7 @@ Do not treat the demos or this file as the implementation contract. When prose a
| `encodeXtermKeyEvent()` or `xtermAdapter()` only | `sdk/node/test-xterm-adapter.mjs` |
| Core debugger query behavior or automation state | `sdk/tests/test-core-browser.mjs` |
| Terminal demo asset references, integrity, or cache policy | Package into a fresh temporary directory and inspect the generated HTML, manifest, and headers |
| Durable sessions: leases, fencing, takeovers, tool reruns, the UI stream | `sdk/tests/test-durable-core.mjs` on memory, local, world, and queue (the core with a scripted harness; world is an app-built World passed to `world()`, queue one whose queue delivers as Vercel Queues does, with `queueLeases`), `sdk/tests/test-durable.mjs` on memory, local, and world, then `sdk/tests/model/check.sh` and `bun sdk/tests/model/drive.mjs send`, `lookup`, `write`, `cutoff` (deadline cut-offs up to their bound), `leased` (a World without liveness, where each worker renews a short lease), and `queue` (a World whose queue shows whether a worker lives, so leases are held without renewals), which walk the real code along the TLC state graphs of `DurableSession.tla`; change the spec with the code. With a Vercel access token, also run both against Vercel's World: `test-durable.mjs vercel native` and `drive.mjs --durability vercel`, configured as `sdk/tests/vercel-storage.mjs` describes |
| Durable sessions: leases, fencing, takeovers, tool reruns, the UI stream | `sdk/tests/test-durable-core.mjs` on memory, local, world, and queue (the core with a scripted harness; world is an app-built World passed to `world()`, queue one whose queue delivers as Vercel Queues does, with `queueLeases`), `sdk/tests/test-durable.mjs` on memory, local, and world, then `sdk/tests/model/check.sh` and `bun sdk/tests/model/drive.mjs send`, `lookup`, `write`, `cutoff` (deadline cut-offs up to their bound), `leased` (a World without liveness, where each worker renews a short lease), and `queue` (a World whose queue shows whether a worker lives, so leases are held without renewals), which walk the real code along the TLC state graphs of `DurableSession.tla`; change the spec with the code. For agent names and routing, also `bun sdk/tests/model/drive-agents.mjs send` and `consume`, which walk `AgentRouting.tla`'s graphs with each process in a worker thread of its own. With a Vercel access token, also run both against Vercel's World: `test-durable.mjs vercel native` and `drive.mjs --durability vercel`, configured as `sdk/tests/vercel-storage.mjs` describes |
| Supported public behavior or setup | Update `sdk/README.md` in the same change |
Use Node.js 24 for SDK tests. CI relies on its JSPI implementation behind `--experimental-wasm-jspi`.
+23 -3
View File
@@ -128,17 +128,36 @@ On `local()`, `vercel()` and `world()`, a session id is a Workflow run id, `wrun
A session stores the conversation and nothing else. Every process passes its own model, credentials, tools, MCP clients and skills when it creates the agent. If a session's saved checkpoint came from a different libfx version, tool set or model, the session still loads, and `onEvent` gets a `checkpoint.mismatch` event naming what changed.
### Several agents
An app can have more than one durable agent, each with its own model, instructions and tools. Give each one a name:
```js
export const support = createFxAgent({ name: "support", instructions: "Answer billing questions." });
export const coding = createFxAgent({ name: "coding", model: "anthropic/claude-opus-5.5", tools: codingTools });
```
A session belongs to the agent that created it, and its turns run only on an agent with that name. A prompt sent through another agent, such as `coding.session(supportSessionId).prompt()`, still runs on `support`, and the sending agent's `onEvent` gets a `session.forwarded` event. An agent created without a name is named `default`.
- Agents with the same name are copies of one agent, such as one on each server, and take turns running its sessions.
- Keep names stable. A session keeps the name it was created with, so renaming an agent leaves its sessions with no agent to run them.
- A process that gets a turn for a name it has no agent for refuses it with an `FX_NO_AGENT` error naming the agent, and the queue delivers it again.
- Agents share sessions through their durability. Give agents that share sessions the same durability, or none: agents created without one share the default.
### On Vercel
Add a route for libfx's queue deliveries:
```js
// app/api/libfx/route.js
import { agent } from "@/lib/agent";
import { wakeHandler } from "libfx";
import "@/lib/agents"; // creates the app's agents, so this function has them
export const POST = agent.wakeHandler();
export const POST = wakeHandler();
```
The route delivers each turn to the agent it belongs to, so one route serves every agent the module it imports creates. With one agent, `agent.wakeHandler()` does the same.
Then subscribe it to libfx's topic in `vercel.json`:
```json
@@ -492,7 +511,8 @@ Every entry point ships TypeScript types, and the package has no runtime depende
- **One agent per process:** create it once at module scope, not per request, and open `agent.session(id)` for each conversation.
- **Session ids:** take them from `turn.accepted` or `agent.newSessionId()`. On every durability except `memory()`, an id that isn't a Workflow run id throws.
- **Side effects:** set `idempotent: true` only on tools that are safe to run twice, and pass `executionId` to services that take an idempotency key.
- **Deployments:** on Vercel, the `wakeHandler()` route and its `vercel.json` trigger are both required, or prompts are stored and never run.
- **Deployments:** on Vercel, the `wakeHandler()` route and its `vercel.json` trigger are both required, or prompts are stored and never run. The route must import the module that creates the agents.
- **Several agents:** give each durable agent its own `name`, and never rename one that has sessions.
- **Credentials:** keep keys and ChatGPT or Grok credentials on the server, and the object `startSignIn()` returns too.
- **Checking a setup:** `getBackendInfo()` reports which backend would load without creating anything or making requests.
- **Changing libfx:** the source is in the [fx repository](https://github.com/vercel-labs/fx) under `sdk/`. Its `AGENTS.md` files name the tests each change needs.
+154 -26
View File
@@ -170,6 +170,10 @@ export function foldSessionLog(entries, { now = 0, alive = () => null } = {}) {
// A checkpoint taken where a turn yielded leaves that turn open.
let openTurn = typeof base?.openTurn?.id === "string" ? { id: base.openTurn.id, at: Number(base.openTurn.at) || 0, context: base.openTurn.context ?? null, settings: base.openTurn.settings ?? null } : null;
let lastTurnId = base?.lastTurnId ?? null;
// The agent the session belongs to: the one its first stored input, or the
// checkpoint it was restored from, names. A session stored before agents
// had names belongs to the default agent.
let agent = typeof base?.agent === "string" ? base.agent : base && base.through != null ? defaultAgentName : null;
let yielded = base?.yielded === true;
// The context each prompt's or steer's caller gave, and the settings its
// session object set, which its turn runs with: an untaken steer becomes a
@@ -201,6 +205,7 @@ export function foldSessionLog(entries, { now = 0, alive = () => null } = {}) {
if (position > maxCursor) maxCursor = position;
switch (entry?.k) {
case "input":
agent ??= typeof entry.agent === "string" ? entry.agent : defaultAgentName;
if (seenInputs.has(entry.key)) break;
seenInputs.add(entry.key);
inputs.push({ ...entry, cursor: position });
@@ -317,6 +322,7 @@ export function foldSessionLog(entries, { now = 0, alive = () => null } = {}) {
cutoffs: openTurn === null ? 0 : cut.cutoffs,
openFailed,
lastTurnId,
agent,
yielded,
pending,
steers,
@@ -371,9 +377,9 @@ function redelivers(delivery, lease) {
function createMemoryBackend({ maxDurationMs, reserveMs } = {}) {
const sessions = new Map();
const holders = liveHolders;
// Each agent sharing the backend takes deliveries in turn.
const handlers = [];
let turnOf = 0;
// The agents sharing the backend, by name: an agent's copies take the
// deliveries for its name in turn.
const handlers = new Map();
const timers = new Set();
const sessionFor = (id) => {
let session = sessions.get(id);
@@ -389,8 +395,14 @@ function createMemoryBackend({ maxDurationMs, reserveMs } = {}) {
};
// A message keeps its id across its deliveries, as a queue's does.
const deliver = (message, attempt, messageId = newId("qmsg")) => {
if (handlers.length === 0) return later(() => deliver(message, attempt, messageId), 10);
const handler = handlers[turnOf++ % handlers.length];
if (handlers.size === 0) return later(() => deliver(message, attempt, messageId), 10);
const group = handlers.get(agentOf(message));
// No agent of that name here yet: tried again, as a failed delivery is.
if (!group) {
if (attempt < 48) later(() => deliver(message, attempt + 1, messageId), 1000);
return;
}
const handler = group.members[group.turn++ % group.members.length];
Promise.resolve()
.then(() => handler(message, { messageId, attempt }))
.then((result) => {
@@ -481,12 +493,11 @@ function createMemoryBackend({ maxDurationMs, reserveMs } = {}) {
later(() => deliver(message, 1), delaySeconds * 1000);
},
},
listen(fn) {
handlers.push(fn);
listen(fn, name = defaultAgentName) {
const group = joinGroup(handlers, name, fn);
return () => {
const index = handlers.indexOf(fn);
if (index >= 0) handlers.splice(index, 1);
if (handlers.length > 0) return;
leaveGroup(handlers, name, group, fn);
if (handlers.size > 0) return;
// Nobody is left to deliver to.
for (const timer of timers) clearTimeout(timer);
timers.clear();
@@ -503,6 +514,12 @@ const logStep = "fx.log";
// "libfx-ui".
const uiStreamOf = (runId) => `${runId.replace(/^wrun_/, "strm_")}_user_bGliZngtdWk`;
const queuePrefix = "__libfx_wkf_workflow_";
// The name of an agent created without one, and of whatever a message or a
// session from before names belonged to.
const defaultAgentName = "default";
const agentNameRule = /^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$/;
// The agent a queued message is for.
const agentOf = (message) => (typeof message?.agent === "string" ? message.agent : defaultAgentName);
const queueName = `${queuePrefix}session`;
const eventIdPattern = /^[a-z]+_(\d{26})$/;
@@ -561,9 +578,70 @@ const laterPageEvents = 100;
// deliveries in turn, as servers sharing a queue do.
const listenersOf = new WeakMap();
// Adds `member` to the group for `name` in `groups`, a Map of name to
// `{ members, turn }`, and returns the group; leaveGroup takes it out again.
function joinGroup(groups, name, member) {
let group = groups.get(name);
if (!group) groups.set(name, group = { members: [], turn: 0 });
group.members.push(member);
return group;
}
function leaveGroup(groups, name, group, member) {
const index = group.members.indexOf(member);
if (index >= 0) group.members.splice(index, 1);
if (group.members.length === 0 && groups.get(name) === group) groups.delete(name);
}
function noAgentNamed(name) {
return Object.assign(new Error(`no libfx agent named ${JSON.stringify(name)} runs in this process; the queue delivers this again`), { code: "FX_NO_AGENT", agent: name });
}
// The durable agents in this process, by name, from when each is created
// until it closes, each as `{ open, durability }`: a route delivers each
// message to the agent it names, and a function that only receives queue
// messages has its agents here before any of them is used. Shared by every
// copy of this module in the process.
const agentsByName = (globalThis[Symbol.for("libfx.durableAgents")] ??= new Map());
// The backend of an agent named `name` in this process on `durability`, the
// agent's copies in turn, or null. Agents share sessions only through the
// durability they were created with, so one on another holds other sessions.
async function agentSharing(name, durability) {
const group = agentsByName.get(name);
if (!group) return null;
const { members } = group;
const first = group.turn++;
for (let index = 0; index < members.length; index += 1) {
const member = members[(first + index) % members.length];
if ((await member.durability()) !== durability) continue;
const created = await member.open();
if (typeof created.deliverHere === "function") return created;
}
return null;
}
/**
* The request handler a deployment mounts for queue deliveries, for every
* durable agent in the process: each delivery goes to the agent its message
* names. One route and one queue trigger serve any number of agents.
*/
export function wakeHandler() {
return async (request) => {
for (const group of agentsByName.values()) {
for (const member of group.members) {
const created = await member.open();
if (created.viaRoute === true) return created.queueHandler(request);
}
}
return new Response("no libfx agent in this process takes queue deliveries from a route", { status: 503 });
};
}
// libfx starts and closes a World it `owned`, having created it; a World the
// app passed in stays the app's.
function createWorldBackend(world, { name, reserveMs, pollMs, leaseMs, alive, holderInfo, livenessKnown = false, queueLeases = false, runInRequest = queueLeases, waitUntil, queueDurable = false, maxDurationMs, owned = true }) {
function createWorldBackend(world, durability) {
const { name, reserveMs, pollMs, leaseMs, alive, holderInfo, livenessKnown = false, queueLeases = false, runInRequest = queueLeases, waitUntil, queueDurable = false, maxDurationMs, owned = true } = durability;
const spec = world.specVersion === undefined ? {} : { specVersion: world.specVersion };
const created = new Set();
// Runs being created now: writes that go out together create a run once.
@@ -595,8 +673,14 @@ function createWorldBackend(world, { name, reserveMs, pollMs, leaseMs, alive, ho
if (result?.redeliver === true) throw new Error("libfx asks the queue to deliver this message again");
return Number.isFinite(result?.timeoutSeconds) ? { timeoutSeconds: result.timeoutSeconds } : undefined;
};
// The route a deployment mounts delivers to this agent.
const queueHandler = world.createQueueHandler(queuePrefix, (message, meta) => deliverTo(handler, message, meta));
// The route a deployment mounts delivers to the agent in this process the
// message names, whichever agent's route it is.
const queueHandler = world.createQueueHandler(queuePrefix, async (message, meta) => {
const named = agentOf(message?.input);
const agent = await agentSharing(named, durability);
if (!agent) throw noAgentNamed(named);
return agent.deliverHere(message, meta);
});
const backend = {
name,
livenessKnown,
@@ -896,23 +980,37 @@ function createWorldBackend(world, { name, reserveMs, pollMs, leaseMs, alive, ho
// The route a deployment mounts for queue deliveries, until the World
// can deliver to an in-process handler itself.
queueHandler,
listen(fn) {
durability,
// Whether the queue delivers to queueHandler, mounted by the deployment.
viaRoute: route,
// Delivers a routed message to this backend's agent.
deliverHere: (message, meta) => deliverTo(handler, message, meta),
listen(fn, name = defaultAgentName) {
handler = fn;
if (route) return () => { if (handler === fn) handler = null; };
// The agents on this World, by name: an agent's copies take the
// deliveries for its name in turn. A name none of them has goes to an
// agent of that name on the same durability elsewhere in the process,
// as each agent has a World of its own, and one the process has none
// of is left for the queue to deliver again.
let listeners = listenersOf.get(world);
if (!listeners) {
listeners = { handlers: [], turn: 0 };
listeners = new Map();
listenersOf.set(world, listeners);
world.registerHandler(queuePrefix, world.createQueueHandler(queuePrefix, (message, meta) => {
const { handlers } = listeners;
return deliverTo(handlers.length === 0 ? null : handlers[listeners.turn++ % handlers.length], message, meta);
world.registerHandler(queuePrefix, world.createQueueHandler(queuePrefix, async (message, meta) => {
const named = agentOf(message?.input);
const group = listeners.get(named);
if (group) return deliverTo(group.members[group.turn++ % group.members.length], message, meta);
const agent = await agentSharing(named, durability);
if (agent) return agent.deliverHere(message, meta);
if (listeners.size === 0) return deliverTo(null, message, meta);
throw noAgentNamed(named);
}));
}
listeners.handlers.push(fn);
const group = joinGroup(listeners, name, fn);
return () => {
if (handler === fn) handler = null;
const index = listeners.handlers.indexOf(fn);
if (index >= 0) listeners.handlers.splice(index, 1);
leaveGroup(listeners, name, group, fn);
};
},
// Closing is best effort: a runtime whose HTTP agent cannot close, as
@@ -972,8 +1070,9 @@ const isFenced = (error) => error?.code === "FX_FENCED";
// Nothing the log already holds. Pure.
function messageEntries(state, message) {
const entries = [];
const named = typeof message.agent === "string" ? { agent: message.agent } : {};
if (message.seed && state.checkpoint === null && state.records.length === 0) {
entries.push({ k: "checkpoint", through: null, data: message.seed });
entries.push({ k: "checkpoint", through: null, data: message.seed, ...named });
}
const key = `${message.type}:${message.messageId}`;
const known = state.inputKeys.has(key) || state.consumed.has(message.messageId);
@@ -983,6 +1082,7 @@ function messageEntries(state, message) {
key,
type: message.type,
messageId: message.messageId,
...named,
...(message.input === undefined ? {} : { input: message.input }),
...(typeof message.target === "string" ? { target: message.target } : {}),
...((message.type === "prompt" || message.type === "steer") && message.context !== undefined ? { context: message.context } : {}),
@@ -1086,6 +1186,17 @@ class SessionWorker {
const deadline = this.backend.deadline() ?? (this.backend.maxDurationMs === null ? null : Date.now() + this.backend.maxDurationMs);
let entries = await log.read();
let state = this.fold(entries, delivery);
// A session runs only on the agent it belongs to, which a new one is the
// agent that sent its first message. A message that reached another
// agent, as one sent to another agent's session does, goes on to it.
const owner = state.agent ?? agentOf(message);
if (owner !== this.agent.name) {
await this.backend.queue.send({ ...message, agent: owner }, {
idempotencyKey: `${message.sessionId}:${message.type}:${message.messageId}:to:${owner}`,
});
this.agent.emit("session.forwarded", { sessionId: this.sessionId, agent: owner });
return undefined;
}
for (let claims = 0; claims < 8; claims += 1) {
// A session with no open turn starts its next prompt at once: the
// first model request goes out while the input and the lease are
@@ -1292,6 +1403,7 @@ class SessionWorker {
pending: now.unconsumed.filter((input) => input.cursor <= Number(through)),
recent: now.recent,
lastTurnId: now.lastTurnId,
...(now.agent === null ? {} : { agent: now.agent }),
...(now.openTurn ? { openTurn: now.openTurn, yielded: now.yielded } : {}),
lease: { holder: this.holder, epoch: lease.epoch, expiresAt: leaseExpiresAt, pid: lease.pid, host: lease.host, ...(lease.delivery ? { delivery: lease.delivery } : {}) },
});
@@ -1910,16 +2022,20 @@ export function createDurableAgentFactory({ harness, defaultDurability, name = "
if (!options || typeof options !== "object" || Array.isArray(options)) {
throw new TypeError(`${name}() options must be an object`);
}
const { durability, ...harnessOptions } = options;
const { durability, name: agentName = defaultAgentName, ...harnessOptions } = options;
if (durability !== undefined && !isDurability(durability)) {
throw new TypeError("durability must come from memory(), local(), vercel() or world()");
}
if (typeof agentName !== "string" || !agentNameRule.test(agentName)) {
throw new TypeError(`${name}() name must be 1 to 64 letters, digits, ".", "_" or "-", starting with a letter or digit`);
}
const plugged = harness(harnessOptions);
const seed = plugged.seed === undefined ? undefined : toBase64(plugged.seed);
const emit = (type, detail = {}) => {
try { options.onEvent?.({ type, timestamp: performance.now(), ...detail }); } catch {}
};
const agent = {
name: agentName,
emit,
// Work that outlives the delivery that started it, which close() waits
// for: an engine that shuts down after its worker let go.
@@ -1937,6 +2053,7 @@ export function createDurableAgentFactory({ harness, defaultDurability, name = "
}),
};
let backendPromise = null;
let durabilityPromise = null;
let chosenDurability = null;
let unlisten = null;
let closed = false;
@@ -1945,15 +2062,21 @@ export function createDurableAgentFactory({ harness, defaultDurability, name = "
const backend = () => {
if (closed) return Promise.reject(new Error(`${label} is closed`));
backendPromise ??= (async () => {
const chosen = durability ?? await defaultDurability();
const chosen = await durabilityOf();
chosenDurability = chosen;
const created = await createBackend(chosen);
await created.start?.();
unlisten = created.listen((message, meta) => receive(created, message, deliveryOf(meta)));
unlisten = created.listen((message, meta) => receive(created, message, deliveryOf(meta)), agentName);
return created;
})();
return backendPromise;
};
// The durability the agent's sessions live in, chosen once.
const durabilityOf = () => (durabilityPromise ??= durability ? Promise.resolve(durability) : defaultDurability());
// In the process's agents from now, so a route that receives a message
// for this agent finds it before anything uses it.
const member = { open: backend, durability: durabilityOf };
const registered = joinGroup(agentsByName, agentName, member);
// One delivery of `message` to the session's worker here: from the
// queue, with `delivery` naming the queue's message and attempt, or with
// `inRequest` from the request that took the prompt.
@@ -2016,7 +2139,7 @@ export function createDurableAgentFactory({ harness, defaultDurability, name = "
...(context === undefined ? {} : { context }),
...(settings === undefined ? {} : { settings }),
};
const full = { ...message, ...extra, sessionId };
const full = { ...message, ...extra, agent: agentName, sessionId };
// A queue that ends with the process could lose the message, so its
// input reaches the log before the caller hears it was accepted.
if (!created.queueDurable) {
@@ -2284,6 +2407,10 @@ export function createDurableAgentFactory({ harness, defaultDurability, name = "
async newSessionId() {
return (await backend()).newSessionId();
},
/** The agent's name: its sessions' turns run only on agents with this name. */
get name() {
return agentName;
},
/** The agent's own session, for code that holds one conversation. */
get sessionId() {
return defaultSession?.id ?? null;
@@ -2311,6 +2438,7 @@ export function createDurableAgentFactory({ harness, defaultDurability, name = "
async close() {
if (closed) return;
closed = true;
leaveGroup(agentsByName, agentName, registered, member);
while (active.size > 0) await Promise.allSettled([...active]);
if (backendPromise) {
const created = await backendPromise.catch(() => null);
+21 -5
View File
@@ -8,7 +8,7 @@ import { fileURLToPath, pathToFileURL } from "node:url";
import { CoreOutput } from "./core-output.js";
import { loadModule, withModuleFailure } from "./wasm-module.js";
import { fetchCleanupAction, cleanupTimeoutMs, cleanupByteLimit } from "./fetch-cleanup.js";
import { createDurableAgentFactory, memory } from "./durable.js";
import { createDurableAgentFactory, memory, wakeHandler } from "./durable.js";
import { fxHarness } from "./fx-harness.js";
import {
createFxEngine as createWasmAgent,
@@ -24,7 +24,7 @@ import {
xtermAdapter,
} from "./fx-sdk.js";
export { createMemoryPersistence, encodeXtermKeyEvent, FxFencedError, FxJournalVersionError, fxSdkApiVersion, listModels, memory, supportsJspi, xtermAdapter };
export { createMemoryPersistence, encodeXtermKeyEvent, FxFencedError, FxJournalVersionError, fxSdkApiVersion, listModels, memory, supportsJspi, wakeHandler, xtermAdapter };
export const libfxApiVersion = 3;
const nativeCoreApiVersion = 4;
@@ -676,14 +676,30 @@ export async function createFxEngine(options = {}) {
// on Vercel, otherwise files in the temporary directory. Each is its own
// module, loaded only here: a bundler copies the file the URL names and
// leaves the import itself to Node.
async function environmentDurability() {
// Agents created without a durability share one for each place sessions
// live, as agents passed one durability do, so they can send each other's
// prompts on.
const environments = new Map();
function environmentDurability() {
const dir = process.env.VERCEL ? null : process.env.FX_SESSIONS_DIR || resolve(tmpdir(), "libfx", "sessions");
const key = dir ?? "vercel";
if (!environments.has(key)) {
environments.set(key, chooseEnvironmentDurability(dir).catch((error) => {
environments.delete(key);
throw error;
}));
}
return environments.get(key);
}
async function chooseEnvironmentDurability(dir) {
const load = (asset) => import(/* webpackIgnore: true */ /* turbopackIgnore: true */ (asset.protocol === "" ? bundledAssetUrl(asset) : asset).href);
if (process.env.VERCEL) {
if (dir === null) {
const { vercel } = await load(new URL("./durable/vercel.mjs", import.meta.url));
return vercel();
}
const { local } = await load(new URL("./durable/local.mjs", import.meta.url));
return local({ dir: process.env.FX_SESSIONS_DIR || resolve(tmpdir(), "libfx", "sessions") });
return local({ dir });
}
/**
+15
View File
@@ -0,0 +1,15 @@
CONSTANTS
Names = {"S", "C"}
Sessions = {"s1", "s2"}
Procs = {"p1", "p2"}
P2Lacks = "C"
MaxSends = 3
RecordAtSend = FALSE
ByName = TRUE
CheckOwner = TRUE
SPECIFICATION Spec
INVARIANTS
TypeOK
RunsOnOwner
MissOnlyWhereLacking
PROPERTY OwnerKept
@@ -0,0 +1,11 @@
CONSTANTS
Names = {"S", "C"}
Sessions = {"s1", "s2"}
Procs = {"p1", "p2"}
P2Lacks = "C"
MaxSends = 3
RecordAtSend = FALSE
ByName = TRUE
CheckOwner = TRUE
SPECIFICATION FairSpec
PROPERTY Drains
@@ -0,0 +1,11 @@
CONSTANTS
Names = {"S", "C"}
Sessions = {"s1", "s2"}
Procs = {"p1", "p2"}
P2Lacks = "C"
MaxSends = 3
RecordAtSend = TRUE
ByName = TRUE
CheckOwner = TRUE
SPECIFICATION FairSpec
PROPERTY Drains
+15
View File
@@ -0,0 +1,15 @@
CONSTANTS
Names = {"S", "C"}
Sessions = {"s1", "s2"}
Procs = {"p1", "p2"}
P2Lacks = "C"
MaxSends = 3
RecordAtSend = TRUE
ByName = TRUE
CheckOwner = TRUE
SPECIFICATION Spec
INVARIANTS
TypeOK
RunsOnOwner
MissOnlyWhereLacking
PROPERTY OwnerKept
+15
View File
@@ -0,0 +1,15 @@
CONSTANTS
Names = {"S", "C", "D"}
Sessions = {"s1", "s2", "s3"}
Procs = {"p1", "p2"}
P2Lacks = "C"
MaxSends = 6
RecordAtSend = FALSE
ByName = TRUE
CheckOwner = TRUE
SPECIFICATION Spec
INVARIANTS
TypeOK
RunsOnOwner
MissOnlyWhereLacking
PROPERTY OwnerKept
+123
View File
@@ -0,0 +1,123 @@
----------------------------- MODULE AgentRouting -----------------------------
(***************************************************************************)
(* Which agent runs a durable session's turns when an app has several. *)
(* *)
(* Every agent queues its turns on one topic. A queued message names the *)
(* agent that sent it. The queue delivers a message to a process, which *)
(* hands it to its agent of that name; a process with no such agent fails *)
(* the delivery, naming the agent, and the queue keeps the message to *)
(* deliver again. The agent's worker stores the message's input, which *)
(* names the sender, and runs the turn when the session is its own. A *)
(* session belongs to the agent its first stored input names, so a prompt *)
(* one agent sends to another agent's session is sent on to that agent. *)
(* *)
(* An input is stored by the sender before it queues the message on a *)
(* queue that ends with the process (memory, local: RecordAtSend), and by *)
(* the worker that takes it on one that keeps messages (Vercel). *)
(* *)
(* With ByName = FALSE a process hands each delivery to any of its agents, *)
(* as libfx did before names (Witness-NoNames). With CheckOwner = FALSE a *)
(* worker runs any session it is handed, so a prompt sent to another *)
(* agent's session runs on the sender (Witness-NoOwnerCheck). *)
(* *)
(* A turn here is one step: what this spec checks is which agent runs it. *)
(* DurableSession.tla checks what happens within one agent's session. *)
(***************************************************************************)
EXTENDS Naturals, FiniteSets
CONSTANTS
Names, \* the agents' names
Sessions,
Procs, \* processes the queue delivers to; each has an agent of every name but P2Lacks
P2Lacks, \* a name process "p2" has no agent for, or "none"
MaxSends, \* prompts sent in all
RecordAtSend, \* the sender stores a prompt's input before it queues it
ByName, \* a delivery goes to the agent the message names
CheckOwner \* a worker runs only its own agent's sessions
None == "none"
Hosts == [p \in Procs |-> IF p = "p2" THEN Names \ {P2Lacks} ELSE Names]
VARIABLES
queue, \* messages queued and not yet taken: [id, s, n]
owner, \* the agent each session belongs to, once an input is stored
ran, \* <<session, agent>>: an agent ran one of the session's turns
missed, \* names a delivery found no agent for
sends, \* prompts sent so far, which also numbers them
act \* the step just taken, for the driver
vars == <<queue, owner, ran, missed, sends, act>>
Msg == [id: 0..MaxSends, s: Sessions, n: Names]
TypeOK ==
/\ queue \subseteq Msg
/\ owner \in [Sessions -> Names \cup {None}]
/\ ran \subseteq (Sessions \X Names)
/\ missed \subseteq Names
/\ sends \in 0..MaxSends
Init ==
/\ queue = {}
/\ owner = [s \in Sessions |-> None]
/\ ran = {}
/\ missed = {}
/\ sends = 0
/\ act = [name |-> "Init"]
\* A session's owner once an input naming n is stored: the first one stays.
Stored(o, n) == IF o = None THEN n ELSE o
\* Agent n prompts session s: its message names n.
Send(n, s) ==
/\ sends < MaxSends
/\ queue' = queue \cup {[id |-> sends, s |-> s, n |-> n]}
/\ sends' = sends + 1
/\ owner' = IF RecordAtSend THEN [owner EXCEPT ![s] = Stored(@, n)] ELSE owner
/\ act' = [name |-> "Send", n |-> n, s |-> s, id |-> sends]
/\ UNCHANGED <<ran, missed>>
\* The queue delivers m to process p. The agent a that takes it stores the
\* input, then runs the turn or sends the message on to the session's owner.
Deliver(p, m) ==
/\ m \in queue
/\ act' = [name |-> "Deliver", p |-> p, id |-> m.id]
/\ IF ByName /\ m.n \notin Hosts[p]
THEN /\ missed' = missed \cup {m.n}
/\ UNCHANGED <<queue, owner, ran, sends>>
ELSE \E a \in (IF ByName THEN {m.n} ELSE Hosts[p]):
LET o == Stored(owner[m.s], m.n) IN
/\ owner' = [owner EXCEPT ![m.s] = o]
/\ IF CheckOwner /\ o # a
THEN /\ queue' = (queue \ {m}) \cup {[m EXCEPT !.n = o]}
/\ UNCHANGED ran
ELSE /\ queue' = queue \ {m}
/\ ran' = ran \cup {<<m.s, a>>}
/\ UNCHANGED <<missed, sends>>
Next ==
\/ \E n \in Names, s \in Sessions: Send(n, s)
\/ \E p \in Procs, m \in queue: Deliver(p, m)
Spec == Init /\ [][Next]_vars
\* The queue keeps delivering each message, and not only to processes
\* without its agent: one that has it gets it eventually.
FairSpec == Spec /\ SF_vars(\E p \in Procs, m \in queue: m.n \in Hosts[p] /\ Deliver(p, m))
-----------------------------------------------------------------------------
(* Properties *)
\* Every turn of a session runs on the agent the session belongs to.
RunsOnOwner == \A r \in ran: r[2] = owner[r[1]]
\* A delivery finds no agent only in a process that lacks that name.
MissOnlyWhereLacking == \A n \in missed: \E p \in Procs: n \notin Hosts[p]
\* A session's owner never changes once set.
OwnerKept == [][\A s \in Sessions: owner[s] # None => owner'[s] = owner[s]]_vars
\* Every queued message is eventually taken, so no prompt is lost, even when
\* a process lacks its agent.
Drains == <>[](queue = {})
=============================================================================
+11
View File
@@ -0,0 +1,11 @@
CONSTANTS
Names = {"S", "C"}
Sessions = {"s1", "s2"}
Procs = {"p1", "p2"}
P2Lacks = "C"
MaxSends = 3
RecordAtSend = TRUE
ByName = FALSE
CheckOwner = FALSE
SPECIFICATION Spec
INVARIANT RunsOnOwner
+11
View File
@@ -0,0 +1,11 @@
CONSTANTS
Names = {"S", "C"}
Sessions = {"s1", "s2"}
Procs = {"p1", "p2"}
P2Lacks = "C"
MaxSends = 3
RecordAtSend = FALSE
ByName = TRUE
CheckOwner = FALSE
SPECIFICATION Spec
INVARIANT RunsOnOwner
+133
View File
@@ -0,0 +1,133 @@
// One process of drive-agents.mjs, in a worker thread so it has a module
// instance and a globalThis of its own, as a process does: its agents, on
// one durability over the walk's world-local directory that makes each of
// them a World of its own, as the durability of agents created without one
// does. Every message its agents queue goes to the driver, which holds them
// all until a step delivers one here, to the queue handler of the World the
// driver picks.
import { parentPort, workerData } from "node:worker_threads";
import { createDurableAgentFactory, foldSessionLog } from "../../durable.js";
import { world } from "../../durable/world.mjs";
const { names, mode } = workerData;
const { createWorld } = await import(new URL("../../durable/node_modules/@workflow/world-local/dist/index.js", import.meta.url).href);
const encoder = new TextEncoder();
const inner = Symbol("handler");
// The turns this process ran since the driver last asked, as [sessionId, agent name].
let ran = [];
// A harness whose every turn is one step, recorded with the agent's name.
const createAgent = createDurableAgentFactory({
name: "createRoutingAgent",
defaultDurability: async () => { throw new Error("the driver passes a durability"); },
harness: ({ tag }) => ({
async open({ sessionId, store }) {
await store.load();
return {
prompt(_input, { turnId }) {
const result = (async () => {
await store.append({ idempotencyKey: `${turnId}:start`, data: encoder.encode(JSON.stringify({ turnId, start: true })), marks: [{ start: turnId }] });
ran.push([sessionId, tag]);
await store.append({ idempotencyKey: `${turnId}:end`, data: encoder.encode(JSON.stringify({ turnId, end: true })), marks: [{ end: true }] });
return { stopReason: "end_turn" };
})();
return { result, steer: async () => {}, cancel() {}, async *[Symbol.asyncIterator]() { await result; } };
},
resume() {
throw new Error("no turn of this harness is ever left open");
},
get openTurn() {
return null;
},
settled: async () => {},
saveCheckpoint: async () => {},
exportCheckpoint: async () => new Uint8Array(),
close: async () => {},
};
},
}),
});
// Calls to the driver, which answers each by its id.
const waiting = new Map();
let asked = 0;
const ask = (message) => new Promise((resolve) => {
asked += 1;
waiting.set(asked, resolve);
parentPort.postMessage({ ...message, id: asked });
});
let walk = null;
const methods = {
// A walk over `dir`: this process's agents, each listening.
async open({ dir }) {
const handlers = [];
// Each agent's World, which libfx starts and closes with the agent.
const createInstance = () => {
const index = handlers.push(null) - 1;
return {
...createWorld({ dataDir: dir, recoverActiveRuns: false }),
queue: (queueName, payload, { idempotencyKey } = {}) => ask({ type: "send", queueName, payload, idempotencyKey }),
createQueueHandler(_prefix, handler) {
const route = async () => new Response("this test World delivers in process", { status: 404 });
route[inner] = handler;
return route;
},
registerHandler(_prefix, route) {
handlers[index] = route[inner];
},
};
};
const durability = world(createInstance, { name: "routing", pollMs: 250, queueDurable: mode === "consume", runInRequest: false });
const agents = Object.fromEntries(names.map((name) => [name, createAgent({ durability, name, tag: name })]));
walk = { handlers, agents };
ran = [];
},
newSessionId: () => walk.agents[names[0]].newSessionId(),
// Opens every agent's backend, so each listens, as one does once it has
// served a request.
async listen({ sessionId }) {
for (const agent of Object.values(walk.agents)) await agent[Symbol.for("libfx.durableInternals")].openLog(sessionId);
},
async prompt({ agent, sessionId, input, messageId }) {
await walk.agents[agent].session(sessionId).prompt(input, { messageId }).accepted;
},
// Delivers a queued message to the World `world` of this process, by
// index; a name this process has no agent for is a miss.
async deliver({ world: index, payload, meta }) {
try {
await walk.handlers[index % walk.handlers.length](payload, meta);
return { delivered: true, ran: ran.splice(0) };
} catch (error) {
if (error?.code !== "FX_NO_AGENT") throw error;
return { delivered: false, missed: error.agent, ran: ran.splice(0) };
}
},
// The agent each session belongs to, as its log has it.
async owners({ sessionIds }) {
const internals = walk.agents[names[0]][Symbol.for("libfx.durableInternals")];
const owners = {};
for (const sessionId of sessionIds) owners[sessionId] = foldSessionLog(await (await internals.openLog(sessionId)).read()).agent ?? "none";
return owners;
},
async close() {
if (!walk) return;
for (const agent of Object.values(walk.agents)) await agent.close().catch(() => {});
walk = null;
},
};
parentPort.on("message", async (message) => {
if (message.type === "answer") {
waiting.get(message.id)?.(message.value);
waiting.delete(message.id);
return;
}
try {
parentPort.postMessage({ type: "result", id: message.id, value: await methods[message.method](message.args ?? {}) });
} catch (error) {
parentPort.postMessage({ type: "result", id: message.id, error: { message: String(error?.stack ?? error), code: error?.code } });
}
});
+40
View File
@@ -12,6 +12,7 @@
# bun drive.mjs send && bun drive.mjs lookup && bun drive.mjs write && bun drive.mjs cutoff
# bun drive.mjs leased # the Heartbeat graph, on a World without liveness
# bun drive.mjs queue # the QueueHeld graph, on a World whose queue shows it
# bun drive-agents.mjs send && bun drive-agents.mjs consume # AgentRouting
set -eu
cd "$(dirname "$0")"
@@ -62,4 +63,43 @@ for pair in UiInOrder:UiInOrder NoStaleShown:NoStaleShown NotTwoActive:NotTwoAct
echo "== witness $witness NOT reached" >&2; status=1
fi
done
# AgentRouting.tla: which agent runs a session's turns when an app has
# several. Its graphs are what drive-agents.mjs walks: inputs stored when
# sent (memory, local) and when delivered (Vercel), with process p2 lacking
# one agent. Drains is checked under fairness, a simulation runs a larger
# config, and each witness breaks RunsOnOwner: NoNames is routing without
# names, as before them, and NoOwnerCheck a worker that runs any session it
# is handed.
for config in send consume; do
echo "== AgentRouting ($config)"
if ! "$java" -XX:+UseParallelGC -cp "$jar" tlc2.TLC -deadlock -workers 1 -cleanup \
-metadir "$work/agents-$config" -dump dot,actionlabels "graphs/agents-$config" \
-config "AgentRouting-$config.cfg" AgentRouting.tla > "$work/agents-$config.log" 2>&1; then
cat "$work/agents-$config.log"; status=1; continue
fi
grep -E 'distinct states|depth of' "$work/agents-$config.log"
done
for config in drains-send drains-consume; do
if "$java" -cp "$jar" tlc2.TLC -deadlock -workers auto -cleanup -metadir "$work/agents-$config" \
-config "AgentRouting-$config.cfg" AgentRouting.tla > "$work/agents-$config.log" 2>&1; then
echo "== AgentRouting ($config): Drains holds"
else
cat "$work/agents-$config.log"; status=1
fi
done
if "$java" -XX:+UseParallelGC -cp "$jar" tlc2.TLC -simulate num=20000 -depth 50 -deadlock -workers auto -cleanup \
-metadir "$work/agents-sim" -config AgentRouting-sim.cfg AgentRouting.tla > "$work/agents-sim.log" 2>&1; then
echo "== AgentRouting (simulation): $(grep -E 'states checked' "$work/agents-sim.log" | tail -1)"
else
cat "$work/agents-sim.log"; status=1
fi
for witness in NoNames NoOwnerCheck; do
if "$java" -cp "$jar" tlc2.TLC -deadlock -workers auto -cleanup -metadir "$work/w-$witness" \
-config "Witness-$witness.cfg" AgentRouting.tla 2>&1 | grep -q -E "Invariant RunsOnOwner is violated"; then
echo "== witness $witness reached"
else
echo "== witness $witness NOT reached" >&2; status=1
fi
done
exit "$status"
+201
View File
@@ -0,0 +1,201 @@
// Drives real libfx durable agents along the TLC state graphs of
// AgentRouting.tla, and checks after every step that the code is where the
// model allows.
//
// bun sdk/tests/model/drive-agents.mjs [send|consume] [--walks N] [--depth N] [--seed S]
//
// Two processes, p1 and p2, are two worker threads (agent-process.mjs), each
// with a module instance and a globalThis of its own: p1 has the agents S
// and C, and p2 only S, all over one world-local directory, each agent on a
// World of its own, as agents created without a durability have. The
// queue they share is the driver's: it holds every message until the walk
// delivers it to a process, to one of its Worlds picked at random, as the
// graph's Send and Deliver steps say. With "send" a prompt's input is stored
// before it is queued, as on local(); with "consume" the worker that takes
// it stores it, as on a World whose queue keeps its messages (Vercel).
//
// After each step the driver reads back each session's owner from its log,
// the turns each agent ran, the names a process had no agent for, and the
// messages still queued, and moves to the successor with that state. A step
// with no such successor is a conformance failure. Each walk ends by draining
// the queue through p1, as the queue's fairness does in the model.
import { strict as assert } from "node:assert";
import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import { Worker } from "node:worker_threads";
import { loadGraph } from "./graph.mjs";
const args = process.argv.slice(2);
const option = (name, fallback) => {
const index = args.indexOf(`--${name}`);
return index >= 0 ? args[index + 1] : fallback;
};
const mode = args.find((arg) => arg === "send" || arg === "consume") ?? "send";
const walks = Number(option("walks", 40));
const depth = Number(option("depth", 24));
let seed = Number(option("seed", Date.now() % 100000));
const firstSeed = seed;
// A small deterministic generator, so a failing walk can be run again.
const random = () => {
seed = (seed * 1103515245 + 12345) % 2147483648;
return seed / 2147483648;
};
const pick = (items) => items[Math.floor(random() * items.length)];
const here = fileURLToPath(new URL(".", import.meta.url));
const graph = await loadGraph(resolve(here, "graphs", `agents-${mode}.dot`));
const hosts = { p1: ["S", "C"], p2: ["S"] };
// The queue the processes share: every message, until a step delivers it.
const held = [];
const keys = new Map();
let sentCount = 0;
function enqueue({ queueName, payload, idempotencyKey }) {
if (idempotencyKey !== undefined && keys.has(idempotencyKey)) return { messageId: keys.get(idempotencyKey) };
sentCount += 1;
const message = { id: `msg_${String(sentCount).padStart(6, "0")}`, queueName, payload, attempt: 0 };
if (idempotencyKey !== undefined) keys.set(idempotencyKey, message.id);
held.push(message);
return { messageId: message.id };
}
// A process: calls to its worker, which also sends to the queue.
function startProcess(name) {
const worker = new Worker(new URL("./agent-process.mjs", import.meta.url), { workerData: { names: hosts[name], mode } });
const calls = new Map();
let called = 0;
worker.on("message", (message) => {
if (message.type === "send") return worker.postMessage({ type: "answer", id: message.id, value: enqueue(message) });
const call = calls.get(message.id);
calls.delete(message.id);
if (message.error) call.reject(Object.assign(new Error(`${name}: ${message.error.message}`), { code: message.error.code }));
else call.resolve(message.value);
});
worker.on("error", (error) => { for (const call of calls.values()) call.reject(error); });
return {
call: (method, args) => new Promise((resolve, reject) => {
called += 1;
calls.set(called, { resolve, reject });
worker.postMessage({ method, args, id: called });
}),
stop: () => worker.terminate(),
};
}
const processes = Object.fromEntries(Object.keys(hosts).map((name) => [name, startProcess(name)]));
// The parts of a model state the driver observes, in one canonical form:
// records by key, and the sets queue, ran and missed in sorted order.
const canonical = (value) => {
if (Array.isArray(value)) return `<<${value.map(canonical).join(",")}>>`;
if (value && typeof value === "object") return `[${Object.keys(value).sort().map((key) => `${key}:${canonical(value[key])}`).join(",")}]`;
return JSON.stringify(value);
};
const set = (items) => `{${items.map(canonical).sort().join(",")}}`;
const viewOf = ({ owner, queue, ran, missed, sends }) => `owner ${canonical(owner)} queue ${set(queue)} ran ${set(ran)} missed ${set(missed)} sends ${sends}`;
async function openWalk() {
const dir = await mkdtemp(join(tmpdir(), "libfx-agents-"));
held.length = 0;
keys.clear();
for (const process of Object.values(processes)) await process.call("open", { dir });
const sessions = { s1: await processes.p1.call("newSessionId"), s2: await processes.p1.call("newSessionId") };
for (const process of Object.values(processes)) await process.call("listen", { sessionId: sessions.s1 });
const close = async () => {
for (const process of Object.values(processes)) await process.call("close");
await rm(dir, { recursive: true, force: true });
};
return { sessions, ran: new Set(), missed: new Set(), sends: 0, close };
}
// The driver's view of the code, in the model's terms.
async function observe(walk) {
const nameOf = new Map(Object.entries(walk.sessions).map(([name, id]) => [id, name]));
const owners = await processes.p1.call("owners", { sessionIds: Object.values(walk.sessions) });
const owner = Object.fromEntries(Object.entries(walk.sessions).map(([name, id]) => [name, owners[id]]));
const queue = held.map(({ payload }) => ({ id: Number(payload.input.messageId.slice(1)), s: nameOf.get(payload.input.sessionId), n: payload.input.agent }));
const ran = [...walk.ran].map((item) => JSON.parse(item)).map(([id, agent]) => [nameOf.get(id), agent]);
return { owner, queue, ran, missed: [...walk.missed], sends: walk.sends };
}
// Performs the step `act` names on the code.
async function perform(walk, act) {
if (act.name === "Send") {
await processes.p1.call("prompt", { agent: act.n, sessionId: walk.sessions[act.s], input: `prompt ${act.id}`, messageId: `m${act.id}` });
walk.sends += 1;
return;
}
const message = held.find(({ payload }) => payload.input.messageId === `m${act.id}`);
assert.ok(message, `message ${act.id} is queued`);
message.attempt += 1;
const outcome = await processes[act.p].call("deliver", {
world: Math.floor(random() * hosts[act.p].length),
payload: message.payload,
meta: { queueName: message.queueName, messageId: message.id, attempt: message.attempt },
});
for (const item of outcome.ran) walk.ran.add(JSON.stringify(item));
if (outcome.delivered) held.splice(held.indexOf(message), 1);
else walk.missed.add(outcome.missed);
}
const seenStates = new Set([graph.init]);
const seenEdges = new Set();
const outcomes = { send: 0, run: 0, forward: 0, miss: 0 };
let steps = 0;
// What a step from `from` to `to` did, as the model has it.
function outcomeOf(from, to) {
if (to.act.name === "Send") return "send";
if (to.queue.length === from.queue.length) return set(to.queue) === set(from.queue) ? "miss" : "forward";
return "run";
}
async function walkOnce(number) {
const walk = await openWalk();
let at = graph.init;
const path = [];
const step = async (choose) => {
// A self-loop is a step too: a second miss in p2 leaves the state as is.
const successors = [...(graph.next.get(at) ?? [])];
const chosen = choose(successors);
if (chosen === undefined) return false;
const act = graph.states.get(chosen).act;
path.push(JSON.stringify(act));
await perform(walk, act);
const seen = viewOf(await observe(walk));
const match = successors.find((id) => canonical(graph.states.get(id).act) === canonical(act) && viewOf(graph.states.get(id)) === seen);
if (match === undefined) {
const expected = successors.filter((id) => canonical(graph.states.get(id).act) === canonical(act)).map((id) => viewOf(graph.states.get(id)));
throw new Error(`walk ${number} (seed ${firstSeed}) left the model after ${path.join(" ")}\n code: ${seen}\n model: ${expected.join("\n ")}`);
}
seenEdges.add(`${at}>${match}`);
seenStates.add(match);
outcomes[outcomeOf(graph.states.get(at), graph.states.get(match))] += 1;
at = match;
steps += 1;
return true;
};
try {
for (let taken = 0; taken < depth; taken += 1) if (!(await step((successors) => (successors.length ? pick(successors) : undefined)))) break;
// The queue's fairness: every message reaches a process that has its
// agent, so the queue drains.
for (let left = 32; left > 0 && graph.states.get(at).queue.length > 0; left -= 1) {
await step((successors) => successors.find((id) => graph.states.get(id).act.name === "Deliver" && graph.states.get(id).act.p === "p1"));
}
assert.deepEqual(graph.states.get(at).queue, [], `walk ${number} drains its queue`);
assert.equal(held.length, 0, `walk ${number} leaves nothing queued in the code`);
} finally {
await walk.close();
}
}
try {
for (let number = 1; number <= walks; number += 1) await walkOnce(number);
} finally {
for (const process of Object.values(processes)) await process.stop();
}
const edges = [...graph.next.values()].reduce((total, successors) => total + successors.size, 0);
console.log(`${walks} walks (${mode}, seed ${firstSeed}), ${steps} steps (${Object.entries(outcomes).map(([name, count]) => `${count} ${name}`).join(", ")}): ${seenStates.size} of ${graph.states.size} states and ${seenEdges.size} of ${edges} edges, every step one the model allows`);
assert.ok(outcomes.forward > 0 && outcomes.miss > 0, "the walks forward a message and miss an agent at least once");
+1 -96
View File
@@ -50,6 +50,7 @@ import { fileURLToPath } from "node:url";
import { createFxAgent } from "../../node.js";
import { local } from "../../durable/local.mjs";
import { holdWrites } from "../hold-writes.mjs";
import { loadGraph } from "./graph.mjs";
const args = process.argv.slice(2);
const option = (name, fallback) => {
@@ -94,102 +95,6 @@ const addon = resolve(here, "../../../zig-out/lib/libfx.node");
const wasm = backend === "wasm" ? await readFile(resolve(here, "../../../zig-out/bin/fx-core.wasm")) : undefined;
const Workers = ["A", "B"];
// ---------------------------------------------------------------------------
// The state graph TLC wrote.
// A TLA+ value as TLC prints it: numbers, strings, booleans, records (and
// functions over names, which print as records), and sequences.
function parseValue(text) {
let at = 0;
const space = () => { while (/\s/.test(text[at] ?? "")) at += 1; };
const expect = (token) => {
space();
if (!text.startsWith(token, at)) throw new Error(`expected ${token} at ${at} in ${text}`);
at += token.length;
};
const value = () => {
space();
if (text.startsWith("<<", at)) {
at += 2;
const items = [];
space();
if (text.startsWith(">>", at)) { at += 2; return items; }
for (;;) {
items.push(value());
space();
if (text[at] === ",") { at += 1; continue; }
expect(">>");
return items;
}
}
if (text[at] === "[") {
at += 1;
const record = {};
for (;;) {
space();
const key = /^[A-Za-z_][A-Za-z0-9_]*/.exec(text.slice(at))?.[0];
if (!key) throw new Error(`expected a field at ${at} in ${text}`);
at += key.length;
expect("|->");
record[key] = value();
space();
if (text[at] === ",") { at += 1; continue; }
expect("]");
return record;
}
}
if (text[at] === "\"") {
const end = text.indexOf("\"", at + 1);
const string = text.slice(at + 1, end);
at = end + 1;
return string;
}
for (const [token, result] of [["TRUE", true], ["FALSE", false]]) {
if (text.startsWith(token, at)) { at += token.length; return result; }
}
const number = /^-?\d+/.exec(text.slice(at))?.[0];
if (number) { at += number.length; return Number(number); }
throw new Error(`cannot read a value at ${at} in ${text}`);
};
const result = value();
space();
if (at !== text.length) throw new Error(`trailing text at ${at} in ${text}`);
return result;
}
async function loadGraph(path) {
const dot = await readFile(path, "utf8").catch(() => {
throw new Error(`${path} is missing; run sdk/tests/model/check.sh first`);
});
const states = new Map();
const next = new Map();
let init = null;
const unescape = (label) => label.replace(/\\(.)/g, (_, char) => (char === "n" ? "\n" : char));
for (const line of dot.split("\n")) {
const edge = /^(-?\d+) -> (-?\d+) \[label="(\w+)"/.exec(line);
if (edge) {
const [, from, to] = edge;
if (!next.has(from)) next.set(from, new Set());
next.get(from).add(to);
continue;
}
const node = /^(-?\d+) \[label="((?:[^"\\]|\\.)*)"/.exec(line);
if (node) {
const [, id, label] = node;
const state = {};
// One entry per variable; TLC wraps a long value over several lines.
for (const entry of unescape(label).split(/\n(?=\/\\ )/)) {
const match = /^\/\\ (\w+) = ([\s\S]*)$/.exec(entry.trim());
if (match) state[match[1]] = parseValue(match[2].trim());
}
states.set(id, state);
if (state.act?.name === "Init") init = id;
}
}
assert.ok(init, "the graph has an initial state");
return { states, next, init };
}
// The epochs of the lines every reader shows, by the spec's ShownAt: each
// line no earlier line outranks.
const shownEpochs = (lines) => {
+99
View File
@@ -0,0 +1,99 @@
// The state graphs check.sh has TLC write, as the model drivers read them.
import { strict as assert } from "node:assert";
import { readFile } from "node:fs/promises";
// A TLA+ value as TLC prints it: numbers, strings, booleans, records (and
// functions over names, which print as records), sequences, and sets, read
// as arrays in TLC's order.
export function parseValue(text) {
let at = 0;
const space = () => { while (/\s/.test(text[at] ?? "")) at += 1; };
const expect = (token) => {
space();
if (!text.startsWith(token, at)) throw new Error(`expected ${token} at ${at} in ${text}`);
at += token.length;
};
const value = () => {
space();
for (const [open, close] of [["<<", ">>"], ["{", "}"]]) {
if (!text.startsWith(open, at)) continue;
at += open.length;
const items = [];
space();
if (text.startsWith(close, at)) { at += close.length; return items; }
for (;;) {
items.push(value());
space();
if (text[at] === ",") { at += 1; continue; }
expect(close);
return items;
}
}
if (text[at] === "[") {
at += 1;
const record = {};
for (;;) {
space();
const key = /^[A-Za-z_][A-Za-z0-9_]*/.exec(text.slice(at))?.[0];
if (!key) throw new Error(`expected a field at ${at} in ${text}`);
at += key.length;
expect("|->");
record[key] = value();
space();
if (text[at] === ",") { at += 1; continue; }
expect("]");
return record;
}
}
if (text[at] === "\"") {
const end = text.indexOf("\"", at + 1);
const string = text.slice(at + 1, end);
at = end + 1;
return string;
}
for (const [token, result] of [["TRUE", true], ["FALSE", false]]) {
if (text.startsWith(token, at)) { at += token.length; return result; }
}
const number = /^-?\d+/.exec(text.slice(at))?.[0];
if (number) { at += number.length; return Number(number); }
throw new Error(`cannot read a value at ${at} in ${text}`);
};
const result = value();
space();
if (at !== text.length) throw new Error(`trailing text at ${at} in ${text}`);
return result;
}
/** The graph check.sh wrote to `path`: each state by id, its successors, and the initial state. */
export async function loadGraph(path) {
const dot = await readFile(path, "utf8").catch(() => {
throw new Error(`${path} is missing; run sdk/tests/model/check.sh first`);
});
const states = new Map();
const next = new Map();
let init = null;
const unescape = (label) => label.replace(/\\(.)/g, (_, char) => (char === "n" ? "\n" : char));
for (const line of dot.split("\n")) {
const edge = /^(-?\d+) -> (-?\d+) \[label="(\w+)"/.exec(line);
if (edge) {
const [, from, to] = edge;
if (!next.has(from)) next.set(from, new Set());
next.get(from).add(to);
continue;
}
const node = /^(-?\d+) \[label="((?:[^"\\]|\\.)*)"/.exec(line);
if (node) {
const [, id, label] = node;
const state = {};
// One entry per variable; TLC wraps a long value over several lines.
for (const entry of unescape(label).split(/\n(?=\/\\ )/)) {
const match = /^\/\\ (\w+) = ([\s\S]*)$/.exec(entry.trim());
if (match) state[match[1]] = parseValue(match[2].trim());
}
states.set(id, state);
if (state.act?.name === "Init") init = id;
}
}
assert.ok(init, "the graph has an initial state");
return { states, next, init };
}
+159 -1
View File
@@ -8,7 +8,7 @@ import { strict as assert } from "node:assert";
import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { createDurableAgentFactory, foldSessionLog, memory } from "../durable.js";
import { createDurableAgentFactory, foldSessionLog, memory, wakeHandler } from "../durable.js";
import { world } from "../durable/world.mjs";
const durabilityKind = process.argv[2] || "memory";
@@ -218,6 +218,18 @@ function chain(before, next) {
const createAgent = createDurableAgentFactory({ harness: scriptedHarness(), defaultDurability: async () => memory(), name: "createScriptedAgent" });
// Agents that record which of them opened each session's engine, by `tag`.
const openedBy = [];
const createTaggedAgent = createDurableAgentFactory({
harness: ({ tag }) => {
const plugged = scriptedHarness()();
return { open: (args) => { openedBy.push([tag, args.sessionId]); return plugged.open(args); } };
},
defaultDurability: async () => memory(),
name: "createTaggedAgent",
});
const openersOf = (sessionId) => [...new Set(openedBy.filter(([, id]) => id === sessionId).map(([tag]) => tag))];
async function durabilityFor(options = {}) {
if (durabilityKind === "memory") return memory(options);
const dir = await mkdtemp(join(tmpdir(), "libfx-core-"));
@@ -325,6 +337,76 @@ test("two agents prompting one session run its turns one at a time, in one histo
await second.close();
});
test("agents with different names each run only their own sessions' turns", async () => {
const durability = await durabilityFor();
// Each agent's own prompts reach it directly, never through the other.
const forwarded = [];
const onEvent = (event) => { if (event.type === "session.forwarded") forwarded.push(event); };
const support = createTaggedAgent({ durability, name: "support", tag: "support", onEvent });
const coding = createTaggedAgent({ durability, name: "coding", tag: "coding", onEvent });
assert.equal(support.name, "support");
const ours = support.session();
const theirs = coding.session();
for (let round = 0; round < 4; round += 1) {
const [a, b] = await Promise.all([textOf(ours.prompt(`s${round}`)), textOf(theirs.prompt(`c${round}`))]);
assert.equal(a.text, `s${round}-0 s${round}-1 s${round}-2`);
assert.equal(b.text, `c${round}-0 c${round}-1 c${round}-2`);
}
assert.deepEqual(openersOf(ours.id), ["support"]);
assert.deepEqual(openersOf(theirs.id), ["coding"]);
assert.deepEqual(forwarded, [], "every delivery went to the agent its message names");
await support.close();
await coding.close();
});
test("a prompt to another agent's session runs on that session's agent", async () => {
const durability = await durabilityFor();
const events = [];
const support = createTaggedAgent({ durability, name: "support", tag: "support" });
const coding = createTaggedAgent({ durability, name: "coding", tag: "coding", onEvent: (event) => events.push(event) });
const session = support.session();
await textOf(session.prompt("first"));
const { text, result } = await textOf(coding.session(session.id).prompt("second"));
assert.equal(text, "second-0 second-1 second-2");
assert.equal(result.stopReason, "end_turn");
assert.deepEqual(openersOf(session.id), ["support"]);
assert.ok(events.some((event) => event.type === "session.forwarded" && event.sessionId === session.id && event.agent === "support"), "the sender says it sent the message on");
await support.close();
await coding.close();
});
if (durabilityKind === "local") {
// local() makes each agent a World of its own, which delivers only the
// messages it queued, as world-local does.
test("agents on one durability, each on a World of its own, still send each other's prompts on", async () => {
const durability = await durabilityFor();
const support = createTaggedAgent({ durability, name: "support", tag: "support" });
const coding = createTaggedAgent({ durability, name: "coding", tag: "coding" });
const ours = support.session();
const theirs = coding.session();
await Promise.all([textOf(ours.prompt("one")), textOf(theirs.prompt("two"))]);
const [a, b] = await Promise.all([textOf(coding.session(ours.id).prompt("three")), textOf(support.session(theirs.id).prompt("four"))]);
assert.equal(a.text, "three-0 three-1 three-2");
assert.equal(b.text, "four-0 four-1 four-2");
assert.deepEqual(openersOf(ours.id), ["support"]);
assert.deepEqual(openersOf(theirs.id), ["coding"]);
await support.close();
await coding.close();
});
}
test("an agent's name defaults to default, and a malformed one is refused", async () => {
const agent = createAgent({ durability: await durabilityFor() });
assert.equal(agent.name, "default");
await agent.close();
for (const name of ["", "two words", "-dash", "x".repeat(65), 7, null]) {
assert.throws(() => createAgent({ name }), /createScriptedAgent\(\) name must be 1 to 64 letters/, `refuses ${JSON.stringify(name)}`);
}
const named = createAgent({ name: "a.b_c-9" });
assert.equal(named.name, "a.b_c-9");
await named.close();
});
test("a retried message runs its turn once", async () => {
const agent = createAgent({ durability: await durabilityFor() });
const session = agent.session();
@@ -1261,6 +1343,82 @@ if (durabilityKind === "queue") {
});
}
if (durabilityKind === "memory") {
test("a session belongs to the agent its first input or its checkpoint names, and one from before names to default", () => {
const input = (messageId, agent) => ({ k: "input", key: `${messageId}:prompt`, type: "prompt", messageId, ...(agent ? { agent } : {}) });
const ownerOf = (...entries) => foldSessionLog(entries.map((entry, index) => ({ cursor: String(index + 1), entry }))).agent;
assert.equal(ownerOf(), null);
assert.equal(ownerOf(input("m1", "support"), input("m2", "coding")), "support");
assert.equal(ownerOf(input("m1"), input("m2", "coding")), "default");
assert.equal(ownerOf({ k: "checkpoint", through: null, data: "seed", agent: "coding" }, input("m1", "support")), "coding");
assert.equal(ownerOf({ k: "checkpoint", through: null, data: "seed" }, input("m1", "support")), "support");
assert.equal(ownerOf({ k: "checkpoint", through: "7", data: "saved" }), "default");
assert.equal(ownerOf({ k: "checkpoint", through: "7", data: "saved", agent: "support" }, input("m9", "coding")), "support");
});
}
if (durabilityKind === "queue") {
test("one route serves every agent in the process, each delivery to the agent its message names", async () => {
const { createWorld } = await import(new URL("../durable/node_modules/@workflow/world-local/dist/index.js", import.meta.url).href);
const dir = await mkdtemp(join(tmpdir(), "libfx-core-"));
dirs.push(dir);
const stored = createWorld({ dataDir: dir, recoverActiveRuns: false });
await stored.start();
// A World that delivers only over HTTP, to the route the test posts to.
const sent = [];
const instance = {
...stored,
registerHandler: undefined,
queue: async (queueName, payload) => { sent.push({ queueName, payload }); return { messageId: `msg_${sent.length}` }; },
createQueueHandler: (_prefix, handler) => async (request) => {
const { payload, meta } = await request.json();
try {
await handler(payload, meta);
return new Response("ok");
} catch (error) {
return Response.json({ code: error?.code, agent: error?.agent }, { status: 500 });
}
},
};
const durability = world(instance, { name: "route", queueDurable: true });
const support = createTaggedAgent({ durability, name: "support", tag: "support" });
const coding = createTaggedAgent({ durability, name: "coding", tag: "coding" });
let attempt = 0;
const post = (route, { queueName, payload }) => {
attempt += 1;
const meta = { queueName, messageId: `msg_post_${attempt}`, attempt: 1 };
return route(new Request("https://app.example/wake", { method: "POST", body: JSON.stringify({ payload, meta }) }));
};
const ours = support.session();
const theirs = coding.session();
const first = ours.prompt("alpha");
const second = theirs.prompt("beta");
await Promise.all([first.accepted, second.accepted]);
const shared = wakeHandler();
// Both messages reach the one route, whichever agent sent them.
for (const message of sent.splice(0)) assert.equal((await post(shared, message)).status, 200);
assert.equal((await textOf(first)).text, "alpha-0 alpha-1 alpha-2");
assert.equal((await textOf(second)).text, "beta-0 beta-1 beta-2");
// An agent's own route delivers to the agent a message names, too.
const third = ours.prompt("gamma");
await third.accepted;
const [message] = sent.splice(0);
assert.equal((await post(coding.wakeHandler(), message)).status, 200);
assert.equal((await textOf(third)).text, "gamma-0 gamma-1 gamma-2");
assert.deepEqual(openersOf(ours.id), ["support"]);
assert.deepEqual(openersOf(theirs.id), ["coding"]);
// A name no agent here has is refused for the queue to deliver again.
const ghost = { ...message, payload: { ...message.payload, input: { ...message.payload.input, agent: "ghost", messageId: "ghost-1" } } };
const refused = await post(shared, ghost);
assert.equal(refused.status, 500);
assert.deepEqual(await refused.json(), { code: "FX_NO_AGENT", agent: "ghost" });
await support.close();
await coding.close();
assert.equal((await post(shared, message)).status, 503, "with no agent left, the route says so");
await stored.close();
});
}
let failed = 0;
for (const [name, fn] of tests) {
const started = performance.now();
+9
View File
@@ -6,6 +6,7 @@ import {
FxFencedError,
FxJournalVersionError,
memory,
wakeHandler,
type Durability,
type FxAgent,
type FxAgentEvent,
@@ -327,6 +328,7 @@ export function events(): void {
if (event.type === "session.error") console.log(event.sessionId, event.error.message);
if (event.type === "session.deadline" || event.type === "session.fenced") assertType<Equal<typeof event.epoch, number>>();
if (event.type === "session.deadline") assertType<Equal<typeof event.cutoffs, number>>();
if (event.type === "session.forwarded") assertType<Equal<typeof event.agent, string>>();
if (event.type === "checkpoint.mismatch") console.log(event.changed.includes("model"), event.current.toolSchemaHash);
if (event.type === "transport.activity") console.log(event.attempt, event.chunkBytes, event.totalBytes);
if (event.type === "transport.response") console.log(event.status, event.requestId);
@@ -369,6 +371,13 @@ export async function GET(request: Request): Promise<Response> {
export const wake: (request: Request) => Promise<Response> = agent.wakeHandler();
// Several agents: each named, one route for all of them.
export const support: FxAgent = createFxAgent({ name: "support", instructions: "Answer billing questions." });
assertType<Equal<typeof support.name, string>>();
export const wakeAll: (request: Request) => Promise<Response> = wakeHandler();
// @ts-expect-error a name is text
createFxAgent({ name: 7 });
// A durable Codex agent: each process asks the host for the session's credentials.
export const codexAgent: FxAgent = createFxAgent({
provider: "codex",
+17 -1
View File
@@ -292,6 +292,8 @@ export type FxEngineEvent = FxEventBase & (
export type FxSessionEvent = FxEventBase & (
| { type: "session.error"; sessionId: string; error: FxErrorSummary }
| { type: "session.fenced"; sessionId: string; epoch: number }
/** A message for a session of the agent `agent` reached this one, which sent it on. */
| { type: "session.forwarded"; sessionId: string; agent: string }
/** `cutoffs`: the times in a row the deadline cut off the same step, this one included. */
| { type: "session.deadline"; sessionId: string; epoch: number; cutoffs: number }
| { type: "ui.error"; sessionId: string; error: string }
@@ -615,6 +617,12 @@ export type FxAgentOptions = Omit<FxSharedOptions<FxAgentEvent>, "tools"> & FxMo
durability?: Durability | undefined;
/** Checkpoint bytes the agent's new sessions start from. */
checkpoint?: FxBytes;
/**
* The agent's name, 1 to 64 letters, digits, `.`, `_` or `-`, starting with a letter or digit. Default `"default"`.
* A session's turns run only on agents with the name of the agent that created it, so an app can have several
* durable agents. Agents with the same name are copies of one agent and share its sessions' turns.
*/
name?: string;
};
/** Options for `agent.session()`. Each applies to the turns this session object starts. */
@@ -729,6 +737,8 @@ export interface FxSession {
/** An agent: the configuration its sessions run under. Create one per server and open a session per conversation. */
export interface FxAgent {
/** The agent's name, `"default"` unless `createFxAgent()` gave one. */
readonly name: string;
/** Opens the session `id`, or starts a new one without it. Any other value, `null` included, throws. */
session(id?: string, options?: FxSessionOptions): FxSession;
/** An id for a new session, before its first prompt. */
@@ -739,12 +749,18 @@ export interface FxAgent {
prompt(input: FxSessionPromptInput, options?: FxPromptOptions): FxTurn;
/** The history of the session `agent.prompt()` uses, as opaque bytes. */
checkpoint(): Promise<Uint8Array>;
/** The request handler a deployment mounts for queue deliveries. */
/** The request handler a deployment mounts for queue deliveries; it delivers to whichever agent in the process a message names. */
wakeHandler(): (request: Request) => Promise<Response>;
/** Waits for the turns this process is running, then lets their sessions go. */
close(): Promise<void>;
}
/**
* The request handler a deployment mounts for queue deliveries, for every durable agent in the process: each
* delivery goes to the agent its message names. One route and one queue trigger serve any number of agents.
*/
export declare function wakeHandler(): (request: Request) => Promise<Response>;
// ---------------------------------------------------------------------------
// Backend diagnostics (Node.js)
+1
View File
@@ -21,6 +21,7 @@ export {
listModels,
memory,
supportsJspi,
wakeHandler,
xtermAdapter,
} from "./libfx.cjs";