Merge remote-tracking branch 'origin/main' into fazxes/turn-end-saved

# Conflicts:
#	sdk/fx-harness.js
#	sdk/fx-sdk.js
#	sdk/tests/test-types.mjs
This commit is contained in:
Pranit committed 2026-10-18 02:43:39 -04:00
commit a9138f8fd9
80 files changed
+8059 -386

No files matched your search

+4
View File
@@ -390,6 +390,10 @@ jobs:
bun tests/tla/drivers/drive.mjs cutoff --walks 20 --seed 5
bun tests/tla/drivers/drive.mjs leased --walks 40 --seed 12
bun tests/tla/drivers/drive.mjs queue --walks 40 --seed 4
bun tests/tla/drivers/drive-intake.mjs --walks 60 --seed 1
bun tests/tla/drivers/drive-intake.mjs --walks 30 --seed 2 --durability world
bun tests/tla/drivers/drive-images.mjs --walks 40 --seed 1
bun tests/tla/drivers/drive-images.mjs --walks 20 --seed 2 --backend wasm
bun tests/tla/drivers/drive-agents.mjs --walks 60 --seed 11
bun tests/tla/drivers/drive-tenants.mjs --walks 60 --seed 11
bun tests/tla/drivers/drive-steer.mjs ideal --walks 20 --seed 2
+4 -1
View File
@@ -27,6 +27,8 @@ The SDK has two WebAssembly surfaces and one shared JavaScript host layer:
| A durability over a World the app supplies; `local()` and `vercel()` are built on it | `sdk/durable/world.mjs` |
| TypeScript declarations for every entry point | `sdk/types/`, checked by `sdk/tests/test-types.mjs` |
| Checkpoint format, including the libfx version, tool set, and model each checkpoint records | `src/core/agent/runtime/checkpoint.zig` |
| Images a session saves once, by name: naming in journal events and checkpoints, and the two-phase open | `src/core/session/session_codec.zig`, `src/acp/libfx_journal.zig`, and `src/acp/server.zig` |
| A session's image store (the internal `persistenceImages` key) and storing before sending | `sdk/fx-sdk.js`; a durable session's images are World streams in `sdk/durable.js` |
| Commands required by CI | `.github/workflows/ci.yml` |
Do not treat the demos or this file as the implementation contract. When prose and behavior disagree, establish the current behavior from code and focused tests before changing either.
@@ -52,6 +54,7 @@ Do not treat the demos or this file as the implementation contract. When prose a
| Public export, option, event, or result shape | Update `sdk/types/` and run `node sdk/tests/test-types.mjs` after `npm ci --prefix sdk/node` |
| `createFxAgent()`, ACP translation, core session persistence, streaming, or cancellation | Core build, core Node tests, and the browser test when browser behavior is involved |
| Live Gateway request or model-catalog translation | Core tests plus the opt-in live smoke test when a credential is available |
| How a session saves images: naming, the image store, staging at open | Core and terminal builds, `bun sdk/tests/test-journal.mjs native wasm` and `wasm native`, `test-checkpoint.mjs` both ways, the Zig tests of `checkpoint.zig` and `journal.zig`, then `tests/tla/sdk/check.sh` and `bun tests/tla/drivers/drive-images.mjs` with `--backend native` and `wasm`, which walks the graph of `ImageStore.tla`; change the spec with the code. For durable sessions, also the durable sessions row |
| Terminal adapter, input encoding, resize, cleanup, config, or prompt history | Terminal build and the headless terminal suite |
| Terminal session persistence or browser device login | Terminal build plus `sdk/tests/test-term-session-resume.mjs` or `sdk/tests/test-term-login.mjs` |
| Browser workspace metadata, permissions, execution, limits, or cancellation | Terminal build plus `sdk/node/test-term-workspace.mjs` |
@@ -59,7 +62,7 @@ Do not treat the demos or this file as the implementation contract. When prose a
| `encodeXtermKeyEvent()` or `xtermAdapter()` only | `sdk/node/test-xterm-adapter.mjs` |
| Core debugger query behavior or automation state | `sdk/tests/test-core-browser.mjs` |
| Terminal demo asset references, integrity, or cache policy | Package into a fresh temporary directory and inspect the generated HTML, manifest, and headers |
| Durable sessions: leases, fencing, takeovers, tool reruns, the UI stream | `sdk/tests/test-durable-core.mjs` on memory, local, world, and queue (the core with a scripted harness; world is an app-built World passed to `world()`, queue one whose queue delivers as Vercel Queues does, with `queueLeases`), `sdk/tests/test-durable.mjs` on memory, local, and world, then `tests/tla/sdk/check.sh` and `bun tests/tla/drivers/drive.mjs send`, `lookup`, `write`, `cutoff` (deadline cut-offs up to their bound), `leased` (a World without liveness, where each worker renews a short lease), and `queue` (a World whose queue shows whether a worker lives, so leases are held without renewals), which walk the real code along the TLC state graphs of `DurableSession.tla`; change the spec with the code. For agent names and routing, also `bun tests/tla/drivers/drive-agents.mjs`, which walks `AgentRouting.tla`'s graph with each process in a worker thread of its own. For what a delivery may run, on whose agent and store, and input limits, also `bun tests/tla/drivers/drive-tenants.mjs`, which walks `TenantIsolation.tla`'s graph with four tenants and a forger sharing one route. With a Vercel access token, also run both against Vercel's World: `test-durable.mjs vercel native` and `drive.mjs --durability vercel`, configured as `sdk/tests/vercel-storage.mjs` describes |
| Durable sessions: leases, fencing, takeovers, tool reruns, the UI stream | `sdk/tests/test-durable-core.mjs` on memory, local, world, and queue (the core with a scripted harness; world is an app-built World passed to `world()`, queue one whose queue delivers as Vercel Queues does, with `queueLeases`), `sdk/tests/test-durable.mjs` on memory, local, and world, then `tests/tla/sdk/check.sh` and `bun tests/tla/drivers/drive.mjs send`, `lookup`, `write`, `cutoff` (deadline cut-offs up to their bound), `leased` (a World without liveness, where each worker renews a short lease), and `queue` (a World whose queue shows whether a worker lives, so leases are held without renewals), which walk the real code along the TLC state graphs of `DurableSession.tla`; change the spec with the code. For how a session object takes prompts in (images and their image writes, cancels, and closing the agent), also run `bun tests/tla/drivers/drive-intake.mjs` on memory and `--durability world`, which walks the graph of `PromptIntake.tla`. For agent names and routing, also `bun tests/tla/drivers/drive-agents.mjs`, which walks `AgentRouting.tla`'s graph with each process in a worker thread of its own. For what a delivery may run, on whose agent and store, and input limits, also `bun tests/tla/drivers/drive-tenants.mjs`, which walks `TenantIsolation.tla`'s graph with four tenants and a forger sharing one route. With a Vercel access token, also run both against Vercel's World: `test-durable.mjs vercel native` and `drive.mjs --durability vercel`, configured as `sdk/tests/vercel-storage.mjs` describes |
| Steering: interrupts, waits during tools, joins, hand-backs on cancel, image handoffs, steer ids | The libfx steering tests in `src/acp/libfx_steering.zig`, `sdk/tests/test-agent-steering.mjs` and `sdk/tests/test-durable.mjs` on native and wasm, then `tests/tla/sdk/check.sh` and `bun tests/tla/drivers/drive-steer.mjs ideal` and `faults`, with `--backend wasm` too, which walk the real code along the TLC state graphs of `DurableSteer.tla`; change the spec with the code |
| Supported public behavior or setup | Update `sdk/README.md` in the same change |
+3 -2
View File
@@ -55,9 +55,10 @@ agent.session(sessionId).prompt("Start with the schema.");
- **One turn at a time.** A prompt sent while a turn runs waits behind it, even when it reaches a different server.
- **Turns finish on their own.** A turn runs to the end whether or not anyone reads its events, and a turn nobody reads holds no connection open: its view starts reading only once you use its events, `readable` or `result`. Stop it with `session.cancel()`, or with the `signal` you passed to `prompt()`.
- **Steering.** `session.steer(input)` steers the running turn the way the fx CLI does. While the model streams a reply, the steer cuts it short and the turn continues with it. While a tool runs, the steer waits for the tool. A steer sent while a turn waits to start joins that turn, and when no turn runs or waits, the steer becomes the next turn. A steer with images can't join a turn, so the running turn ends at its next safe point and the steer runs next.
- **Steering.** `session.steer(input)` steers the running turn the way the fx CLI does. While the model streams a reply, the steer cuts it short and the turn continues with it. While a tool runs, the steer waits for the tool. A steer sent while a turn waits to start joins that turn, and when no turn runs or waits, the steer becomes the next turn. A steer with images can't join a turn, so the running turn ends at its next safe point and the steer runs next. Its images are saved once with the session before the steer is sent, as a prompt's are, and a steer whose image can't be saved throws the store's error.
- **Retries are safe.** Give a prompt or a steer a `messageId`, and a repeated request with the same id runs nothing new. A repeated prompt follows the original turn, and its result has `repeated: true`. One id names one input, so a steer that reuses a prompt's id is rejected with `FX_MESSAGE_ID_IN_USE`. A steer's `user_message` event carries its id as `id` where it joined a turn.
- **Inputs have a size limit.** A prompt or steer whose input, with the session's `context` and settings, comes to more than 8 MiB of JSON is rejected with `FX_INPUT_TOO_LARGE` before anything is stored. The model can't take a larger one.
- **A refused prompt is never stored.** `prompt()` checks a prompt as the engine's `prompt()` does and throws a `TypeError` or `RangeError` for one libfx would refuse, such as an empty prompt, base64 that isn't canonical, an image whose bytes aren't its type, or more than 8 images. A prompt whose `signal` aborts before the prompt is stored isn't stored either, and its turn ends `cancelled`.
Each conversation can bring its own settings, which apply to the turns that session object starts:
@@ -407,7 +408,7 @@ agent.session().prompt([
]);
```
A session stores each prompt as JSON before it runs, so a session prompt takes image data as a base64 string. `createFxEngine()` also takes bytes, a `Blob` or a `File`. Each prompt takes up to 8 PNG, JPEG, GIF or WebP images, at most 3.75 MiB each and 6 MiB together. Pass `resizeImage` to downscale or convert prompt images before they're sent.
A session stores each prompt as JSON before it runs, so a session prompt takes image data as a base64 string. `createFxEngine()` also takes bytes, a `Blob` or a `File`. A session saves each image once, named by its SHA-256, before it stores the prompt, and the saved conversation refers to it by that name, so images don't count toward the 4 MiB the session's own checkpoints hold. Images a host tool returns are saved the same way; one that can't be saved stays in the tool's result, held inline. A session keeps its newest 4,096 images, up to 512 MiB of them; an older image leaves the conversation, and the model sees a note in its place. A checkpoint you take with `checkpoint()` carries the images' bytes, within its 4 MiB. An engine with a `persistence` store of its own keeps images inside its records and checkpoints, so they count toward that 4 MiB; for a conversation with many images, use a session with [durability](#durability). libfx never deletes a saved image; it lasts as long as the session's storage. Each prompt takes up to 8 PNG, JPEG, GIF or WebP images, at most 3.75 MiB each and 6 MiB together. Pass `resizeImage` to downscale or convert prompt images before they're sent.
## Diagnostics
+328 -35
View File
@@ -75,14 +75,129 @@ const recentIdsKept = 256;
// The longest a message waits for a live holder before it is delivered again.
const maxBackstopSeconds = 300;
// Base64 of a Uint8Array: the runtime's own encoder where it has one, and
// otherwise four letters from the table for each three bytes. Both are fast
// enough for a large image; btoa over String.fromCharCode is not.
const base64Letters = encoder.encode("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/");
const toBase64 = (bytes) => {
let text = "";
for (let index = 0; index < bytes.length; index += 0x8000) {
text += String.fromCharCode(...bytes.subarray(index, index + 0x8000));
if (typeof bytes.toBase64 === "function") return bytes.toBase64();
const letters = new Uint8Array(Math.ceil(bytes.length / 3) * 4);
const whole = bytes.length - (bytes.length % 3);
let at = 0;
for (let index = 0; index < whole; index += 3) {
const group = (bytes[index] << 16) | (bytes[index + 1] << 8) | bytes[index + 2];
letters[at] = base64Letters[group >> 18];
letters[at + 1] = base64Letters[(group >> 12) & 63];
letters[at + 2] = base64Letters[(group >> 6) & 63];
letters[at + 3] = base64Letters[group & 63];
at += 4;
}
return btoa(text);
if (whole < bytes.length) {
// One or two bytes left: two or three letters, then "=" padding.
const two = whole + 1 < bytes.length;
const group = (bytes[whole] << 16) | (two ? bytes[whole + 1] << 8 : 0);
letters[at] = base64Letters[group >> 18];
letters[at + 1] = base64Letters[(group >> 12) & 63];
letters[at + 2] = two ? base64Letters[(group >> 6) & 63] : 61;
letters[at + 3] = 61;
}
return decoder.decode(letters);
};
const fromBase64 = (text) => Uint8Array.from(atob(text), (char) => char.charCodeAt(0));
const fromBase64 = (text) => {
if (typeof Uint8Array.fromBase64 === "function") return Uint8Array.fromBase64(text);
const binary = atob(text);
const bytes = new Uint8Array(binary.length);
for (let index = 0; index < binary.length; index += 1) bytes[index] = binary.charCodeAt(index);
return bytes;
};
// A session keeps each prompt image once, in its image store, named by the
// lowercase hex SHA-256 of its bytes, as libfx's core names it; the log only
// names it.
async function imageName(bytes) {
const digest = new Uint8Array(await globalThis.crypto.subtle.digest("SHA-256", bytes));
let name = "";
for (const byte of digest) name += byte.toString(16).padStart(2, "0");
return name;
}
const base64url = (text) => toBase64(encoder.encode(text)).replace(/\+/g, "-").replace(/\//g, "_").replace(/=+$/, "");
// The key a harness's store gives its image store under, `{ put, get }`:
// `persistenceImages` in fx-sdk.js, which engines read it from.
const persistenceImages = Symbol.for("libfx.persistenceImages");
// A view of the bytes in an ArrayBuffer or typed array, or null.
const byteView = (value) => (value instanceof ArrayBuffer ? new Uint8Array(value)
: ArrayBuffer.isView(value) ? new Uint8Array(value.buffer, value.byteOffset, value.byteLength) : null);
// A prompt's input as the queue and the log hold it: JSON, so image data is
// base64 text. With `named`, the session stores each image in its image
// store and the log names it (`storePromptImages`); `images` says whether any
// image has data to store.
//
// Each image block is copied once and only the copy is read, since an
// accessor can return something else the next time it is read, and the log
// must hold what was checked. The blocks are walked by index for the same
// reason: an Array subclass can make map() return anything. Pure.
function promptInputForLog(input, { named = false } = {}) {
let images = false;
const blocks = [];
for (let index = 0; index < input.length; index += 1) {
const block = input[index];
if (block?.type !== "image") {
blocks.push(block);
continue;
}
const image = { ...block };
if (image.data !== undefined && typeof image.data !== "string") throw new TypeError("a durable prompt carries image data as a base64 string");
if (typeof image.data === "string") images = named;
blocks.push(image);
}
return { input: blocks, images };
}
// Stores each image a prompt's input holds in the session's image store,
// before the input that names it is stored, and returns the input naming
// them: `{ type: "image", mimeType, sourceRef?, data: { sha256 } }`. An older
// libfx refuses data it can't read, so a worker that predates names fails
// the turn rather than run it without its images. A store that fails rejects
// with its own error. With `images` null it only names them, for a prompt the
// log already holds.
async function storePromptImages(images, input) {
const blocks = [];
for (const block of input) {
if (block?.type !== "image" || block.data === undefined) {
blocks.push(block);
continue;
}
const { data, ...rest } = block;
const bytes = fromBase64(data);
const sha256 = await imageName(bytes);
if (images) await images.put(sha256, bytes);
blocks.push({ ...rest, data: { sha256 } });
}
return blocks;
}
// A prompt's input with each named image read back from the session's image
// store, as base64, which is how a harness has always received image data.
// `missing` names the first image the store does not hold.
async function resolvePromptImages(images, input) {
if (!Array.isArray(input)) return { input };
const blocks = [];
for (const block of input) {
const name = block?.type === "image" ? block.data?.sha256 : undefined;
if (typeof name !== "string") {
blocks.push(block);
continue;
}
const bytes = await images.get(name);
if (!bytes) return { missing: name };
// Bytes that aren't the ones named are not the image its caller sent.
if (await imageName(bytes) !== name) return { missing: name, corrupt: true };
blocks.push({ ...block, data: toBase64(bytes) });
}
return { input: blocks };
}
/**
* The durability that keeps sessions in this process only. Agents given the
@@ -451,7 +566,7 @@ function createMemoryBackend({ maxDurationMs, reserveMs } = {}) {
const sessionFor = (id) => {
let session = sessions.get(id);
if (!session) {
session = { entries: [], head: null, ui: [], uiWaiters: new Set(), watchers: new Set() };
session = { entries: [], head: null, ui: [], uiWaiters: new Set(), watchers: new Set(), images: new Map() };
sessions.set(id, session);
}
return session;
@@ -530,6 +645,14 @@ function createMemoryBackend({ maxDurationMs, reserveMs } = {}) {
session.watchers.add(fn);
return () => session.watchers.delete(fn);
},
images: {
async put(name, bytes) {
if (!session.images.has(name)) session.images.set(name, bytes.slice());
},
async get(name) {
return session.images.get(name)?.slice();
},
},
ui: {
async write(lines) {
session.ui.push(...lines);
@@ -580,6 +703,9 @@ const logStep = "fx.log";
// the run's id, then `_user_` and the base64url of the namespace
// "libfx-ui".
const uiStreamOf = (runId) => `${runId.replace(/^wrun_/, "strm_")}_user_bGliZngtdWk`;
// The stream that holds an image, named as the UI stream is, with the
// namespace "libfx-image-" and the image's name.
const imageStreamOf = (runId, name) => `${runId.replace(/^wrun_/, "strm_")}_user_${base64url(`libfx-image-${name}`)}`;
const queuePrefix = "__libfx_wkf_workflow_";
// The name of an agent created without one, and of whatever a message or a
// session from before names belonged to.
@@ -867,6 +993,8 @@ function createWorldBackend(world, durability) {
async session(runId) {
// The newest slot this worker has seen.
let known = 0;
// The images this worker knows the run's image store holds.
const storedImages = new Set();
// Where the World has the in-band writer fence, chained entries are
// written in-band, each expecting the World's count of in-band writes
// at the entry it continues, so the World refuses one that continues a
@@ -1047,6 +1175,44 @@ function createWorldBackend(world, durability) {
set.add(fn);
return () => set.delete(fn);
},
// Each image is a stream of the run, named after it, so it is read by
// name wherever checkpoints fall. Two writers of one image each add
// its bytes once; a read takes the first chunk that matches the name.
images: {
async put(name, bytes) {
if (storedImages.has(name)) return;
// Refused here, before the prompt is stored, rather than by every
// worker that reads the image back.
if (typeof world.streams.getChunks !== "function") throw new TypeError("a World needs streams.getChunks() to keep a session's images");
await ensureRun();
const stream = imageStreamOf(runId, name);
let present = false;
try {
present = Number((await world.streams.getInfo(runId, stream))?.tailIndex ?? -1) >= 0;
} catch (error) {
if (!missingRun(error)) throw error;
}
if (!present) await world.streams.write(runId, stream, bytes);
storedImages.add(name);
},
async get(name) {
let page;
try {
page = await world.streams.getChunks(runId, imageStreamOf(runId, name), { limit: 8 });
} catch (error) {
if (missingRun(error)) return undefined;
throw error;
}
for (const chunk of page?.data ?? []) {
const bytes = byteView(chunk?.data);
if (bytes && bytes.byteLength > 0 && await imageName(bytes) === name) {
storedImages.add(name);
return bytes.slice();
}
}
return undefined;
},
},
ui: {
async write(lines) {
await ensureRun();
@@ -1589,6 +1755,8 @@ class SessionWorker {
const ui = uiWriter(log, (error) => agent.emit("ui.error", { sessionId: this.sessionId, error: error?.name ?? "Error" }), lease.epoch);
// When the lease runs out, as this worker last renewed it.
let leaseExpiresAt = lease.expiresAt;
// An image read that threw while the engine opened.
let imageReadFailed = null;
const store = {
load: async () => ({
...(state.checkpoint?.data ? { checkpoint: { through: state.checkpoint.through ?? "0", data: fromBase64(state.checkpoint.data) } } : {}),
@@ -1597,6 +1765,20 @@ class SessionWorker {
head: head ?? state.head,
}),
append: ({ idempotencyKey, data, marks }) => chained({ k: "record", key: idempotencyKey, data: toBase64(data), marks }).then((cursor) => ({ cursor })),
// The engine stores each prompt image before it sends it; a prompt
// from the log names images already here.
[persistenceImages]: {
put: (name, bytes) => log.images.put(name, bytes),
get: async (name) => {
try {
return await log.images.get(name);
} catch (error) {
// The store failed to answer, which says nothing of the image.
imageReadFailed = error;
throw error;
}
},
},
saveCheckpoint: async ({ through, data }) => {
// What a fold that starts at this checkpoint would otherwise miss:
// inputs before it that no turn took or that the open turn holds
@@ -1686,6 +1868,10 @@ class SessionWorker {
harnessRun = runOf(state.openTurn ?? state.pending[0]);
harness = await agent.openHarness(this.sessionId, store, harnessRun, ready);
} catch (error) {
// The image store failed to answer while the engine read the
// session's images: as when the log can't be read, the delivery
// fails and its message runs again; nothing failed the turn.
if (imageReadFailed) throw imageReadFailed;
// A harness that cannot start fails the turn waiting on it, once:
// the open turn, or after it failed, the next prompt.
const waiting = state.openTurn && !state.openFailed ? state.openTurn.id : state.pending[0]?.messageId;
@@ -1763,8 +1949,21 @@ class SessionWorker {
harness = await agent.openHarness(this.sessionId, store, wanted, ready);
harnessRun = wanted;
}
// A prompt names the images it stored; the engine gets their bytes.
const withImages = await resolvePromptImages(log.images, next.input);
if (withImages.missing) {
// A history without the image is not the one its caller sent.
await log.append({ k: "ended", messageId });
const message = withImages.corrupt
? `the session's image store holds other bytes under image ${withImages.missing}`
: `the session's image store does not hold image ${withImages.missing}`;
ui.push({ type: "turn_end", messageId, stopReason: "error", usage: {}, error: { name: "Error", message } });
await ui.settle();
state = this.fold(await log.read());
continue;
}
try {
turn = harness.prompt(next.input ?? "", { turnId: messageId, yieldAt });
turn = harness.prompt(withImages.input ?? "", { turnId: messageId, yieldAt });
} catch (error) {
await log.append({ k: "stopped", messageId });
return unusable(error);
@@ -2227,6 +2426,9 @@ function turnView({ messageId, resumeRequest = null, start }) {
* sets over the agent's options for the turns it starts, as JSON the core
* stores with each prompt and passes to `open()`, or undefined for none.
* It throws a TypeError for options it refuses.
* - `checkPrompt(input)` (optional): throws a TypeError or RangeError for
* a prompt input the harness would refuse. A session's `prompt()` calls it
* before it stores anything, so a prompt it refuses is never stored.
*
* A harness session has `prompt(input, { turnId, yieldAt })` and
* `resume({ yieldAt, rerun })`, each returning a turn, where `rerun: false`
@@ -2244,7 +2446,8 @@ function turnView({ messageId, resumeRequest = null, start }) {
* end. Either way the steer runs as the next turn. A turn stops before a
* model request once `yieldAt` has passed, with the stop reason `yielded`.
*
* An input the harness refuses is a turn that ends with an error.
* An input the harness refuses only once a turn runs it is a turn that ends
* with an error.
* `prompt()` and `resume()` throw only when the harness cannot run a turn,
* and a turn whose harness stopped under it rejects with a
* `FX_HARNESS_STOPPED` code; either way the same message runs the session
@@ -2442,7 +2645,16 @@ export function createDurableAgentFactory({ harness, defaultDurability, defaultS
await created.queue.send(queued, { idempotencyKey });
seeded = true;
};
const startTurn = (type, messageId, input, retried = false, aborted = false) => {
// Settles once the latest prompt from this session object with images
// to store is stored or has failed; null while none has. Each later
// prompt waits for it, so they are stored in the order made.
let storing = null;
// `ready`, when given, resolves the input once it can be stored; only a
// prompt that is stored calls it.
// `aborted()` says whether the prompt's signal has aborted. One that
// aborted before the prompt's send began never stores it; `progress.sent`
// is set as the send begins, after which an abort cancels the prompt.
const startTurn = (type, messageId, input, retried = false, aborted = () => false, ready = null, progress = null) => {
let accept;
const accepted = new Promise((resolve, reject) => { accept = { resolve, reject }; });
void accepted.catch(() => {});
@@ -2451,20 +2663,27 @@ export function createDurableAgentFactory({ harness, defaultDurability, defaultS
resumeRequest: type === "resume" ? messageId : null,
start: async () => {
try {
const read = input === undefined || ready === null || aborted() ? input : await ready();
const message = { type, messageId, ...(input === undefined ? {} : { input: read }) };
const queue = () => {
if (progress) progress.sent = true;
return send(message, { inRequest: type === "prompt" });
};
const { created, id: sessionId } = await resolved;
const log = await created.session(sessionId);
let attaching = type === "resume";
let from = null;
if (retried && !aborted) {
// The viewer's start is read before the message is queued, so
// every line this message's own turn writes comes after it,
// and an end for this id before it can only be from a turn
// this id already ran. The log, for that answer, is read
// alongside the one queue send.
from = await log.ui.length();
// The viewer's start is read before the message is queued, so
// every line this message's own turn writes comes after it, and
// an end for this id before it can only be from a turn this id
// already ran.
if (retried) from = await log.ui.length();
if (retried && !aborted()) {
// The log, for that answer, is read alongside the one queue
// send.
const [state] = await Promise.all([
log.read().then(foldSessionLog),
send({ type, messageId, ...(input === undefined ? {} : { input }) }, { inRequest: type === "prompt" }).then(() => accept.resolve({ messageId, sessionId })),
queue().then(() => accept.resolve({ messageId, sessionId })),
]);
// One the log does not know yet is this send's own. When the
// log says it ran and nothing shows how, either this send's
@@ -2477,9 +2696,8 @@ export function createDurableAgentFactory({ harness, defaultDurability, defaultS
}
if (retried) {
// A turn this id already ran answers from the log, without
// queueing it again. The viewer's start is read first, so a
// queueing it again. The viewer's start was read first, so a
// turn that ends after it has its end in view.
from = await log.ui.length();
const outcome = await retryOutcome(log, foldSessionLog(await log.read()), messageId, from, { sent: false });
if (outcome.follow) attaching = true;
else if (!outcome.fresh) {
@@ -2488,15 +2706,17 @@ export function createDurableAgentFactory({ harness, defaultDurability, defaultS
return { log, from, attaching: true, endWithinMs: retryEndWaitMs };
}
}
if (aborted && !attaching) {
// A new prompt whose signal had aborted is never stored.
accept.resolve({ messageId, sessionId });
return { settled: { messageId, stopReason: "cancelled", usage: {} } };
}
// The viewer starts where the stream is now, before the
// message can produce anything.
from ??= await log.ui.length();
await send({ type, messageId, ...(input === undefined ? {} : { input }) }, { inRequest: type === "prompt" });
if (aborted()) {
// A prompt whose signal aborted before its send began is never
// stored; a retry of one the log holds only follows its turn.
accept.resolve({ messageId, sessionId });
if (!attaching) return { settled: { messageId, stopReason: "cancelled", usage: {} } };
return { log, from, attaching };
}
await queue();
accept.resolve({ messageId, sessionId });
return { log, from, attaching };
} catch (error) {
@@ -2513,25 +2733,83 @@ export function createDurableAgentFactory({ harness, defaultDurability, defaultS
* Queues `input` as the session's next turn and returns a view of it.
* With `messageId`, a retried call is the same turn: it continues a
* turn still running and answers one that ended, without running it
* again.
* again. Each image is stored once in the session's image store and
* the log names it. A later prompt from this session object waits for
* an earlier one's images. An input the harness would refuse throws
* here and is never stored. A prompt whose signal aborts before it is
* stored is never stored either: it ends cancelled.
*/
prompt(input, promptOptions = {}) {
if (closed) throw new Error(`${label} is closed`);
const messageId = promptOptions.messageId ?? newId("msg");
if (!validId(messageId)) throw new TypeError(`messageId must be ${idRule}`);
if (typeof input !== "string" && !Array.isArray(input)) throw new TypeError("prompt input must be a string or an array of content blocks");
// The input travels through the queue and the log as JSON.
if (Array.isArray(input) && input.some((block) => block?.type === "image" && block.data !== undefined && typeof block.data !== "string")) {
throw new TypeError("a durable prompt carries image data as a base64 string");
}
admit(input);
const signal = promptOptions.signal;
if (signal !== undefined && typeof signal?.addEventListener !== "function") throw new TypeError("prompt signal must be an AbortSignal");
const turn = startTurn("prompt", messageId, input, promptOptions.messageId !== undefined, signal?.aborted === true);
const aborted = signal?.aborted === true;
const retried = promptOptions.messageId !== undefined;
// The input travels through the queue and the log as JSON. Its
// images go to the session's image store first and the log names
// them; a prompt aborted before the call stores nothing, so it
// keeps them as base64.
const named = !aborted;
const stored = Array.isArray(input) ? promptInputForLog(input, { named }) : { input, images: false };
// The copy the log will hold is what is checked: a getter can't
// show the check one value and the log another.
admit(stored.input);
plugged.checkPrompt?.(stored.input);
const before = storing;
const ready = before || stored.images ? async () => {
// The prompt waits for an earlier one and its image writes only
// until its signal aborts; what it waited for may still land, but
// the prompt is not stored.
let release = () => {};
const abortion = signal === undefined ? null : new Promise((resolve) => {
signal.addEventListener("abort", resolve, { once: true });
release = () => signal.removeEventListener("abort", resolve);
});
const unlessAborted = (promise) => {
if (abortion === null) return promise;
void promise.catch(() => {});
return Promise.race([promise, abortion]);
};
try {
await unlessAborted(before ?? Promise.resolve());
if (signal?.aborted) return null;
let read = stored.input;
if (stored.images) {
const { created, id: sessionId } = await resolved;
const log = await created.session(sessionId);
// A retry of a prompt the log already holds is answered from
// the log: its images are named, not written again.
const known = retried && knownMessage(foldSessionLog(await log.read()), { type: "prompt", messageId });
if (signal?.aborted) return null;
read = await unlessAborted(storePromptImages(known ? null : log.images, read));
if (signal?.aborted) return null;
}
// An agent closed meanwhile stores nothing more.
if (closed) throw new Error(`${label} is closed`);
return read;
} finally {
release();
}
} : null;
const progress = { sent: false };
const turn = startTurn("prompt", messageId, stored.input, retried, () => signal?.aborted === true, ready, progress);
if (stored.images && !aborted) {
// A later prompt waits for this one and every one before it: one
// withdrawn while it waited settles before them.
const settled = Promise.all([before, turn.accepted.then(() => {}, () => {})]).then(() => {});
storing = settled;
void settled.then(() => { if (storing === settled) storing = null; });
}
if (signal !== undefined && !signal.aborted) {
// The prompt's own cancel: it stops this turn whether it runs or
// still waits, and never a later one.
const stop = () => { void send({ type: "cancel", messageId: newId("cnl"), target: messageId }).catch(() => {}); };
// still waits, and never a later one. A prompt whose send had not
// begun is never stored, so it needs none.
const stop = () => {
if (progress.sent) void send({ type: "cancel", messageId: newId("cnl"), target: messageId }).catch(() => {});
};
signal.addEventListener("abort", stop, { once: true });
void turn.result.finally(() => signal.removeEventListener("abort", stop)).catch(() => {});
}
@@ -2633,7 +2911,16 @@ export function createDurableAgentFactory({ harness, defaultDurability, defaultS
const messageId = steerOptions.messageId ?? newId("str");
if (!validId(messageId)) throw new TypeError(`messageId must be ${idRule}`);
admit(input);
await send({ type: "steer", messageId, input });
// A steer's images go in the session's image store before the steer
// that names them, as a prompt's do; storing one again under its
// name is harmless, so a retry needs no check.
let named = input;
if (Array.isArray(input) && input.some((block) => block?.type === "image" && typeof block.data === "string")) {
const { created, id: sessionId } = await resolved;
named = await storePromptImages((await created.session(sessionId)).images, input);
if (closed) throw new Error(`${label} is closed`);
}
await send({ type: "steer", messageId, input: named });
return { messageId };
},
/** Cancels the running turn; dropping a turn's view does not. */
@@ -2652,6 +2939,12 @@ export function createDurableAgentFactory({ harness, defaultDurability, defaultS
head: state.head,
}),
append: async () => { throw new Error("a checkpoint reads the session without changing it"); },
// The records name the session's images; the engine reads them
// from its image store to export them.
[persistenceImages]: {
get: (name) => log.images.get(name),
put: async () => { throw new Error("a checkpoint reads the session without changing it"); },
},
};
const harness = await agent.openHarness(sessionId, readOnly, null);
try {
+6 -1
View File
@@ -1,7 +1,7 @@
// fx as a harness of the durable core in durable.js: `createFxAgent` is that
// core running fx engines. Everything the core needs to know about fx is
// here; the core itself knows only the harness contract.
import { coreAnswered, engineInternals, journalMarksWanted, normalizeInstructions, normalizeModelChoice, resumeOutputShown } from "./fx-sdk.js";
import { checkPromptInput, coreAnswered, engineInternals, journalMarksWanted, normalizeInstructions, normalizeModelChoice, resumeOutputShown } from "./fx-sdk.js";
import { createCredentialMemory, credentialMemory } from "./subscription.js";
const encoder = new TextEncoder();
@@ -28,6 +28,11 @@ export function fxHarness({ createEngine, defaultApiKey = async () => undefined
return {
...(checkpoint === undefined ? {} : { seed: snapshotOf(checkpointBytesOf(checkpoint)) }),
settings: sessionSettings,
// What a session's prompt() refuses before storing anything: whatever
// this agent's engines would refuse.
checkPrompt(input) {
checkPromptInput(input, { resizeImage: engineOptions.resizeImage });
},
async open({ sessionId, store, context, settings, durability, ready }) {
const apiKey = engineOptions.apiKey ?? await defaultApiKey(durability);
const engine = await createEngine({
+211 -26
View File
@@ -37,7 +37,10 @@ const maxUnreadEvents = 256;
// Prompt images travel as raw bytes beside the ACP frame and are base64
// encoded only in the model request. An image may use 5 MiB of encoded
// request data, and a prompt's images 8 MiB, so the raw limits are 3/4 of that.
// The kernel still validates content and media type.
// The kernel still validates content and media type, and holds a prompt to
// these limits too: `max_libfx_prompt_images` and
// `max_libfx_prompt_images_bytes` in src/core/images/image_attachments.zig
// must match them.
const maxPromptImages = 8;
const maxPromptImageDataBytes = 5 * 1024 * 1024;
const maxPromptImagesDataBytes = 8 * 1024 * 1024;
@@ -45,6 +48,7 @@ const maxPromptImageBytes = (maxPromptImageDataBytes / 4) * 3;
const maxPromptImagesBytes = (maxPromptImagesDataBytes / 4) * 3;
// Matches the native attachment table: one prompt's images or one checkpoint.
const maxPendingAttachments = 8;
const maxPendingAttachmentBytes = 8 * 1024 * 1024;
const maxOutboundAttachments = 4;
// Matches the core's kernel checkpoint limit (max_checkpoint_bytes).
const maxCheckpointBytes = 4 * 1024 * 1024;
@@ -234,6 +238,14 @@ export function normalizeAgentOptions(value) {
// public surface.
export const engineInternals = Symbol.for("libfx.engineInternals");
export const journalMarksWanted = Symbol.for("libfx.journalMarks");
// A persistence's image store, `{ put(name, bytes), get(name) }`, which keeps
// each image once, named by the SHA-256 of its bytes, so records and
// checkpoints only name it. Durable sessions give one to the engines they
// run; without it, images stay inline.
export const persistenceImages = Symbol.for("libfx.persistenceImages");
// `{ images, bytes }`: a smaller image window than the core's, the newest
// images a session with an image store keeps, so tests can reach its edge.
const imageWindowLimits = Symbol.for("libfx.imageWindow");
// Marks an error the core answered a request with, as opposed to one from a
// core that stopped answering.
export const coreAnswered = Symbol.for("libfx.coreAnswered");
@@ -279,8 +291,26 @@ const journalOpenErrorCodes = new Map([
["libfx journal events are out of order", "FX_JOURNAL_INVALID"],
["libfx journal is too large", "FX_JOURNAL_TOO_LARGE"],
["libfx journal holds more than 1024 turns", "FX_JOURNAL_TOO_LARGE"],
["libfx journal images are too large", "FX_JOURNAL_TOO_LARGE"],
["Invalid libfx journal images", "FX_JOURNAL_INVALID"],
["libfx image does not match its name", "FX_JOURNAL_IMAGE_MISSING"],
]);
// An image's name in a persistence's image store: the lowercase hex SHA-256
// of its bytes, as the core names it.
async function imageName(bytes) {
const digest = new Uint8Array(await globalThis.crypto.subtle.digest("SHA-256", bytes));
let name = "";
for (const byte of digest) name += byte.toString(16).padStart(2, "0");
return name;
}
function imageMissingError(name, cause) {
const error = journalLoadError(`libfx journal names an image the store does not hold: ${name}`, "FX_JOURNAL_IMAGE_MISSING", cause);
error.image = name;
return error;
}
/**
* A persistence store that keeps a session's records and its latest
* checkpoint in memory, for tests and for hosts that copy them elsewhere.
@@ -1986,6 +2016,62 @@ function normalizePromptInput(input, { deferImageLimits = false } = {}) {
return prompt;
}
// The image types libfx's core takes, each known by how its bytes start.
const imageSignatures = [
["image/png", 0, [0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]],
["image/jpeg", 0, [0xff, 0xd8, 0xff]],
["image/gif", 0, [0x47, 0x49, 0x46, 0x38, 0x37, 0x61]],
["image/gif", 0, [0x47, 0x49, 0x46, 0x38, 0x39, 0x61]],
["image/webp", 0, [0x52, 0x49, 0x46, 0x46]],
];
const imageTypes = new Set(imageSignatures.map(([type]) => type));
const webpMarker = [0x57, 0x45, 0x42, 0x50];
// The image type `bytes` start as, or null.
function imageTypeOf(bytes) {
const startsWith = (offset, signature) => bytes.length >= offset + signature.length && signature.every((byte, index) => bytes[offset + index] === byte);
for (const [type, offset, signature] of imageSignatures) {
if (!startsWith(offset, signature)) continue;
if (type === "image/webp" && !startsWith(8, webpMarker)) continue;
return type;
}
return null;
}
// Refuses image bytes that are not the type they declare, as libfx's core does.
function checkImageBytesType(bytes, mimeType, index) {
const type = imageTypeOf(bytes);
if (type === null) throw new TypeError(`image prompt block ${index} is not a PNG, JPEG, GIF or WebP image`);
if (type !== mimeType) throw new TypeError(`image prompt block ${index} holds ${type} data, not ${mimeType}`);
}
/**
* Checks a prompt's input as libfx takes it and returns it normalized:
* every check `prompt()` makes, and the ones libfx's core makes on images,
* which would otherwise refuse the prompt only once it reached the core: a
* supported image type, and bytes or base64 that are that type. A Blob is not
* read, so its bytes are checked once read (`checkImageBytesType`). With
* `resizeImage`, image bytes are checked after it runs instead. Throws a
* TypeError or RangeError.
*/
export function checkPromptInput(input, { resizeImage } = {}) {
const resized = resizeImage !== undefined;
const normalized = normalizePromptInput(input, { deferImageLimits: resized });
// The core takes no prompt without text or an image; a resource adds neither.
if (!normalized.some((block) => block.type === "image" || (block.type === "text" && block.text.length > 0))) {
throw new TypeError("prompt is empty");
}
// resizeImage decides the final image sizes, so they are counted after it runs.
checkPromptFrameSize(normalized, !resized);
normalized.forEach((block, index) => {
if (block.type !== "image" || (resized && block.data !== undefined)) return;
if (!imageTypes.has(block.mimeType)) throw new TypeError(`image prompt block ${index} mimeType must be image/png, image/jpeg, image/gif or image/webp`);
if (block.source === "bytes") checkImageBytesType(block.data, block.mimeType, index);
else if (block.source === "base64") checkImageBytesType(base64ToBytes(block.data.slice(0, 16)), block.mimeType, index);
});
return normalized;
}
// Image bytes travel beside the frame as attachment references, but the model
// request carries them base64 encoded and the native host caps that request at
// 8 MiB. Images therefore count at their encoded size, the same budget they
@@ -2071,6 +2157,11 @@ async function materializePromptImages(blocks, isCancelled, resizeImage) {
}
if (promptFrameSize(prepared) > maxPromptFrameBytes) prepared = omitReferencedImageData(prepared);
checkPromptFrameSize(prepared, true);
// A file's bytes, or what resizeImage made, are checked once there are
// some, after the size limits.
prepared.forEach((image, index) => {
if (image.type === "image" && image.bytes !== undefined) checkImageBytesType(image.bytes, image.mimeType, index);
});
return prepared;
}
@@ -2201,7 +2292,10 @@ export function normalizeInstructions(value) {
return instructions;
}
function hostToolContent(value) {
// `stored` maps an image's base64 data to its name in the image store, for
// the images stored before the result goes to the core: their blocks carry
// the name, so the core names them too.
function hostToolContent(value, stored = null) {
if (value?.type === "libfx.tool-result") {
if (typeof value.text !== "string" || !Array.isArray(value.images) || value.images.length > 8) {
throw new TypeError("invalid typed tool result");
@@ -2213,11 +2307,13 @@ function hostToolContent(value) {
if ((!referenceOnly && typeof image.data !== "string") || typeof image.mimeType !== "string" || image.mimeType.length === 0 || image.mimeType.length > 128 || (image.data?.length > maxPromptImageDataBytes && sourceRef === undefined)) {
throw new TypeError("invalid tool image");
}
const data = !referenceOnly && image.data.length <= maxPromptImageDataBytes ? image.data : undefined;
return {
type: "image",
...(!referenceOnly && image.data.length <= maxPromptImageDataBytes ? { data: image.data } : {}),
...(data === undefined ? {} : { data }),
mimeType: image.mimeType,
...(sourceRef === undefined ? {} : { sourceRef }),
...(data !== undefined && stored?.has(data) ? { stored: stored.get(data) } : {}),
};
});
if (promptImageDataBytes(images) > maxPromptImagesDataBytes) images = omitReferencedImageData(images);
@@ -2228,7 +2324,7 @@ function hostToolContent(value) {
content = JSON.stringify({ text: value.text, images });
}
if (encoder.encode(content).length > maxPromptImagesDataBytes) throw new RangeError("typed tool result exceeds the result limit");
return { content, rich: true, isError: value.isError === true };
return { content, rich: true, isError: value.isError === true, images };
}
if (typeof value === "string") return { content: value, rich: false };
if (value === undefined) return { content: "null", rich: false };
@@ -2317,6 +2413,11 @@ export async function createFxEngine(options = {}) {
// waits for it: the core before an action others can see, such as a tool
// call with effects, or a caller for its input.
const persistence = options.persistence ?? null;
// A persistence with an image store keeps each prompt image once, by
// name, and the core names it in records and checkpoints instead of
// holding its bytes.
const imageStorage = persistence?.[persistenceImages] ?? null;
const imageStore = imageStorage !== null;
const wantsMarks = persistence?.marks === true || persistence?.[journalMarksWanted] === true;
const checkpointAfterBytes = options.checkpointAfterBytes ?? defaultCheckpointAfterBytes;
// The store's last cursor, and the last seq its records hold.
@@ -2625,6 +2726,30 @@ export async function createFxEngine(options = {}) {
}
throw new Error("transport retry exhausted");
};
// A typed tool result's images go in the image store by name before the
// result reaches the core, as a prompt's do, so the session's records name
// them. The tool has already run, so an image the store refuses is not
// lost: it stays in the result, and the records hold it inline.
const storedToolContent = async (value) => {
const normalized = hostToolContent(value);
if (!imageStore || !normalized.images?.some((image) => image.data !== undefined)) return normalized;
const stored = new Map();
for (const image of normalized.images) {
if (image.data === undefined || stored.has(image.data)) continue;
let bytes;
// Data that is not base64 is the core's to refuse.
try { bytes = base64ToBytes(image.data); } catch { continue; }
const name = await imageName(bytes);
try {
await imageStorage.put(name, bytes);
stored.set(image.data, name);
} catch {
// Not stored: the result keeps the image, and the records hold it
// inline.
}
}
return stored.size ? hostToolContent(value, stored) : normalized;
};
const executeHostTool = async (name, input, requestedSessionId, toolCallId) => {
const execute = hostTools.executors.get(name);
const issuerSessionId = requestedSessionId ?? sessionId;
@@ -2658,7 +2783,7 @@ export async function createFxEngine(options = {}) {
});
const value = await Promise.race([execution, aborted]);
if (!controller.signal.aborted) {
const normalized = hostToolContent(value);
const normalized = await storedToolContent(value);
content = normalized.content;
rich = normalized.rich;
isError = normalized.isError === true;
@@ -2667,7 +2792,7 @@ export async function createFxEngine(options = {}) {
isError = true;
if (error?.toolResult?.type === "libfx.tool-result") {
try {
const normalized = hostToolContent(error.toolResult);
const normalized = await storedToolContent(error.toolResult);
content = normalized.content;
rich = normalized.rich;
} catch {
@@ -2746,6 +2871,25 @@ export async function createFxEngine(options = {}) {
if (!(bytes instanceof Uint8Array) || bytes.byteLength === 0) throw new Error("fx returned an invalid checkpoint");
return new Uint8Array(bytes.buffer, bytes.byteOffset, bytes.byteLength);
}
// Stores each prompt image in the persistence's image store, by name,
// before the prompt that carries it is sent: nothing the core writes may
// name an image the store does not hold. Returns the blocks with their
// names, or null when the prompt stopped meanwhile.
async function storePromptImages(blocks, stopped) {
const named = [];
for (const block of blocks) {
if (block.type !== "image" || block.bytes === undefined) {
named.push(block);
continue;
}
const name = await imageName(block.bytes);
if (stopped()) return null;
await imageStorage.put(name, block.bytes);
if (stopped()) return null;
named.push({ ...block, stored: name });
}
return named;
}
const sendPrompt = (blocks, resuming = false, followUp = null, turnId = null, rerun = [], yieldAt = undefined, outputShown = false) => {
// A journaled turn given a time to stop at yields before its first model
// request after it, keeping the turn open.
@@ -2760,7 +2904,8 @@ export async function createFxEngine(options = {}) {
type: "image",
mimeType: block.mimeType,
...(block.sourceRef === undefined ? {} : { sourceRef: block.sourceRef }),
...(block.bytes === undefined ? {} : { _meta: { fx: { attachment: ids[next++] } } }),
// An image the store holds carries its name, so the core names it too.
...(block.bytes === undefined ? {} : { _meta: { fx: { attachment: ids[next++], ...(block.stored === undefined ? {} : { stored: block.stored }) } } }),
});
// A follow-up's turn places it; the core records it as accepted first
// when the host could not. A journaled turn's first progress records its id.
@@ -2878,24 +3023,62 @@ export async function createFxEngine(options = {}) {
}
if (journal) {
const { events, checkpoint } = loaded;
const params = { sessionId, ...(options.inputsDurable === true ? { inputsDurable: true } : {}) };
// One call: attaching again would drop the first attachment.
const attachments = [];
if (events.length > 0) {
const bytes = encoder.encode(JSON.stringify(withoutSupersededProgress(events)));
if (bytes.byteLength > maxJournalBytes) throw journalLoadError("libfx journal is too large", "FX_JOURNAL_TOO_LARGE");
attachments.push(["journalAttachment", bytes]);
}
if (checkpoint) attachments.push(["snapshotAttachment", checkpoint]);
const ids = attachments.length ? attachBytes(attachments.map(([, bytes]) => bytes)) : [];
attachments.forEach(([name], index) => { params[name] = ids[index]; });
const opened = await request("libfx/journal_open", params).catch((error) => {
const journalBytes = events.length > 0 ? encoder.encode(JSON.stringify(withoutSupersededProgress(events))) : null;
if (journalBytes && journalBytes.byteLength > maxJournalBytes) throw journalLoadError("libfx journal is too large", "FX_JOURNAL_TOO_LARGE");
const openFailed = (error) => {
const message = error?.message ?? "";
if (/newer fx/.test(message)) throw new FxJournalVersionError(message);
const code = journalOpenErrorCodes.get(message);
if (code) throw journalLoadError(message, code, error);
throw error;
});
};
const openOnce = () => {
const params = {
sessionId,
...(options.inputsDurable === true ? { inputsDurable: true } : {}),
...(imageStore ? { imageStore: true } : {}),
...(imageStore && options[imageWindowLimits] ? { imageWindow: options[imageWindowLimits] } : {}),
};
// One call: attaching again would drop the first attachment.
const attachments = [];
if (journalBytes) attachments.push(["journalAttachment", journalBytes]);
if (checkpoint) attachments.push(["snapshotAttachment", checkpoint]);
const ids = attachments.length ? attachBytes(attachments.map(([, bytes]) => bytes)) : [];
attachments.forEach(([name], index) => { params[name] = ids[index]; });
return request("libfx/journal_open", params).catch(openFailed);
};
// Stages the images a history names, read from the store, in batches
// the attachment table holds. A store that lacks one fails the open:
// the history without it is not the one saved.
const stageImages = async (names) => {
let batch = [];
let batchBytes = 0;
const flush = async () => {
if (batch.length === 0) return;
const ids = attachBytes(batch.map(({ bytes }) => bytes));
const images = batch.map(({ name }, index) => ({ name, attachment: ids[index] }));
batch = [];
batchBytes = 0;
await request("libfx/journal_images", { sessionId, images }).catch(openFailed);
};
for (const name of new Set(names.map(String))) {
const stored = imageStore ? await imageStorage.get(name) : undefined;
if (stored === undefined || stored === null) throw imageMissingError(name);
const bytes = persistenceBytes(stored, "an image the image store returns");
if (batch.length === maxPendingAttachments || batchBytes + bytes.byteLength > maxPendingAttachmentBytes) await flush();
batch.push({ name, bytes });
batchBytes += bytes.byteLength;
}
await flush();
};
// A history that names images opens once they are staged from the
// store: the core answers with the names it needs, all at once.
let opened = await openOnce();
if (Array.isArray(opened?.imagesNeeded)) {
await stageImages(opened.imagesNeeded);
opened = await openOnce();
if (Array.isArray(opened?.imagesNeeded)) throw imageMissingError(String(opened.imagesNeeded[0]));
}
compareCheckpointMeta(opened?.checkpointMeta);
resumable = opened?.resumable === true;
openTurnId = resumable && validTurnId(opened?.openTurnId) ? opened.openTurnId : null;
@@ -3211,13 +3394,12 @@ export async function createFxEngine(options = {}) {
const resizeImage = options.resizeImage;
const resuming = promptOptions[resumeTurn] === true;
const followUp = promptOptions[followUpInput] ?? null;
const normalized = resuming ? [] : normalizePromptInput(input, { deferImageLimits: resizeImage !== undefined });
const normalized = resuming ? [] : checkPromptInput(input, { resizeImage });
const hasPixels = normalized.some((block) => block.type === "image" && block.data !== undefined);
// Blob reads and resizeImage run before the prompt frame is sent.
const asyncImages = hasPixels && (resizeImage !== undefined ||
// Blob reads, resizeImage and storing images run before the prompt
// frame is sent.
const asyncImages = hasPixels && (resizeImage !== undefined || imageStore ||
normalized.some((block) => block.type === "image" && block.source === "blob"));
// resizeImage decides the final image sizes, so they are counted after it runs.
checkPromptFrameSize(normalized, resizeImage === undefined);
// Snapshot caller-owned bytes, which could change before an async send.
const prompt = asyncImages
? normalized.map((block) => block.type === "image" && block.source === "bytes" ? { ...block, data: block.data.slice() } : block)
@@ -3422,8 +3604,11 @@ export async function createFxEngine(options = {}) {
}) : null;
let response;
if (asyncImages) {
const stopped = () => cancelled || closing;
response = Promise.race([
Promise.resolve().then(() => materializePromptImages(prompt, () => cancelled || closing, resizeImage)),
Promise.resolve()
.then(() => materializePromptImages(prompt, stopped, resizeImage))
.then((prepared) => prepared && imageStore ? storePromptImages(prepared, stopped) : prepared),
imagePrepCancelled,
]).then((prepared) => {
cancelImagePrep = null;
+18 -8
View File
@@ -608,9 +608,9 @@ for (const resizeImage of [undefined, (image) => image]) {
}
// The SDK decodes base64 itself, so non-canonical base64 throws before the
// turn starts. Kernel-side content validation stays authoritative: a sniffed
// media type that contradicts the declaration fails the turn with a typed error
// rather than reaching the model.
// turn starts, and so does an image whose bytes aren't the type it declares,
// or a type libfx doesn't take, as the kernel would refuse it. A Blob's bytes
// are checked once read, which rejects the turn before the prompt is sent.
{
const gateway = mockGateway();
const agent = await createAgent(gateway, { model: "sdk/vision-model" });
@@ -621,17 +621,27 @@ for (const resizeImage of [undefined, (image) => image]) {
);
}
const jpegBytes = Buffer.from([0xff, 0xd8, 0xff, 0xe0, 1, 2, 3, 4]).toString("base64");
const mismatch = agent.prompt([{ type: "image", data: jpegBytes, mimeType: "image/png" }]);
await assert.rejects(mismatch.result, /Invalid image prompt block/);
const mislabeled = (error) => error instanceof TypeError && /holds image\/jpeg data, not image\/png/.test(error.message);
assert.throws(() => agent.prompt([{ type: "image", data: jpegBytes, mimeType: "image/png" }]), mislabeled);
assert.throws(() => agent.prompt([{ type: "image", data: Buffer.from(jpegBytes, "base64"), mimeType: "image/png" }]), mislabeled);
assert.throws(
() => agent.prompt([{ type: "image", data: Buffer.from("not an image"), mimeType: "image/png" }]),
(error) => error instanceof TypeError && /is not a PNG, JPEG, GIF or WebP image/.test(error.message),
);
const blobMismatch = agent.prompt([
{ type: "image", data: new Blob([Buffer.from(jpegBytes, "base64")], { type: "image/png" }) },
]);
await assert.rejects(blobMismatch.result, /Invalid image prompt block/);
await assert.rejects(blobMismatch.result, mislabeled);
for (const mimeType of ["image/svg+xml", "text/plain", "image/png\n"]) {
const referenceOnly = agent.prompt([{ type: "image", mimeType, sourceRef: "host:unsupported-mime" }]);
await assert.rejects(referenceOnly.result, /Invalid image prompt block/);
assert.throws(
() => agent.prompt([{ type: "image", mimeType, sourceRef: "host:unsupported-mime" }]),
(error) => error instanceof TypeError && /mimeType must be image\/png, image\/jpeg, image\/gif or image\/webp/.test(error.message),
);
}
assert.throws(() => agent.prompt(""), (error) => error instanceof TypeError && /prompt is empty/.test(error.message));
assert.equal(gateway.state.chatBodies.length, 0);
// The agent still takes a valid prompt.
assert.equal((await runPrompt(agent, [{ type: "image", data: pngData, mimeType: "image/png" }])).stopReason, "end_turn");
await agent.close();
}
+458 -1
View File
@@ -5,7 +5,7 @@
//
// node sdk/tests/test-durable-core.mjs [memory|local|world|queue]
import { strict as assert } from "node:assert";
import { mkdtemp, rm } from "node:fs/promises";
import { mkdtemp, readdir, readFile, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { createDurableAgentFactory, foldSessionLog, memory, wakeHandler } from "../durable.js";
@@ -964,6 +964,48 @@ if (durabilityKind === "queue") {
await agent.close();
}
});
test("an image prompt run in the request reads its image from the store, and no queued copy holds it", async () => {
const internals = (agent) => agent[Symbol.for("libfx.durableInternals")];
for (const maxDurationMs of [60_000, 300]) {
const dir = await mkdtemp(join(tmpdir(), "libfx-in-request-image-"));
dirs.push(dir);
const waited = [];
const { durability, broker } = await queueWorldAt(dir, { queueLeases: true, runInRequest: true, maxDurationMs, reserveMs: 100, waitUntil: (work) => waited.push(work) });
const { agent, seen } = recordingAgent(durability);
const session = agent.session();
// With the short deadline, a step outlasts the request.
if (maxDurationMs < 1000) {
const slow = holdStep("image-1");
void slow.holding.then(() => setTimeout(slow.release, 600));
}
const messageId = `turn-image-${maxDurationMs}`;
const turn = session.prompt([{ type: "text", text: "look" }, image], { messageId });
assert.equal((await turn.result).stopReason, "end_turn");
assert.equal(waited.length, 1, "the request ran the turn");
assert.deepEqual(seen, [[{ type: "text", text: "look" }, image]], "with its image, read once");
const sent = broker.sent.filter((message) => message.payload?.input?.messageId === messageId);
assert.equal(sent[0].attempt, 0, "the backstop waited");
if (maxDurationMs < 1000) assert.ok(sent.length > 1, "a turn longer than the request went on in a fresh message");
for (const message of sent) {
const payload = JSON.stringify(message.payload);
assert.ok(!payload.includes(png), "a queued copy holds none of the image");
}
await waited[0];
for (let tries = 0; internals(agent).liveWorkers() > 0; tries += 1) {
assert.ok(tries < 200, "the request's worker never let go");
await sleep(10);
}
// The backstop, delivered at last, runs nothing.
broker.lapse(sent[0]);
for (let tries = 0; broker.held(() => true).some((message) => message.id === sent[0].id); tries += 1) {
assert.ok(tries < 200, "the backstop was never acknowledged");
await sleep(10);
}
assert.equal(seen.length, 1, "the backstop ran nothing");
await agent.close();
}
});
}
test("a reconnect reads the stream from its cursor's line, not its start", async () => {
@@ -1052,6 +1094,421 @@ test("each prompt's turn runs with its session object's settings, resumed or not
await agent.close();
});
// A real 1x1 PNG, as a caller would send it.
const png = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+jP0cAAAAASUVORK5CYII=";
const pngBytes = Uint8Array.from(atob(png), (char) => char.charCodeAt(0));
const image = { type: "image", data: png, mimeType: "image/png" };
// An image block of its own: the PNG with byte `n` after it, so its name is
// new and storing it writes it.
const imageOf = (n) => ({ type: "image", data: Buffer.from([...pngBytes, n]).toString("base64"), mimeType: "image/png" });
// An agent whose harness records each prompt input it is handed, and runs an
// image prompt as the word "image"; with `refuseFirst`, its first prompt
// throws, as a harness that cannot run a turn does.
function recordingAgent(durability, { refuseFirst = false } = {}) {
const seen = [];
let refusals = refuseFirst ? 1 : 0;
const agent = createDurableAgentFactory({
harness: () => {
const base = scriptedHarness()();
return {
open: async (options) => {
const session = await base.open(options);
return {
...session,
prompt: (input, opts) => {
seen.push(input);
if (refusals > 0) {
refusals -= 1;
throw new Error("the harness stopped");
}
return session.prompt(typeof input === "string" ? input : "image", opts);
},
get openTurn() { return session.openTurn; },
};
},
};
},
defaultDurability: async () => memory(),
})({ durability });
return { agent, seen };
}
const kinds = (seen) => seen.map((input) => (typeof input === "string" ? input : "image"));
test("a durable prompt carries image data as base64: bytes and files throw from prompt(), and nothing is queued", async () => {
const { agent, seen } = recordingAgent(await durabilityFor());
const session = agent.session();
const refused = [
{ type: "image", data: 42, mimeType: "image/png" },
{ type: "image", data: new Uint8Array(pngBytes), mimeType: "image/png" },
{ type: "image", data: new Uint8Array(pngBytes).buffer, mimeType: "image/png" },
{ type: "image", data: new Blob([pngBytes], { type: "image/png" }) },
{ type: "image", data: new File([pngBytes], "shot.png", { type: "image/png" }) },
];
for (const block of refused) {
assert.throws(
() => session.prompt([{ type: "text", text: "look" }, block]),
(error) => error instanceof TypeError && /^a durable prompt carries image data as a base64 string$/.test(error.message),
);
}
const turn = session.prompt([{ type: "text", text: "look" }, image, { ...image, sourceRef: "uploads:shot" }]);
assert.equal((await turn.result).stopReason, "end_turn");
assert.deepEqual(seen, [[{ type: "text", text: "look" }, image, { ...image, sourceRef: "uploads:shot" }]], "base64 passes as given");
await agent.close();
});
test("a prompt storing its image is stored before the prompts made after it, and a failed image write stores nothing", async () => {
const fault = { writeMs: 150 };
const { agent, seen } = recordingAgent(faultyImages(await durabilityFor(), fault));
const session = agent.session();
const first = session.prompt([imageOf(1)]);
const second = session.prompt("second");
assert.deepEqual([(await first.result).stopReason, (await second.result).stopReason], ["end_turn", "end_turn"]);
assert.deepEqual(kinds(seen), ["image", "second"]);
fault.failWrites = 1;
const failing = session.prompt([imageOf(2)]);
const after = session.prompt("after");
await assert.rejects(failing.accepted, /^Error: the store is full$/);
await assert.rejects(failing.result, /^Error: the store is full$/);
assert.equal((await after.result).stopReason, "end_turn");
assert.deepEqual(kinds(seen), ["image", "second", "after"], "the failed write stored no turn, and the prompt behind it ran");
await agent.close();
});
test("a prompt's signal or the agent closing while its image is stored keeps its turn from running", async () => {
const fault = { writeMs: 400 };
const { agent, seen } = recordingAgent(faultyImages(await durabilityFor(), fault));
const session = agent.session();
const controller = new AbortController();
const cancelled = session.prompt([imageOf(3)], { messageId: "turn-store", signal: controller.signal });
await sleep(20);
const aborted = Date.now();
controller.abort();
await cancelled.accepted;
assert.equal((await cancelled.result).stopReason, "cancelled");
assert.ok(Date.now() - aborted < 300, "the prompt stopped waiting for its image write at once");
// Neither the prompt nor a cancel naming it was stored: its id runs anew.
assert.equal((await session.prompt("after", { messageId: "turn-store" }).result).stopReason, "end_turn");
assert.deepEqual(kinds(seen), ["after"]);
fault.writeMs = 150;
const closing = session.prompt([imageOf(4)]);
await agent.close();
await assert.rejects(closing.accepted, /agent is closed/);
await assert.rejects(closing.result, /agent is closed/);
assert.deepEqual(kinds(seen), ["after"]);
});
test("a prompt withdrawn while it waits keeps the prompts behind it waiting for the ones before it", async () => {
const { agent, seen } = recordingAgent(faultyImages(await durabilityFor(), { writeMs: 300 }));
const session = agent.session();
const first = session.prompt([imageOf(5)], { messageId: "turn-first" });
const controller = new AbortController();
const second = session.prompt([imageOf(6)], { messageId: "turn-second", signal: controller.signal });
const third = session.prompt("third", { messageId: "turn-third" });
await sleep(20);
controller.abort();
assert.equal((await second.result).stopReason, "cancelled");
assert.equal((await first.result).stopReason, "end_turn");
assert.equal((await third.result).stopReason, "end_turn");
assert.deepEqual(kinds(seen), ["image", "third"], "the prompt behind the withdrawn one still came after the one before it");
await agent.close();
});
test("an image prompt its first worker could not start runs from the log with the same base64, once", async () => {
const { agent, seen } = recordingAgent(await durabilityFor(), { refuseFirst: true });
const session = agent.session();
const shot = { ...image, sourceRef: "uploads:shot" };
assert.equal((await session.prompt([shot], { messageId: "turn-img" }).result).stopReason, "end_turn");
assert.deepEqual(seen, [[shot], [shot]]);
const again = await agent.session(session.id).prompt([shot], { messageId: "turn-img" }).result;
assert.equal(again.repeated, true);
assert.equal(seen.length, 2, "the retry ran nothing");
await agent.close();
});
test("an image block is read once, so the log holds what was checked", async () => {
const { agent, seen } = recordingAgent(await durabilityFor());
const session = agent.session();
// Accessors that answer one way when checked and another way after.
let dataReads = 0;
let mimeReads = 0;
const shifting = {
type: "image",
get data() { dataReads += 1; return dataReads === 1 ? png : new Uint8Array([1, 2, 3]); },
mimeType: "image/png",
sourceRef: "uploads:shifting",
};
const relabelled = {
type: "image",
data: png,
get mimeType() { mimeReads += 1; return mimeReads === 1 ? "image/png" : undefined; },
};
assert.equal((await session.prompt([{ type: "text", text: "look" }, shifting, relabelled]).result).stopReason, "end_turn");
assert.deepEqual(seen.at(-1), [{ type: "text", text: "look" }, { ...image, sourceRef: "uploads:shifting" }, image]);
assert.deepEqual([dataReads, mimeReads], [1, 1]);
// An Array subclass whose map() returns something else still stores its blocks.
class Remapped extends Array { map() { return "not the caller's blocks"; } }
const blocks = Remapped.from([{ type: "text", text: "remapped" }, image]);
assert.equal((await session.prompt(blocks).result).stopReason, "end_turn");
assert.deepEqual(seen.at(-1), [{ type: "text", text: "remapped" }, image]);
await agent.close();
});
const pngName = Buffer.from(await crypto.subtle.digest("SHA-256", pngBytes)).toString("hex");
// Every image stream's name holds this: "_user_", then the base64url of the
// namespace "libfx-image-".
const imageStreamTag = `_user_${Buffer.from("libfx-image-").toString("base64url")}`;
// The durability with a faulty image store: with `fault.lose`, every image
// reads back as missing; with `fault.corrupt`, every image reads back with
// its last byte changed; `fault.failWrites` fails that many image writes;
// `fault.writeMs` makes each image write wait that long first. The test may
// change `fault` as it goes. A World's faults are in its streams, under the
// code that names images.
function faultyImages(durability, fault = {}) {
fault.failWrites ??= 0;
const slowly = async () => { if (fault.writeMs) await sleep(fault.writeMs); };
const refuse = () => {
fault.failWrites -= 1;
throw new Error("the store is full");
};
const flipped = (bytes) => {
if (!(bytes instanceof Uint8Array) || bytes.byteLength === 0) return bytes;
const copy = bytes.slice();
copy[copy.length - 1] ^= 0xff;
return copy;
};
if (typeof durability.create === "function") {
return {
...durability,
create() {
const backend = durability.create();
return {
...backend,
async session(id) {
const log = await backend.session(id);
return {
...log,
images: {
put: async (name, bytes) => {
await slowly();
return fault.failWrites > 0 ? refuse() : log.images.put(name, bytes);
},
get: async (name) => {
if (fault.lose) return undefined;
const bytes = await log.images.get(name);
return fault.corrupt ? flipped(bytes) : bytes;
},
},
};
},
};
},
};
}
const isImage = (name) => String(name).includes(imageStreamTag);
const bound = (target, value) => (typeof value === "function" ? value.bind(target) : value);
return {
...durability,
async world() {
const world = await durability.world();
const streams = new Proxy(world.streams, {
get(target, prop) {
const value = Reflect.get(target, prop);
if (prop === "write") {
return async (runId, name, chunk) => {
if (!isImage(name)) return value.call(target, runId, name, chunk);
await slowly();
return fault.failWrites > 0 ? refuse() : value.call(target, runId, name, chunk);
};
}
if (prop === "getChunks") {
return async (runId, name, options) => {
if (fault.lose && isImage(name)) return { data: [], cursor: null, hasMore: false, done: false };
const result = await value.call(target, runId, name, options);
if (!fault.corrupt || !isImage(name) || !Array.isArray(result?.data)) return result;
return { ...result, data: result.data.map((chunk) => ({ ...chunk, data: flipped(chunk?.data) })) };
};
}
return bound(target, value);
},
});
return new Proxy(world, { get: (target, prop) => (prop === "streams" ? streams : bound(target, Reflect.get(target, prop))) });
},
};
}
// Every file under a World's folder, for checking what it stored. A lock or
// temporary file gone by the time it is read is skipped.
async function filesUnder(dir) {
const files = [];
for (const entry of await readdir(dir, { withFileTypes: true, recursive: true })) {
if (!entry.isFile()) continue;
try {
files.push(await readFile(join(entry.parentPath ?? entry.path, entry.name)));
} catch (error) {
if (error?.code !== "ENOENT") throw error;
}
}
return files;
}
test("a durable prompt stores each image once in the session's image store, and the log only names it", async () => {
const durability = await durabilityFor();
const { agent, seen } = recordingAgent(durability);
const session = agent.session();
const turn = session.prompt([
{ type: "text", text: "look" },
image,
{ ...image, sourceRef: "uploads:shot" },
image,
]);
assert.equal((await turn.result).stopReason, "end_turn");
assert.deepEqual(seen, [[{ type: "text", text: "look" }, image, { ...image, sourceRef: "uploads:shot" }, image]]);
if (typeof durability.create === "function") {
const log = await durability.create().session(session.id);
const entries = (await log.readSince("0")).map(({ entry }) => entry);
assert.deepEqual(entries.find((entry) => entry.k === "input")?.input, [
{ type: "text", text: "look" },
{ type: "image", mimeType: "image/png", data: { sha256: pngName } },
{ type: "image", mimeType: "image/png", sourceRef: "uploads:shot", data: { sha256: pngName } },
{ type: "image", mimeType: "image/png", data: { sha256: pngName } },
]);
assert.ok(!JSON.stringify(entries).includes(png), "no entry holds the image's base64");
assert.deepEqual(await log.images.get(pngName), pngBytes);
}
await agent.close();
if (typeof durability.create !== "function") {
// On disk: the image's bytes once, in its stream, and its base64 nowhere.
const files = await filesUnder(dirs.at(-1));
assert.ok(!files.some((bytes) => bytes.includes(Buffer.from(png))), "no file holds the image's base64");
assert.equal(files.filter((bytes) => bytes.includes(Buffer.from(pngBytes))).length, 1, "one file holds the image's bytes");
}
});
test("an image the session's store no longer holds fails its turn, naming it", async () => {
const { agent, seen } = recordingAgent(faultyImages(await durabilityFor(), { lose: true }));
const session = agent.session();
const result = await session.prompt([{ type: "text", text: "look" }, image]).result;
assert.equal(result.stopReason, "error");
assert.match(result.error.message, new RegExp(`does not hold image ${pngName}`));
assert.deepEqual(seen, [], "the harness never ran it");
assert.equal((await session.prompt("after").result).stopReason, "end_turn");
await agent.close();
});
test("an image whose stored bytes changed fails its turn, naming it", async () => {
const { agent, seen } = recordingAgent(faultyImages(await durabilityFor(), { corrupt: true }));
const session = agent.session();
const result = await session.prompt([{ type: "text", text: "look" }, image]).result;
assert.equal(result.stopReason, "error");
// A World's read skips bytes that don't match the name, so to it the image
// is missing; a store that returns them fails the check after the read.
assert.match(result.error.message, new RegExp(`(holds other bytes under|does not hold) image ${pngName}`));
assert.deepEqual(seen, [], "the harness never ran it");
assert.equal((await session.prompt("after").result).stopReason, "end_turn");
await agent.close();
});
test("a World whose streams lack getChunks refuses an image prompt before storing it, and text still runs", async () => {
const durability = await durabilityFor();
// memory() keeps images itself; the others keep them in World streams.
if (typeof durability.world !== "function") return;
const bound = (target, value) => (typeof value === "function" ? value.bind(target) : value);
const withoutGetChunks = {
...durability,
async world() {
const world = await durability.world();
const streams = new Proxy(world.streams, { get: (target, prop) => (prop === "getChunks" ? undefined : bound(target, Reflect.get(target, prop))) });
return new Proxy(world, { get: (target, prop) => (prop === "streams" ? streams : bound(target, Reflect.get(target, prop))) });
},
};
const { agent, seen } = recordingAgent(withoutGetChunks);
const session = agent.session();
const refused = session.prompt([{ type: "text", text: "look" }, image]);
await assert.rejects(refused.accepted, (error) => error instanceof TypeError && /getChunks/.test(error.message));
await assert.rejects(refused.result);
assert.equal((await session.prompt("text only").result).stopReason, "end_turn");
assert.deepEqual(kinds(seen), ["text only"]);
await agent.close();
});
test("a failed image write rejects the prompt before anything is stored", async () => {
const { agent, seen } = recordingAgent(faultyImages(await durabilityFor(), { failWrites: 1 }));
const session = agent.session();
const refused = session.prompt([{ type: "text", text: "look" }, image], { messageId: "refused-img" });
// The store's own error.
const storeFailed = (error) => error.message === "the store is full";
await assert.rejects(refused.accepted, storeFailed);
await assert.rejects(refused.result, storeFailed);
assert.equal((await session.prompt([{ type: "text", text: "again" }, image]).result).stopReason, "end_turn");
assert.deepEqual(kinds(seen), ["image"], "only the second prompt was stored and ran");
// The refused prompt was never stored, so its retry is a new turn.
const retried = await session.prompt([{ type: "text", text: "look" }, image], { messageId: "refused-img" }).result;
assert.equal(retried.stopReason, "end_turn");
assert.notEqual(retried.repeated, true);
assert.deepEqual(kinds(seen), ["image", "image"]);
await agent.close();
});
test("a failed image write rejects a steer before it is sent", async () => {
const fault = {};
const { agent, seen } = recordingAgent(faultyImages(await durabilityFor(), fault));
const session = agent.session();
assert.equal((await session.prompt("first").result).stopReason, "end_turn");
fault.failWrites = 1;
// The store's own error.
await assert.rejects(session.steer([{ type: "text", text: "look" }, image]), (error) => error.message === "the store is full");
// The steer was never sent, so no turn ran for it before the next prompt.
assert.equal((await session.prompt("after").result).stopReason, "end_turn");
assert.deepEqual(kinds(seen), ["first", "after"]);
await agent.close();
});
test("a retried image prompt the log already holds is answered without writing its images again", async () => {
const fault = {};
const { agent, seen } = recordingAgent(faultyImages(await durabilityFor(), fault));
const session = agent.session();
assert.equal((await session.prompt([{ type: "text", text: "look" }, image], { messageId: "img-once" }).result).stopReason, "end_turn");
// From now on every image write fails, so a retry that wrote again would fail.
fault.failWrites = Infinity;
const again = await agent.session(session.id).prompt([{ type: "text", text: "look" }, image], { messageId: "img-once" }).result;
assert.equal(again.repeated, true);
assert.equal(again.stopReason, "end_turn");
assert.deepEqual(kinds(seen), ["image"], "the retry ran nothing");
await agent.close();
});
test("base64 is the same with and without the runtime's own encoder and decoder", async () => {
const reference = (bytes) => btoa(String.fromCharCode(...bytes));
const inputs = [...Array.from({ length: 9 }, (_, n) => pngBytes.subarray(0, n + 1)), Uint8Array.from({ length: 3000 }, (_, i) => (i * 131) & 255)];
const run = async () => {
const { agent, seen } = recordingAgent(await durabilityFor());
const session = agent.session();
// Each image is decoded to be stored, and encoded again for the harness.
for (const bytes of inputs) assert.equal((await session.prompt([{ type: "image", data: reference(bytes), mimeType: "image/png" }]).result).stopReason, "end_turn");
// Each turn's records went through the log as base64 and back.
const words = JSON.parse(decoder.decode(await session.checkpoint()));
await agent.close();
return { seen, words };
};
const native = await run();
const encode = Uint8Array.prototype.toBase64;
const decode = Uint8Array.fromBase64;
delete Uint8Array.prototype.toBase64;
delete Uint8Array.fromBase64;
let fallback;
try {
fallback = await run();
} finally {
if (encode) Uint8Array.prototype.toBase64 = encode;
if (decode) Uint8Array.fromBase64 = decode;
}
for (const result of [native, fallback]) {
assert.deepEqual(result.seen.map((input) => input[0].data), inputs.map(reference));
assert.equal(result.words.length, inputs.length * 3);
}
});
test("a worker lives only while a delivery runs it", async () => {
const agent = createAgent({ durability: await durabilityFor() });
const internals = agent[Symbol.for("libfx.durableInternals")];
+348 -11
View File
@@ -5,7 +5,8 @@
// node --experimental-wasm-jspi sdk/tests/test-durable.mjs [memory|local] [native|wasm]
import { strict as assert } from "node:assert";
import { spawn } from "node:child_process";
import { mkdtemp, readFile, rm } from "node:fs/promises";
import { createHash } from "node:crypto";
import { mkdtemp, readdir, readFile, rm } from "node:fs/promises";
import { createServer } from "node:http";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
@@ -61,6 +62,8 @@ const crashSteer = (kind) => (kind === "image"
? [{ type: "text", text: "use lookup with this" }, { type: "image", data: onePixelPng, mimeType: "image/png" }]
: "use lookup");
const resumeNotice = "Resuming from unexpected session interruption.";
// The user message the gateway adds after a tool result's images.
const toolImageFollowUp = "Attached image(s) from the tool result.";
const textOf = (message) => typeof message.content === "string"
? message.content
@@ -81,7 +84,7 @@ const answer = (text) => [{ type: "text-delta", id: "answer", delta: text }, fin
// lookup call's input a little at a time for a few seconds; anything else is
// echoed.
function framesFor(prompt) {
const users = userTexts(prompt).filter((text) => text !== resumeNotice);
const users = userTexts(prompt).filter((text) => text !== resumeNotice && text !== toolImageFollowUp);
const text = users.at(-1) ?? "";
const results = toolResults(prompt);
const turnResults = results.filter((part) => String(part.toolCallId).startsWith(`call-${users.length}-`));
@@ -121,8 +124,11 @@ const server = createServer((request, response) => {
request.on("end", () => {
if (request.method === "GET") {
response.writeHead(200, { "content-type": "application/json" });
// It takes images, for the steers that carry one.
response.end(JSON.stringify({ object: "list", data: [{ id: "durable/model", type: "language", tags: ["vision", "file-input"] }] }));
response.end(JSON.stringify({ object: "list", data: [
// It takes images, for the steers that carry one.
{ id: "durable/model", type: "language", tags: ["vision", "file-input"] },
{ id: "durable/vision-model", type: "language", tags: ["tool-use", "vision", "file-input"] },
] }));
return;
}
const prompt = JSON.parse(body).prompt;
@@ -1671,19 +1677,332 @@ test("an option the backend rejects fails the first turn once", async () => {
await agent.close();
});
test("a prompt the engine refuses ends its own turn once, and the session goes on", async () => {
const agent = createFxAgent(agentOptions(await durabilityFor()));
// A 1x1 PNG, as bytes, for the image tests below.
const pngBase64 = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+jP0cAAAAASUVORK5CYII=";
const pngBytes = () => Uint8Array.from(atob(pngBase64), (char) => char.charCodeAt(0));
test("a prompt the engine would refuse throws, stores nothing, and the session goes on", async () => {
const agent = createFxAgent(agentOptions(await durabilityFor(), { model: "durable/vision-model" }));
const session = agent.session();
const empty = await session.prompt("").result;
assert.equal(empty.stopReason, "error");
const malformed = await session.prompt([{ type: "text" }]).result;
assert.equal(malformed.stopReason, "error");
const { text, result } = await collect(session.prompt("hello again"));
const image = (data, mimeType = "image/png") => ({ type: "image", data, mimeType });
for (const [input, refusal] of [
["", /prompt is empty/],
[[{ type: "text" }], /text prompt block 0 requires text/],
[[{ type: "text", text: "look" }, image("not base64!")], /image prompt block 1 requires canonical base64 data/],
[[{ type: "text", text: "look" }, image(pngBytes())], /a durable prompt carries image data as a base64 string/],
[[{ type: "text", text: "look" }, image(pngBase64, "image/jpeg")], /image prompt block 1 holds image\/png data, not image\/jpeg/],
[[{ type: "text", text: "look" }, image(pngBase64, "image/gif")], /image prompt block 1 holds image\/png data, not image\/gif/],
[[{ type: "text", text: "look" }, image(Buffer.from([1, 2, 3, 4]).toString("base64"))], /image prompt block 1 is not a PNG, JPEG, GIF or WebP image/],
[[{ type: "text", text: "look" }, image(pngBase64, "image/bmp")], /image prompt block 1 mimeType must be image\/png, image\/jpeg, image\/gif or image\/webp/],
[[{ type: "text", text: "look" }, ...Array.from({ length: 9 }, () => image(pngBase64))], /prompt cannot contain more than 8 images/],
]) {
assert.throws(() => session.prompt(input, { messageId: "refused" }), refusal);
}
// Nothing was stored under that id, so a prompt with it runs.
const { text, result } = await collect(session.prompt("hello again", { messageId: "refused" }));
assert.equal(result.stopReason, "end_turn");
assert.equal(text, "echo: hello again");
await agent.close();
});
test("a durable prompt's images reach the model as the base64 it was given", async () => {
const png = pngBase64;
const agent = createFxAgent(agentOptions(await durabilityFor(), { model: "durable/vision-model" }));
const { text, result } = await collect(agent.session().prompt([
{ type: "text", text: "look" },
{ type: "image", data: png, mimeType: "image/png" },
{ type: "image", data: png, mimeType: "image/png", sourceRef: "uploads:shot" },
]));
assert.equal(result.stopReason, "end_turn");
// The model hears each image's placeholder alongside its text.
assert.equal(text, "echo: look\n[Image #1]\n[Image #2]");
const files = chats.at(-1).prompt
.filter((message) => message.role === "user" && Array.isArray(message.content))
.flatMap((message) => message.content)
.filter((part) => part.type === "file");
const file = { type: "file", mediaType: "image/png", data: { type: "data", data: png } };
assert.deepEqual(files, [file, file]);
await agent.close();
});
test("an image read that fails while the engine opens runs the turn again instead of failing it", async () => {
const base = await durabilityFor();
// memory() hands out its sessions directly, so one image read can fail.
if (typeof base.create !== "function") return;
const fault = { failReads: 0 };
const durability = {
...base,
create() {
const backend = base.create();
return {
...backend,
async session(id) {
const log = await backend.session(id);
return {
...log,
images: {
put: (name, bytes) => log.images.put(name, bytes),
get: async (name) => {
if (fault.failReads > 0) {
fault.failReads -= 1;
throw new Error("the image store did not answer");
}
return log.images.get(name);
},
},
};
},
};
},
};
const png = pngBase64;
const first = createFxAgent(agentOptions(durability, { model: "durable/vision-model" }));
const session = first.session();
assert.equal((await session.prompt([{ type: "text", text: "look" }, { type: "image", data: png, mimeType: "image/png" }]).result).stopReason, "end_turn");
await first.close();
// A new agent's engine reads the session's image as it opens; that read
// fails once.
fault.failReads = 1;
const second = createFxAgent(agentOptions(durability, { model: "durable/vision-model" }));
const { result } = await collect(second.session(session.id).prompt("and now"));
assert.equal(result.stopReason, "end_turn");
assert.equal(fault.failReads, 0, "the failed read happened");
const files = chats.at(-1).prompt
.filter((message) => message.role === "user" && Array.isArray(message.content))
.flatMap((message) => message.content)
.filter((part) => part.type === "file");
assert.deepEqual(files.map((part) => part.data?.data), [png]);
await second.close();
});
test("an engine given an image the log names refuses it, so a worker that can't read names fails the turn", async () => {
// The log holds a stored image as `data: { sha256 }`, which an engine that
// predates names can't read, with or without a sourceRef: such a worker
// fails the turn instead of running it without the image.
const { durability: _durability, tools: _tools, ...engineOptions } = agentOptions(null, { model: "durable/vision-model" });
const engine = await createFxEngine(engineOptions);
const named = { type: "image", mimeType: "image/png", data: { sha256: "a".repeat(64) } };
for (const block of [named, { ...named, sourceRef: "uploads:shot" }]) {
assert.throws(() => engine.prompt([{ type: "text", text: "look" }, block]), /requires base64 data, bytes, or a Blob/);
}
await engine.close();
});
test("a durable prompt checks the copy its log holds, so a getter can't pass one image and store another", async () => {
const agent = createFxAgent(agentOptions(await durabilityFor(), { model: "durable/vision-model" }));
let reads = 0;
const block = { type: "image", mimeType: "image/png" };
Object.defineProperty(block, "data", { enumerable: true, get: () => (reads++ === 0 ? pngBase64 : Buffer.from([1, 2, 3, 4]).toString("base64")) });
const { result } = await collect(agent.session().prompt([{ type: "text", text: "look" }, block]));
assert.equal(result.stopReason, "end_turn");
assert.equal(reads, 1, "the block was read once");
const files = chats.at(-1).prompt.flatMap((message) => Array.isArray(message.content) ? message.content : []).filter((part) => part.type === "file");
assert.deepEqual(files.map((part) => part.data?.data), [pngBase64]);
await agent.close();
});
test("a durable prompt cancelled while its image is stored is never stored, and sends no cancel", async () => {
const base = await durabilityFor();
// memory() hands out its sessions directly, so its image writes can be held.
if (typeof base.create !== "function") return;
let release;
const held = new Promise((resolveHeld) => { release = resolveHeld; });
const durability = {
...base,
create() {
const backend = base.create();
return {
...backend,
async session(id) {
const log = await backend.session(id);
return { ...log, images: { put: async (name, bytes) => { await held; return log.images.put(name, bytes); }, get: (name) => log.images.get(name) } };
},
};
},
};
const agent = createFxAgent(agentOptions(durability, { model: "durable/vision-model" }));
const session = agent.session();
const controller = new AbortController();
const turn = session.prompt([{ type: "text", text: "look" }, { type: "image", data: pngBase64, mimeType: "image/png" }], { messageId: "withdrawn", signal: controller.signal });
await new Promise((resolveWait) => setTimeout(resolveWait, 20));
controller.abort();
// The prompt stops waiting for its image write at once.
await turn.accepted;
assert.equal((await turn.result).stopReason, "cancelled");
release();
// Neither the prompt nor a cancel naming it was stored: one stored would
// answer a prompt with its id from the log, and a cancel would end it.
const { text, result } = await collect(session.prompt("hello again", { messageId: "withdrawn" }));
assert.equal(result.stopReason, "end_turn");
assert.equal(text, "echo: hello again");
await agent.close();
});
test("an engine refuses the same prompts in prompt(), before its core sees them", async () => {
const { durability: _durability, tools: _tools, ...engineOptions } = agentOptions(null, { model: "durable/vision-model" });
const engine = await createFxEngine(engineOptions);
assert.throws(() => engine.prompt(""), /prompt is empty/);
assert.throws(() => engine.prompt([{ type: "text", text: "look" }, { type: "image", data: pngBytes(), mimeType: "image/jpeg" }]), /holds image\/png data, not image\/jpeg/);
await assert.rejects(engine.prompt([{ type: "text", text: "look" }, { type: "image", data: new File([pngBytes()], "shot.jpg", { type: "image/jpeg" }) }]).result, /holds image\/png data, not image\/jpeg/);
const { result } = await collect(engine.prompt("hello"));
assert.equal(result.stopReason, "end_turn");
await engine.close();
});
test("a durable session saves each image once, and a new agent sends it again", async () => {
// A PNG header, then filler, so its bytes are easy to look for.
const header = Uint8Array.from(atob("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+jP0cAAAAASUVORK5CYII="), (char) => char.charCodeAt(0));
const bytes = new Uint8Array(300_000);
bytes.set(header);
for (let index = header.length; index < bytes.length; index += 1) bytes[index] = (index * 31 + 7) & 255;
const base64 = Buffer.from(bytes).toString("base64");
const durability = await durabilityFor();
const holdsImage = (data) => data.includes(Buffer.from(base64.slice(1000, 1100))) || data.includes(Buffer.from(bytes.subarray(1000, 1064)));
const savedData = (entry) => (entry.k === "record" || entry.k === "checkpoint" ? Buffer.from(entry.data, "base64") : Buffer.from(JSON.stringify(entry)));
// A remote World keeps nothing on this machine, so what reaches it is
// watched instead: each entry the log appends, and each stream write.
const sent = { entries: [], imageWrites: 0 };
if (remote) {
const open = durability.world.bind(durability);
durability.world = async () => {
const world = await open();
const create = world.events.create.bind(world.events);
world.events.create = async (runId, event, ...rest) => {
const input = event?.eventData?.input;
if (input instanceof Uint8Array) {
try {
const entry = JSON.parse(Buffer.from(input).toString("utf8"));
if (typeof entry?.k === "string") sent.entries.push(entry);
} catch {}
}
return create(runId, event, ...rest);
};
const write = world.streams.write.bind(world.streams);
world.streams.write = async (runId, stream, data, ...rest) => {
if (data instanceof Uint8Array && Buffer.from(data).equals(Buffer.from(bytes))) sent.imageWrites += 1;
return write(runId, stream, data, ...rest);
};
return world;
};
}
const first = createFxAgent(agentOptions(durability, { model: "durable/vision-model" }));
const session = first.session();
// Records that only name the image stay small, so the second prompt is
// `bulky`: its turn's end saves a checkpoint, which must name the images too.
for (const text of ["look", bulky]) {
const { result } = await collect(session.prompt([{ type: "text", text }, { type: "image", data: base64, mimeType: "image/png" }]));
assert.equal(result.stopReason, "end_turn");
}
// A turn's checkpoint is saved after its end, and close() waits for it.
await first.close();
if (durabilityKind === "memory" || remote) {
// Every entry, records and checkpoints decoded: names only.
const entries = remote ? sent.entries : (await (await durability.create().session(session.id)).readSince("0")).map(({ entry }) => entry);
assert.ok(entries.some((entry) => entry.k === "checkpoint"));
assert.ok(!entries.map(savedData).some(holdsImage), "no entry holds the image");
}
if (remote) {
assert.equal(sent.imageWrites, 1, "the World got the image's bytes once");
} else if (durabilityKind !== "memory") {
const files = [];
for (const entry of await readdir(dirs.at(-1), { withFileTypes: true, recursive: true })) {
if (!entry.isFile()) continue;
try {
files.push(await readFile(join(entry.parentPath ?? entry.path, entry.name)));
} catch (error) {
if (error?.code !== "ENOENT") throw error;
}
}
assert.ok(!files.some((data) => data.includes(Buffer.from(base64.slice(1000, 1100)))), "no file holds the image as base64");
assert.equal(files.filter((data) => data.includes(Buffer.from(bytes.subarray(1000, 1064)))).length, 1, "one file holds the image's bytes");
}
const second = createFxAgent(agentOptions(durabilityKind === "memory" ? durability : await durabilityFor({ dir: dirs.at(-1) }), { model: "durable/vision-model" }));
const { result } = await collect(second.session(session.id).prompt("and now"));
assert.equal(result.stopReason, "end_turn");
const files = chats.at(-1).prompt
.filter((message) => message.role === "user" && Array.isArray(message.content))
.flatMap((message) => message.content)
.filter((part) => part.type === "file");
assert.deepEqual(files.map((part) => part.data?.data), [base64, base64]);
await second.close();
});
// A PNG of 300 KB with filler after its header, so its bytes are easy to
// look for, as base64.
const shotBytes = (() => {
const header = Uint8Array.from(atob(onePixelPng), (char) => char.charCodeAt(0));
const bytes = new Uint8Array(300_000);
bytes.set(header);
for (let index = header.length; index < bytes.length; index += 1) bytes[index] = (index * 17 + 3) & 255;
return bytes;
})();
const shotBase64 = Buffer.from(shotBytes).toString("base64");
const shotName = createHash("sha256").update(shotBytes).digest("hex");
const holdsShot = (data) => data.includes(Buffer.from(shotBase64.slice(1000, 1100))) || data.includes(Buffer.from(shotBytes.subarray(1000, 1064)));
const entryData = (entry) => (entry.k === "record" || entry.k === "checkpoint" ? Buffer.from(entry.data, "base64") : Buffer.from(JSON.stringify(entry)));
// Returns the screenshot as a typed result.
const screenshot = {
name: "screenshot",
description: "Takes a screenshot",
idempotent: true,
inputSchema: { type: "object", properties: { key: { type: "string" } } },
execute: async () => ({ type: "libfx.tool-result", text: "the screen", images: [{ type: "image", data: shotBase64, mimeType: "image/png" }] }),
};
const requestFiles = (prompt) => prompt
.filter((message) => message.role === "user" && Array.isArray(message.content))
.flatMap((message) => message.content)
.filter((part) => part.type === "file")
.map((part) => part.data?.data ?? part.data);
test("a durable session saves a tool's image once, its log only names it, and a new agent sends it again", async () => {
const durability = await durabilityFor();
const withShots = { model: "durable/vision-model", tools: [lookup, send, save, screenshot] };
const first = createFxAgent(agentOptions(durability, withShots));
const session = first.session();
const { result } = await collect(session.prompt("use screenshot"));
assert.equal(result.stopReason, "end_turn");
assert.deepEqual(requestFiles(chats.at(-1).prompt), [shotBase64], "the model saw the screenshot");
await first.close();
if (durabilityKind === "memory") {
const log = await durability.create().session(session.id);
const entries = (await log.readSince("0")).map(({ entry }) => entry);
assert.ok(!entries.map(entryData).some(holdsShot), "no entry holds the screenshot");
assert.ok(Buffer.from(await log.images.get(shotName)).equals(Buffer.from(shotBytes)), "the image store holds it by name");
}
const second = createFxAgent(agentOptions(durabilityKind === "memory" ? durability : await durabilityFor({ dir: dirs.at(-1) }), withShots));
const { result: after } = await collect(second.session(session.id).prompt("and now"));
assert.equal(after.stopReason, "end_turn");
assert.deepEqual(requestFiles(chats.at(-1).prompt), [shotBase64], "a new agent sends the screenshot again");
await second.close();
});
test("a durable steer saves its image once, and its log entry only names it", async () => {
const durability = await durabilityFor();
const agent = createFxAgent(agentOptions(durability, { model: "durable/vision-model" }));
const session = agent.session();
const before = slow.length;
const turn = session.prompt("slow reply");
await turn.accepted;
await until(() => slow.length > before && slow.at(-1).sent > 0, "the reply to stream");
await session.steer([{ type: "text", text: "match the shot" }, { type: "image", data: shotBase64, mimeType: "image/png" }], { messageId: "m-shot" });
await turn.result;
await linesUntil(session, (line) => line.type === "turn_end" && line.messageId === "m-shot", "the steer's turn");
const steered = chats.findLast(({ prompt }) => userTexts(prompt).some((text) => text.includes("match the shot")));
assert.deepEqual(requestFiles(steered.prompt), [shotBase64], "the steer's image reached the model");
await agent.close();
if (durabilityKind === "memory") {
const log = await durability.create().session(session.id);
const entries = (await log.readSince("0")).map(({ entry }) => entry);
const entry = entries.find((item) => item.k === "input" && item.messageId === "m-shot");
assert.ok(entry, "the steer is in the log");
assert.deepEqual(entry.input.find((block) => block.type === "image").data, { sha256: shotName });
assert.ok(!entries.map(entryData).some(holdsShot), "no entry holds the image");
assert.ok(Buffer.from(await log.images.get(shotName)).equals(Buffer.from(shotBytes)), "the image store holds it by name");
}
});
test("the fx harness tells a refused turn from one whose core exited under it", async () => {
const { fxHarness } = await import("../fx-harness.js");
const { coreAnswered, engineInternals } = await import("../fx-sdk.js");
@@ -1840,6 +2159,24 @@ test("a checkpoint restores the conversation in a new agent", async () => {
await restored.close();
});
test("a checkpoint after an image prompt holds the image, and a new agent sends it", async () => {
const png = pngBase64;
const agent = createFxAgent(agentOptions(await durabilityFor(), { model: "durable/vision-model" }));
const look = [{ type: "text", text: "look" }, { type: "image", data: png, mimeType: "image/png" }];
assert.equal((await agent.prompt(look).result).stopReason, "end_turn");
// The session's records name the image; the checkpoint carries its bytes.
const checkpoint = await agent.checkpoint();
await agent.close();
const restored = createFxAgent(agentOptions(await durabilityFor(), { checkpoint, model: "durable/vision-model" }));
assert.equal((await restored.prompt("what was in it").result).stopReason, "end_turn");
const files = chats.at(-1).prompt
.filter((message) => message.role === "user" && Array.isArray(message.content))
.flatMap((message) => message.content)
.filter((part) => part.type === "file");
assert.deepEqual(files.map((part) => part.data?.data), [png]);
await restored.close();
});
// LIBFX_TEST_ONLY runs only the tests whose names contain it.
const only = process.env.LIBFX_TEST_ONLY;
const selected = only ? tests.filter(([name]) => name.includes(only)) : tests;
+590 -5
View File
@@ -19,7 +19,10 @@ const usage = { inputTokens: { total: 1 }, outputTokens: { total: 1 } };
const textOf = (message) => typeof message.content === "string"
? message.content
: message.content.filter((part) => part.type === "text").map((part) => part.text).join("");
const lastUserText = (prompt) => textOf(prompt.filter((message) => message.role === "user").at(-1));
// The user message the gateway adds after a tool result's images.
const toolImageFollowUp = "Attached image(s) from the tool result.";
const isPrompt = (message) => message.role === "user" && textOf(message) !== toolImageFollowUp;
const lastUserText = (prompt) => textOf(prompt.filter(isPrompt).at(-1));
const toolResults = (prompt) => prompt
.flatMap((message) => Array.isArray(message.content) ? message.content : [])
.filter((part) => part.type === "tool-result");
@@ -70,8 +73,13 @@ function persisted(journal) {
// "gather everything" calls bulky bulkySteps times, one call per response.
// Any other prompt is answered with its own text.
const bulkySteps = 24;
// An image prompt's text ends with a marker per image, which the scripts
// ignore.
function scriptText(prompt) {
return lastUserText(prompt).replace(/(\n\[Image #\d+\])+$/u, "");
}
function framesFor(prompt) {
const text = lastUserText(prompt);
const text = scriptText(prompt);
// "keep using the tool" asks for a lookup until three results are in, so an
// agent that ignored a fence would keep calling it.
if (text === "keep using the tool") {
@@ -113,6 +121,35 @@ function framesFor(prompt) {
{ type: "finish", finishReason: { unified: "stop", raw: "stop" }, usage },
];
}
// "album N" shows N album pages at once in one response, then says it saw
// them.
const pages = /^album (\d+)$/u.exec(text);
if (pages) {
if (toolResults(prompt.slice(prompt.findLastIndex(isPrompt) + 1)).length === 0) {
return [
...Array.from({ length: Number(pages[1]) }, (_, page) => ({ type: "tool-call", toolCallId: `call-page-${page}`, toolName: "album", input: { page } })),
{ type: "finish", finishReason: { unified: "tool-calls", raw: "tool-calls" }, usage },
];
}
return [
{ type: "text-delta", id: "answer", delta: "saw the album" },
{ type: "finish", finishReason: { unified: "stop", raw: "stop" }, usage },
];
}
// "screenshot N" takes screenshot N once in its turn, then says it saw it.
const shot = /^screenshot (\d+)$/u.exec(text);
if (shot) {
if (toolResults(prompt.slice(prompt.findLastIndex(isPrompt) + 1)).length === 0) {
return [
{ type: "tool-call", toolCallId: `call-shot-${shot[1]}`, toolName: "screenshot", input: { shot: Number(shot[1]) } },
{ type: "finish", finishReason: { unified: "tool-calls", raw: "tool-calls" }, usage },
];
}
return [
{ type: "text-delta", id: "answer", delta: `saw ${shot[1]}` },
{ type: "finish", finishReason: { unified: "stop", raw: "stop" }, usage },
];
}
if (text === "use the tool" && toolResults(prompt).length === 0) {
return [
{ type: "tool-call", toolCallId: "call-1", toolName: "lookup", input: { key: "alpha" } },
@@ -145,7 +182,7 @@ const server = createServer((request, response) => {
Object.defineProperty(prompt, "toolChoice", { value: toolChoice?.type ?? null });
requests.push(prompt);
sessionHeaders.push([request.headers["x-session-id"], request.headers["x-session-affinity"]]);
if (lastUserText(prompt) === "fail please") {
if (scriptText(prompt) === "fail please") {
response.writeHead(400, { "content-type": "application/json" });
response.end(JSON.stringify({ error: { message: "refused for the test", type: "invalid_request_error" } }));
return;
@@ -178,6 +215,35 @@ const bulky = {
inputSchema: { type: "object", properties: { part: { type: "number" } } },
execute: async ({ part }) => `${part}:${"x".repeat(60_000)}`,
};
// Returns screenshot `shot` as a typed result: a PNG distinct per shot.
const screenshot = {
name: "screenshot",
description: "Takes a screenshot",
inputSchema: { type: "object", properties: { shot: { type: "number" } }, required: ["shot"] },
execute: async ({ shot }) => ({
type: "libfx.tool-result",
text: `shot ${shot}`,
images: [{ type: "image", data: base64Of(screenshotOf(shot)), mimeType: "image/png" }],
}),
};
// Returns album page `page`: a result's 8 images, small and distinct from
// every other page's.
const albumImage = (index) => {
const bytes = new Uint8Array(71);
bytes.set(Buffer.from(pngData, "base64"));
new DataView(bytes.buffer).setUint32(67, index);
return bytes;
};
const album = {
name: "album",
description: "Shows an album page",
inputSchema: { type: "object", properties: { page: { type: "number" } }, required: ["page"] },
execute: async ({ page }) => ({
type: "libfx.tool-result",
text: `page ${page}`,
images: Array.from({ length: 8 }, (_, index) => ({ type: "image", data: base64Of(albumImage(page * 8 + index)), mimeType: "image/png" })),
}),
};
const options = (backend, journal, extra = {}) => ({
backend,
nativeAddon: addon,
@@ -805,7 +871,7 @@ test("a journal that does not fold is refused", async () => {
const invalid = (pattern) => (error) => pattern.test(error.message) && error.code === "FX_JOURNAL_INVALID";
const duplicated = eventJournal([...journal.events, journal.events.at(-1)]);
await assert.rejects(createFxEngine(options(targetBackend, duplicated)), invalid(/libfx journal events are out of order/));
const newer = eventJournal([{ ...journal.events[0], v: 3 }]);
const newer = eventJournal([{ ...journal.events[0], v: 4 }]);
await assert.rejects(createFxEngine(options(targetBackend, newer)), /libfx journal was written by a newer fx/);
const garbled = eventJournal([{ v: 1, seq: 1, turn: 1, type: "turn_committed", data: { kind: "nope" } }]);
await assert.rejects(createFxEngine(options(targetBackend, garbled)), invalid(/Invalid libfx journal/));
@@ -1181,7 +1247,7 @@ test("a turn left open resumes under the tools the new agent has", async () => {
});
test("a journal from a newer libfx is refused", async () => {
const future = eventJournal([{ v: 3, seq: 1, turn: 1, type: "turn_progress", data: {} }]);
const future = eventJournal([{ v: 4, seq: 1, turn: 1, type: "turn_progress", data: {} }]);
const newer = (error) => error instanceof FxJournalVersionError && error.code === "FX_JOURNAL_VERSION";
await assert.rejects(createFxEngine(options(sourceBackend, future)), newer);
const record = (format) => ({
@@ -1454,6 +1520,487 @@ test("an image prompt after a crashed image turn gets the next image id", async
assert.match(lastUserText(requests.at(-1)), /\[Image #2\]$/);
});
// Images for the image store's tests: a PNG header, then filler that differs
// per salt, so each image has its own name.
const visionCatalog = { modelCatalog: [{ id: "journal/model", type: "language", tags: ["tool-use", "vision", "file-input"] }] };
const pngOf = (size, salt) => {
const bytes = new Uint8Array(size);
bytes.set(Buffer.from(pngData, "base64"));
for (let index = 67; index < size; index += 1) bytes[index] = (index * 31 + salt) & 255;
return bytes;
};
const base64Of = (bytes) => Buffer.from(bytes).toString("base64");
const imageBlock = (bytes) => ({ type: "image", data: bytes, mimeType: "image/png" });
const nameOf = async (bytes) => Buffer.from(await crypto.subtle.digest("SHA-256", bytes)).toString("hex");
// Whether saved bytes hold any of an image, raw or as base64.
const holdsImage = (data, bytes) => Buffer.from(data).includes(Buffer.from(bytes.subarray(200, 264))) ||
Buffer.from(data).includes(Buffer.from(base64Of(bytes).slice(400, 480)));
const savedData = (store) => [...store.records.map((record) => record.data), ...(store.checkpoint ? [store.checkpoint.data] : [])];
const requestImages = (prompt) => prompt
.flatMap((message) => Array.isArray(message.content) ? message.content : [])
.filter((part) => part.type === "file")
.map((part) => part.data?.data ?? part.data);
// libfx's own key for a persistence's image store, which durable sessions
// give the engines they run.
const persistenceImages = Symbol.for("libfx.persistenceImages");
// `persistence` with an image store that keeps each image once, by name, in
// `images`, as a durable session gives its engines.
const withImageStore = (persistence, images = new Map()) => Object.assign(persistence, {
images,
[persistenceImages]: {
put: async (name, bytes) => { if (!images.has(name)) images.set(name, bytes.slice()); },
get: async (name) => images.get(name)?.slice(),
},
});
const createImagePersistence = () => withImageStore(createMemoryPersistence());
test("a store keeps each prompt image once, and records and checkpoints only name it", async () => {
const store = createImagePersistence();
const first = pngOf(300_000, 1);
const second = pngOf(200_000, 2);
const source = await createFxEngine(options(sourceBackend, undefined, { ...visionCatalog, persistence: store, checkpointAfterBytes: 0 }));
await run(source, [{ type: "text", text: "one" }, imageBlock(first)]);
await run(source, [{ type: "text", text: "two" }, imageBlock(second), imageBlock(first)]);
// A turn without images: its events name none.
await run(source, "no image here");
await source.close();
assert.equal(store.images.size, 2);
assert.deepEqual([...store.images.keys()].sort(), [await nameOf(first), await nameOf(second)].sort());
assert.ok(store.checkpoint, "a checkpoint was saved");
for (const data of savedData(store)) {
assert.ok(!holdsImage(data, first) && !holdsImage(data, second), "saved data holds no image bytes");
}
// Events that name an image are version 3. A progress that extends another
// (`base`) is version 2 and leaves out the user's turn, so it names none;
// every other event stays version 1.
const versions = storedEvents(store).map((event) => [JSON.stringify(event).includes("inline_ref"), event.v, event.base !== undefined || event.type === "tool_result"]);
assert.ok(versions.some(([named]) => named), "some events name images");
for (const [named, version, delta] of versions) {
assert.ok(!(named && delta), "a delta names no image");
assert.equal(version, named ? 3 : delta ? 2 : 1);
}
assert.ok(versions.some(([named]) => !named), "the text-only turn's events name none");
const target = await createFxEngine(options(targetBackend, undefined, { ...visionCatalog, persistence: store }));
requests.length = 0;
const { result } = await run(target, "four");
await target.close();
assert.equal(result.stopReason, "end_turn");
assert.deepEqual(requestImages(requests.at(-1)), [base64Of(first), base64Of(second), base64Of(first)]);
});
test("a session whose store lost an image is refused, naming it", async () => {
const store = createImagePersistence();
const kept = pngOf(50_000, 4);
const lost = pngOf(60_000, 5);
const source = await createFxEngine(options(sourceBackend, undefined, { ...visionCatalog, persistence: store, checkpointAfterBytes: 0 }));
await run(source, [{ type: "text", text: "one" }, imageBlock(kept), imageBlock(lost)]);
await source.close();
const name = await nameOf(lost);
store.images.delete(name);
await assert.rejects(
createFxEngine(options(targetBackend, undefined, { ...visionCatalog, persistence: store })),
(error) => error.code === "FX_JOURNAL_IMAGE_MISSING" && error.image === name && error.message.includes(name),
);
});
test("a persistence without an image store keeps writing images inline", async () => {
const image = pngOf(20_000, 6);
const journal = eventJournal();
const agent = await createFxEngine(options(sourceBackend, journal, visionCatalog));
await run(agent, [{ type: "text", text: "inline" }, imageBlock(image)]);
await agent.close();
// No event is the version that names stored images; a progress that
// extends another is version 2, every other event version 1.
for (const event of journal.events) assert.equal(event.v, event.base !== undefined || event.type === "tool_result" ? 2 : 1);
assert.ok(JSON.stringify(journal.events).includes(base64Of(image)));
assert.ok(!JSON.stringify(journal.events).includes("inline_ref"));
});
test("an image turn that crashed after a delta reopens with its image", async () => {
// The turn's first progress names the image. The one after the tool step
// extends it and leaves out the user's turn, so a crash right after that
// one brings the image back from the first.
const image = pngOf(50_000, 8);
const images = new Map();
const journal = eventJournal();
const source = await createFxEngine(options(sourceBackend, undefined, { ...visionCatalog, persistence: withImageStore(persisted(journal), images) }));
assert.equal((await run(source, [{ type: "text", text: "use the tool" }, imageBlock(image)])).text, "tool said beta");
await source.close();
const progress = journal.events.filter((event) => event.type === "turn_progress");
assert.equal(progress[0].v, 3, "the first progress names the image");
const stepped = progress.find((event) => event.steps === 1);
assert.ok(stepped?.base !== undefined && stepped.v === 2, "the progress after the tool extends it");
assert.ok(!JSON.stringify(stepped).includes("inline_ref"), "and names no image");
const survived = journal.events.slice(0, journal.events.indexOf(stepped) + 1);
const resumed = await createFxEngine(options(targetBackend, undefined, { ...visionCatalog, persistence: withImageStore(persisted(eventJournal(survived)), images) }));
requests.length = 0;
const turn = resumed.resume();
for await (const _ of turn) {}
assert.equal((await turn.result).stopReason, "end_turn");
await resumed.close();
assert.deepEqual(requestImages(requests[0]), [base64Of(image)], "the image came back from the first progress");
assert.equal(toolResults(requests[0]).length, 1, "the step came back from the delta");
});
// A smaller image window than the core's, so a test reaches its edge.
const imageWindow = Symbol.for("libfx.imageWindow");
const droppedNote = /\[Image #1 is no longer in this session, which keeps only its newest images\.\]/;
for (const [label, limits] of [["images", { images: 2 }], ["bytes", { bytes: 2 * 40_000 }]]) {
test(`a session keeps its newest images within its window of ${label}, and an older one leaves as a note`, async () => {
const store = createImagePersistence();
const pictures = [pngOf(40_000, 11), pngOf(40_000, 12), pngOf(40_000, 13)];
const window = { [imageWindow]: limits };
const source = await createFxEngine(options(sourceBackend, undefined, { ...visionCatalog, persistence: store, checkpointAfterBytes: 0, ...window }));
for (const [index, picture] of pictures.entries()) {
await run(source, [{ type: "text", text: `look ${index}` }, imageBlock(picture)]);
}
// The third turn's request still carries every image; once it entered the
// history, its image pushed the first out, and the next request carries
// the two newest and a note where the first was.
assert.deepEqual(requestImages(requests.at(-1)), pictures.map(base64Of));
await run(source, "and then?");
assert.deepEqual(requestImages(requests.at(-1)), [base64Of(pictures[1]), base64Of(pictures[2])]);
assert.match(JSON.stringify(requests.at(-1)), droppedNote);
await source.close();
assert.ok(savedData(store).some((data) => Buffer.from(data).includes("\"dropped\":true")), "the checkpoint keeps the first image's name alone");
// A store that lost the first image still opens the session, which reads
// only the images its window keeps.
store.images.delete(await nameOf(pictures[0]));
const read = [];
const { get } = store[persistenceImages];
store[persistenceImages].get = async (name) => {
read.push(name);
return get(name);
};
const target = await createFxEngine(options(targetBackend, undefined, { ...visionCatalog, persistence: store, ...window }));
await run(target, "and now?");
await target.close();
assert.deepEqual(read.sort(), [await nameOf(pictures[1]), await nameOf(pictures[2])].sort());
assert.deepEqual(requestImages(requests.at(-1)), [base64Of(pictures[1]), base64Of(pictures[2])]);
assert.match(JSON.stringify(requests.at(-1)), droppedNote);
});
}
test("a turn that fails before it enters the history drops no image", async () => {
const store = createImagePersistence();
const pictures = [pngOf(40_000, 21), pngOf(40_000, 22), pngOf(40_000, 23)];
const agent = await createFxEngine(options(sourceBackend, undefined, { ...visionCatalog, persistence: store, [imageWindow]: { images: 2 } }));
await run(agent, [{ type: "text", text: "look 0" }, imageBlock(pictures[0])]);
await run(agent, [{ type: "text", text: "look 1" }, imageBlock(pictures[1])]);
const failed = await run(agent, [{ type: "text", text: "fail please" }, imageBlock(pictures[2])]);
assert.notEqual(failed.result.stopReason, "end_turn");
// The failed turn is not in the history, so the window still holds both.
await run(agent, "and then?");
assert.deepEqual(requestImages(requests.at(-1)), [base64Of(pictures[0]), base64Of(pictures[1])]);
assert.doesNotMatch(JSON.stringify(requests.at(-1)), /no longer in this session/);
await agent.close();
});
test("a turn that crashed with a full image window reopens with all of its images", async () => {
// An open stages the whole window and the open turn's images, which the
// window leaves out: one prompt's, up to 8 of them.
const tiny = (n) => {
const bytes = pngOf(80, 0);
new DataView(bytes.buffer).setUint32(70, n);
return bytes;
};
let made = 0;
const prompt = (text) => [{ type: "text", text }, ...Array.from({ length: 8 }, () => imageBlock(tiny(made++)))];
const images = new Map();
const journal = eventJournal();
const source = await createFxEngine(options(sourceBackend, undefined, { ...visionCatalog, persistence: withImageStore(persisted(journal), images) }));
for (let turn = 0; turn < 4096 / 8; turn += 1) await run(source, prompt(`look ${turn}`));
const open = prompt("use the tool");
assert.equal((await run(source, open)).text, "tool said beta");
await source.close();
assert.equal(images.size, 4096 + 8);
// A crash after the last turn's tool step leaves it open, with the window
// full of the turns before it.
const stepped = journal.events.findLast((event) => event.type === "turn_progress" && event.steps === 1);
const survived = journal.events.slice(0, journal.events.indexOf(stepped) + 1);
const read = new Set();
const reading = withImageStore(persisted(eventJournal(survived)), images);
const { get } = reading[persistenceImages];
reading[persistenceImages].get = async (name) => { read.add(name); return get(name); };
const resumed = await createFxEngine(options(targetBackend, undefined, { ...visionCatalog, persistence: reading }));
requests.length = 0;
const turn = resumed.resume();
for await (const _ of turn) {}
assert.equal((await turn.result).stopReason, "end_turn");
await resumed.close();
assert.equal(read.size, 4096 + 8, "the open read the window and the open turn's images");
const sent = requestImages(requests[0]);
for (const block of open.slice(1)) assert.ok(sent.includes(base64Of(block.data)), "the open turn kept its images");
});
test("a failed image write fails the prompt before it is sent", async () => {
const store = createImagePersistence();
store[persistenceImages].put = async () => { throw new Error("disk full"); };
const agent = await createFxEngine(options(sourceBackend, undefined, { ...visionCatalog, persistence: store }));
requests.length = 0;
const turn = agent.prompt([{ type: "text", text: "look" }, imageBlock(pngOf(1_000, 7))]);
// The store's own error.
await assert.rejects(turn.result, (error) => error.message === "disk full");
assert.equal(requests.length, 0, "the prompt never reached the model");
assert.equal(storedEvents(store).length, 0, "nothing was recorded");
const { result } = await run(agent, "text only");
assert.equal(result.stopReason, "end_turn");
await agent.close();
});
test("a session saved inline opens behind an image store, and only new images are stored", async () => {
const old = pngOf(40_000, 8);
const fresh = pngOf(30_000, 9);
const journal = eventJournal();
const before = await createFxEngine(options(sourceBackend, journal, visionCatalog));
await run(before, [{ type: "text", text: "old" }, imageBlock(old)]);
await before.close();
const images = new Map();
const upgraded = withImageStore(persisted(journal), images);
const after = await createFxEngine(options(targetBackend, undefined, { ...visionCatalog, persistence: upgraded }));
requests.length = 0;
await run(after, [{ type: "text", text: "new" }, imageBlock(fresh)]);
await after.close();
assert.deepEqual(requestImages(requests.at(-1)), [base64Of(old), base64Of(fresh)]);
assert.deepEqual([...images.keys()], [await nameOf(fresh)]);
const again = await createFxEngine(options(sourceBackend, undefined, { ...visionCatalog, persistence: upgraded }));
requests.length = 0;
await run(again, "again");
await again.close();
assert.deepEqual(requestImages(requests.at(-1)), [base64Of(old), base64Of(fresh)]);
});
test("images too large for a checkpoint are named instead", async () => {
// Two 2.1 MiB images: more than one checkpoint holds raw.
const images = [pngOf(2_200_000, 10), pngOf(2_200_000, 11)];
const prompt = [{ type: "text", text: "large" }, ...images.map(imageBlock)];
// Without an image store, the core refuses the checkpoint.
const inline = createMemoryPersistence();
const without = await createFxEngine(options(sourceBackend, undefined, { ...visionCatalog, persistence: inline, checkpointAfterBytes: 0 }));
await run(without, prompt);
await without.close();
assert.equal(inline.checkpoint, null);
const store = createImagePersistence();
const saved = [];
const named = await createFxEngine(options(sourceBackend, undefined, {
...visionCatalog, persistence: store, checkpointAfterBytes: 0,
onEvent: (event) => { if (event.type === "checkpoint.save") saved.push(event.type); },
}));
await run(named, prompt);
await named.close();
assert.deepEqual(saved, ["checkpoint.save"]);
assert.ok(store.checkpoint.data.byteLength < 64 * 1024, `checkpoint is ${store.checkpoint.data.byteLength} bytes`);
const reopened = await createFxEngine(options(targetBackend, undefined, { ...visionCatalog, persistence: store }));
requests.length = 0;
const { result } = await run(reopened, "and now");
await reopened.close();
assert.equal(result.stopReason, "end_turn");
assert.deepEqual(requestImages(requests.at(-1)), images.map(base64Of));
});
test("a session holds more images than a checkpoint could, turn after turn", async () => {
const store = createImagePersistence();
const images = Array.from({ length: 5 }, (_, index) => pngOf(1024 * 1024, 20 + index));
let saves = 0;
const countSaves = { onEvent: (event) => { if (event.type === "checkpoint.save") saves += 1; } };
for (const [index, image] of images.entries()) {
// Each turn reopens the session, as a new process would.
const agent = await createFxEngine(options(index % 2 ? targetBackend : sourceBackend, undefined, { ...visionCatalog, ...countSaves, persistence: store, checkpointAfterBytes: 0 }));
const { result } = await run(agent, [{ type: "text", text: `turn ${index}` }, imageBlock(image)]);
await agent.close();
assert.equal(result.stopReason, "end_turn");
}
assert.equal(saves, images.length, "every turn saved a checkpoint");
const last = await createFxEngine(options(targetBackend, undefined, { ...visionCatalog, persistence: store }));
requests.length = 0;
await run(last, "all of them");
await last.close();
assert.deepEqual(requestImages(requests.at(-1)), images.map(base64Of));
});
const screenshotOf = (shot) => pngOf(300_000, 100 + shot);
const withScreenshots = { ...visionCatalog, tools: [lookup, screenshot] };
const toolDroppedNote = /\[1 image from this result is no longer in this session, which keeps only its newest images\.\]/;
test("a store keeps each tool image once, and records and checkpoints only name it", async () => {
const store = createImagePersistence();
const source = await createFxEngine(options(sourceBackend, undefined, { ...withScreenshots, persistence: store, checkpointAfterBytes: 0 }));
for (const shot of [1, 2, 3]) assert.equal((await run(source, `screenshot ${shot}`)).text, `saw ${shot}`);
await source.close();
const shots = [1, 2, 3].map(screenshotOf);
assert.deepEqual([...store.images.keys()].sort(), (await Promise.all(shots.map(nameOf))).sort());
assert.ok(store.checkpoint, "a checkpoint was saved");
for (const data of savedData(store)) assert.ok(shots.every((bytes) => !holdsImage(data, bytes)), "saved data holds no tool image bytes");
// An event that names an image is version 3, a progress that extends
// another and names a new tool image too.
const events = storedEvents(store);
assert.ok(events.some((event) => event.base !== undefined && JSON.stringify(event).includes("inline_ref")), "a delta names its tool image");
for (const event of events) {
const named = JSON.stringify(event).includes("inline_ref");
assert.equal(event.v, named ? 3 : event.base !== undefined || event.type === "tool_result" ? 2 : 1);
}
const target = await createFxEngine(options(targetBackend, undefined, { ...withScreenshots, persistence: store }));
requests.length = 0;
const { result } = await run(target, "after the shots");
await target.close();
assert.equal(result.stopReason, "end_turn");
assert.deepEqual(requestImages(requests.at(-1)), shots.map(base64Of));
});
test("a tool image leaves the window oldest first, as a note, and stays out after a reopen", async () => {
const store = createImagePersistence();
const window = { [imageWindow]: { images: 2 } };
const shots = [1, 2, 3].map(screenshotOf);
const source = await createFxEngine(options(sourceBackend, undefined, { ...withScreenshots, ...window, persistence: store, checkpointAfterBytes: 0 }));
for (const shot of [1, 2, 3]) await run(source, `screenshot ${shot}`);
requests.length = 0;
await run(source, "what is left");
await source.close();
assert.deepEqual(requestImages(requests.at(-1)), [shots[1], shots[2]].map(base64Of));
assert.match(JSON.stringify(requests.at(-1)), toolDroppedNote);
assert.ok(savedData(store).some((data) => Buffer.from(data).includes("\"dropped\":true")), "the first shot keeps its name alone");
const target = await createFxEngine(options(targetBackend, undefined, { ...withScreenshots, ...window, persistence: store }));
requests.length = 0;
await run(target, "and after reopening");
await target.close();
assert.deepEqual(requestImages(requests.at(-1)), [shots[1], shots[2]].map(base64Of));
assert.match(JSON.stringify(requests.at(-1)), toolDroppedNote);
});
test("a tool image the store refuses stays in the session inline", async () => {
const store = createImagePersistence();
store[persistenceImages].put = async () => { throw new Error("store is full"); };
const agent = await createFxEngine(options(sourceBackend, undefined, { ...withScreenshots, persistence: store }));
requests.length = 0;
const { text, result } = await run(agent, "screenshot 4");
await agent.close();
assert.equal(result.stopReason, "end_turn");
assert.equal(text, "saw 4");
// The tool ran, so its image is not lost: the model saw it, and the
// records hold it inline.
assert.deepEqual(requestImages(requests.at(-1)), [base64Of(screenshotOf(4))]);
assert.ok(savedData(store).some((data) => holdsImage(data, screenshotOf(4))), "the records hold the image inline");
assert.equal(store.images.size, 0);
const reopened = await createFxEngine(options(targetBackend, undefined, { ...withScreenshots, persistence: store }));
requests.length = 0;
await run(reopened, "after the refusal");
await reopened.close();
assert.deepEqual(requestImages(requests.at(-1)), [base64Of(screenshotOf(4))]);
});
test("a turn that crashed after its screenshot's progress reopens with the screenshot", async () => {
const journal = eventJournal();
const images = new Map();
const source = await createFxEngine(options(sourceBackend, undefined, { ...withScreenshots, persistence: withImageStore(persisted(journal), images) }));
assert.equal((await run(source, "screenshot 5")).text, "saw 5");
await source.close();
// The progress after the tool step extends the first, and names the shot.
const stepped = journal.events.find((event) => event.type === "turn_progress" && event.steps === 1);
assert.ok(stepped?.base !== undefined && stepped.v === 3 && JSON.stringify(stepped).includes("inline_ref"), "the delta names the screenshot");
assert.ok(!JSON.stringify(stepped).includes(base64Of(screenshotOf(5)).slice(0, 400)), "and does not hold it");
const survived = journal.events.slice(0, journal.events.indexOf(stepped) + 1);
const resumed = await createFxEngine(options(targetBackend, undefined, { ...withScreenshots, persistence: withImageStore(persisted(eventJournal(survived)), images) }));
requests.length = 0;
const turn = resumed.resume();
for await (const _ of turn) {}
assert.equal((await turn.result).stopReason, "end_turn");
await resumed.close();
assert.deepEqual(requestImages(requests[0]), [base64Of(screenshotOf(5))], "the screenshot came back from the store");
});
test("a turn that crashed right after its screenshot's result reopens with the screenshot, without running the tool again", async () => {
const journal = eventJournal();
const images = new Map();
let shots = 0;
const counting = { ...screenshot, execute: async (input) => { shots += 1; return screenshot.execute(input); } };
const withCounting = { ...visionCatalog, tools: [lookup, counting] };
const source = await createFxEngine(options(sourceBackend, undefined, { ...withCounting, persistence: withImageStore(persisted(journal), images) }));
assert.equal((await run(source, "screenshot 9")).text, "saw 9");
await source.close();
// The result names the screenshot as the call finished, before any
// progress holds the step.
const result = journal.events.find((event) => event.type === "tool_result");
assert.ok(result?.v === 3 && result.data.images?.[0]?.inline_ref === await nameOf(screenshotOf(9)), "the result names the screenshot");
assert.ok(!JSON.stringify(result).includes(base64Of(screenshotOf(9)).slice(0, 400)), "and does not hold it");
const survived = journal.events.slice(0, journal.events.indexOf(result) + 1);
assert.ok(!survived.some((event) => event.type === "turn_progress" && event.steps === 1), "no progress holds the step yet");
const resumed = await createFxEngine(options(targetBackend, undefined, { ...withCounting, persistence: withImageStore(persisted(eventJournal(survived)), images) }));
requests.length = 0;
const turn = resumed.resume();
for await (const _ of turn) {}
assert.equal((await turn.result).stopReason, "end_turn");
await resumed.close();
assert.equal(shots, 1, "the tool ran once");
assert.deepEqual(requestImages(requests[0]), [base64Of(screenshotOf(9))], "the screenshot came back from the store");
});
test("a tool image the store refused leaves the window like any other, as a note", async () => {
const store = createImagePersistence();
const refusedName = await nameOf(screenshotOf(6));
const { put } = store[persistenceImages];
store[persistenceImages].put = async (name, bytes) => {
if (name === refusedName) throw new Error("store is full");
return put(name, bytes);
};
const window = { [imageWindow]: { images: 2 } };
const source = await createFxEngine(options(sourceBackend, undefined, { ...withScreenshots, ...window, persistence: store, checkpointAfterBytes: 0 }));
for (const shot of [6, 7, 8]) assert.equal((await run(source, `screenshot ${shot}`)).text, `saw ${shot}`);
requests.length = 0;
await run(source, "what is left");
await source.close();
// The refused shot, held inline, is the oldest image: it leaves first.
assert.deepEqual(requestImages(requests.at(-1)), [7, 8].map((shot) => base64Of(screenshotOf(shot))));
assert.match(JSON.stringify(requests.at(-1)), toolDroppedNote);
assert.ok(!holdsImage(store.checkpoint.data, screenshotOf(6)), "the checkpoint no longer holds it");
const target = await createFxEngine(options(targetBackend, undefined, { ...withScreenshots, ...window, persistence: store }));
requests.length = 0;
await run(target, "and after reopening");
await target.close();
assert.deepEqual(requestImages(requests.at(-1)), [7, 8].map((shot) => base64Of(screenshotOf(shot))));
assert.match(JSON.stringify(requests.at(-1)), toolDroppedNote);
});
test("a turn whose tools returned more images than a window and a prompt hold reopens with all of them", async () => {
// 514 pages of 8: 4,112 images, more than the 4,096 a session keeps and
// the 8 of one prompt, all an open stages unless it reports more missing.
const pages = 514;
const journal = eventJournal();
const images = new Map();
let staged = 0;
const withImages = (persistence) => {
const counted = withImageStore(persistence, images);
const { get } = counted[persistenceImages];
counted[persistenceImages].get = async (name) => { staged += 1; return get(name); };
return counted;
};
const withAlbum = { ...visionCatalog, tools: [album] };
const source = await createFxEngine(options(sourceBackend, undefined, { ...withAlbum, persistence: withImages(persisted(journal)) }));
assert.equal((await run(source, `album ${pages}`)).text, "saw the album");
await source.close();
assert.equal(images.size, pages * 8);
const stepped = journal.events.find((event) => event.type === "turn_progress" && event.steps === 1);
assert.ok(stepped, "a progress holds the album's step");
const survived = journal.events.slice(0, journal.events.indexOf(stepped) + 1);
const resumed = await createFxEngine(options(targetBackend, undefined, { ...withAlbum, persistence: withImages(persisted(eventJournal(survived))) }));
assert.equal(staged, pages * 8, "the open staged every image of the open turn");
const turn = resumed.resume();
for await (const _ of turn) {}
assert.equal((await turn.result).stopReason, "end_turn");
await resumed.close();
});
test("a journal past the turn limit is refused by name", async () => {
const journal = eventJournal();
const agent = await createFxEngine(options(sourceBackend, journal));
@@ -1619,6 +2166,44 @@ test("a tool receives the model's call id as its execution id, and its session's
await agent.close();
assert.deepEqual(seen, [{ keys: ["executionId", "sessionId", "signal"], executionId: "call-1", sessionId, signal: true }]);
});
test("a yielded turn resumed in place after its checkpoint reopens from any cut", async () => {
// The store keeps a copy of itself after each append, as a crash there
// would leave it.
const store = createMemoryPersistence();
const states = [];
const append = store.append.bind(store);
store.append = async (input) => {
const out = await append(input);
states.push({
records: store.records.map((record) => ({ ...record, data: record.data.slice() })),
checkpoint: store.checkpoint && { through: store.checkpoint.through, data: store.checkpoint.data.slice() },
});
return out;
};
const agent = await createFxEngine(options(sourceBackend, undefined, { persistence: store, checkpointAfterBytes: 0 }));
const first = agent.prompt("use the tool", { yieldAt: Date.now() });
for await (const _ of first) {}
assert.equal((await first.result).stopReason, "yielded");
for (let wait = 0; wait < 100 && !store.checkpoint; wait += 1) await new Promise((resolve) => setTimeout(resolve, 20));
assert.ok(store.checkpoint, "a checkpoint holding the open turn was saved at the yield");
const before = states.length;
const turn = agent.resume();
for await (const _ of turn) {}
assert.equal((await turn.result).stopReason, "end_turn");
await agent.close();
const cuts = states.slice(before).filter((state) => state.checkpoint);
assert.ok(cuts.length > 0, "the resumed turn appended after the checkpoint");
for (const [index, state] of cuts.entries()) {
const copy = createMemoryPersistence();
copy.records.push(...state.records);
await copy.saveCheckpoint(state.checkpoint);
const reopened = await createFxEngine(options(targetBackend, undefined, { persistence: copy })).catch((error) => {
throw new Error(`the cut after resumed append ${index + 1} (checkpoint through ${state.checkpoint.through}, ${state.records.length} records) did not reopen: ${error.code}`);
});
await reopened.close();
}
});
try {
for (const { name, body } of cases) {
await body();
+20 -4
View File
@@ -51,6 +51,18 @@ try {
{ type: "tool-call", toolCallId: "images1", toolName: "screenshots", input: {} },
{ type: "finish", finishReason: { unified: "tool-calls", raw: "tool-calls" } },
);
if (requests === 3) {
// The bridge carries at most 8 MiB, so this request left out the
// older image instead of failing, with a note, and kept the newer.
assert.ok(init.body.length <= 8 * 1024 * 1024 - 144 * 1024);
const body = Buffer.from(init.body).toString("utf8");
assert.equal(body.split(data).length - 1, 1, "only the newer image stays");
assert.match(body, /not sent to keep the request under the provider's size limit/);
return sse(
{ type: "text-delta", delta: "read the text" },
{ type: "finish", finishReason: { unified: "stop", raw: "stop" } },
);
}
assert.equal(requests, 2);
assert.ok(init.body.length < 8 * 1024 * 1024);
assert.ok(Buffer.from(init.body).toString("base64").length > 8 * 1024 * 1024);
@@ -76,10 +88,14 @@ try {
assert.equal(toolCalls, 1);
assert.equal(requests, 2);
// The retained images plus this prompt exceed the unchanged raw request budget.
await assert.rejects(agent.prompt("x".repeat(2 * 1024 * 1024)).result, /HostStreamBackpressure/);
assert.equal(requests, 2, "an oversized raw request must not reach host fetch");
console.log("native image framing passed: accepted images survive base64 framing; oversized raw requests stay bounded");
// The retained images plus this prompt exceed what the bridge carries: the
// oldest images leave the request, which reaches host fetch within it.
assert.equal((await agent.prompt("x".repeat(2 * 1024 * 1024)).result).stopReason, "end_turn");
assert.equal(requests, 3);
// A request past the bridge even without its images stays bounded.
await assert.rejects(agent.prompt("z".repeat(6.5 * 1024 * 1024)).result, /HostStreamBackpressure/);
assert.equal(requests, 3, "an oversized request must not reach host fetch");
console.log("native image framing passed: accepted images survive base64 framing; requests past the bridge leave out their oldest images");
} finally {
clearTimeout(timer);
await agent?.close();
+1 -1
View File
@@ -27,7 +27,7 @@ const sdkDir = fileURLToPath(new URL("..", import.meta.url));
const fixturesDir = fileURLToPath(new URL("./types/", import.meta.url));
// fx-sdk.js exports these for libfx's own modules; they are not public API.
const wasmInternals = new Set(["coreAnswered", "engineInternals", "journalMarksWanted", "normalizeAgentOptions", "normalizeInstructions", "normalizeModelChoice", "resumeOutputShown"]);
const wasmInternals = new Set(["checkPromptInput", "coreAnswered", "engineInternals", "journalMarksWanted", "normalizeAgentOptions", "normalizeInstructions", "normalizeModelChoice", "persistenceImages", "resumeOutputShown"]);
// Each specifier as a consumer resolves it, the declaration file it must
// reach, and the module whose runtime exports that file must match. node.cjs
+1
View File
@@ -205,6 +205,7 @@ export async function worlds(): Promise<void> {
async write() {},
async writeMulti() {},
async get() { return new ReadableStream<Uint8Array>(); },
async getChunks() { return { data: [], cursor: null, hasMore: false }; },
async getInfo() { return { tailIndex: -1 }; },
},
async queue() { return { messageId: null }; },
+4 -1
View File
@@ -15,9 +15,12 @@ export interface World {
list(params: unknown): unknown;
};
readonly streams: {
write(runId: string, name: string, chunk: string): unknown;
/** A session's view is written as text; each prompt image as one chunk of bytes. */
write(runId: string, name: string, chunk: string | Uint8Array): unknown;
writeMulti?(runId: string, name: string, chunks: string[]): unknown;
get(runId: string, name: string, startIndex?: number): unknown;
/** Read for a session's images: a page of chunks, `{ data: { data }[] }`. */
getChunks(runId: string, name: string, options?: { limit?: number; cursor?: string }): unknown;
getInfo(runId: string, name: string): unknown;
};
queue(queueName: string, message: unknown, options?: unknown): unknown;
+5 -1
View File
@@ -863,7 +863,11 @@ export type FxStreamLine = (
export interface FxSession {
/** The session's id; for a new session, null until its first prompt is accepted. */
readonly id: string | null;
/** Queues `input` as the session's next turn and returns a view of that turn. */
/**
* Queues `input` as the session's next turn and returns a view of that turn. Throws a
* TypeError or RangeError for an input libfx refuses, and stores nothing. A prompt whose
* `signal` aborts before it is stored is never stored, and ends `cancelled`.
*/
prompt(input: FxSessionPromptInput, options?: FxPromptOptions): FxTurn;
/** Continues a turn a stopped process left open; its result has the stop reason `idle` when there was none. */
resume(): FxResumeTurn;
+516 -6
View File
@@ -13,6 +13,7 @@ const debug_trace = @import("../core/shared/debug_trace.zig");
const session_adapter = @import("../core/session/session_adapter.zig");
const checkpoint_codec = @import("../core/agent/runtime/checkpoint.zig");
const execution_memory = @import("../core/agent/execution_memory.zig");
const runtime_execution_memory = @import("../core/agent/runtime/execution_memory.zig");
const tool_dispatch = @import("../core/tooling/tool_dispatch.zig");
const Allocator = std.mem.Allocator;
@@ -20,7 +21,7 @@ const Allocator = std.mem.Allocator;
pub const Event = journal.Event;
pub const max_load_bytes = journal.max_load_bytes;
/// What the model is told when libfx continues a turn a crash left open.
const resume_notice = "Resuming from unexpected session interruption.";
const resume_notice = runtime_execution_memory.resume_notice;
/// A host's id for a turn, kept in place so copying a journal copies it.
pub const TurnId = struct {
@@ -98,13 +99,23 @@ pub const Journal = struct {
/// The pending resume is a turn that yielded: it continues without being
/// told it was interrupted.
pending_resume_yielded: bool = false,
/// The host keeps an image store: events and snapshots name the images
/// in `stored_images` instead of holding their bytes.
image_store: bool = false,
/// The newest images the session keeps when the host keeps a store;
/// older ones keep only their names (`dropOutsideWindow`).
image_window: session_codec.ImageWindow = .{},
/// The images the host's store holds: those it said it stored before it
/// sent their prompt, and those an open resolved from it. Owned by the
/// session's allocator.
stored_images: session_codec.StoredImages = .{},
/// The calls of the pending resume's answering step that no result was
/// recorded for: the ones answered as possibly run. Owned by the
/// session's allocator.
pending_resume_ambiguous: []types.ToolCall = &.{},
/// The last progress this journal sent in the open turn, so the next can
/// carry only what is new. Its hashes are owned by the session's
/// allocator; `stale` once its turn ended.
/// allocator; `stale` once its turn ended or a snapshot took the turn in.
last_progress: ?LastProgress = null,
const LastProgress = struct {
@@ -114,6 +125,7 @@ pub const Journal = struct {
};
pub fn deinit(self: *Journal, alloc: Allocator) void {
self.stored_images.deinit(alloc);
if (self.last_progress) |*last| last.hashes.deinit(alloc);
self.last_progress = null;
self.dropPendingResume(alloc);
@@ -308,7 +320,10 @@ pub const Journal = struct {
try params.writer.writeAll("{\"sessionId\":");
try jsonrpc.writeJsonStr(session_id, &params.writer);
try params.writer.writeAll(",\"events\":[");
const next = try self.cursor.write(&params.writer, event);
const next = if (self.storedImages()) |stored|
try self.cursor.writeNamingImages(&params.writer, event, stored)
else
try self.cursor.write(&params.writer, event);
try params.writer.writeByte(']');
if (barrier) try params.writer.writeAll(",\"barrier\":true");
try params.writer.writeByte('}');
@@ -335,6 +350,80 @@ pub const Journal = struct {
}
}
/// The images events and snapshots name, when the host keeps a store.
pub fn storedImages(self: *const Journal) ?*const session_codec.StoredImages {
return if (self.image_store) &self.stored_images else null;
}
/// Records a prompt image the host stored before it sent it, under
/// `name`, which counts only when it is the image's digest. A name that
/// does not match leaves the image inline.
pub fn noteStoredImage(
self: *Journal,
alloc: Allocator,
image: types.ImageAttachment,
name: session_codec.ImageName,
) Allocator.Error!void {
if (!self.image_store) return;
const digest = image.snapshot_sha256 orelse return;
if (image.inline_data == null or !std.mem.eql(u8, digest, &name)) {
debug_trace.logf("session", "event=libfx_image_not_stored id={d} reason=name_mismatch", .{image.id});
return;
}
try self.stored_images.add(alloc, name);
}
/// Keeps the session within its image window when the host keeps a
/// store: each image of `history` (owned by `history_alloc`) older than
/// the window loses its bytes and keeps its name. A turn's images join the
/// window once the turn enters the history; until then they stay outside
/// it, as an open keeps the open turn's (`keepNeededImages`), so a session
/// opened at any point holds the images the running one held.
pub fn dropOutsideWindow(
self: *Journal,
alloc: Allocator,
history_alloc: Allocator,
history: []types.HistoryTurn,
) Allocator.Error!void {
if (!self.image_store) return;
var held: std.ArrayList(HeldImage) = .empty;
defer held.deinit(alloc);
var sizes: std.ArrayList(?usize) = .empty;
defer sizes.deinit(alloc);
for (history) |*turn| {
if (userOf(turn)) |user| for (user.images) |*image| {
if (try countSize(alloc, &sizes, windowSize(image.*))) try held.append(alloc, .{ .prompt = image });
};
if (executionOf(turn)) |execution| for (execution.tool_steps) |step| for (step.tool_results) |result| for (result.tool_images) |*image| {
if (try countSize(alloc, &sizes, toolWindowSize(image.*))) try held.append(alloc, .{ .tool = image });
};
}
const outside = self.image_window.oldestOutside(sizes.items);
var dropped: usize = 0;
for (held.items[0..outside]) |image| switch (image) {
.prompt => |prompt| {
const bytes = prompt.inline_data orelse continue;
history_alloc.free(bytes);
prompt.inline_data = null;
dropped += 1;
},
.tool => |tool| {
if (tool.droppedFromSession()) continue;
if (tool.name == null) {
tool.name = try inlineToolImageName(alloc, tool.data) orelse {
debug_trace.logf("session", "event=libfx_image_kept reason=inline_tool_image_not_base64 bytes={d}", .{tool.data.len});
continue;
};
}
const empty = try history_alloc.alloc(u8, 0);
history_alloc.free(tool.data);
tool.data = empty;
dropped += 1;
},
};
if (dropped > 0) debug_trace.logf("session", "event=libfx_images_dropped count={d} reason=image_window max_images={d} max_bytes={d}", .{ dropped, self.image_window.max_images, self.image_window.max_bytes });
}
/// Records that a call of the running batch finished, with what the
/// model sees of it. Nothing waits for it to be stored: a crash before
/// the next progress answers the call with it instead of with "may have
@@ -369,6 +458,14 @@ pub const Journal = struct {
self.steers_waiting == 0 and self.follow_ups_waiting == 0;
}
/// Records that a snapshot holding the open turn went to the host. A load
/// from it starts the turn from the snapshot, without the progress record
/// the snapshot folded, so the turn's next progress carries all of it
/// instead of extending that record.
pub fn noteOpenTurnSnapshot(self: *Journal) void {
if (self.last_progress) |*last| last.stale = true;
}
/// Records that the open turn ended without a history entry. A turn
/// with no progress on record needs no event.
pub fn clearProgress(self: *Journal, alloc: Allocator, session_id: []const u8) !void {
@@ -410,12 +507,189 @@ fn dupePendingInputs(alloc: Allocator, inputs: []const journal.PendingInput) All
return owned;
}
fn userOf(turn: *types.HistoryTurn) ?*types.UserTurn {
return switch (turn.*) {
.assistant => |*entry| &entry.user,
.interrupted => |*entry| &entry.user,
.compacted_summary => null,
};
}
fn executionOf(turn: *types.HistoryTurn) ?*types.ExecutionMemory {
return switch (turn.*) {
.assistant => |*entry| &entry.execution,
.interrupted => |*entry| &entry.execution,
.compacted_summary => null,
};
}
/// An image the window counts: a prompt's, or one a tool returned, counted
/// together in the order they entered the session.
const HeldImage = union(enum) {
prompt: *types.ImageAttachment,
tool: *types.ToolImage,
};
const MissingImages = std.array_hash_map.Auto(session_codec.ImageName, usize);
/// What the image window counts for one image.
const WindowSize = union(enum) {
/// Its bytes: those it holds, or for an image an open has yet to stage,
/// the size its record gives (`unstaged`).
bytes: usize,
/// One the session dropped, which keeps every older image out.
dropped,
/// One with neither bytes nor a name, which only a host reference
/// describes, and so holds nothing.
none,
};
/// What the window counts for `image`, as the session holds it.
fn windowSize(image: types.ImageAttachment) WindowSize {
if (image.inline_data) |bytes| return .{ .bytes = bytes.len };
return if (image.droppedFromSession()) .dropped else .none;
}
/// What the window counts for a tool image: its decoded bytes, whether the
/// host stored it or the session holds it inline, as one the store refused
/// or an older libfx saved. One with only a host reference holds nothing.
fn toolWindowSize(image: types.ToolImage) WindowSize {
if (image.droppedFromSession()) return .dropped;
if (image.data.len == 0) return .none;
return .{ .bytes = std.base64.standard.Decoder.calcSizeForSlice(image.data) catch image.data.len };
}
/// The name of a tool image's bytes, which one held inline takes when it
/// leaves the window, so it is saved dropped like a stored one; null for
/// bytes that are not base64.
fn inlineToolImageName(alloc: Allocator, data: []const u8) Allocator.Error!?session_codec.ImageName {
const size = std.base64.standard.Decoder.calcSizeForSlice(data) catch return null;
const bytes = try alloc.alloc(u8, size);
defer alloc.free(bytes);
std.base64.standard.Decoder.decode(bytes, data) catch return null;
return session_codec.imageNameOf(bytes);
}
/// An image an open has yet to stage, with the size its record gives.
const Unstaged = struct { name: session_codec.ImageName, size: usize };
/// The name and size of `image` when it is one an open has yet to stage: it
/// reads such an image as one without bytes, which `missing` names.
fn unstaged(image: types.ImageAttachment, missing: *const MissingImages) ?Unstaged {
if (!image.droppedFromSession()) return null;
const name = session_codec.imageName(image.snapshot_sha256.?) orelse return null;
return .{ .name = name, .size = missing.get(name) orelse return null };
}
/// `unstaged` for a tool image, which an open reads by name without bytes.
fn unstagedTool(image: types.ToolImage, missing: *const MissingImages) ?Unstaged {
if (!image.droppedFromSession()) return null;
const name = image.name.?;
return .{ .name = name, .size = missing.get(name) orelse return null };
}
/// Appends what the window counts for one image to `sizes`, oldest first,
/// and returns whether it counted.
fn countSize(alloc: Allocator, sizes: *std.ArrayList(?usize), size: WindowSize) Allocator.Error!bool {
switch (size) {
.bytes => |bytes| try sizes.append(alloc, bytes),
.dropped => try sizes.append(alloc, null),
.none => return false,
}
return true;
}
/// Leaves in `images.missing` only the names of images the window keeps, so
/// the host stages none the session would drop. The window counts the
/// history alone, as it did when the last turn entered it
/// (`Journal.dropOutsideWindow`); the open turn's images, its prompt's and
/// its tools', are always needed.
fn keepNeededImages(
alloc: Allocator,
images: *session_codec.ImageResolver,
history: []const types.HistoryTurn,
open_user: ?types.UserTurn,
open_execution: ?types.ExecutionMemory,
open_finished: []const journal.Finished,
window: session_codec.ImageWindow,
) Allocator.Error!void {
var sizes: std.ArrayList(?usize) = .empty;
defer sizes.deinit(alloc);
var names: std.ArrayList(?session_codec.ImageName) = .empty;
defer names.deinit(alloc);
for (history) |turn| {
const user, const execution = switch (turn) {
.assistant => |entry| .{ entry.user, entry.execution },
.interrupted => |entry| .{ entry.user, entry.execution },
.compacted_summary => continue,
};
try countOpenedImages(alloc, &sizes, &names, user.images, &images.missing);
try countOpenedToolImages(alloc, &sizes, &names, execution, &images.missing);
}
const outside = window.oldestOutside(sizes.items);
var needed: MissingImages = .empty;
errdefer needed.deinit(images.alloc);
for (names.items[outside..], sizes.items[outside..]) |name, size| {
if (name) |value| try needed.put(images.alloc, value, size.?);
}
if (open_user) |user| for (user.images) |image| {
if (unstaged(image, &images.missing)) |found| try needed.put(images.alloc, found.name, found.size);
};
if (open_execution) |execution| for (execution.tool_steps) |step| for (step.tool_results) |result| for (result.tool_images) |image| {
if (unstagedTool(image, &images.missing)) |found| try needed.put(images.alloc, found.name, found.size);
};
// A result the open turn recorded before a crash answers its call again.
for (open_finished) |finished| for (finished.images) |image| {
if (unstagedTool(image, &images.missing)) |found| try needed.put(images.alloc, found.name, found.size);
};
if (outside > 0) debug_trace.logf("session", "event=libfx_image_window_open outside={d} needed={d}", .{ outside, needed.count() });
images.missing.deinit(images.alloc);
images.missing = needed;
}
/// Counts the images of `user_images` an open read, each with its name when
/// it is one the host has yet to stage.
fn countOpenedImages(
alloc: Allocator,
sizes: *std.ArrayList(?usize),
names: *std.ArrayList(?session_codec.ImageName),
user_images: []const types.ImageAttachment,
missing: *const MissingImages,
) Allocator.Error!void {
for (user_images) |image| {
const pending = unstaged(image, missing);
if (!try countSize(alloc, sizes, if (pending) |found| .{ .bytes = found.size } else windowSize(image))) continue;
try names.append(alloc, if (pending) |found| found.name else null);
}
}
/// Counts the tool images of `execution` an open read, as `countOpenedImages`
/// counts a prompt's.
fn countOpenedToolImages(
alloc: Allocator,
sizes: *std.ArrayList(?usize),
names: *std.ArrayList(?session_codec.ImageName),
execution: types.ExecutionMemory,
missing: *const MissingImages,
) Allocator.Error!void {
for (execution.tool_steps) |step| for (step.tool_results) |result| for (result.tool_images) |image| {
const pending = unstagedTool(image, missing);
if (!try countSize(alloc, sizes, if (pending) |found| .{ .bytes = found.size } else toolWindowSize(image))) continue;
try names.append(alloc, if (pending) |found| found.name else null);
};
}
/// Rebuilds a fresh session from the host's events, after `snapshot` when
/// the host stored one, and starts its journal after them. A turn a crash
/// left open becomes the pending resume, with every call it left running
/// answered as possibly run, for the model to decide about unless the host
/// has the resume run it again (`rerunCalls`); `session_alloc` owns it.
/// With `inputs_durable`, the host stores every input before handing it in.
/// Named images resolve through `images`, which holds what the host staged;
/// when one is missing the session stays unopened and `error.ImagesMissing`
/// leaves every missing name in `images.missing`. With `image_store` the
/// journal names stored images from then on, and the session keeps only the
/// newest images `window` holds: an older one is neither needed nor kept.
pub fn open(
alloc: Allocator,
session_alloc: Allocator,
@@ -424,16 +698,19 @@ pub fn open(
events_json: []const u8,
snapshot: ?[]const u8,
inputs_durable: bool,
images: *session_codec.ImageResolver,
image_store: bool,
window: session_codec.ImageWindow,
) !struct { Journal, Opened } {
if (!runtime.agent.fresh or runtime.agent.history.items.len != 0) return error.AgentNotFresh;
var base: journal.Base = .{};
var base: journal.Base = .{ .images = images };
var decoded: ?checkpoint_codec.Decoded = null;
defer if (decoded) |*value| value.deinit(alloc);
var saved_meta: checkpoint_codec.Meta = .{};
errdefer saved_meta.free(alloc);
if (snapshot) |bytes| {
const parsed = try journal.parseSnapshot(bytes);
decoded = checkpoint_codec.decode(alloc, parsed.checkpoint) catch |err| switch (err) {
decoded = checkpoint_codec.decodeWithImages(alloc, parsed.checkpoint, images) catch |err| switch (err) {
error.OutOfMemory => return error.OutOfMemory,
error.UnsupportedCheckpointVersion => return error.UnsupportedJournalVersion,
else => return error.InvalidJournal,
@@ -444,10 +721,19 @@ pub fn open(
.cursor = .{ .next_seq = parsed.seq + 1, .turn = parsed.turn },
.last_turn_id = parsed.last_turn_id,
.open_turn = parsed.open_turn,
.images = images,
};
}
var folded = try journal.foldFrom(alloc, events_json, base);
defer folded.deinit(alloc);
if (image_store and images.missing.count() > 0) {
try keepNeededImages(alloc, images, folded.history, if (folded.open_turn) |turn| turn.user else null, if (folded.open_turn) |turn| turn.execution else null, folded.finished, window);
}
// A history missing an image it keeps is not the one saved: open none.
if (images.missing.count() > 0) {
debug_trace.logf("session", "event=libfx_journal_images_missing count={d}", .{images.missing.count()});
return error.ImagesMissing;
}
if (folded.history.len > 0) try runtime.agent.restoreHistory(alloc, folded.history);
if (decoded) |value| runtime.agent.turn_usage = value.usage;
var state: Journal = .{
@@ -460,10 +746,16 @@ pub fn open(
.last_turn_id = TurnId.of(folded.last_turn_id),
.inputs_durable = inputs_durable,
.steers_waiting = folded.pending_inputs.len,
.image_store = image_store,
.image_window = window,
};
errdefer state.deinit(session_alloc);
const checkpoint = folded.open_turn orelse
// An image the open resolved is in the store; one it read inline is not.
if (image_store) for (images.resolved.keys()) |name| try state.stored_images.add(session_alloc, name);
try state.dropOutsideWindow(alloc, alloc, runtime.agent.history.items);
const checkpoint = folded.open_turn orelse {
return .{ state, .{ .turns = folded.history.len, .resumable = false, .last_turn_id = state.last_turn_id, .checkpoint_meta = saved_meta } };
};
debug_trace.logf("session", "event=libfx_journal_open_turn turn={d} running_calls={d} finished_calls={d} resolved=pending_resume", .{ folded.cursor.turn, folded.running_calls.len, folded.finished.len });
var scratch = std.heap.ArenaAllocator.init(alloc);
@@ -571,6 +863,7 @@ fn answerRunning(scratch: Allocator, turn: *session_codec.RecoveryCheckpoint, ca
.tool_name = @constCast(call.name),
.status = if (recorded) |entry| (if (entry.failed) .failure else .success) else .failure,
.output = @constCast(output),
.tool_images = if (recorded) |entry| entry.images else &.{},
.output_bytes = output.len,
.stored_output_bytes = output.len,
};
@@ -835,6 +1128,29 @@ test "a progress after its turn's first carries only what is new, and the next t
try std.testing.expect(std.mem.find(u8, capture.frames.items[next..], "\"user\"") != null);
}
test "a progress after a snapshot that holds its turn carries all of it" {
const alloc = std.testing.allocator;
var capture: TestCapture = .{};
defer capture.frames.deinit(alloc);
var writer = jsonrpc.Writer.initCallback(&capture, TestCapture.write);
var session: Journal = .{ .writer = &writer };
defer session.deinit(alloc);
var progress = try testProgress(alloc);
defer progress.deinit(alloc);
try session.appendProgress(alloc, alloc, "session", progress, "fake/model");
// A load from the snapshot has no record for the next progress to extend.
session.noteOpenTurnSnapshot();
const next = capture.frames.items.len;
try session.appendProgress(alloc, alloc, "session", progress, "fake/model");
try std.testing.expect(std.mem.find(u8, capture.frames.items[next..], "\"base\"") == null);
try std.testing.expect(std.mem.find(u8, capture.frames.items[next..], "\"user\"") != null);
// The progress after that one extends it again.
const after = capture.frames.items.len;
try session.appendProgress(alloc, alloc, "session", progress, "fake/model");
try std.testing.expect(std.mem.find(u8, capture.frames.items[after..], "\"base\":2,") != null);
}
test "a resumed turn runs again the calls the host asks for and leaves the rest to the model" {
const alloc = std.testing.allocator;
var calls = [_]types.ToolCall{
@@ -883,3 +1199,197 @@ test "a resume cancelled during a rerun keeps that call and the later ones answe
// A step index past the turn's steps runs nothing.
try std.testing.expectEqual(@as(usize, 0), try rerunCalls(alloc, &answered, 5, TestRerunner{ .ids = &ids, .ran = &ran }, null));
}
/// An inline image of `size` bytes for the window tests, every field owned by
/// `alloc`; `types.freeImageAttachment` frees it.
fn windowTestImage(alloc: Allocator, id: usize, size: usize) !types.ImageAttachment {
const bytes = try alloc.alloc(u8, size);
errdefer alloc.free(bytes);
@memset(bytes, 'x');
const path = try alloc.dupe(u8, "inline://image");
errdefer alloc.free(path);
const media_type = try alloc.dupe(u8, "image/png");
errdefer alloc.free(media_type);
const digest = session_codec.imageNameOf(bytes);
return .{ .id = id, .path = path, .media_type = media_type, .snapshot_sha256 = try alloc.dupe(u8, &digest), .inline_data = bytes };
}
test "the image window drops the oldest history images, and not for an open turn's" {
const alloc = std.testing.allocator;
var capture: TestCapture = .{};
defer capture.frames.deinit(alloc);
var writer = jsonrpc.Writer.initCallback(&capture, TestCapture.write);
var session: Journal = .{ .writer = &writer, .image_store = true, .image_window = .{ .max_images = 2 } };
defer session.deinit(alloc);
var images: [3]types.ImageAttachment = undefined;
for (&images, 0..) |*image, index| image.* = try windowTestImage(alloc, index + 1, 10);
defer for (images) |image| types.freeImageAttachment(alloc, image);
var history: [3]types.HistoryTurn = undefined;
for (&history, &images) |*turn, *image| {
turn.* = .{ .assistant = .{ .user = .{ .text = @constCast("look"), .images = image[0..1] }, .assistant = @constCast("ok") } };
}
// Two images at most: the two newest of history.
try session.dropOutsideWindow(alloc, alloc, &history);
try std.testing.expect(images[0].droppedFromSession());
try std.testing.expect(images[1].inline_data != null and images[2].inline_data != null);
// An open turn has not entered the history, so its image moves nothing.
var pending = try testProgress(alloc);
pending.user.images = try alloc.alloc(types.ImageAttachment, 1);
pending.user.images[0] = try windowTestImage(alloc, 4, 10);
session.pending_resume = pending;
try session.dropOutsideWindow(alloc, alloc, &history);
try std.testing.expect(images[1].inline_data != null and images[2].inline_data != null);
try std.testing.expect(session.pending_resume.?.user.images[0].inline_data != null);
}
/// A one-image slice a tool returned, `size` bytes as base64, named as the
/// host stored it unless `named` is false; `types.freeToolImages` frees it.
fn windowTestToolImages(alloc: Allocator, size: usize, named: bool) ![]types.ToolImage {
const bytes = try alloc.alloc(u8, size);
defer alloc.free(bytes);
@memset(bytes, 't');
const encoded = try alloc.alloc(u8, std.base64.standard.Encoder.calcSize(size));
errdefer alloc.free(encoded);
_ = std.base64.standard.Encoder.encode(encoded, bytes);
const mime_type = try alloc.dupe(u8, "image/png");
errdefer alloc.free(mime_type);
const images = try alloc.alloc(types.ToolImage, 1);
images[0] = .{ .data = encoded, .mime_type = mime_type, .name = if (named) session_codec.imageNameOf(bytes) else null };
return images;
}
fn windowTestToolStep(results: *[1]types.PersistedToolResult, steps: *[1]types.ToolExecutionStep, images: []types.ToolImage) types.ExecutionMemory {
results.* = .{.{
.tool_images = images,
.tool_call_id = @constCast("call"),
.tool_name = @constCast("screenshot"),
.status = .success,
.output = @constCast("took it"),
.output_bytes = 7,
.stored_output_bytes = 7,
}};
steps.* = .{.{ .tool_results = results }};
return .{ .tool_steps = steps };
}
test "the image window counts a turn's tool images after its prompt's, inline ones too, and drops them by name" {
const alloc = std.testing.allocator;
var capture: TestCapture = .{};
defer capture.frames.deinit(alloc);
var writer = jsonrpc.Writer.initCallback(&capture, TestCapture.write);
var session: Journal = .{ .writer = &writer, .image_store = true, .image_window = .{ .max_images = 2 } };
defer session.deinit(alloc);
var prompts: [2]types.ImageAttachment = undefined;
for (&prompts, 0..) |*image, index| image.* = try windowTestImage(alloc, index + 1, 10);
defer for (prompts) |image| types.freeImageAttachment(alloc, image);
var tools: [2][]types.ToolImage = undefined;
for (&tools) |*images| images.* = try windowTestToolImages(alloc, 10, true);
defer for (tools) |images| types.freeToolImages(alloc, images);
// A tool image the session holds inline, as one the store refused, the
// oldest tool image here.
const unstored = try windowTestToolImages(alloc, 12, false);
defer types.freeToolImages(alloc, unstored);
var unstored_bytes: [12]u8 = undefined;
try std.base64.standard.Decoder.decode(&unstored_bytes, unstored[0].data);
var results: [3][1]types.PersistedToolResult = undefined;
var steps: [3][1]types.ToolExecutionStep = undefined;
var history: [3]types.HistoryTurn = undefined;
history[0] = .{ .assistant = .{ .user = .{ .text = @constCast("one"), .images = prompts[0..1] }, .assistant = @constCast("ok"), .execution = windowTestToolStep(&results[0], &steps[0], unstored) } };
history[1] = .{ .assistant = .{ .user = .{ .text = @constCast("two"), .images = prompts[1..2] }, .assistant = @constCast("ok"), .execution = windowTestToolStep(&results[1], &steps[1], tools[0]) } };
history[2] = .{ .assistant = .{ .user = .{ .text = @constCast("three") }, .assistant = @constCast("ok"), .execution = windowTestToolStep(&results[2], &steps[2], tools[1]) } };
// Two images at most: the newest two tool images, in the order they
// entered; every older one leaves, keeping its name. The inline one
// counts as any image does, and takes the name of its bytes.
try session.dropOutsideWindow(alloc, alloc, &history);
try std.testing.expect(prompts[0].droppedFromSession());
try std.testing.expect(unstored[0].droppedFromSession());
try std.testing.expectEqualSlices(u8, &session_codec.imageNameOf(&unstored_bytes), &unstored[0].name.?);
try std.testing.expect(prompts[1].droppedFromSession());
try std.testing.expect(!tools[0][0].droppedFromSession() and tools[0][0].data.len > 0);
try std.testing.expect(!tools[1][0].droppedFromSession() and tools[1][0].data.len > 0);
}
test "an open needs the newest tool images its window keeps, and every one of the open turn" {
const alloc = std.testing.allocator;
var staged: session_codec.StagedImages = .{};
defer staged.deinit(alloc);
var resolver: session_codec.ImageResolver = .{ .alloc = alloc, .staged = &staged };
defer resolver.deinit();
// Three tool images the host has not staged yet, read as names alone.
var names: [3]session_codec.ImageName = undefined;
var images: [3][1]types.ToolImage = undefined;
for (&names, &images, 0..) |*name, *image, index| {
name.* = session_codec.imageNameOf(&.{@as(u8, @intCast(index))});
image.* = .{.{ .data = @constCast(&[_]u8{}), .mime_type = @constCast("image/png"), .name = name.* }};
try resolver.missing.put(alloc, name.*, 10);
}
var results: [3][1]types.PersistedToolResult = undefined;
var steps: [3][1]types.ToolExecutionStep = undefined;
var history: [2]types.HistoryTurn = undefined;
for (&history, 0..) |*turn, index| {
turn.* = .{ .assistant = .{ .user = .{ .text = @constCast("look") }, .assistant = @constCast("ok"), .execution = windowTestToolStep(&results[index], &steps[index], &images[index]) } };
}
const open_execution = windowTestToolStep(&results[2], &steps[2], &images[2]);
try keepNeededImages(alloc, &resolver, &history, null, open_execution, &.{}, .{ .max_images = 1 });
try std.testing.expectEqual(@as(usize, 2), resolver.missing.count());
try std.testing.expect(!resolver.missing.contains(names[0]));
try std.testing.expect(resolver.missing.contains(names[1]) and resolver.missing.contains(names[2]));
}
test "an open needs the images of a result its open turn recorded before a crash" {
const alloc = std.testing.allocator;
var staged: session_codec.StagedImages = .{};
defer staged.deinit(alloc);
var resolver: session_codec.ImageResolver = .{ .alloc = alloc, .staged = &staged };
defer resolver.deinit();
const name = session_codec.imageNameOf("recorded shot");
try resolver.missing.put(alloc, name, 13);
var images = [_]types.ToolImage{.{ .data = @constCast(&[_]u8{}), .mime_type = @constCast("image/png"), .name = name }};
const finished = [_]journal.Finished{.{ .id = @constCast("call-1"), .failed = false, .output = @constCast("took it"), .images = &images }};
try keepNeededImages(alloc, &resolver, &.{}, null, null, &finished, .{ .max_images = 1 });
try std.testing.expectEqual(@as(usize, 1), resolver.missing.count());
try std.testing.expectEqual(@as(?usize, 13), resolver.missing.get(name));
}
test "an open needs only the named images its window keeps, and every image of the open turn" {
const alloc = std.testing.allocator;
var staged: session_codec.StagedImages = .{};
defer staged.deinit(alloc);
var resolver: session_codec.ImageResolver = .{ .alloc = alloc, .staged = &staged };
defer resolver.deinit();
// Four named images the host has not staged yet, oldest first.
var names: [4]session_codec.ImageName = undefined;
var images: [4]types.ImageAttachment = undefined;
for (&names, &images, 0..) |*name, *image, index| {
name.* = session_codec.imageNameOf(&.{@as(u8, @intCast(index))});
image.* = .{ .id = index + 1, .path = @constCast("inline://image"), .media_type = @constCast("image/png"), .snapshot_sha256 = name };
}
const reset = struct {
fn run(target: *session_codec.ImageResolver, all: []const session_codec.ImageName) !void {
target.missing.clearRetainingCapacity();
for (all) |name| try target.missing.put(target.alloc, name, 10);
}
}.run;
var history: [3]types.HistoryTurn = undefined;
for (&history, images[0..3]) |*turn, *image| {
turn.* = .{ .assistant = .{ .user = .{ .text = @constCast("look"), .images = image[0..1] }, .assistant = @constCast("ok") } };
}
// The window counts the history alone: its two newest, and the open
// turn's image as well.
try reset(&resolver, &names);
try keepNeededImages(alloc, &resolver, &history, .{ .text = @constCast("now"), .images = images[3..4] }, null, &.{}, .{ .max_images = 2 });
try std.testing.expectEqual(@as(usize, 3), resolver.missing.count());
try std.testing.expect(!resolver.missing.contains(names[0]));
// An open turn larger than the window keeps all of its images.
try reset(&resolver, &names);
try keepNeededImages(alloc, &resolver, history[0..2], .{ .text = @constCast("now"), .images = images[2..4] }, null, &.{}, .{ .max_images = 1 });
try std.testing.expectEqual(@as(usize, 3), resolver.missing.count());
try std.testing.expect(!resolver.missing.contains(names[0]));
}
+1 -1
View File
@@ -140,7 +140,7 @@ fn prepareHost(
defer inherited.deinit();
const h = mcp_host.Host.create(alloc, io_mod.getIo(), configs, &inherited, .{
.client_version = build_options.app_version,
.credentials = .detect(alloc, home),
.credentials = .{ .detect = home },
.questions = questions,
.url_completions = url_completions,
}, builtin_tools.registry) catch |err| switch (err) {
+74 -3
View File
@@ -134,10 +134,48 @@ fn promptInputFailure(err: anyerror) anyerror!TerminalOutcome {
.code = ErrorCode.invalid_params,
.message = "Image prompt exceeds size limit",
} },
error.PromptImagesTooLarge => .{ .rpc_error = .{
.code = ErrorCode.invalid_params,
.message = "Prompt images exceed the libfx limit",
} },
else => err,
};
}
/// Whether `images` fit one libfx prompt: at most
/// `image_attachments.max_libfx_prompt_images` of them, holding at most
/// `max_libfx_prompt_images_bytes` together. An open stages its open turn's
/// images on top of the session's image window, and counts on this.
fn fitsLibfxPrompt(images: []const PendingPromptImage) bool {
if (images.len > image_attachments.max_libfx_prompt_images) return false;
var bytes: usize = 0;
for (images) |image| bytes += image.bytes.len;
return bytes <= image_attachments.max_libfx_prompt_images_bytes;
}
test "a libfx prompt holds at most its limit of images and image bytes" {
const alloc = std.testing.allocator;
var media_type = "image/png".*;
var byte: [1]u8 = .{0x89};
var images: [image_attachments.max_libfx_prompt_images + 1]PendingPromptImage = @splat(.{ .id = 1, .bytes = &byte, .media_type = &media_type });
try std.testing.expect(fitsLibfxPrompt(images[0..image_attachments.max_libfx_prompt_images]));
try std.testing.expect(!fitsLibfxPrompt(&images));
// Reference-only images count toward the number too.
var references = images;
for (&references) |*image| image.bytes = &.{};
try std.testing.expect(!fitsLibfxPrompt(&references));
const bytes = try alloc.alloc(u8, image_attachments.max_libfx_prompt_images_bytes + 1);
defer alloc.free(bytes);
const exact = [_]PendingPromptImage{.{ .id = 1, .bytes = bytes[1..], .media_type = &media_type }};
try std.testing.expect(fitsLibfxPrompt(&exact));
const over = [_]PendingPromptImage{
.{ .id = 1, .bytes = bytes[0..1024], .media_type = &media_type },
.{ .id = 2, .bytes = bytes[1024..], .media_type = &media_type },
};
try std.testing.expect(!fitsLibfxPrompt(&over));
}
fn promptExecutionFailure(err: anyerror) anyerror!TerminalOutcome {
return switch (err) {
error.ModelImageCapabilityUnavailable,
@@ -880,8 +918,10 @@ pub fn handlePrompt(
if (session.store == null and session.wasm_state == null and session.v2 == null) {
// libfx kernel session: images stay in memory and ride the kernel
// checkpoint, so no filesystem snapshot backend is needed.
if (!fitsLibfxPrompt(prompt_input.pending_images)) return promptInputFailure(error.PromptImagesTooLarge);
prompt_input.captureImagesInline(alloc) catch |err|
return promptInputFailure(err);
if (session.journal) |*journal| try prompt_input.noteStoredImages(state.alloc, journal);
} else {
if (comptime host_target.is_wasm) return promptInputFailure(error.UnsupportedPromptImage);
var temporary_snapshot_dir: ?[]u8 = null;
@@ -1488,6 +1528,8 @@ const PendingPromptImage = struct {
bytes: []u8,
media_type: []u8,
source_ref: ?[]u8 = null,
/// The name the host stored the bytes under before it sent them.
stored_name: ?session_codec.ImageName = null,
fn deinit(self: *PendingPromptImage, alloc: Allocator) void {
alloc.free(self.bytes);
@@ -1587,6 +1629,16 @@ const ParsedPromptInput = struct {
self.images = images;
}
/// Tells `journal` which captured images the host stored before it sent
/// them, so its events name them. `captureImagesInline` keeps the order.
fn noteStoredImages(self: *const ParsedPromptInput, session_alloc: Allocator, journal: *libfx_journal.Journal) Allocator.Error!void {
std.debug.assert(self.images.len == self.pending_images.len);
for (self.images, self.pending_images) |image, pending| {
const name = pending.stored_name orelse continue;
try journal.noteStoredImage(session_alloc, image, name);
}
}
fn retainImageSnapshots(self: *ParsedPromptInput) void {
self.retain_image_snapshots = true;
}
@@ -1770,6 +1822,11 @@ fn parsePromptInputWithFirstImageId(
break :bytes try alloc.alloc(u8, 0);
};
errdefer alloc.free(decoded);
// A host with an image store names the bytes it stored first.
const stored_name: ?session_codec.ImageName = if (acp_types.fxMetaField(block.object, "stored")) |value| name: {
if (value != .string or decoded.len == 0) return error.InvalidPromptImage;
break :name session_codec.imageName(value.string) orelse return error.InvalidPromptImage;
} else null;
const image_id = std.math.add(usize, first_image_id, pending_images.items.len) catch
return error.ImageIdOverflow;
@@ -1788,6 +1845,7 @@ fn parsePromptInputWithFirstImageId(
.bytes = decoded,
.media_type = media_type,
.source_ref = source_ref,
.stored_name = stored_name,
});
} else if (std.mem.eql(u8, block_type.string, "resource")) {
if (block.object.get("resource")) |resource| {
@@ -2609,9 +2667,9 @@ fn completeAuthorizedToolCallTransport(
}
/// Records a finished call's result in a journaled libfx session, so a
/// crash before the next progress answers it with what the model sees
/// instead of with "may have partly run". Nothing waits for it. A cancelled
/// call did not finish and stays unanswered; a result's images are not kept.
/// crash before the next progress answers it with what the model sees,
/// its images too, instead of with "may have partly run". Nothing waits for
/// it. A cancelled call did not finish and stays unanswered.
fn recordJournalToolResult(ctx: *AcpContext, call: ToolCall, result: ToolExecutionResult) void {
if (result.cancelled) return;
const session = if (ctx.state.active_session) |*value| value else return;
@@ -2623,6 +2681,7 @@ fn recordJournalToolResult(ctx: *AcpContext, call: ToolCall, result: ToolExecuti
.name = call.name,
.failed = result.status == .failure,
.output = result.model_output,
.images = if (result.tool_result_memory) |memory| memory.tool_images else &.{},
}) catch |err| {
debug_trace.logf("session", "event=libfx_journal_tool_result_dropped call={s} err={s} answered_by=next_progress", .{ call.id, @errorName(err) });
};
@@ -2946,6 +3005,7 @@ fn persistAcpHistoryTurn(
session.session_rt.commitPreparedHistoryEntry(alloc, prepared);
prepared_owned = false;
if (current_prompt_input) |prompt_input| prompt_input.retainImageSnapshots();
try keepImageWindow(alloc, session);
if (session.wasm_state != null) try sessions.commitWasmSessionLocked(alloc, session);
return;
}
@@ -2965,6 +3025,7 @@ fn persistAcpHistoryTurn(
session.session_rt.commitPreparedHistoryEntry(alloc, prepared);
prepared_owned = false;
if (current_prompt_input) |prompt_input| prompt_input.retainImageSnapshots();
try keepImageWindow(alloc, session);
return;
};
try writable.prepareHistoryTurnForCommit(alloc, &prepared);
@@ -2988,6 +3049,15 @@ fn persistAcpHistoryTurn(
if (current_prompt_input) |prompt_input| prompt_input.retainImageSnapshots();
}
/// Keeps a libfx session within its image window once a turn enters its
/// history (owned by `alloc`): its images are the newest, and older ones past
/// the window give up their bytes. A turn that fails before it is recorded
/// drops nothing, so a reopen keeps the same images.
fn keepImageWindow(alloc: Allocator, session: *server.ActiveSessionState) Allocator.Error!void {
const journal = if (session.journal) |*value| value else return;
try journal.dropOutsideWindow(alloc, alloc, session.session_rt.agent.history.items);
}
fn commitContextCompaction(
raw_ctx: *anyopaque,
summary: types.CompactedSummaryHistoryTurn,
@@ -3254,6 +3324,7 @@ fn pushToolLifecycle(raw_ctx: *anyopaque, event: types.ToolLifecycleEvent) !void
.id = started.id.call_id,
.name = started.tool_name,
.arguments_json = started.arguments_json orelse "{}",
.resolved_display_target = started.resolved_display_target,
});
},
.terminal => |terminal| {
+188 -5
View File
@@ -51,6 +51,7 @@ const host_tool_runtime = @import("../core/tooling/host_tool_runtime.zig");
const tool_dispatch = @import("../core/tooling/tool_dispatch.zig");
const agent_checkpoint = @import("../core/agent/runtime/checkpoint.zig");
const host_attachments = @import("../core/hosts/host_attachments.zig");
const image_attachments = @import("../core/images/image_attachments.zig");
const libfx_steering = @import("libfx_steering.zig");
const journal_events = @import("../core/agent/runtime/journal.zig");
const libfx_journal = @import("libfx_journal.zig");
@@ -87,6 +88,7 @@ const AcpMethod = enum {
libfx_follow_up,
libfx_snapshot,
libfx_journal_open,
libfx_journal_images,
mcp_message,
unknown,
@@ -112,6 +114,7 @@ const AcpMethod = enum {
if (std.mem.eql(u8, method, "libfx/follow_up")) return .libfx_follow_up;
if (std.mem.eql(u8, method, "libfx/snapshot")) return .libfx_snapshot;
if (std.mem.eql(u8, method, "libfx/journal_open")) return .libfx_journal_open;
if (std.mem.eql(u8, method, "libfx/journal_images")) return .libfx_journal_images;
if (std.mem.eql(u8, method, "mcp/message")) return .mcp_message;
return .unknown;
}
@@ -140,6 +143,7 @@ const AcpMethod = enum {
.libfx_restore,
.libfx_snapshot,
.libfx_journal_open,
.libfx_journal_images,
.mcp_message,
.unknown,
=> true,
@@ -148,7 +152,7 @@ const AcpMethod = enum {
fn isLibfx(self: AcpMethod) bool {
return switch (self) {
.libfx_checkpoint, .libfx_restore, .libfx_new, .libfx_steer, .libfx_withdraw, .libfx_handoff, .libfx_follow_up, .libfx_snapshot, .libfx_journal_open => true,
.libfx_checkpoint, .libfx_restore, .libfx_new, .libfx_steer, .libfx_withdraw, .libfx_handoff, .libfx_follow_up, .libfx_snapshot, .libfx_journal_open, .libfx_journal_images => true,
else => false,
};
}
@@ -268,6 +272,13 @@ pub const ActiveSessionState = struct {
steering: libfx_steering.Runtime = .{},
/// Set when the host passed a journal; every state change is sent to it.
journal: ?libfx_journal.Journal = null,
/// Images the host staged from its store for the journal it opens next.
/// Owned by the server allocator; cleared once the journal opens.
staged_images: session_codec.StagedImages = .{},
/// What the last open that found images missing needs staged, all told:
/// an open turn's tool images can take a session past one window and one
/// prompt, and what a running session held must reopen.
staging_needed: StagedTotals = .{},
pub fn retainGrant(self: *ActiveSessionState, alloc: Allocator, tool_name: []const u8, target_path: []const u8) !void {
for (self.session_grants) |grant| {
@@ -753,6 +764,7 @@ fn destroyActiveSession(state: *ServerState) void {
active.tool_identities.deinit(state.alloc);
active.steering.deinit(state.alloc);
if (active.journal) |*journal| journal.deinit(state.alloc);
active.staged_images.deinit(state.alloc);
types.freePermissionGrantSlice(state.alloc, active.session_grants);
if (comptime !host_target.is_wasm) {
if (active.mcp) |runtime| {
@@ -1448,6 +1460,7 @@ fn dispatch(state: *ServerState, alloc: Allocator, msg: *jsonrpc.Message) !void
.libfx_follow_up => handleKernelFollowUp(state, alloc, msg),
.libfx_snapshot => handleKernelSnapshot(state, alloc, msg),
.libfx_journal_open => handleKernelJournalOpen(state, alloc, msg),
.libfx_journal_images => handleKernelJournalImages(state, alloc, msg),
else => state.writer.writeError(alloc, msg.id, .{
.code = ErrorCode.method_not_found,
.message = "Method not available in the web core yet",
@@ -1473,6 +1486,7 @@ fn dispatch(state: *ServerState, alloc: Allocator, msg: *jsonrpc.Message) !void
.libfx_follow_up => handleKernelFollowUp(state, alloc, msg),
.libfx_snapshot => handleKernelSnapshot(state, alloc, msg),
.libfx_journal_open => handleKernelJournalOpen(state, alloc, msg),
.libfx_journal_images => handleKernelJournalImages(state, alloc, msg),
.initialize,
.request_cancel,
.session_cancel,
@@ -1891,6 +1905,21 @@ fn handleKernelJournalOpen(
.message = "Invalid libfx journal params",
}),
} else false;
// A host with an image store stores each prompt image before it sends
// it, so the journal names stored images instead of holding their bytes.
const image_store = if (parsed.value.object.get("imageStore")) |value| switch (value) {
.bool => |enabled| enabled,
else => return state.writer.writeError(alloc, msg.id, .{
.code = ErrorCode.invalid_params,
.message = "Invalid libfx journal params",
}),
} else false;
const window = imageWindowParam(parsed.value.object.get("imageWindow")) orelse return state.writer.writeError(alloc, msg.id, .{
.code = ErrorCode.invalid_params,
.message = "Invalid libfx journal params",
});
var images: session_codec.ImageResolver = .{ .alloc = alloc, .staged = &active.staged_images };
defer images.deinit();
const journal, const opened = libfx_journal.open(
alloc,
state.alloc,
@@ -1899,8 +1928,21 @@ fn handleKernelJournalOpen(
events orelse "[]",
snapshot,
inputs_durable,
&images,
image_store,
window,
) catch |err| switch (err) {
error.OutOfMemory => return error.OutOfMemory,
// The host stages them from its store, then opens again.
error.ImagesMissing => {
var needed: StagedTotals = .{ .images = active.staged_images.images.count(), .bytes = active.staged_images.bytes };
for (images.missing.values()) |size| {
needed.images +|= 1;
needed.bytes +|= size;
}
active.staging_needed = needed;
return writeImagesNeeded(state, alloc, msg.id, images.missing.keys());
},
error.AgentNotFresh => return state.writer.writeError(alloc, msg.id, .{
.code = ErrorCode.invalid_request,
.message = "A libfx journal opens only on a new session",
@@ -1922,6 +1964,9 @@ fn handleKernelJournalOpen(
};
defer opened.checkpoint_meta.free(alloc);
active.journal = journal;
// The history holds its own copies of every staged image it named.
active.staged_images.clear(state.alloc);
active.staging_needed = .{};
// The host runs the follow-ups the journal held, after any resume.
const follow_ups = active.journal.?.takeFollowUps();
defer journal_events.freePendingInputs(state.alloc, follow_ups);
@@ -1977,6 +2022,124 @@ fn handleKernelJournalOpen(
try state.writer.writeResponse(alloc, msg.id, response.written());
}
/// Answers an open whose saved history names images the host has not
/// staged: every such name, so one round of staging serves the next open.
fn writeImagesNeeded(state: *ServerState, alloc: Allocator, id: ?jsonrpc.RequestId, names: []const session_codec.ImageName) !void {
var response: std.Io.Writer.Allocating = .init(alloc);
defer response.deinit();
try response.writer.writeAll("{\"imagesNeeded\":[");
for (names, 0..) |name, index| {
if (index > 0) try response.writer.writeByte(',');
try response.writer.print("\"{s}\"", .{&name});
}
try response.writer.writeAll("]}");
try state.writer.writeResponse(alloc, id, response.written());
}
/// The most images, and image bytes, a host may stage for one open: those of
/// the largest image window, the newest images a session keeps, and those of
/// the open turn, which the window leaves out and which holds one prompt's
/// images at most.
/// An open that needs more, because its open turn's tools returned images
/// beyond one prompt's, may stage what it reported missing
/// (`ActiveSessionState.staging_needed`).
const max_staged_images = (session_codec.ImageWindow{}).max_images + image_attachments.max_libfx_prompt_images;
const max_staged_image_bytes = (session_codec.ImageWindow{}).max_bytes + image_attachments.max_libfx_prompt_images_bytes;
const StagedTotals = struct { images: usize = 0, bytes: usize = 0 };
/// The image window an open asks for, `{images, bytes}`, each at most the
/// default: a host may keep fewer images, never more than an open stages.
/// Null when the value is not one.
fn imageWindowParam(value: ?std.json.Value) ?session_codec.ImageWindow {
const defaults: session_codec.ImageWindow = .{};
const object = switch (value orelse return defaults) {
.object => |object| object,
else => return null,
};
var window = defaults;
var iterator = object.iterator();
while (iterator.next()) |entry| {
const limit = switch (entry.value_ptr.*) {
.integer => |integer| std.math.cast(usize, integer) orelse return null,
else => return null,
};
if (limit == 0) return null;
if (std.mem.eql(u8, entry.key_ptr.*, "images")) {
window.max_images = @min(limit, defaults.max_images);
} else if (std.mem.eql(u8, entry.key_ptr.*, "bytes")) {
window.max_bytes = @min(limit, defaults.max_bytes);
} else return null;
}
return window;
}
/// Stages images from the host's store for the journal the session opens
/// next. `images` is `[{name, attachment}]`; each attachment holds the bytes
/// whose SHA-256 is the name.
fn handleKernelJournalImages(
state: *ServerState,
alloc: Allocator,
msg: *const jsonrpc.Message,
) !void {
const invalid: jsonrpc.RpcError = .{
.code = ErrorCode.invalid_params,
.message = "Invalid libfx journal images",
};
const too_large: jsonrpc.RpcError = .{
.code = ErrorCode.invalid_params,
.message = "libfx journal images are too large",
};
var parsed = libfxSessionId(alloc, msg) catch return state.writer.writeError(alloc, msg.id, invalid);
defer parsed.deinit();
const active = activeLibfxSession(state, parsed.value) orelse
return state.writer.writeError(alloc, msg.id, .{
.code = ErrorCode.invalid_params,
.message = "Unknown libfx session",
});
const store = state.cfg.host_attachments orelse return state.writer.writeError(alloc, msg.id, invalid);
const list = parsed.value.object.get("images") orelse return state.writer.writeError(alloc, msg.id, invalid);
if (list != .array or list.array.items.len == 0) return state.writer.writeError(alloc, msg.id, invalid);
active.session_write_mutex.lockUncancelable(io_mod.getIo());
defer active.session_write_mutex.unlock(io_mod.getIo());
if (active.journal != null) return state.writer.writeError(alloc, msg.id, .{
.code = ErrorCode.invalid_request,
.message = "libfx journal is already open",
});
for (list.array.items) |item| {
if (item != .object) return state.writer.writeError(alloc, msg.id, invalid);
const name_value = item.object.get("name") orelse return state.writer.writeError(alloc, msg.id, invalid);
if (name_value != .string) return state.writer.writeError(alloc, msg.id, invalid);
const name = session_codec.imageName(name_value.string) orelse return state.writer.writeError(alloc, msg.id, invalid);
const reference = item.object.get("attachment") orelse return state.writer.writeError(alloc, msg.id, invalid);
const attachment = host_attachments.idFromJson(reference) orelse return state.writer.writeError(alloc, msg.id, invalid);
if (active.staged_images.images.count() >= @max(max_staged_images, active.staging_needed.images)) return state.writer.writeError(alloc, msg.id, too_large);
const bytes = store.take(state.alloc, attachment, image_attachments.max_image_bytes) catch |err| switch (err) {
error.OutOfMemory => return error.OutOfMemory,
error.AttachmentUnavailable => return state.writer.writeError(alloc, msg.id, invalid),
error.AttachmentTooLarge => return state.writer.writeError(alloc, msg.id, too_large),
};
if (bytes.len > @max(max_staged_image_bytes, active.staging_needed.bytes) -| active.staged_images.bytes) {
state.alloc.free(bytes);
return state.writer.writeError(alloc, msg.id, too_large);
}
active.staged_images.add(state.alloc, name, bytes) catch |err| {
state.alloc.free(bytes);
return switch (err) {
error.OutOfMemory => error.OutOfMemory,
error.ImageNameMismatch => state.writer.writeError(alloc, msg.id, .{
.code = ErrorCode.invalid_params,
.message = "libfx image does not match its name",
}),
};
};
}
const written = try alloc.print("{{\"staged\":{d}}}", .{active.staged_images.images.count()});
defer alloc.free(written);
try state.writer.writeResponse(alloc, msg.id, written);
}
/// A snapshot of the session for its journal to store, when it is quiet (no
/// turn open and no follow-up waiting) or at a turn that yielded with nothing
/// waiting for it, which the snapshot then carries. `null` otherwise; the
@@ -2009,8 +2172,14 @@ fn handleKernelSnapshot(
const yielded = journal.yieldedQuiet();
if (!journal.quiet() and !yielded) return state.writer.writeResponse(alloc, msg.id, none);
// A session past a snapshot's bounds gets none until compaction brings it
// back within them; the host learns why.
const checkpoint = active.session_rt.agent.checkpoint(alloc, meta) catch |err| switch (err) {
// back within them; the host learns why. Images the host stored are
// named, not held.
const stored = journal.storedImages();
const encoded = if (stored) |names|
active.session_rt.agent.checkpointNamingImages(alloc, meta, names)
else
active.session_rt.agent.checkpoint(alloc, meta);
const checkpoint = encoded catch |err| switch (err) {
error.OutOfMemory => return error.OutOfMemory,
error.CheckpointTooLarge => return writeSnapshotSkipped(state, alloc, msg.id, if (active.session_rt.agent.history.items.len > journal_events.max_history_turns) .too_many_turns else .too_large),
else => return writeSnapshotSkipped(state, alloc, msg.id, .invalid),
@@ -2019,8 +2188,21 @@ fn handleKernelSnapshot(
const at_seq = journal.cursor.next_seq - 1;
var open_turn: std.Io.Writer.Allocating = .init(alloc);
defer open_turn.deinit();
if (yielded) try journal_events.writeOpenTurn(&open_turn.writer, journal.open_turn_id.slice(), journal.pending_resume.?);
const bytes = journal_events.encodeSnapshot(alloc, at_seq, journal.cursor.turn, journal.last_turn_id.slice(), if (yielded) open_turn.written() else null, checkpoint) catch |err| switch (err) {
var open_turn_names_images = false;
if (yielded) {
if (stored) |names| {
open_turn_names_images = try journal_events.writeOpenTurnNamingImages(&open_turn.writer, journal.open_turn_id.slice(), journal.pending_resume.?, names);
} else {
try journal_events.writeOpenTurn(&open_turn.writer, journal.open_turn_id.slice(), journal.pending_resume.?);
// An image the session dropped is written by name even here.
open_turn_names_images = session_codec.checkpointNamesImages(journal.pending_resume.?, null, null);
}
}
const snapshot_bytes = if (open_turn_names_images)
journal_events.encodeSnapshotNamingImages(alloc, at_seq, journal.cursor.turn, journal.last_turn_id.slice(), open_turn.written(), checkpoint)
else
journal_events.encodeSnapshot(alloc, at_seq, journal.cursor.turn, journal.last_turn_id.slice(), if (yielded) open_turn.written() else null, checkpoint);
const bytes = snapshot_bytes catch |err| switch (err) {
error.OutOfMemory => return error.OutOfMemory,
error.SnapshotTooLarge => return writeSnapshotSkipped(state, alloc, msg.id, .too_large),
};
@@ -2031,6 +2213,7 @@ fn handleKernelSnapshot(
error.OutOfMemory => return error.OutOfMemory,
error.AttachmentStoreFull => return state.writer.writeResponse(alloc, msg.id, none),
};
if (yielded) journal.noteOpenTurnSnapshot();
const written = try alloc.print("{{\"snapshotAttachment\":{d},\"atSeq\":{d}}}", .{ attachment, at_seq });
defer alloc.free(written);
try state.writer.writeResponse(alloc, msg.id, written);
+200 -23
View File
@@ -26,6 +26,7 @@ const session_runtime = @import("../core/session/session.zig");
const agent_execution_memory = @import("../core/agent/execution_memory.zig");
const tool_dispatch = @import("../core/tooling/tool_dispatch.zig");
const tool_call_presentation = @import("tool_call_presentation.zig");
const runtime_tool_presentation = @import("../core/agent/runtime/tool_presentation.zig");
const mcp_runtime = @import("../core/mcp/mcp_runtime.zig");
const mcp_host = @import("../core/mcp_host/host.zig");
const mcp_contract = @import("../core/mcp/mcp_contract.zig");
@@ -729,7 +730,9 @@ pub fn handleLoadWasmSession(state: *server.ServerState, alloc: Allocator, msg:
if (state.active_session) |*active| {
if (sameSessionId(active.session_id, session_id)) {
for (active.session_rt.agent.history.items) |turn| try sendHistoryTurnAsUpdates(state, alloc, session_id, turn);
var sources: runtime_tool_presentation.ReplayedSavedOutputSources = .{};
defer sources.deinit(alloc);
for (active.session_rt.agent.history.items) |turn| try sendHistoryTurnAsUpdates(state, alloc, session_id, &sources, turn);
try sendActiveSessionInfoUpdate(state, alloc);
return writeLoadSessionResponse(state, alloc, msg, active.model, observer_hooks);
}
@@ -792,7 +795,9 @@ pub fn handleLoadWasmSession(state: *server.ServerState, alloc: Allocator, msg:
sid_owned = false;
model_owned = false;
session_rt_owned = false;
for (state.active_session.?.session_rt.agent.history.items) |turn| try sendHistoryTurnAsUpdates(state, alloc, session_id, turn);
var sources: runtime_tool_presentation.ReplayedSavedOutputSources = .{};
defer sources.deinit(alloc);
for (state.active_session.?.session_rt.agent.history.items) |turn| try sendHistoryTurnAsUpdates(state, alloc, session_id, &sources, turn);
try sendActiveSessionInfoUpdate(state, alloc);
try writeLoadSessionResponse(state, alloc, msg, state.active_session.?.model, observer_hooks);
}
@@ -972,13 +977,17 @@ fn handleRestoreSession(
}
if (previous_host) |h| h.destroy();
server.enableSubagentHost(state);
// One map, so a recovered read can name a call from an earlier turn.
var sources: runtime_tool_presentation.ReplayedSavedOutputSources = .{};
defer sources.deinit(alloc);
if (kind.replaysHistory()) {
try sendActiveHistoryUpdates(state, alloc, session_id);
try sendActiveHistoryUpdates(state, alloc, session_id, &sources);
}
try sendPendingRecoveryUpdate(
state,
alloc,
session_id,
&sources,
if (active.writable) |*writable| writable.state.recovery_checkpoint else null,
);
try sendActiveSessionInfoUpdate(state, alloc);
@@ -1138,13 +1147,17 @@ fn handleRestoreSession(
session_rt_owned = false;
session_mcp_owned = false;
session_host_owned = false;
// One map, so a recovered read can name a call from an earlier turn.
var sources: runtime_tool_presentation.ReplayedSavedOutputSources = .{};
defer sources.deinit(alloc);
if (kind.replaysHistory()) {
try sendActiveHistoryUpdates(state, alloc, session_id);
try sendActiveHistoryUpdates(state, alloc, session_id, &sources);
}
try sendPendingRecoveryUpdate(
state,
alloc,
session_id,
&sources,
if (state.active_session.?.writable) |*value| value.state.recovery_checkpoint else null,
);
try sendActiveSessionInfoUpdate(state, alloc);
@@ -1291,11 +1304,12 @@ fn sendPendingRecoveryUpdate(
state: *server.ServerState,
alloc: Allocator,
session_id: []const u8,
sources: *runtime_tool_presentation.ReplayedSavedOutputSources,
checkpoint: ?session_codec.RecoveryCheckpoint,
) !void {
const recovery = checkpoint orelse return;
try sendUserHistoryTurn(state, alloc, session_id, recovery.user);
try sendExecutionHistory(state, alloc, session_id, recovery.execution);
try sendExecutionHistory(state, alloc, session_id, sources, recovery.execution);
if (recovery.assistant_source.len > 0) {
try sendAgentHistoryChunk(state, alloc, session_id, recovery.assistant_source);
}
@@ -1880,19 +1894,25 @@ fn parseListCursor(raw: []const u8) !session_store.ResumableSessionContinuation
return .{ .updated_at_ms = updated_at_ms, .id = id };
}
fn sendActiveHistoryUpdates(state: *server.ServerState, alloc: Allocator, session_id: []const u8) !void {
fn sendActiveHistoryUpdates(
state: *server.ServerState,
alloc: Allocator,
session_id: []const u8,
sources: *runtime_tool_presentation.ReplayedSavedOutputSources,
) !void {
const active = &state.active_session.?;
if (active.store) |store| {
const Visitor = struct {
state: *server.ServerState,
alloc: Allocator,
session_id: []const u8,
sources: *runtime_tool_presentation.ReplayedSavedOutputSources,
pub fn append(self: *@This(), turn: types.HistoryTurn) !void {
try sendHistoryTurnAsUpdates(self.state, self.alloc, self.session_id, turn);
try sendHistoryTurnAsUpdates(self.state, self.alloc, self.session_id, self.sources, turn);
}
};
var visitor = Visitor{ .state = state, .alloc = alloc, .session_id = session_id };
var visitor = Visitor{ .state = state, .alloc = alloc, .session_id = session_id, .sources = sources };
return store.visitConversationHistory(alloc, session_id, &visitor);
}
// Every saved turn, as v1 replays it, not only those in memory.
@@ -1901,20 +1921,27 @@ fn sendActiveHistoryUpdates(state: *server.ServerState, alloc: Allocator, sessio
state: *server.ServerState,
alloc: Allocator,
session_id: []const u8,
sources: *runtime_tool_presentation.ReplayedSavedOutputSources,
pub fn append(self: *@This(), turn: types.HistoryTurn) !void {
try sendHistoryTurnAsUpdates(self.state, self.alloc, self.session_id, turn);
try sendHistoryTurnAsUpdates(self.state, self.alloc, self.session_id, self.sources, turn);
}
};
var visitor = Visitor{ .state = state, .alloc = alloc, .session_id = session_id };
var visitor = Visitor{ .state = state, .alloc = alloc, .session_id = session_id, .sources = sources };
return v2.visitHistory(alloc, &visitor);
}
for (active.session_rt.agent.history.items) |turn| {
try sendHistoryTurnAsUpdates(state, alloc, session_id, turn);
try sendHistoryTurnAsUpdates(state, alloc, session_id, sources, turn);
}
}
fn sendHistoryTurnAsUpdates(state: *server.ServerState, alloc: Allocator, session_id: []const u8, turn: types.HistoryTurn) !void {
fn sendHistoryTurnAsUpdates(
state: *server.ServerState,
alloc: Allocator,
session_id: []const u8,
sources: *runtime_tool_presentation.ReplayedSavedOutputSources,
turn: types.HistoryTurn,
) !void {
switch (turn) {
.assistant => |assistant| try sendUserHistoryTurn(state, alloc, session_id, assistant.user),
.interrupted => |interrupted| try sendUserHistoryTurn(state, alloc, session_id, interrupted.user),
@@ -1923,11 +1950,11 @@ fn sendHistoryTurnAsUpdates(state: *server.ServerState, alloc: Allocator, sessio
switch (turn) {
.assistant => |assistant| {
try sendExecutionHistory(state, alloc, session_id, assistant.execution);
try sendExecutionHistory(state, alloc, session_id, sources, assistant.execution);
try sendAgentHistoryChunk(state, alloc, session_id, assistant.assistant);
},
.interrupted => |i| {
try sendExecutionHistory(state, alloc, session_id, i.execution);
try sendExecutionHistory(state, alloc, session_id, sources, i.execution);
if (i.assistant) |assistant| {
if (assistant.len > 0) try sendAgentHistoryChunk(state, alloc, session_id, assistant);
}
@@ -2012,6 +2039,7 @@ fn sendExecutionHistory(
state: *server.ServerState,
alloc: Allocator,
session_id: []const u8,
sources: *runtime_tool_presentation.ReplayedSavedOutputSources,
execution: types.ExecutionMemory,
) !void {
if (execution.isEmpty()) return;
@@ -2022,6 +2050,11 @@ fn sendExecutionHistory(
tool_call_presentation.activeToolRegistry(state),
activeSessionMcp(state),
if (state.active_session) |*active| &active.tool_identities else null,
.{
.alloc = alloc,
.workspace_root = if (state.active_session) |*active| active.workspace_root else state.workspace_root,
.sources = sources,
},
execution,
);
for (frames) |frame| {
@@ -2066,14 +2099,37 @@ fn findToolResultIndex(results: []const types.PersistedToolResult, call_id: []co
return null;
}
/// Calls whose saved output later replayed reads name, kept across the turns
/// of one replay.
const ReplaySources = struct {
alloc: Allocator,
workspace_root: []const u8,
sources: *runtime_tool_presentation.ReplayedSavedOutputSources,
};
fn planToolCallFrame(
arena: Allocator,
registry: tool_dispatch.Registry,
mcp: ?*mcp_runtime.McpRuntime,
identities: ?*tool_call_identities.Record,
call: types.ToolCall,
replay_sources: ReplaySources,
replayed_call: types.ToolCall,
result: ?types.PersistedToolResult,
) !ReplayToolCall {
var call = replayed_call;
call.resolved_display_target = try runtime_tool_presentation.readToolResultDisplayTarget(
arena,
registry,
call,
replay_sources.sources,
);
if (result) |saved| try replay_sources.sources.record(
replay_sources.alloc,
registry,
replay_sources.workspace_root,
call,
saved,
);
const name = tool_call_presentation.acpToolName(call.name);
// MCP identity comes from the live catalog, or from the identity recorded
// when the session made the call; it is never guessed from the exposed
@@ -2136,6 +2192,7 @@ fn planExecutionReplay(
registry: tool_dispatch.Registry,
mcp: ?*mcp_runtime.McpRuntime,
identities: ?*tool_call_identities.Record,
replay_sources: ReplaySources,
execution: types.ExecutionMemory,
) ![]ReplayFrame {
var frames: std.ArrayList(ReplayFrame) = .empty;
@@ -2151,7 +2208,7 @@ fn planExecutionReplay(
matched[i] = true;
break :blk step.tool_results[i];
} else null;
try frames.append(arena, .{ .tool_call = try planToolCallFrame(arena, registry, mcp, identities, call, result) });
try frames.append(arena, .{ .tool_call = try planToolCallFrame(arena, registry, mcp, identities, replay_sources, call, result) });
}
// Results without a surviving call record (e.g. trimmed history) still
// replay as completed frames so the transcript stays faithful.
@@ -2162,7 +2219,7 @@ fn planExecutionReplay(
.name = result.tool_name,
.arguments_json = "{}",
};
try frames.append(arena, .{ .tool_call = try planToolCallFrame(arena, registry, mcp, identities, call, result) });
try frames.append(arena, .{ .tool_call = try planToolCallFrame(arena, registry, mcp, identities, replay_sources, call, result) });
}
steering_index = try appendSteeringReplay(arena, &frames, execution.steering, steering_index, step_index + 1);
}
@@ -2701,7 +2758,7 @@ test "ACP history excludes typed summaries without filtering original user text"
var state = try initAcpSessionTestState(arena, workspace, capture);
defer state.deinit();
try sendHistoryTurnAsUpdates(&state, arena, "session-1", .{ .compacted_summary = .{
try sendTestHistoryTurn(&state, arena, "session-1", .{ .compacted_summary = .{
.summary = @constCast("internal summary"),
.removed_turn_count = 1,
.compaction_count = 1,
@@ -2709,7 +2766,7 @@ test "ACP history excludes typed summaries without filtering original user text"
try std.testing.expectEqual(@as(u64, 0), try capture.length(io_mod.getIo()));
const original = "Explain <context_handoff> without hiding my question.";
try sendHistoryTurnAsUpdates(&state, arena, "session-1", .{ .assistant = .{
try sendTestHistoryTurn(&state, arena, "session-1", .{ .assistant = .{
.user = .{ .text = @constCast(original) },
.assistant = @constCast("original reply"),
} });
@@ -2743,7 +2800,7 @@ test "ACP interrupted history replay hides model-only abort context" {
var state = try initAcpSessionTestState(arena, workspace, capture);
defer state.deinit();
var completed_tool_names = [_][]u8{@constCast("read_file")};
try sendHistoryTurnAsUpdates(&state, arena, "session-1", .{ .interrupted = .{
try sendTestHistoryTurn(&state, arena, "session-1", .{ .interrupted = .{
.user = .{ .text = @constCast("inspect the project") },
.completed_tool_names = completed_tool_names[0..],
} });
@@ -2809,7 +2866,7 @@ test "ACP history replay emits structured tool call frames" {
.tool_calls = calls[0..],
.tool_results = results[0..],
}};
try sendHistoryTurnAsUpdates(&state, arena, "session-1", .{ .assistant = .{
try sendTestHistoryTurn(&state, arena, "session-1", .{ .assistant = .{
.user = .{ .text = @constCast("read and write") },
.assistant = @constCast("All done."),
.execution = .{ .tool_steps = steps[0..] },
@@ -2857,7 +2914,7 @@ test "ACP history replay leaves resultless tool calls pending" {
.{ .id = "call_orphan", .name = "read_file", .arguments_json = "{\"path\":\"a.txt\"}" },
};
var steps = [_]types.ToolExecutionStep{.{ .tool_calls = calls[0..] }};
try sendHistoryTurnAsUpdates(&state, arena, "session-1", .{ .interrupted = .{
try sendTestHistoryTurn(&state, arena, "session-1", .{ .interrupted = .{
.user = .{ .text = @constCast("inspect") },
.execution = .{ .tool_steps = steps[0..] },
} });
@@ -2895,7 +2952,7 @@ test "ACP history replay redacts sensitive tool arguments" {
.stored_output_bytes = 2,
}};
var steps = [_]types.ToolExecutionStep{.{ .tool_calls = calls[0..], .tool_results = results[0..] }};
try sendHistoryTurnAsUpdates(&state, arena, "session-1", .{ .assistant = .{
try sendTestHistoryTurn(&state, arena, "session-1", .{ .assistant = .{
.user = .{ .text = @constCast("run it") },
.assistant = @constCast("done"),
.execution = .{ .tool_steps = steps[0..] },
@@ -2940,11 +2997,14 @@ test "execution replay plan interleaves assistant text and orphan results" {
.tool_results = results[0..],
}};
var sources: runtime_tool_presentation.ReplayedSavedOutputSources = .{};
defer sources.deinit(arena);
const frames = try planExecutionReplay(
arena,
@import("../builtins/tools.zig").registry,
null,
null,
.{ .alloc = arena, .workspace_root = "/workspace", .sources = &sources },
.{ .tool_steps = steps[0..] },
);
try std.testing.expectEqual(@as(usize, 3), frames.len);
@@ -2958,6 +3018,123 @@ test "execution replay plan interleaves assistant text and orphan results" {
try std.testing.expect(frames[2].tool_call.raw_input != null);
}
fn sendTestHistoryTurn(state: *server.ServerState, alloc: Allocator, session_id: []const u8, turn: types.HistoryTurn) !void {
var sources: runtime_tool_presentation.ReplayedSavedOutputSources = .{};
defer sources.deinit(alloc);
return sendHistoryTurnAsUpdates(state, alloc, session_id, &sources, turn);
}
test "execution replay titles reads with the call whose saved output they read" {
const alloc = std.testing.allocator;
var arena_state = std.heap.ArenaAllocator.init(alloc);
defer arena_state.deinit();
const arena = arena_state.allocator();
const replay = "fx-command-replay-0123.bin";
var run_calls = [_]types.ToolCall{
.{ .id = "run", .name = "shell", .arguments_json = "{\"action\":\"run\",\"command\":\"zig build test\"}" },
};
var run_results = [_]types.PersistedToolResult{.{
.tool_call_id = @constCast("run"),
.tool_name = @constCast("shell"),
.status = .success,
.output = @constCast("{\"exit_code\":0}"),
.output_bytes = 15,
.stored_output_bytes = 15,
.command_output_replay = .{ .available = .{ .handle = replay, .framed_bytes = 64 } },
}};
var read_calls = [_]types.ToolCall{
.{ .id = "read", .name = "read_tool_result", .arguments_json = "{\"request\":{\"handle\":\"" ++ replay ++ "\",\"query\":\"error:\"}}" },
};
var read_results = [_]types.PersistedToolResult{.{
.tool_call_id = @constCast("read"),
.tool_name = @constCast("read_tool_result"),
.status = .success,
.output = @constCast("error: x"),
.output_bytes = 8,
.stored_output_bytes = 8,
}};
var steps = [_]types.ToolExecutionStep{
.{ .tool_calls = run_calls[0..], .tool_results = run_results[0..] },
.{ .tool_calls = read_calls[0..], .tool_results = read_results[0..] },
};
var sources: runtime_tool_presentation.ReplayedSavedOutputSources = .{};
defer sources.deinit(alloc);
const registry = @import("../builtins/tools.zig").registry;
const replay_sources: ReplaySources = .{ .alloc = alloc, .workspace_root = "/workspace", .sources = &sources };
const frames = try planExecutionReplay(arena, registry, null, null, replay_sources, .{ .tool_steps = steps[0..] });
try std.testing.expectEqual(@as(usize, 2), frames.len);
try std.testing.expectEqualStrings("Searching zig build test output for \"error:\"", frames[1].tool_call.title);
// A later turn replayed with the same sources still names the command.
var later_calls = [_]types.ToolCall{
.{ .id = "later", .name = "read_tool_result", .arguments_json = "{\"request\":{\"handle\":\"" ++ replay ++ "\"}}" },
};
var later_steps = [_]types.ToolExecutionStep{.{ .tool_calls = later_calls[0..] }};
const later = try planExecutionReplay(arena, registry, null, null, replay_sources, .{ .tool_steps = later_steps[0..] });
try std.testing.expectEqualStrings("Reading more output from zig build test", later[0].tool_call.title);
}
test "ACP load titles a recovered read with the call from an earlier turn" {
const alloc = std.testing.allocator;
var tmp = std.testing.tmpDir(.{});
defer tmp.cleanup();
var arena_state = std.heap.ArenaAllocator.init(alloc);
defer arena_state.deinit();
const arena = arena_state.allocator();
const workspace = try io_mod.dirRealpathAlloc(alloc, tmp.dir, ".");
defer alloc.free(workspace);
var capture = try tmp.dir.createFile(io_mod.getIo(), "history.jsonl", .{ .read = true });
defer capture.close(io_mod.getIo());
var state = try initAcpSessionTestState(arena, workspace, capture);
defer state.deinit();
const replay = "fx-command-replay-0123.bin";
var run_calls = [_]types.ToolCall{
.{ .id = "run", .name = "shell", .arguments_json = "{\"action\":\"run\",\"command\":\"zig build test\"}" },
};
var run_results = [_]types.PersistedToolResult{.{
.tool_call_id = @constCast("run"),
.tool_name = @constCast("shell"),
.status = .success,
.output = @constCast("{\"exit_code\":0}"),
.output_bytes = 15,
.stored_output_bytes = 15,
.command_output_replay = .{ .available = .{ .handle = replay, .framed_bytes = 64 } },
}};
var run_steps = [_]types.ToolExecutionStep{.{ .tool_calls = run_calls[0..], .tool_results = run_results[0..] }};
var read_calls = [_]types.ToolCall{
.{ .id = "read", .name = "read_tool_result", .arguments_json = "{\"request\":{\"handle\":\"" ++ replay ++ "\",\"query\":\"error:\"}}" },
};
var read_steps = [_]types.ToolExecutionStep{.{ .tool_calls = read_calls[0..] }};
var sources: runtime_tool_presentation.ReplayedSavedOutputSources = .{};
defer sources.deinit(arena);
try sendHistoryTurnAsUpdates(&state, arena, "session-1", &sources, .{ .assistant = .{
.user = .{ .text = @constCast("run the tests") },
.assistant = @constCast("done"),
.execution = .{ .tool_steps = run_steps[0..] },
} });
try sendPendingRecoveryUpdate(&state, arena, "session-1", &sources, .{
.turn_id = 2,
.user = .{ .text = @constCast("find the failure") },
.assistant_source = @constCast(""),
.execution = .{ .tool_steps = read_steps[0..] },
.cause = .response_interrupted,
.action = .paused,
.authority = .{ .provider = .gateway, .model = @constCast("test-model") },
.requested_fast_mode = false,
.fast_mode = false,
.max_provider_attempts = 1,
.consumed_provider_attempts = 1,
});
const captured = try readCaptured(&capture, alloc);
defer alloc.free(captured);
try std.testing.expect(std.mem.find(u8, captured, "Searching zig build test output for \\\"error:\\\"") != null);
}
test "ACP load maps one-off child denial to invalid params" {
const alloc = std.testing.allocator;
var tmp = std.testing.tmpDir(.{});
+25 -1
View File
@@ -38,8 +38,13 @@ pub fn mapToolKind(tool_name: []const u8) acp_types.ToolCallKind {
}
pub fn describeToolTitle(registry: tool_dispatch.Registry, arena: Allocator, call: ToolCall) ![]const u8 {
if (registry.lookup(call.name) != null) {
if (registry.lookup(call.name)) |tool| {
if (try tool_presentation.formatSubagentPlainAction(arena, call, .identity)) |title| return title;
// A read's input is an opaque handle, so its verb alone says nothing
// about what it reads.
if (tool.executor_kind == .read_tool_result) {
return tool_presentation.formatPlainAction(arena, .{ .tool_registry = registry, .call = call });
}
}
if (tool_presentation.isProviderSearchAlias(call.name)) {
return tool_presentation.formatPlainAction(arena, .{
@@ -161,6 +166,25 @@ test "describeToolCall flags internal discovery and names MCP identity" {
try std.testing.expect(!titled.meta.internal);
}
test "read titles say what they read while other tools keep their verb" {
var arena_state = std.heap.ArenaAllocator.init(std.testing.allocator);
defer arena_state.deinit();
const arena = arena_state.allocator();
const registry = builtin_tools.registry;
const read: ToolCall = .{
.id = "read",
.name = "read_tool_result",
.arguments_json = "{\"request\":{\"handle\":\"fx-command-replay-0123.bin\",\"query\":\"error:\"}}",
.resolved_display_target = "zig build test output for \"error:\"",
};
try std.testing.expectEqualStrings("Searching zig build test output for \"error:\"", describeToolCall(registry, arena, read, null).title);
var unresolved = read;
unresolved.resolved_display_target = null;
try std.testing.expectEqualStrings("Searching command output", describeToolCall(registry, arena, unresolved, null).title);
const file: ToolCall = .{ .id = "file", .name = "read_file", .arguments_json = "{\"path\":\"a.txt\"}" };
try std.testing.expectEqualStrings("Reading", describeToolCall(registry, arena, file, null).title);
}
test "toolUpdateContentText clips long output and guards unsafe bytes" {
const long = text_utils.repeat("x", 500);
const clipped = toolUpdateContentText(false, long);
+1 -1
View File
@@ -666,7 +666,7 @@ pub fn loadHost(
}
const h = try mcp_host.Host.create(alloc, io_mod.getIo(), loaded.configs.items, &inherited, .{
.client_version = build_options.app_version,
.credentials = .detect(alloc, home),
.credentials = .{ .detect = home },
.questions = extras.questions,
.notify = extras.notify,
}, builtins);
+5 -4
View File
@@ -830,8 +830,8 @@ pub const read_tool_result = ToolSpec{
.requires_approval = false,
.action_label = "Reading",
.completed_action_label = "Read",
.label_arg_kind = .path,
.label_arg_default = "tool result",
.label_arg_default = "saved output",
.presentation_fn = read_tool_result_impl.presentation,
.permission_target_kind = .none,
.decode = read_tool_result_impl.decode,
.validate = read_tool_result_impl.validate,
@@ -1707,8 +1707,9 @@ test "built-in read_tool_result owns product metadata schema and callbacks" {
try std.testing.expect(!read_tool_result.requires_approval);
try std.testing.expectEqualStrings("Reading", read_tool_result.action_label);
try std.testing.expectEqualStrings("Read", read_tool_result.completed_action_label);
try std.testing.expectEqual(tool_dispatch.LabelArgKind.path, read_tool_result.label_arg_kind);
try std.testing.expectEqualStrings("tool result", read_tool_result.label_arg_default);
try std.testing.expectEqual(tool_dispatch.LabelArgKind.none, read_tool_result.label_arg_kind);
try std.testing.expectEqualStrings("saved output", read_tool_result.label_arg_default);
try std.testing.expect(read_tool_result.presentation_fn.? == read_tool_result_impl.presentation);
try std.testing.expectEqual(tool_dispatch.PermissionTargetKind.none, read_tool_result.permission_target_kind);
try std.testing.expect(read_tool_result.decode == read_tool_result_impl.decode);
try std.testing.expect(read_tool_result.validate.? == read_tool_result_impl.validate);
+12
View File
@@ -2,6 +2,7 @@ const std = @import("std");
const testing_allocator = @import("../../shared/testing_allocator.zig");
const types = @import("../../shared/types.zig");
const checkpoint_codec = @import("checkpoint.zig");
const session_codec = @import("../../session/session_codec.zig");
const runtime_prompt_context = @import("prompt_context.zig");
const debug_trace = @import("../../shared/debug_trace.zig");
const text_utils = @import("../../shared/text_utils.zig");
@@ -44,6 +45,17 @@ pub const Agent = struct {
return checkpoint_codec.encode(alloc, self.history.items, self.turn_usage, meta);
}
/// Like `checkpoint`, but names the images a libfx host's store holds
/// instead of holding their bytes.
pub fn checkpointNamingImages(
self: *const Agent,
alloc: Allocator,
meta: checkpoint_codec.Meta,
stored: *const session_codec.StoredImages,
) checkpoint_codec.Error![]u8 {
return checkpoint_codec.encodeNamingImages(alloc, self.history.items, self.turn_usage, meta, stored);
}
/// Restores `bytes` into this fresh agent. Returns what the checkpoint
/// recorded about the agent that saved it, owned by `alloc`
/// (`Meta.free`).
+298 -12
View File
@@ -11,6 +11,12 @@ const magic = "FXCP";
/// Its little-endian payload is:
/// u32 json_len | json | u32 blob_count | (u32 blob_len | blob)*
const version: u16 = 2;
/// Version 3 has version 2's layout, and its JSON may name images instead of
/// holding their bytes: ones the host stored (`inline_ref`), or ones the
/// session dropped. Only a checkpoint that names one is version 3, so an fx
/// that reads only version 2 reports a newer checkpoint instead of an
/// invalid one.
const naming_version: u16 = 3;
/// Version 1 embedded inline image bytes as base64 inside a JSON-only payload.
const json_only_version: u16 = 1;
const header_bytes: usize = 4 + 2 + 2 + 4 + Sha256.digest_length;
@@ -89,12 +95,34 @@ pub const Decoded = struct {
};
/// `meta` is recorded when any of its fields is set; each must fit
/// `max_meta_field_bytes`.
/// `max_meta_field_bytes`. The checkpoint holds every image's bytes.
pub fn encode(
alloc: Allocator,
history: []const types.HistoryTurn,
usage: types.Usage,
meta: Meta,
) Error![]u8 {
return encodeWith(alloc, history, usage, meta, null);
}
/// Like `encode`, but names the images `stored` holds instead of holding
/// their bytes.
pub fn encodeNamingImages(
alloc: Allocator,
history: []const types.HistoryTurn,
usage: types.Usage,
meta: Meta,
stored: *const session_codec.StoredImages,
) Error![]u8 {
return encodeWith(alloc, history, usage, meta, stored);
}
fn encodeWith(
alloc: Allocator,
history: []const types.HistoryTurn,
usage: types.Usage,
meta: Meta,
stored: ?*const session_codec.StoredImages,
) Error![]u8 {
std.debug.assert(meta.fits());
if (history.len > max_history_turns) return error.CheckpointTooLarge;
@@ -105,12 +133,18 @@ pub fn encode(
defer blobs.deinit(alloc);
const image_blobs: session_codec.ImageBlobs = .{ .alloc = alloc, .items = &blobs };
var blob_section_bytes: usize = 0;
var names_images = false;
json.writer.writeAll("{\"history\":[") catch return error.OutOfMemory;
for (history, 0..) |turn, index| {
if (index > 0) json.writer.writeByte(',') catch return error.OutOfMemory;
const first_new_blob = blobs.items.len;
session_codec.writeHistoryTurnWithImageBlobs(&json.writer, turn, image_blobs) catch |err| switch (err) {
names_images = names_images or session_codec.turnNamesImages(turn, stored);
const written = if (stored) |names|
session_codec.writeHistoryTurnNamingImages(&json.writer, turn, .{ .stored = names, .blobs = image_blobs })
else
session_codec.writeHistoryTurnWithImageBlobs(&json.writer, turn, image_blobs);
written catch |err| switch (err) {
error.InvalidSessionFormat => return error.InvalidCheckpoint,
else => return error.OutOfMemory,
};
@@ -138,7 +172,7 @@ pub fn encode(
std.debug.assert(cursor == out.len);
@memcpy(out[0..magic.len], magic);
std.mem.writeInt(u16, out[4..6], version, .little);
std.mem.writeInt(u16, out[4..6], if (names_images) naming_version else version, .little);
std.mem.writeInt(u16, out[6..8], 0, .little);
std.mem.writeInt(u32, out[8..12], @intCast(payload_len), .little);
Sha256.hash(out[header_bytes..], out[12..header_bytes], .{});
@@ -161,12 +195,19 @@ fn writeSection(out: []u8, cursor: *usize, bytes: []const u8) void {
}
pub fn decode(alloc: Allocator, bytes: []const u8) Error!Decoded {
return decodeWithImages(alloc, bytes, null);
}
/// Like `decode`, but resolves named images through `names`. Without it a
/// checkpoint that names images is invalid. A name `names` cannot resolve is
/// recorded there, and the decoded images it names have no bytes.
pub fn decodeWithImages(alloc: Allocator, bytes: []const u8, names: ?*session_codec.ImageResolver) Error!Decoded {
if (bytes.len < header_bytes or bytes.len > max_checkpoint_bytes) {
return error.CorruptCheckpoint;
}
if (!std.mem.eql(u8, bytes[0..magic.len], magic)) return error.CorruptCheckpoint;
const checkpoint_version = std.mem.readInt(u16, bytes[4..6], .little);
if (checkpoint_version != version and checkpoint_version != json_only_version) {
if (checkpoint_version != version and checkpoint_version != json_only_version and checkpoint_version != naming_version) {
return error.UnsupportedCheckpointVersion;
}
if (std.mem.readInt(u16, bytes[6..8], .little) != 0) {
@@ -180,7 +221,7 @@ pub fn decode(alloc: Allocator, bytes: []const u8) Error!Decoded {
return error.CorruptCheckpoint;
}
const payload = bytes[header_bytes..];
if (checkpoint_version == json_only_version) return decodeJson(alloc, payload, null);
if (checkpoint_version == json_only_version) return decodeJson(alloc, payload, .{});
var reader: PayloadReader = .{ .bytes = payload };
const json = reader.section() orelse return error.CorruptCheckpoint;
@@ -192,7 +233,11 @@ pub fn decode(alloc: Allocator, bytes: []const u8) Error!Decoded {
for (blobs) |*blob| blob.* = reader.section() orelse return error.CorruptCheckpoint;
if (reader.remaining() != 0) return error.CorruptCheckpoint;
var blob_reader: session_codec.ImageBlobReader = .{ .blobs = blobs };
return decodeJson(alloc, json, &blob_reader);
// Only a version 3 checkpoint names images.
return decodeJson(alloc, json, .{
.blobs = &blob_reader,
.names = if (checkpoint_version == naming_version) names else null,
});
}
/// Reads length-prefixed sections from a version 2 payload. Returned slices
@@ -221,7 +266,7 @@ const PayloadReader = struct {
}
};
fn decodeJson(alloc: Allocator, json: []const u8, image_blobs: ?*session_codec.ImageBlobReader) Error!Decoded {
fn decodeJson(alloc: Allocator, json: []const u8, images: session_codec.ImageSources) Error!Decoded {
const parsed = std.json.parseFromSlice(
std.json.Value,
alloc,
@@ -244,13 +289,13 @@ fn decodeJson(alloc: Allocator, json: []const u8, image_blobs: ?*session_codec.I
alloc.free(history);
}
for (history_value.array.items, 0..) |turn_value, index| {
history[index] = session_codec.parseHistoryTurnWithImageBlobs(alloc, turn_value, image_blobs) catch |err| switch (err) {
history[index] = session_codec.parseHistoryTurnWithImages(alloc, turn_value, images) catch |err| switch (err) {
error.OutOfMemory => return error.OutOfMemory,
else => return error.InvalidCheckpoint,
};
decoded_count += 1;
}
if (image_blobs) |blobs| {
if (images.blobs) |blobs| {
if (!blobs.consumedAll()) return error.InvalidCheckpoint;
}
const usage = std.json.parseFromValueLeaky(types.Usage, alloc, usage_value, .{}) catch
@@ -327,9 +372,9 @@ test "kernel checkpoint refuses meta that is not short text" {
"{\"history\":[],\"usage\":{},\"meta\":{\"model\":7}}",
"{\"history\":[],\"usage\":{},\"meta\":{\"model\":\"" ++ text_utils.repeat("m", max_meta_field_bytes + 1) ++ "\"}}",
};
for (cases) |json| try std.testing.expectError(error.InvalidCheckpoint, decodeJson(alloc, json, null));
for (cases) |json| try std.testing.expectError(error.InvalidCheckpoint, decodeJson(alloc, json, .{}));
// Fields it does not know are left alone.
var decoded = try decodeJson(alloc, "{\"history\":[],\"usage\":{},\"meta\":{\"model\":\"m\",\"later\":1}}", null);
var decoded = try decodeJson(alloc, "{\"history\":[],\"usage\":{},\"meta\":{\"model\":\"m\",\"later\":1}}", .{});
defer decoded.deinit(alloc);
try std.testing.expectEqualStrings("m", decoded.meta.model);
try std.testing.expectEqualStrings("", decoded.meta.libfx_version);
@@ -425,7 +470,7 @@ test "kernel checkpoint rejects corruption and unsupported versions" {
const unsupported = try alloc.dupe(u8, bytes);
defer alloc.free(unsupported);
std.mem.writeInt(u16, unsupported[4..6], version + 1, .little);
std.mem.writeInt(u16, unsupported[4..6], naming_version + 1, .little);
try std.testing.expectError(
error.UnsupportedCheckpointVersion,
decode(alloc, unsupported),
@@ -577,3 +622,244 @@ test "kernel checkpoint requires each image blob once, in order" {
resealForTest(unreferenced);
try std.testing.expectError(error.InvalidCheckpoint, decode(alloc, unreferenced));
}
test "kernel checkpoint names stored images and keeps the bytes of the rest" {
const alloc = std.testing.allocator;
const stored_png = "\x89PNG\r\n\x1a\nstored-in-host";
const kept_png = "\x89PNG\r\n\x1a\nnever-stored";
var stored_image: TestImageHistory = undefined;
stored_image.init(stored_png);
var kept_image: TestImageHistory = undefined;
kept_image.init(kept_png);
const history = [_]types.HistoryTurn{ stored_image.history[0], kept_image.history[0] };
var stored: session_codec.StoredImages = .{};
defer stored.deinit(alloc);
try stored.add(alloc, stored_image.digest_hex);
const bytes = try encodeNamingImages(alloc, &history, .{}, .{}, &stored);
defer alloc.free(bytes);
try std.testing.expectEqual(naming_version, std.mem.readInt(u16, bytes[4..6], .little));
// The stored image is named, not held; the other keeps its raw blob.
try std.testing.expect(std.mem.find(u8, bytes, stored_png) == null);
try std.testing.expect(std.mem.find(u8, bytes, "\"inline_ref\":\"" ++ stored_image.digest_hex ++ "\"") != null);
try std.testing.expect(std.mem.endsWith(u8, bytes, kept_png));
// Without the host's images it cannot be read.
try std.testing.expectError(error.InvalidCheckpoint, decode(alloc, bytes));
var staged: session_codec.StagedImages = .{};
defer staged.deinit(alloc);
try staged.add(alloc, stored_image.digest_hex, try alloc.dupe(u8, stored_png));
var resolver: session_codec.ImageResolver = .{ .alloc = alloc, .staged = &staged };
defer resolver.deinit();
var decoded = try decodeWithImages(alloc, bytes, &resolver);
defer decoded.deinit(alloc);
try std.testing.expectEqualStrings(stored_png, decoded.history[0].assistant.user.images[0].inline_data.?);
try std.testing.expectEqualStrings(kept_png, decoded.history[1].assistant.user.images[0].inline_data.?);
try std.testing.expectEqual(@as(usize, 0), resolver.missing.count());
try std.testing.expect(resolver.resolved.contains(stored_image.digest_hex));
}
test "kernel checkpoint names a stored image with its size" {
const alloc = std.testing.allocator;
const png = "\x89PNG\r\n\x1a\nsized-in-host";
var fixture: TestImageHistory = undefined;
fixture.init(png);
var stored: session_codec.StoredImages = .{};
defer stored.deinit(alloc);
try stored.add(alloc, fixture.digest_hex);
const bytes = try encodeNamingImages(alloc, &fixture.history, .{}, .{}, &stored);
defer alloc.free(bytes);
try std.testing.expect(std.mem.find(u8, bytes, std.fmt.comptimePrint("\"inline_size\":{d}", .{png.len})) != null);
// A staged image whose size is not the one recorded is refused.
const sized = "\"inline_size\":";
const resized = try alloc.dupe(u8, bytes);
defer alloc.free(resized);
const at = std.mem.find(u8, resized, sized).? + sized.len;
resized[at] += 1;
resealForTest(resized);
var staged: session_codec.StagedImages = .{};
defer staged.deinit(alloc);
try staged.add(alloc, fixture.digest_hex, try alloc.dupe(u8, png));
var resolver: session_codec.ImageResolver = .{ .alloc = alloc, .staged = &staged };
defer resolver.deinit();
try std.testing.expectError(error.InvalidCheckpoint, decodeWithImages(alloc, resized, &resolver));
}
test "kernel checkpoint keeps a dropped image's name alone, at the naming version" {
const alloc = std.testing.allocator;
var fixture: TestImageHistory = undefined;
fixture.init("\x89PNG\r\n\x1a\ndropped-from-the-session");
fixture.images[0].inline_data = null;
try std.testing.expect(fixture.images[0].droppedFromSession());
// Even where no image is named, a dropped one needs the naming version.
const bytes = try encode(alloc, &fixture.history, .{}, .{});
defer alloc.free(bytes);
try std.testing.expectEqual(naming_version, std.mem.readInt(u16, bytes[4..6], .little));
try std.testing.expect(std.mem.find(u8, bytes, "\"dropped\":true") != null);
try std.testing.expect(std.mem.find(u8, bytes, "\"inline_ref\"") == null);
var decoded = try decode(alloc, bytes);
defer decoded.deinit(alloc);
const image = decoded.history[0].assistant.user.images[0];
try std.testing.expect(image.droppedFromSession());
try std.testing.expectEqualStrings(&fixture.digest_hex, image.snapshot_sha256.?);
}
/// A history of one turn whose one tool call returned one image, held as
/// base64 and named as the host stored it.
const TestToolImageHistory = struct {
digest_hex: [Sha256.digest_length * 2]u8,
encoded: [64]u8,
images: [1]types.ToolImage,
calls: [1]types.ToolCall,
results: [1]types.PersistedToolResult,
steps: [1]types.ToolExecutionStep,
history: [1]types.HistoryTurn,
fn init(self: *TestToolImageHistory, png: []const u8) void {
var digest: [Sha256.digest_length]u8 = undefined;
Sha256.hash(png, &digest, .{});
self.digest_hex = std.fmt.bytesToHex(digest, .lower);
const data = std.base64.standard.Encoder.encode(&self.encoded, png);
self.images = .{.{ .data = @constCast(data), .mime_type = @constCast("image/png"), .name = self.digest_hex }};
self.calls = .{.{ .id = "call-1", .name = "screenshot", .arguments_json = "{}" }};
self.results = .{.{
.tool_images = &self.images,
.tool_call_id = @constCast("call-1"),
.tool_name = @constCast("screenshot"),
.status = .success,
.output = @constCast("took it"),
.output_bytes = 7,
.stored_output_bytes = 7,
}};
self.steps = .{.{ .tool_calls = &self.calls, .tool_results = &self.results }};
self.history = .{.{ .assistant = .{
.user = .{ .text = @constCast("look") },
.assistant = @constCast("seen"),
.execution = .{ .tool_steps = &self.steps },
} }};
}
};
test "kernel checkpoint names a stored tool image and reads it back from the host's copy" {
const alloc = std.testing.allocator;
const png = "\x89PNG\r\n\x1a\ntool-image-in-host";
var fixture: TestToolImageHistory = undefined;
fixture.init(png);
var stored: session_codec.StoredImages = .{};
defer stored.deinit(alloc);
const bytes = try encodeNamingImages(alloc, &fixture.history, .{}, .{}, &stored);
defer alloc.free(bytes);
try std.testing.expectEqual(naming_version, std.mem.readInt(u16, bytes[4..6], .little));
// Named and sized, never held.
try std.testing.expect(std.mem.find(u8, bytes, fixture.images[0].data) == null);
try std.testing.expect(std.mem.find(u8, bytes, "\"inline_ref\":\"" ++ fixture.digest_hex ++ "\"") != null);
try std.testing.expect(std.mem.find(u8, bytes, std.fmt.comptimePrint("\"inline_size\":{d}", .{png.len})) != null);
// Without the host's images it cannot be read.
try std.testing.expectError(error.InvalidCheckpoint, decode(alloc, bytes));
var staged: session_codec.StagedImages = .{};
defer staged.deinit(alloc);
try staged.add(alloc, fixture.digest_hex, try alloc.dupe(u8, png));
var resolver: session_codec.ImageResolver = .{ .alloc = alloc, .staged = &staged };
defer resolver.deinit();
var decoded = try decodeWithImages(alloc, bytes, &resolver);
defer decoded.deinit(alloc);
const image = decoded.history[0].assistant.execution.tool_steps[0].tool_results[0].tool_images[0];
try std.testing.expectEqualStrings(fixture.images[0].data, image.data);
try std.testing.expectEqualStrings(&fixture.digest_hex, &image.name.?);
try std.testing.expectEqual(@as(usize, 0), resolver.missing.count());
}
test "kernel checkpoint without a store keeps a tool image's bytes, and a dropped one's name alone" {
const alloc = std.testing.allocator;
var fixture: TestToolImageHistory = undefined;
fixture.init("\x89PNG\r\n\x1a\ntool-image-kept");
const bytes = try encode(alloc, &fixture.history, .{}, .{});
defer alloc.free(bytes);
try std.testing.expectEqual(version, std.mem.readInt(u16, bytes[4..6], .little));
try std.testing.expect(std.mem.find(u8, bytes, fixture.images[0].data) != null);
try std.testing.expect(std.mem.find(u8, bytes, "\"inline_ref\"") == null);
fixture.images[0].data = fixture.images[0].data[0..0];
try std.testing.expect(fixture.images[0].droppedFromSession());
const dropped = try encode(alloc, &fixture.history, .{}, .{});
defer alloc.free(dropped);
try std.testing.expectEqual(naming_version, std.mem.readInt(u16, dropped[4..6], .little));
try std.testing.expect(std.mem.find(u8, dropped, "\"dropped\":true") != null);
try std.testing.expect(std.mem.find(u8, dropped, "\"inline_size\"") == null);
var decoded = try decode(alloc, dropped);
defer decoded.deinit(alloc);
const image = decoded.history[0].assistant.execution.tool_steps[0].tool_results[0].tool_images[0];
try std.testing.expect(image.droppedFromSession());
try std.testing.expectEqualStrings(&fixture.digest_hex, &image.name.?);
}
test "kernel checkpoint naming no stored image stays version 2, byte for byte" {
const alloc = std.testing.allocator;
var fixture: TestImageHistory = undefined;
fixture.init("\x89PNG\r\n\x1a\nnot-in-the-store");
var stored: session_codec.StoredImages = .{};
defer stored.deinit(alloc);
try stored.add(alloc, session_codec.imageNameOf("some other image"));
const plain = try encode(alloc, &fixture.history, .{}, .{});
defer alloc.free(plain);
const named = try encodeNamingImages(alloc, &fixture.history, .{}, .{}, &stored);
defer alloc.free(named);
try std.testing.expectEqualSlices(u8, plain, named);
}
test "kernel checkpoint records a named image the host did not stage" {
const alloc = std.testing.allocator;
var fixture: TestImageHistory = undefined;
fixture.init("\x89PNG\r\n\x1a\nlost-from-the-store");
var stored: session_codec.StoredImages = .{};
defer stored.deinit(alloc);
try stored.add(alloc, fixture.digest_hex);
const bytes = try encodeNamingImages(alloc, &fixture.history, .{}, .{}, &stored);
defer alloc.free(bytes);
var staged: session_codec.StagedImages = .{};
defer staged.deinit(alloc);
var resolver: session_codec.ImageResolver = .{ .alloc = alloc, .staged = &staged };
defer resolver.deinit();
var decoded = try decodeWithImages(alloc, bytes, &resolver);
defer decoded.deinit(alloc);
try std.testing.expect(resolver.missing.contains(fixture.digest_hex));
try std.testing.expect(decoded.history[0].assistant.user.images[0].inline_data == null);
}
test "kernel checkpoint rejects a name that is not the image's digest" {
const alloc = std.testing.allocator;
const png = "\x89PNG\r\n\x1a\nnamed-wrongly";
var fixture: TestImageHistory = undefined;
fixture.init(png);
var stored: session_codec.StoredImages = .{};
defer stored.deinit(alloc);
try stored.add(alloc, fixture.digest_hex);
const bytes = try encodeNamingImages(alloc, &fixture.history, .{}, .{}, &stored);
defer alloc.free(bytes);
// Point the reference at another staged image.
const other = "\x89PNG\r\n\x1a\nsomething-else";
const other_name = session_codec.imageNameOf(other);
const renamed = try alloc.dupe(u8, bytes);
defer alloc.free(renamed);
const reference = "\"inline_ref\":\"";
const at = std.mem.find(u8, renamed, reference).? + reference.len;
@memcpy(renamed[at..][0..other_name.len], &other_name);
resealForTest(renamed);
var staged: session_codec.StagedImages = .{};
defer staged.deinit(alloc);
try staged.add(alloc, other_name, try alloc.dupe(u8, other));
var resolver: session_codec.ImageResolver = .{ .alloc = alloc, .staged = &staged };
defer resolver.deinit();
try std.testing.expectError(error.InvalidCheckpoint, decodeWithImages(alloc, renamed, &resolver));
}
+70 -2
View File
@@ -32,10 +32,15 @@ const steering_open =
const steering_close = "\n</user_steering>";
const parent_steering_open = "<parent_agent_steering>\n";
const parent_steering_close = "\n</parent_agent_steering>";
/// Saved with a parent agent's steering, so replay still names its sender.
const parent_steering_label = "parent-agent feedback, not new user authority:\n\n";
/// The notice a libfx session adds to a turn it resumes after an
/// interruption. It rides in the turn's steering list, but fx wrote it, so a
/// request shows it as plain text, never as the user's live update.
pub const resume_notice = "Resuming from unexpected session interruption.";
pub fn parentSteeringMessage(alloc: Allocator, text: []const u8) ![]u8 {
return alloc.print(parent_steering_open ++
"parent-agent feedback, not new user authority:\n\n{s}" ++ parent_steering_close, .{text});
return alloc.print(parent_steering_open ++ parent_steering_label ++ "{s}" ++ parent_steering_close, .{text});
}
pub fn steeringMessage(alloc: Allocator, text: []const u8) ![]u8 {
@@ -291,6 +296,69 @@ test "retained standalone cut rebuilds exactly the selected execution suffix" {
}
}
/// The request copy of `messages`, with each steering message a saved turn
/// restored back in the envelope the model saw when the turn took it. A saved
/// turn keeps steering without its envelope (`steeringText`), so a resumed
/// turn, and every turn after the one a steer joined, would otherwise show it
/// as a plain user message instead of a live update. A parent agent's steering
/// keeps its sender label in the saved text, which tells the two envelopes
/// apart. Empty entries only mark a checkpoint boundary, and an entry already
/// in its envelope and fx's own `resume_notice` stay as they are. Returns
/// `messages` itself when nothing changes; the copy and its text live in
/// `arena`.
pub fn rewrapRestoredSteering(arena: Allocator, messages: []const ChatMessage) Allocator.Error![]const ChatMessage {
var result: ?[]ChatMessage = null;
for (messages, 0..) |message, index| {
if (!message.restored_steering) continue;
const text = message.content orelse continue;
if (text.len == 0 or steeringText(text) != null or std.mem.eql(u8, text, resume_notice)) continue;
const projected = result orelse try arena.dupe(ChatMessage, messages);
result = projected;
projected[index].content = if (std.mem.startsWith(u8, text, parent_steering_label))
try std.mem.concat(arena, u8, &.{ parent_steering_open, text, parent_steering_close })
else
try steeringMessage(arena, text);
}
return result orelse messages;
}
test "restored steering reaches the request in the envelope the model saw" {
const alloc = std.testing.allocator;
const session = @import("../../session/session.zig");
var arena_state = std.heap.ArenaAllocator.init(alloc);
defer arena_state.deinit();
const arena = arena_state.allocator();
const user = try steeringMessage(arena, "focus on rendering");
const parent = try parentSteeringMessage(arena, "review this change");
const live = [_]ChatMessage{
.{ .role = .user, .content = user },
.{ .role = .user, .content = parent },
};
const memory = try buildExecutionMemory(alloc, &live);
defer types.freeExecutionMemory(alloc, memory);
// A saved turn comes back as steering's bare text, flagged as steering.
var restored: std.ArrayList(ChatMessage) = .empty;
try session.appendExecutionMemoryChatMessages(arena, &restored, memory);
try std.testing.expectEqual(@as(usize, 2), restored.items.len);
try std.testing.expect(restored.items[0].restored_steering and steeringText(restored.items[0].content.?) == null);
const projected = try rewrapRestoredSteering(arena, restored.items);
try std.testing.expectEqualStrings(user, projected[0].content.?);
try std.testing.expectEqualStrings(parent, projected[1].content.?);
// The session's own messages keep the bare text a saved turn holds.
try std.testing.expectEqualStrings("focus on rendering", restored.items[0].content.?);
// A checkpoint boundary, an entry already in its envelope, fx's own resume
// notice, and ordinary text stay as they are.
const unchanged = [_]ChatMessage{
.{ .role = .user, .content = "", .restored_steering = true },
.{ .role = .user, .content = user, .restored_steering = true },
.{ .role = .user, .content = resume_notice, .restored_steering = true },
.{ .role = .user, .content = "plain prompt" },
};
try std.testing.expect((try rewrapRestoredSteering(arena, &unchanged)).ptr == (&unchanged).ptr);
}
/// Steering the user typed during the turn, which a cancelled turn keeps.
fn isSteering(message: ChatMessage) bool {
if (message.restored_steering) return true;
+369 -37
View File
@@ -36,6 +36,9 @@
//! such as a compaction during the turn. Events that use `base` or are a
//! `tool_result` are version 2; every other event stays version 1, so an fx
//! that reads only version 1 still loads a journal without them.
//! An event that names an image instead of holding its bytes, one the host
//! stored or one the session dropped, is version 3, so an fx that reads only
//! up to version 2 refuses it as newer.
//!
//! A `turn_progress` event lists the inputs its model request carries for
//! the first time in `inputs`, so placement is stored with the text it
@@ -47,15 +50,23 @@ const std = @import("std");
const types = @import("../../shared/types.zig");
const testing_allocator = @import("../../shared/testing_allocator.zig");
const session_codec = @import("../../session/session_codec.zig");
const image_data = @import("../../images/image_data.zig");
const checkpoint_codec = @import("checkpoint.zig");
const compactor = @import("../../compactor/compactor.zig");
const Allocator = std.mem.Allocator;
/// The newest event version this fx reads and writes.
pub const version: u8 = 2;
/// The version of an event that needs none of version 2's additions.
pub const version: u8 = 3;
/// The version of an event that needs none of the later additions.
const plain_version: u8 = 1;
/// The version of a `tool_result` and of a progress that extends another.
const delta_version: u8 = 2;
/// The version of an event that names an image instead of holding its
/// bytes, one the host stored (`inline_ref`) or one the session dropped, so an
/// fx that reads only up to version 2 reports a newer journal instead of an
/// invalid one.
const naming_version: u8 = 3;
/// The most event bytes one load accepts, and the most bytes one snapshot
/// holds: each travels as one host attachment, and an attachment holds at
/// most one kernel checkpoint. Longer sessions need snapshots.
@@ -112,6 +123,9 @@ pub const ToolResult = struct {
name: []const u8,
failed: bool,
output: []const u8,
/// The images the result returned, which a reopened turn gives back to
/// the call with its output.
images: []const types.ToolImage = &.{},
};
/// The longest turn id a host may give.
@@ -130,11 +144,13 @@ pub const Event = union(EventType) {
turn_yielded,
tool_result: ToolResult,
/// The version `Cursor.write` gives the event.
fn eventVersion(self: Event) u8 {
/// The version `Cursor.write` gives the event: `naming_version` when it
/// names an image, one `stored` holds or one the session dropped.
fn eventVersion(self: Event, stored: ?*const session_codec.StoredImages) u8 {
if (eventNamesImages(self, stored)) return naming_version;
return switch (self) {
.tool_result => version,
.turn_progress => |progress| if (progress.delta != null) version else plain_version,
.tool_result => delta_version,
.turn_progress => |progress| if (progress.delta != null) delta_version else plain_version,
else => plain_version,
};
}
@@ -154,8 +170,18 @@ pub const Cursor = struct {
/// Writes `event` as one JSON object and returns the cursor after it.
pub fn write(self: Cursor, writer: *std.Io.Writer, event: Event) !Cursor {
return self.writeWith(writer, event, null);
}
/// Like `write`, but names the images `stored` holds instead of writing
/// their bytes.
pub fn writeNamingImages(self: Cursor, writer: *std.Io.Writer, event: Event, stored: *const session_codec.StoredImages) !Cursor {
return self.writeWith(writer, event, stored);
}
fn writeWith(self: Cursor, writer: *std.Io.Writer, event: Event, stored: ?*const session_codec.StoredImages) !Cursor {
try writer.print("{{\"v\":{d},\"seq\":{d},\"turn\":{d},\"type\":\"{s}\"", .{
event.eventVersion(),
event.eventVersion(stored),
self.next_seq,
self.turn,
@tagName(event),
@@ -163,14 +189,22 @@ pub const Cursor = struct {
switch (event) {
.turn_committed => |turn| {
try writer.writeAll(",\"data\":");
try session_codec.writeHistoryTurn(writer, turn);
if (stored) |names|
try session_codec.writeHistoryTurnNamingImages(writer, turn, .{ .stored = names })
else
try session_codec.writeHistoryTurn(writer, turn);
},
.turn_progress => |progress| {
try writer.writeAll(",\"data\":");
if (progress.delta) |delta| {
if (delta.base_seq >= self.next_seq) return error.InvalidDurableField;
try session_codec.writeRecoveryCheckpointDelta(writer, progress.checkpoint, delta.from);
if (stored) |names|
try session_codec.writeRecoveryCheckpointDeltaNamingImages(writer, progress.checkpoint, delta.from, .{ .stored = names })
else
try session_codec.writeRecoveryCheckpointDelta(writer, progress.checkpoint, delta.from);
try writer.print(",\"base\":{d},\"steps\":{d}", .{ delta.base_seq, progress.checkpoint.execution.tool_steps.len });
} else if (stored) |names| {
try session_codec.writeRecoveryCheckpointNamingImages(writer, progress.checkpoint, .{ .stored = names });
} else {
try session_codec.writeRecoveryCheckpoint(writer, progress.checkpoint);
}
@@ -197,7 +231,10 @@ pub const Cursor = struct {
try writer.writeAll(",\"data\":[");
for (history, 0..) |turn, index| {
if (index > 0) try writer.writeByte(',');
try session_codec.writeHistoryTurn(writer, turn);
if (stored) |names|
try session_codec.writeHistoryTurnNamingImages(writer, turn, .{ .stored = names })
else
try session_codec.writeHistoryTurn(writer, turn);
}
try writer.writeByte(']');
},
@@ -211,13 +248,17 @@ pub const Cursor = struct {
},
.turn_yielded => {},
.tool_result => |result| {
try writer.writeAll(",\"data\":");
try std.json.Stringify.value(.{
.id = result.id,
.name = result.name,
.status = if (result.failed) "failure" else "success",
.output = result.output,
}, .{}, writer);
try writer.writeAll(",\"data\":{\"id\":");
try std.json.Stringify.value(result.id, .{}, writer);
try writer.writeAll(",\"name\":");
try std.json.Stringify.value(result.name, .{}, writer);
try writer.print(",\"status\":\"{s}\",\"output\":", .{if (result.failed) "failure" else "success"});
try std.json.Stringify.value(result.output, .{}, writer);
if (result.images.len > 0) {
try writer.writeAll(",\"images\":");
try session_codec.writeToolImages(writer, result.images, stored != null);
}
try writer.writeByte('}');
},
}
try writer.writeByte('}');
@@ -228,6 +269,22 @@ pub const Cursor = struct {
}
};
/// Whether writing `event` names an image: one `stored` holds, or one the
/// session dropped.
fn eventNamesImages(event: Event, stored: ?*const session_codec.StoredImages) bool {
return switch (event) {
.turn_committed => |turn| session_codec.turnNamesImages(turn, stored),
// A delta leaves out the user's turn: it names only what its new
// tool results do.
.turn_progress => |progress| session_codec.checkpointNamesImages(progress.checkpoint, if (progress.delta) |delta| delta.from else null, stored),
.history_replaced => |history| for (history) |turn| {
if (session_codec.turnNamesImages(turn, stored)) break true;
} else false,
.tool_result => |result| session_codec.toolImagesNameImages(result.images, stored),
.tool_intent, .turn_progress_cleared, .input_accepted, .input_withdrawn, .turn_yielded => false,
};
}
/// A snapshot of a quiet prefix: `FXSN`, a version byte, the last `seq` it
/// covers and the turn after it (each a little-endian u64), the id of the
/// last turn (a length byte, zero for none, then its bytes), then a
@@ -239,6 +296,9 @@ pub const Cursor = struct {
const snapshot_magic = "FXSN";
const snapshot_version_quiet: u8 = 1;
const snapshot_version: u8 = 2;
/// Version 2's layout, with an open turn that names images: ones the host
/// stored, or ones the session dropped.
const snapshot_naming_version: u8 = 3;
const snapshot_header_bytes = snapshot_magic.len + 1 + 8 + 8 + 1;
pub const Snapshot = struct {
@@ -264,6 +324,31 @@ pub fn encodeSnapshot(
last_turn_id: ?[]const u8,
open_turn: ?[]const u8,
checkpoint: []const u8,
) error{ OutOfMemory, SnapshotTooLarge }![]u8 {
return encodeSnapshotWith(alloc, seq, turn, last_turn_id, open_turn, checkpoint, false);
}
/// Like `encodeSnapshot`, for an `open_turn` that names at least one image
/// (`session_codec.namesImages`).
pub fn encodeSnapshotNamingImages(
alloc: Allocator,
seq: u64,
turn: u64,
last_turn_id: ?[]const u8,
open_turn: []const u8,
checkpoint: []const u8,
) error{ OutOfMemory, SnapshotTooLarge }![]u8 {
return encodeSnapshotWith(alloc, seq, turn, last_turn_id, open_turn, checkpoint, true);
}
fn encodeSnapshotWith(
alloc: Allocator,
seq: u64,
turn: u64,
last_turn_id: ?[]const u8,
open_turn: ?[]const u8,
checkpoint: []const u8,
open_turn_names_images: bool,
) error{ OutOfMemory, SnapshotTooLarge }![]u8 {
const id = last_turn_id orelse "";
std.debug.assert(id.len <= max_turn_id_bytes);
@@ -271,7 +356,12 @@ pub fn encodeSnapshot(
if (open_turn) |open| if (open.len > std.math.maxInt(u32)) return error.SnapshotTooLarge;
const bytes = try alloc.alloc(u8, snapshot_header_bytes + id.len + open_bytes + checkpoint.len);
@memcpy(bytes[0..snapshot_magic.len], snapshot_magic);
bytes[snapshot_magic.len] = if (open_turn == null) snapshot_version_quiet else snapshot_version;
bytes[snapshot_magic.len] = if (open_turn == null)
snapshot_version_quiet
else if (open_turn_names_images)
snapshot_naming_version
else
snapshot_version;
std.mem.writeInt(u64, bytes[snapshot_magic.len + 1 ..][0..8], seq, .little);
std.mem.writeInt(u64, bytes[snapshot_magic.len + 9 ..][0..8], turn, .little);
bytes[snapshot_header_bytes - 1] = @intCast(id.len);
@@ -291,8 +381,9 @@ pub fn parseSnapshot(bytes: []const u8) LoadError!Snapshot {
return error.InvalidJournal;
}
const found_version = bytes[snapshot_magic.len];
if (found_version > snapshot_version) return error.UnsupportedJournalVersion;
if (found_version != snapshot_version and found_version != snapshot_version_quiet) return error.InvalidJournal;
if (found_version > snapshot_naming_version) return error.UnsupportedJournalVersion;
if (found_version == 0) return error.InvalidJournal;
const has_open_turn = found_version != snapshot_version_quiet;
const seq = std.mem.readInt(u64, bytes[snapshot_magic.len + 1 ..][0..8], .little);
const turn = std.mem.readInt(u64, bytes[snapshot_magic.len + 9 ..][0..8], .little);
if (seq == 0 or turn == 0 or seq > max_safe_position or turn > max_safe_position) return error.InvalidJournal;
@@ -300,7 +391,7 @@ pub fn parseSnapshot(bytes: []const u8) LoadError!Snapshot {
if (id_len > max_turn_id_bytes or bytes.len - snapshot_header_bytes < id_len) return error.InvalidJournal;
var at: usize = snapshot_header_bytes + id_len;
var open_turn: ?[]const u8 = null;
if (found_version == snapshot_version) {
if (has_open_turn) {
if (bytes.len - at < 4) return error.InvalidJournal;
const open_len = std.mem.readInt(u32, bytes[at..][0..4], .little);
at += 4;
@@ -320,6 +411,28 @@ pub fn parseSnapshot(bytes: []const u8) LoadError!Snapshot {
/// Writes the turn that yielded, for a version 2 snapshot: its id, when the
/// host gave one, and its progress as a `turn_progress` carries it.
pub fn writeOpenTurn(writer: *std.Io.Writer, turn_id: ?[]const u8, progress: session_codec.RecoveryCheckpoint) !void {
return writeOpenTurnWith(writer, turn_id, progress, null);
}
/// Like `writeOpenTurn`, but names the images `stored` holds instead of
/// writing their bytes. Returns whether it named any, a dropped one too,
/// which makes the snapshot version 3 (`encodeSnapshotNamingImages`).
pub fn writeOpenTurnNamingImages(
writer: *std.Io.Writer,
turn_id: ?[]const u8,
progress: session_codec.RecoveryCheckpoint,
stored: *const session_codec.StoredImages,
) !bool {
try writeOpenTurnWith(writer, turn_id, progress, stored);
return session_codec.checkpointNamesImages(progress, null, stored);
}
fn writeOpenTurnWith(
writer: *std.Io.Writer,
turn_id: ?[]const u8,
progress: session_codec.RecoveryCheckpoint,
stored: ?*const session_codec.StoredImages,
) !void {
try writer.writeByte('{');
if (turn_id) |id| {
std.debug.assert(id.len > 0 and id.len <= max_turn_id_bytes);
@@ -328,7 +441,10 @@ pub fn writeOpenTurn(writer: *std.Io.Writer, turn_id: ?[]const u8, progress: ses
try writer.writeByte(',');
}
try writer.writeAll("\"progress\":");
try session_codec.writeRecoveryCheckpoint(writer, progress);
if (stored) |names|
try session_codec.writeRecoveryCheckpointNamingImages(writer, progress, .{ .stored = names })
else
try session_codec.writeRecoveryCheckpoint(writer, progress);
try writer.writeByte('}');
}
@@ -387,14 +503,22 @@ pub const Finished = struct {
id: []u8,
failed: bool,
output: []u8,
/// Owned. An image the host has yet to stage has no bytes, and the
/// fold's resolver records it as missing.
images: []types.ToolImage = &.{},
};
fn freeFinished(alloc: Allocator, finished: []Finished) void {
freeFinishedEntries(alloc, finished);
alloc.free(finished);
}
fn freeFinishedEntries(alloc: Allocator, finished: []const Finished) void {
for (finished) |entry| {
alloc.free(entry.id);
alloc.free(entry.output);
types.freeToolImages(alloc, entry.images);
}
alloc.free(finished);
}
pub fn freePendingInputs(alloc: Allocator, inputs: []PendingInput) void {
@@ -443,14 +567,12 @@ fn announcedCall(calls: []const std.json.Value, id: []const u8) bool {
return false;
}
/// Owned copies of the newest recorded result of each call in `calls`.
fn finishedOf(alloc: Allocator, calls: []const types.ToolCall, results: []const std.json.Value) LoadError![]Finished {
/// Owned copies of the newest recorded result of each call in `calls`, with
/// the images each names resolved through `sources`.
fn finishedOf(alloc: Allocator, calls: []const types.ToolCall, results: []const std.json.Value, sources: session_codec.ImageSources) LoadError![]Finished {
var finished: std.ArrayList(Finished) = .empty;
errdefer {
for (finished.items) |entry| {
alloc.free(entry.id);
alloc.free(entry.output);
}
freeFinishedEntries(alloc, finished.items);
finished.deinit(alloc);
}
for (calls) |call| {
@@ -463,10 +585,19 @@ fn finishedOf(alloc: Allocator, calls: []const types.ToolCall, results: []const
errdefer alloc.free(id);
const output = try alloc.dupe(u8, result.object.get("output").?.string);
errdefer alloc.free(output);
const images: []types.ToolImage = if (result.object.get("images")) |list|
session_codec.parseToolImages(alloc, list.array.items, sources) catch |err| switch (err) {
error.OutOfMemory => return error.OutOfMemory,
else => return error.InvalidJournal,
}
else
&.{};
errdefer types.freeToolImages(alloc, images);
try finished.append(alloc, .{
.id = id,
.failed = std.mem.eql(u8, result.object.get("status").?.string, "failure"),
.output = output,
.images = images,
});
}
return finished.toOwnedSlice(alloc);
@@ -489,6 +620,10 @@ pub const Base = struct {
last_turn_id: ?[]const u8 = null,
/// The turn that yielded, as `writeOpenTurn` wrote it.
open_turn: ?[]const u8 = null,
/// Resolves the images events name. Without it an event that names one
/// is invalid; a name it cannot resolve is recorded there, and the fold
/// is then only good for learning which images the host must stage.
images: ?*session_codec.ImageResolver = null,
};
/// Rebuilds a session from `events_json`, a JSON array of events in order.
@@ -551,7 +686,7 @@ pub fn foldFrom(alloc: Allocator, events_json: []const u8, base: Base) LoadError
if (event.kind != .turn_yielded) yielded = false;
switch (event.kind) {
.turn_committed => {
const turn = try parseTurn(alloc, event.data orelse return error.InvalidJournal);
const turn = try parseTurn(alloc, event.data orelse return error.InvalidJournal, base.images);
// The same rule the session applies when it commits the turn.
if (replacesHistory(turn)) {
for (history.items) |old| types.freeHistoryTurn(alloc, old);
@@ -629,6 +764,7 @@ pub fn foldFrom(alloc: Allocator, events_json: []const u8, base: Base) LoadError
if (!std.mem.eql(u8, status, "success") and !std.mem.eql(u8, status, "failure")) return error.InvalidJournal;
const output = data.object.get("output") orelse return error.InvalidJournal;
if (output != .string) return error.InvalidJournal;
if (data.object.get("images")) |images| if (images != .array) return error.InvalidJournal;
// One for a call no intent announced answers nothing.
if (announcedCall(running.items, id)) try results.append(alloc, data);
}
@@ -666,7 +802,7 @@ pub fn foldFrom(alloc: Allocator, events_json: []const u8, base: Base) LoadError
replacement.deinit(alloc);
}
try replacement.ensureTotalCapacity(alloc, data.array.items.len);
for (data.array.items) |item| replacement.appendAssumeCapacity(try parseTurn(alloc, item));
for (data.array.items) |item| replacement.appendAssumeCapacity(try parseTurn(alloc, item, base.images));
for (history.items) |turn| types.freeHistoryTurn(alloc, turn);
history.deinit(alloc);
history = replacement;
@@ -680,7 +816,7 @@ pub fn foldFrom(alloc: Allocator, events_json: []const u8, base: Base) LoadError
}
var open_turn: ?session_codec.RecoveryCheckpoint = null;
if (open) |data| open_turn = if (data == .null) return error.InvalidJournal else session_codec.parseRecoveryCheckpoint(alloc, data) catch |err| switch (err) {
if (open) |data| open_turn = if (data == .null) return error.InvalidJournal else session_codec.parseRecoveryCheckpointWithImages(alloc, data, .{ .names = base.images }) catch |err| switch (err) {
error.OutOfMemory => return error.OutOfMemory,
else => return error.InvalidJournal,
};
@@ -697,7 +833,7 @@ pub fn foldFrom(alloc: Allocator, events_json: []const u8, base: Base) LoadError
return err;
};
errdefer types.freeToolCallSlice(alloc, unanswered);
const finished = try finishedOf(alloc, unanswered, results.items);
const finished = try finishedOf(alloc, unanswered, results.items, .{ .names = base.images });
errdefer freeFinished(alloc, finished);
const pending = try pendingInputsOf(alloc, if (open_turn != null) inputs.items else &.{}, .steer);
errdefer freePendingInputs(alloc, pending);
@@ -863,8 +999,8 @@ fn replacesHistory(turn: types.HistoryTurn) bool {
};
}
fn parseTurn(alloc: Allocator, value: std.json.Value) LoadError!types.HistoryTurn {
return session_codec.parseHistoryTurn(alloc, value) catch |err| switch (err) {
fn parseTurn(alloc: Allocator, value: std.json.Value, images: ?*session_codec.ImageResolver) LoadError!types.HistoryTurn {
return session_codec.parseHistoryTurnWithImages(alloc, value, .{ .names = images }) catch |err| switch (err) {
error.OutOfMemory => error.OutOfMemory,
else => error.InvalidJournal,
};
@@ -1277,7 +1413,7 @@ test "a journal from a newer fx or with a malformed event is rejected" {
const alloc = std.testing.allocator;
try std.testing.expectError(
error.UnsupportedJournalVersion,
fold(alloc, "[{\"v\":3,\"seq\":1,\"turn\":1,\"type\":\"turn_progress_cleared\"}]"),
fold(alloc, "[{\"v\":4,\"seq\":1,\"turn\":1,\"type\":\"turn_progress_cleared\"}]"),
);
const malformed = [_][]const u8{
"{}",
@@ -1656,7 +1792,7 @@ test "a snapshot that is not one, or comes from a newer fx, is refused" {
try std.testing.expectError(error.InvalidJournal, parseSnapshot(zero));
const newer = try encodeSnapshot(alloc, 3, 2, null, null, "");
defer alloc.free(newer);
newer[snapshot_magic.len] = snapshot_version + 1;
newer[snapshot_magic.len] = snapshot_naming_version + 1;
try std.testing.expectError(error.UnsupportedJournalVersion, parseSnapshot(newer));
}
@@ -1698,3 +1834,199 @@ test "a turn's first progress names it, and the turn keeps that name until it en
defer alloc.free(wrong);
try std.testing.expectError(error.InvalidJournal, fold(alloc, wrong));
}
fn testImageUser(alloc: Allocator, png: []const u8) !types.UserTurn {
const name = session_codec.imageNameOf(png);
const images = try alloc.alloc(types.ImageAttachment, 1);
images[0] = .{
.id = 1,
.path = try alloc.dupe(u8, "inline://image-1"),
.media_type = try alloc.dupe(u8, "image/png"),
.snapshot_sha256 = try alloc.dupe(u8, &name),
.inline_data = try alloc.dupe(u8, png),
};
return .{ .text = try alloc.dupe(u8, "[Image #1]"), .images = images };
}
fn testImageTurn(alloc: Allocator, png: []const u8) !types.HistoryTurn {
return .{ .assistant = .{ .user = try testImageUser(alloc, png), .assistant = try alloc.dupe(u8, "seen") } };
}
test "a journal names the images the host stored and folds them back from its copies" {
const alloc = std.testing.allocator;
const png = "\x89PNG\r\n\x1a\njournal-named";
const name = session_codec.imageNameOf(png);
const turn = try testImageTurn(alloc, png);
defer types.freeHistoryTurn(alloc, turn);
var stored: session_codec.StoredImages = .{};
defer stored.deinit(alloc);
try stored.add(alloc, name);
var out: std.Io.Writer.Allocating = .init(alloc);
defer out.deinit();
try out.writer.writeByte('[');
_ = try (Cursor{}).writeNamingImages(&out.writer, .{ .turn_committed = turn }, &stored);
try out.writer.writeByte(']');
const events = out.written();
// The event names the image and holds none of its bytes.
try std.testing.expect(std.mem.startsWith(u8, events, "[{\"v\":3,"));
try std.testing.expect(std.mem.find(u8, events, "\"inline_ref\":\"" ++ name ++ "\"") != null);
try std.testing.expect(std.mem.find(u8, events, "journal-named") == null);
try std.testing.expect(std.mem.find(u8, events, "\"inline_data\"") == null);
try std.testing.expectError(error.InvalidJournal, fold(alloc, events));
var staged: session_codec.StagedImages = .{};
defer staged.deinit(alloc);
try staged.add(alloc, name, try alloc.dupe(u8, png));
var resolver: session_codec.ImageResolver = .{ .alloc = alloc, .staged = &staged };
defer resolver.deinit();
var folded = try foldFrom(alloc, events, .{ .images = &resolver });
defer folded.deinit(alloc);
try std.testing.expectEqualStrings(png, folded.history[0].assistant.user.images[0].inline_data.?);
try std.testing.expectEqual(@as(usize, 0), resolver.missing.count());
try std.testing.expect(resolver.resolved.contains(name));
}
test "an event that names no image is version 1, byte for byte" {
const alloc = std.testing.allocator;
const turn = try testImageTurn(alloc, "\x89PNG\r\n\x1a\nnot-stored");
defer types.freeHistoryTurn(alloc, turn);
var stored: session_codec.StoredImages = .{};
defer stored.deinit(alloc);
try stored.add(alloc, session_codec.imageNameOf("another image"));
var plain: std.Io.Writer.Allocating = .init(alloc);
defer plain.deinit();
_ = try (Cursor{}).write(&plain.writer, .{ .turn_committed = turn });
var named: std.Io.Writer.Allocating = .init(alloc);
defer named.deinit();
_ = try (Cursor{}).writeNamingImages(&named.writer, .{ .turn_committed = turn }, &stored);
try std.testing.expectEqualStrings(plain.written(), named.written());
try std.testing.expect(std.mem.startsWith(u8, named.written(), "{\"v\":1,"));
}
test "one fold names every image the host did not stage" {
const alloc = std.testing.allocator;
const first_png = "\x89PNG\r\n\x1a\nfirst-missing";
const second_png = "\x89PNG\r\n\x1a\nsecond-missing";
var stored: session_codec.StoredImages = .{};
defer stored.deinit(alloc);
try stored.add(alloc, session_codec.imageNameOf(first_png));
try stored.add(alloc, session_codec.imageNameOf(second_png));
var out: std.Io.Writer.Allocating = .init(alloc);
defer out.deinit();
try out.writer.writeByte('[');
var cursor: Cursor = .{};
const first = try testImageTurn(alloc, first_png);
defer types.freeHistoryTurn(alloc, first);
cursor = try cursor.writeNamingImages(&out.writer, .{ .turn_committed = first }, &stored);
try out.writer.writeByte(',');
var progress = try testProgress(alloc, "");
defer progress.deinit(alloc);
types.freeUserTurn(alloc, progress.user);
progress.user = try testImageUser(alloc, second_png);
_ = try cursor.writeNamingImages(&out.writer, .{ .turn_progress = .{ .checkpoint = progress } }, &stored);
try out.writer.writeByte(']');
var staged: session_codec.StagedImages = .{};
defer staged.deinit(alloc);
var resolver: session_codec.ImageResolver = .{ .alloc = alloc, .staged = &staged };
defer resolver.deinit();
var folded = try foldFrom(alloc, out.written(), .{ .images = &resolver });
defer folded.deinit(alloc);
try std.testing.expectEqual(@as(usize, 2), resolver.missing.count());
try std.testing.expect(resolver.missing.contains(session_codec.imageNameOf(first_png)));
try std.testing.expect(resolver.missing.contains(session_codec.imageNameOf(second_png)));
}
test "a recorded tool result keeps its images, naming the stored one" {
const alloc = std.testing.allocator;
const stored_png = image_data.testPngHeader(1, 1);
const inline_png = image_data.testPngHeader(2, 1);
const name = session_codec.imageNameOf(&stored_png);
var stored: session_codec.StoredImages = .{};
defer stored.deinit(alloc);
try stored.add(alloc, name);
var stored_data: [std.base64.standard.Encoder.calcSize(stored_png.len)]u8 = undefined;
var inline_data: [std.base64.standard.Encoder.calcSize(inline_png.len)]u8 = undefined;
const images = [_]types.ToolImage{
.{ .data = @constCast(std.base64.standard.Encoder.encode(&stored_data, &stored_png)), .mime_type = @constCast("image/png"), .name = name },
.{ .data = @constCast(std.base64.standard.Encoder.encode(&inline_data, &inline_png)), .mime_type = @constCast("image/png") },
};
var progress = try testProgress(alloc, "take two shots");
defer progress.deinit(alloc);
const calls = [_]types.ToolCall{.{ .id = "call-a", .name = "screenshot", .arguments_json = "{}" }};
var out: std.Io.Writer.Allocating = .init(alloc);
defer out.deinit();
try out.writer.writeByte('[');
var cursor: Cursor = .{};
cursor = try cursor.writeNamingImages(&out.writer, .{ .turn_progress = .{ .checkpoint = progress } }, &stored);
try out.writer.writeByte(',');
cursor = try cursor.writeNamingImages(&out.writer, .{ .tool_intent = &calls }, &stored);
try out.writer.writeByte(',');
_ = try cursor.writeNamingImages(&out.writer, .{ .tool_result = .{ .id = "call-a", .name = "screenshot", .failed = false, .output = "two shots", .images = &images } }, &stored);
try out.writer.writeByte(']');
const bytes = out.written();
// It names the stored image, which makes it a naming event, and holds the
// other's bytes.
try std.testing.expect(std.mem.find(u8, bytes, "{\"v\":3,\"seq\":3,\"turn\":1,\"type\":\"tool_result\"") != null);
try std.testing.expect(std.mem.find(u8, bytes, images[0].data) == null);
try std.testing.expect(std.mem.find(u8, bytes, images[1].data) != null);
// An open that has yet to stage the stored image names it as missing.
var staged: session_codec.StagedImages = .{};
defer staged.deinit(alloc);
var discover: session_codec.ImageResolver = .{ .alloc = alloc, .staged = &staged };
defer discover.deinit();
var unstaged = try foldFrom(alloc, bytes, .{ .images = &discover });
defer unstaged.deinit(alloc);
try std.testing.expect(discover.missing.contains(name));
// Staged, the call is answered with both images.
try staged.add(alloc, name, try alloc.dupe(u8, &stored_png));
var resolver: session_codec.ImageResolver = .{ .alloc = alloc, .staged = &staged };
defer resolver.deinit();
var folded = try foldFrom(alloc, bytes, .{ .images = &resolver });
defer folded.deinit(alloc);
try std.testing.expectEqual(@as(usize, 0), resolver.missing.count());
try std.testing.expectEqual(@as(usize, 1), folded.finished.len);
try std.testing.expectEqualStrings("two shots", folded.finished[0].output);
try std.testing.expectEqual(@as(usize, 2), folded.finished[0].images.len);
try std.testing.expectEqualStrings(images[0].data, folded.finished[0].images[0].data);
try std.testing.expectEqualStrings(images[1].data, folded.finished[0].images[1].data);
}
test "a yielded snapshot names its open turn's stored images" {
const alloc = std.testing.allocator;
const png = "\x89PNG\r\n\x1a\nyielded-turn";
const name = session_codec.imageNameOf(png);
var stored: session_codec.StoredImages = .{};
defer stored.deinit(alloc);
try stored.add(alloc, name);
var progress = try testProgress(alloc, "");
defer progress.deinit(alloc);
types.freeUserTurn(alloc, progress.user);
progress.user = try testImageUser(alloc, png);
var open_out: std.Io.Writer.Allocating = .init(alloc);
defer open_out.deinit();
try std.testing.expect(try writeOpenTurnNamingImages(&open_out.writer, "turn-y", progress, &stored));
try std.testing.expect(std.mem.find(u8, open_out.written(), "yielded-turn") == null);
const snapshot = try encodeSnapshotNamingImages(alloc, 5, 2, null, open_out.written(), "");
defer alloc.free(snapshot);
try std.testing.expectEqual(snapshot_naming_version, snapshot[snapshot_magic.len]);
const parsed = try parseSnapshot(snapshot);
var staged: session_codec.StagedImages = .{};
defer staged.deinit(alloc);
try staged.add(alloc, name, try alloc.dupe(u8, png));
var resolver: session_codec.ImageResolver = .{ .alloc = alloc, .staged = &staged };
defer resolver.deinit();
var folded = try foldFrom(alloc, "[]", .{ .cursor = .{ .next_seq = 6, .turn = 2 }, .open_turn = parsed.open_turn, .images = &resolver });
defer folded.deinit(alloc);
try std.testing.expectEqualStrings(png, folded.open_turn.?.user.images[0].inline_data.?);
try std.testing.expect(folded.yielded);
}
+70 -22
View File
@@ -4622,24 +4622,24 @@ test "context overflow recovery is typed safe and bounded" {
));
}
/// Size retries allowed in one turn. Each lowers the learned limit, so the
/// next request leaves out more of its oldest images.
const max_request_size_retries: u8 = 2;
/// A size rejection is retried without compaction only when the rejected
/// request carried images the next one can leave out. Anything else falls
/// through to context overflow recovery.
/// A size rejection is retried without compaction while the rejected request
/// carried images, and fewer than the request refused before it, if any.
/// Each retry lowers the learned limit below the size just refused, which
/// only leaving out more images can meet, so the count falls with every
/// retry and the retries end. Anything else falls through to context
/// overflow recovery.
fn shouldRetryWithFewerImages(
failure: agent_stream_provider.Failure,
images_sent: usize,
retries_used: u8,
images_last_refused: ?usize,
replay_safe: bool,
cancelled: bool,
) bool {
const fewer = if (images_last_refused) |last| images_sent < last else true;
return !cancelled and
replay_safe and
images_sent > 0 and
retries_used < max_request_size_retries and
fewer and
failure.kind == .request_too_large;
}
@@ -4687,15 +4687,20 @@ test "size rejections retry with fewer images only while images remain" {
const cases = [_]struct {
failure: agent_stream_provider.Failure,
images_sent: usize = 3,
retries_used: u8 = 0,
images_last_refused: ?usize = null,
replay_safe: bool = true,
cancelled: bool = false,
expected: bool,
}{
.{ .failure = too_large, .expected = true },
.{ .failure = too_large, .retries_used = max_request_size_retries - 1, .expected = true },
.{ .failure = too_large, .retries_used = max_request_size_retries, .expected = false },
// Any number of retries, while each leaves out more images.
.{ .failure = too_large, .images_sent = 3, .images_last_refused = 5, .expected = true },
.{ .failure = too_large, .images_sent = 1, .images_last_refused = 2, .expected = true },
// One that left out none stops, so the retries always end.
.{ .failure = too_large, .images_sent = 3, .images_last_refused = 3, .expected = false },
.{ .failure = too_large, .images_sent = 4, .images_last_refused = 3, .expected = false },
.{ .failure = too_large, .images_sent = 0, .expected = false },
.{ .failure = too_large, .images_sent = 0, .images_last_refused = 1, .expected = false },
.{ .failure = too_large, .replay_safe = false, .expected = false },
.{ .failure = too_large, .cancelled = true, .expected = false },
.{ .failure = overflow_text, .expected = false },
@@ -4703,7 +4708,7 @@ test "size rejections retry with fewer images only while images remain" {
for (cases) |case| try std.testing.expectEqual(case.expected, shouldRetryWithFewerImages(
case.failure,
case.images_sent,
case.retries_used,
case.images_last_refused,
case.replay_safe,
case.cancelled,
));
@@ -7265,7 +7270,9 @@ fn processQueuedPromptLoop(
var assistant_prefill_recovery_used = false;
var skip_next_preflight_refresh = false;
var context_overflow_recovery: ContextOverflowRecoveryState = .ready;
var request_size_retries: u8 = 0;
var request_size_retries: usize = 0;
// The images the last request refused for its size carried.
var request_images_last_refused: ?usize = null;
var recovery_has_unexecuted_tool_start = false;
var successful_recovery_strategy: ?model_response_recovery.Strategy = null;
defer {
@@ -7459,11 +7466,12 @@ fn processQueuedPromptLoop(
subagent_request_messages.len,
});
}
const result_request_messages = try project_read_tool_result_request_messages(
// Images the session dropped leave every request as notes.
const result_request_messages = try image_attachments.withholdDroppedImages(overlay_arena, try project_read_tool_result_request_messages(
overlay_arena,
read_tool_result_request_eligible,
subagent_request_messages,
);
));
if (!tool_image_capabilities_resolved and request_capabilities.image_input_support == .unknown) {
for (result_request_messages) |message| {
const memory = message.tool_result_memory orelse continue;
@@ -7487,9 +7495,12 @@ fn processQueuedPromptLoop(
break :native projection.messages;
} else result_request_messages;
const image_projection = try runtime_gateway_step.projectToolImageMessages(overlay_arena, materialized_messages, request_capabilities.image_input_support, vision_policy.route == .fallback, config.max_tool_result_bytes);
const request_messages = try with_replyable_conversation_tail(overlay_arena, image_projection.messages);
if (request_messages.ptr != image_projection.messages.ptr) {
debug_trace.eventf("history", "assistant_tail_continued", step_ctx, "messages={d}", .{image_projection.messages.len});
// Saved turns keep steering without its envelope; the model gets it
// back as the live update it saw when the turn took it.
const steering_messages = try runtime_execution_memory.rewrapRestoredSteering(overlay_arena, image_projection.messages);
const request_messages = try with_replyable_conversation_tail(overlay_arena, steering_messages);
if (request_messages.ptr != steering_messages.ptr) {
debug_trace.eventf("history", "assistant_tail_continued", step_ctx, "messages={d}", .{steering_messages.len});
}
// Tool images withheld because the catalog could not confirm image
// input are invisible in the transcript otherwise. Tell the user
@@ -7543,6 +7554,20 @@ fn processQueuedPromptLoop(
.max_output_tokens = model_capabilities.requestOutputTokens(request_capabilities),
.budget = .{ .cancel_flag = config.cancel_flag },
};
// The request re-sends every image the session keeps, and
// building it whole would hold all of them, encoded, at once.
// Images that pass every limit by their own size leave before the
// build; `fitRequestImages` fits the rest to this model's limit.
const past_limit = try request_image_budget.withholdImagesPastLimit(
overlay_arena,
request_data.messages,
request_image_budget.largestRequestBytes(agent.requestByteLimit(gateway_model)),
config.max_tool_result_bytes,
);
if (past_limit.withheld > 0) {
debug_trace.eventf("images", "request_images_withheld_before_build", step_ctx, "withheld={d}", .{past_limit.withheld});
request_data.messages = past_limit.messages;
}
var prepared_request_body: ?[]const u8 = null;
var request_cost_for_attempt: ?runtime_prompt_context.RequestCost = null;
var request_images_for_attempt: usize = 0;
@@ -7555,7 +7580,10 @@ fn processQueuedPromptLoop(
deps.agent_stream_provider,
&request_data,
built_request_body,
.{ .max_request_bytes = agent.requestByteLimit(gateway_model) orelse request_image_budget.default_max_request_bytes },
.{ .max_request_bytes = request_image_budget.maxRequestBytes(
agent.requestByteLimit(gateway_model),
deps.agent_stream_provider.max_request_bytes,
) },
config.max_tool_result_bytes,
step_ctx,
);
@@ -7719,7 +7747,14 @@ fn processQueuedPromptLoop(
.tool_steps = finalization.compacted_execution.tool_steps + active_cut.tool_steps,
.steering = finalization.compacted_execution.steering + active_cut.steering,
};
if (overflow_pending) context_overflow_recovery = .used;
if (overflow_pending) {
context_overflow_recovery = .used;
// The shorter history can fit more images than the
// request last refused for its size carried, so size
// retries count again from the next refusal. This
// happens once per step, so the retries still end.
request_images_last_refused = null;
}
compactor.traceEvent(
step_ctx,
.installed,
@@ -8365,12 +8400,13 @@ fn processQueuedPromptLoop(
if (request_cost_for_attempt) |rejected| if (shouldRetryWithFewerImages(
failure,
request_images_for_attempt,
request_size_retries,
request_images_last_refused,
streamReplaySafe(&stream_ctx),
config.cancel_flag.load(.seq_cst),
)) {
const limit = loweredRequestByteLimit(rejected.serialized_bytes);
agent.lowerRequestByteLimit(gateway_model, limit);
request_images_last_refused = request_images_for_attempt;
request_size_retries += 1;
debug_trace.eventf("images", "request_size_retry", step_ctx, "model={s} rejected_bytes={d} images_sent={d} limit_bytes={d} retry={d}", .{
gateway_model,
@@ -9862,8 +9898,19 @@ fn processQueuedPromptLoop(
ToolCall,
prepared_tool_calls.len,
);
const saved_output_sources: runtime_tool_presentation.TurnSavedOutputSources = .{
.registry = deps.tool_registry,
.workspace_root = config.workspace_root,
.turns = &.{ history_messages.items, within_turn_suffix.items },
};
for (prepared_tool_calls, 0..) |prepared_call, i| {
effective_tool_calls[i] = prepared_call.call();
effective_tool_calls[i].resolved_display_target = try runtime_tool_presentation.readToolResultDisplayTarget(
arena,
deps.tool_registry,
effective_tool_calls[i],
saved_output_sources,
);
}
const pending_assistant: ChatMessage = .{
.role = .assistant,
@@ -10136,6 +10183,7 @@ fn processQueuedPromptLoop(
for (prepared_tool_calls, 0..) |prepared_tool_call, tool_call_index| {
if (tool_call_index < parallel_skip_until) continue;
var tool_call = prepared_tool_call.call();
tool_call.resolved_display_target = effective_tool_calls[tool_call_index].resolved_display_target;
const root_live_permission_mode = snapshotRootPermissionMode(deps);
const root_action_permission_mode = permissionModeForAction(
job.permission_mode,
@@ -24,6 +24,13 @@ pub const Limits = struct {
max_images: usize = default_max_request_images,
};
/// The size a request must fit: the limit learned from the provider, else
/// the default, and never more than the transport can carry.
pub fn maxRequestBytes(learned: ?usize, transport: ?usize) usize {
const limit = learned orelse default_max_request_bytes;
return if (transport) |cap| @min(limit, cap) else limit;
}
/// Returns how many of the oldest images must leave a request of
/// `request_bytes` for it to fit `limits`. `image_bytes` holds each image's
/// serialized size, oldest first. Returns `image_bytes.len` when the request
@@ -54,6 +61,37 @@ pub fn withholdOldestImages(
limits: Limits,
tool_text_limit: usize,
) Allocator.Error!Projection {
const sizes = try imageSizes(arena, messages);
return withhold(arena, messages, oldestImagesToWithhold(sizes, request_bytes, limits), tool_text_limit);
}
/// A size no request is fit below: the default, or a larger limit learned
/// from a provider's rejection. A transport may lower either, never raise it.
pub fn largestRequestBytes(learned: ?usize) usize {
return @max(default_max_request_bytes, learned orelse 0);
}
/// Leaves out the oldest images a request of at most `max_request_bytes`
/// can't carry by their serialized size alone, before the request is built.
/// A request re-sends every image the session keeps, so building it whole
/// would hold all of them, encoded, at once. A body is never smaller than
/// its images, so this withholds no image `withholdOldestImages` would keep
/// for a limit of at most `max_request_bytes`.
pub fn withholdImagesPastLimit(
arena: Allocator,
messages: []const ChatMessage,
max_request_bytes: usize,
tool_text_limit: usize,
) Allocator.Error!Projection {
const sizes = try imageSizes(arena, messages);
var images_bytes: usize = 0;
for (sizes) |size| images_bytes +|= size;
const withheld = oldestImagesToWithhold(sizes, images_bytes, .{ .max_request_bytes = max_request_bytes });
return withhold(arena, messages, withheld, tool_text_limit);
}
/// Each image's serialized size, oldest first, allocated in `arena`.
fn imageSizes(arena: Allocator, messages: []const ChatMessage) Allocator.Error![]const usize {
var sizes: std.ArrayList(usize) = .empty;
for (messages) |message| {
for (message.images) |image| {
@@ -65,7 +103,11 @@ pub fn withholdOldestImages(
for (memory.tool_images) |image| try sizes.append(arena, image.data.len);
}
}
const withheld = oldestImagesToWithhold(sizes.items, request_bytes, limits);
return sizes.items;
}
/// `messages` with its `withheld` oldest images replaced by notes.
fn withhold(arena: Allocator, messages: []const ChatMessage, withheld: usize, tool_text_limit: usize) Allocator.Error!Projection {
if (withheld == 0) return .{ .messages = messages };
const projected = try arena.dupe(ChatMessage, messages);
@@ -127,6 +169,15 @@ test "oldest images leave until the request fits" {
try std.testing.expectEqual(@as(usize, 0), oldestImagesToWithhold(&.{}, 50_000, limits));
}
test "a request fits the transport's limit as well as the provider's" {
try std.testing.expectEqual(default_max_request_bytes, maxRequestBytes(null, null));
try std.testing.expectEqual(@as(usize, 4000), maxRequestBytes(4000, null));
// A host transport that carries less lowers either limit.
try std.testing.expectEqual(@as(usize, 8000), maxRequestBytes(null, 8000));
try std.testing.expectEqual(@as(usize, 4000), maxRequestBytes(4000, 8000));
try std.testing.expectEqual(@as(usize, 3000), maxRequestBytes(4000, 3000));
}
test "oldest images leave when a request carries too many" {
const limits = Limits{ .max_request_bytes = 1_000_000, .max_images = 2 };
const sizes = [_]usize{ 10, 10, 10, 10 };
@@ -186,6 +237,46 @@ test "withholding keeps history and names how to reload each image" {
try std.testing.expectEqual(@as([*]const ChatMessage, &messages), unchanged.messages.ptr);
}
test "images no request can carry leave before it is built, and none it can" {
var arena_state = std.heap.ArenaAllocator.init(std.testing.allocator);
defer arena_state.deinit();
const arena = arena_state.allocator();
var tool_images: [6]types.ToolImage = undefined;
for (&tool_images) |*image| image.* = try testToolImage(arena, 4000);
const messages = [_]ChatMessage{
.{ .role = .tool, .content = "<path>a.png</path>", .tool_result_memory = .{ .tool_images = tool_images[0..3], .tool_image_handle = "image-result-a" } },
.{ .role = .tool, .content = "<path>b.png</path>", .tool_result_memory = .{ .tool_images = tool_images[3..], .tool_image_handle = "image-result-b" } },
};
// 24,000 bytes of images against 10,000: the oldest four leave, whatever
// the text adds, and the newest two stay for the exact fit.
const before = try withholdImagesPastLimit(arena, &messages, 10_000, 4096);
try std.testing.expectEqual(@as(usize, 4), before.withheld);
try std.testing.expectEqual(@as(usize, 0), before.messages[0].tool_result_memory.?.tool_images.len);
try std.testing.expectEqual(@as(usize, 2), before.messages[1].tool_result_memory.?.tool_images.len);
// Fitting the whole body withholds at least as many.
const sizes = [_]usize{ 4000, 4000, 4000, 4000, 4000, 4000 };
for ([_]usize{ 24_000, 24_100, 30_000 }) |body_bytes| {
try std.testing.expect(oldestImagesToWithhold(&sizes, body_bytes, .{ .max_request_bytes = 10_000 }) >= before.withheld);
}
// Images that fit by size stay for the exact fit to decide.
const fits = try withholdImagesPastLimit(arena, &messages, 30_000, 4096);
try std.testing.expectEqual(@as(usize, 0), fits.withheld);
try std.testing.expectEqual(@as([*]const ChatMessage, &messages), fits.messages.ptr);
// The image count limit applies too.
const many: [default_max_request_images + 5]usize = @splat(1);
try std.testing.expectEqual(@as(usize, 5), oldestImagesToWithhold(&many, many.len, .{ .max_request_bytes = largestRequestBytes(null) }));
}
test "no learned limit lowers the bound a request is fit below" {
try std.testing.expectEqual(default_max_request_bytes, largestRequestBytes(null));
try std.testing.expectEqual(default_max_request_bytes, largestRequestBytes(1000));
// A rejected request that was mostly text can teach a larger limit.
try std.testing.expectEqual(default_max_request_bytes + 1, largestRequestBytes(default_max_request_bytes + 1));
}
test "withheld snapshot attachments point at their saved file" {
var arena_state = std.heap.ArenaAllocator.init(std.testing.allocator);
defer arena_state.deinit();
@@ -225,9 +225,12 @@ test "terminal assistant completion continues with steering admitted during the
resumed_job,
);
// The steer comes back as the live update the model saw, not as a plain
// user message.
try expectBodyContainsInOrder(&resumed_gateway, 0, &.{
"user prompt",
"Original answer",
"user_steering",
"change direction",
"Updated answer",
"follow up",
@@ -14,6 +14,7 @@ const debug_trace = @import("../../shared/debug_trace.zig");
const diff = @import("../../output/diff.zig");
const file_mutation_contract = @import("../../tooling/file_mutation_contract.zig");
const tool_result_errors = @import("../../tooling/tool_result_errors.zig");
const read_result_presentation = @import("../../tooling/read_tool_result_presentation.zig");
const command_result_mapping = if (@import("builtin").is_test)
@import("../../tooling/command_result_mapping.zig")
else
@@ -751,6 +752,198 @@ pub fn fileMutationDisplayPath(
};
}
/// The handles under which a tool result keeps its full output or images.
fn savedOutputHandles(
output_handle: ?[]const u8,
tool_image_handle: ?[]const u8,
command_output_replay: ?types.CommandOutputReplay,
) [3]?[]const u8 {
return .{
output_handle,
tool_image_handle,
if (command_output_replay) |replay| switch (replay) {
.available => |descriptor| descriptor.handle,
.unavailable => null,
} else null,
};
}
/// Finds which call saved an output among a running prompt's messages: the
/// newest tool result that keeps the handle, and the call that produced it.
pub const TurnSavedOutputSources = struct {
registry: tool_dispatch.Registry,
workspace_root: []const u8,
/// Message lists in conversation order, searched newest first.
turns: []const []const types.ChatMessage,
/// Returns the terminal-safe name of the call that saved `handle`, or null
/// when no message still holds it. The arena owns the returned text.
pub fn find(self: TurnSavedOutputSources, arena: Allocator, handle: []const u8) !?[]const u8 {
var turn_index = self.turns.len;
while (turn_index > 0) {
turn_index -= 1;
const messages = self.turns[turn_index];
var message_index = messages.len;
while (message_index > 0) {
message_index -= 1;
const message = messages[message_index];
if (message.role != .tool) continue;
const memory = message.tool_result_memory orelse continue;
if (!holdsHandle(savedOutputHandles(memory.output_handle, memory.tool_image_handle, memory.command_output_replay), handle)) continue;
const source = self.findCall(message.tool_call_id orelse return null) orelse return null;
return tooling_presentation.savedOutputSourceLabel(arena, self.registry, self.workspace_root, source);
}
}
return null;
}
fn findCall(self: TurnSavedOutputSources, call_id: []const u8) ?ToolCall {
var turn_index = self.turns.len;
while (turn_index > 0) {
turn_index -= 1;
const messages = self.turns[turn_index];
var message_index = messages.len;
while (message_index > 0) {
message_index -= 1;
const message = messages[message_index];
if (message.role != .assistant) continue;
for (message.tool_calls) |call| {
if (std.mem.eql(u8, call.id, call_id)) return call;
}
}
}
return null;
}
};
/// Names of the calls whose saved output later `read_tool_result` calls read,
/// recorded while a saved conversation replays in order and keyed by output
/// handle. Owns its keys and names; free them with `deinit`.
pub const ReplayedSavedOutputSources = struct {
map: std.StringHashMapUnmanaged([]const u8) = .empty,
pub fn deinit(self: *ReplayedSavedOutputSources, alloc: Allocator) void {
var it = self.map.iterator();
while (it.next()) |entry| {
alloc.free(entry.key_ptr.*);
alloc.free(entry.value_ptr.*);
}
self.map.deinit(alloc);
}
/// Records `call` under each handle that keeps `result`'s output. A later
/// call saving the same handle wins.
pub fn record(
self: *ReplayedSavedOutputSources,
alloc: Allocator,
registry: tool_dispatch.Registry,
workspace_root: []const u8,
call: ToolCall,
result: types.PersistedToolResult,
) Allocator.Error!void {
const handles = savedOutputHandles(result.output_handle, result.tool_image_handle, result.command_output_replay);
for (handles) |maybe_handle| {
if (maybe_handle != null) break;
} else return;
const label = (try tooling_presentation.savedOutputSourceLabel(alloc, registry, workspace_root, call)) orelse return;
defer alloc.free(label);
for (handles) |maybe_handle| {
const handle = maybe_handle orelse continue;
const owned_handle = try alloc.dupe(u8, handle);
errdefer alloc.free(owned_handle);
const owned_label = try alloc.dupe(u8, label);
errdefer alloc.free(owned_label);
const gop = try self.map.getOrPut(alloc, owned_handle);
if (gop.found_existing) {
alloc.free(owned_handle);
alloc.free(gop.value_ptr.*);
}
gop.value_ptr.* = owned_label;
}
}
/// The name of the call that saved `handle`, borrowed from the map, for
/// `readToolResultDisplayTarget`.
pub fn find(self: *const ReplayedSavedOutputSources, _: Allocator, handle: []const u8) error{}!?[]const u8 {
return self.map.get(handle);
}
};
fn holdsHandle(handles: [3]?[]const u8, handle: []const u8) bool {
for (handles) |candidate| {
if (std.mem.eql(u8, candidate orelse continue, handle)) return true;
}
return false;
}
/// Row text for a `read_tool_result` call: what it reads, and the earlier
/// call whose output that is when `sources.find(arena, handle)` still knows
/// it. Returns null for other tools. The arena owns the returned text.
pub fn readToolResultDisplayTarget(
arena: Allocator,
registry: tool_dispatch.Registry,
call: ToolCall,
sources: anytype,
) !?[]const u8 {
const spec = registry.lookup(call.name) orelse return null;
if (spec.executor_kind != .read_tool_result) return null;
const source = if (try read_result_presentation.requestedHandle(arena, call.arguments_json)) |handle|
try sources.find(arena, handle)
else
null;
return try read_result_presentation.displayTarget(arena, call.arguments_json, source);
}
test "read_tool_result rows name the earlier call whose output they read" {
var arena_state = std.heap.ArenaAllocator.init(std.testing.allocator);
defer arena_state.deinit();
const arena = arena_state.allocator();
const registry = test_builtin_tools.advertisement_set.registry;
const replay = "fx-command-replay-0123.bin";
const earlier_turn = [_]types.ChatMessage{
.{ .role = .assistant, .tool_calls = &.{.{
.id = "read-1",
.name = "read_file",
.arguments_json = "{\"path\":\"src/main.zig\"}",
}} },
.{ .role = .tool, .tool_call_id = "read-1", .tool_result_memory = .{ .output_handle = "result-read_file-1.txt" } },
};
const current_turn = [_]types.ChatMessage{
.{ .role = .assistant, .tool_calls = &.{.{
.id = "run-1",
.name = "shell",
.arguments_json = "{\"action\":\"run\",\"command\":\"zig build test\"}",
}} },
.{ .role = .tool, .tool_call_id = "run-1", .tool_result_memory = .{
.command_output_replay = .{ .available = .{ .handle = replay, .framed_bytes = 64 } },
} },
};
const sources: TurnSavedOutputSources = .{
.registry = registry,
.workspace_root = "/workspace",
.turns = &.{ earlier_turn[0..], current_turn[0..] },
};
const cases = [_]struct { arguments_json: []const u8, expected: ?[]const u8 }{
.{ .arguments_json = "{\"request\":{\"handle\":\"" ++ replay ++ "\",\"start_byte\":9}}", .expected = "more output from zig build test" },
.{ .arguments_json = "{\"request\":{\"handle\":\"" ++ replay ++ "\",\"query\":\"error:\"}}", .expected = "zig build test output for \"error:\"" },
// The model may drop the `.txt` that `validate` restores.
.{ .arguments_json = "{\"request\":{\"handle\":\"result-read_file-1\"}}", .expected = "more output from src/main.zig" },
.{ .arguments_json = "{\"request\":{\"handle\":\"fx-command-replay-ffff.bin\"}}", .expected = null },
.{ .arguments_json = "{\"request\":{\"handle\":\"T5\"}}", .expected = "earlier tool call T5" },
};
for (cases) |case| {
const call: ToolCall = .{ .id = "page", .name = "read_tool_result", .arguments_json = case.arguments_json };
const target = try readToolResultDisplayTarget(arena, registry, call, sources);
if (case.expected) |expected| {
try std.testing.expectEqualStrings(expected, target orelse return error.TestExpectedDisplayTarget);
} else {
try std.testing.expect(target == null);
}
}
const other: ToolCall = .{ .id = "grep", .name = "grep_files", .arguments_json = "{\"pattern\":\"x\"}" };
try std.testing.expect(try readToolResultDisplayTarget(arena, registry, other, sources) == null);
}
pub noinline fn startToolVisibleLifecycle(
hooks: *const AgentRuntimeDeps,
arena: Allocator,
@@ -778,6 +971,7 @@ pub noinline fn startToolVisibleLifecycle(
.activity_kind = activity_kind,
.arguments_json = redacted_arguments,
.place_after_current_transcript = false,
.resolved_display_target = call.resolved_display_target,
} });
if (activity_line) |line| {
try hooks.push_tool_lifecycle(hooks.ctx, .{ .progress = .{
+3
View File
@@ -465,6 +465,9 @@ pub const Provider = struct {
build_request_fn: ?BuildRequestFn = null,
/// Pure provider-owned slicing when one reply becomes separate history units.
project_replay_fn: ?*const fn (Allocator, ?types.ProviderReplay, []const types.ToolCall, bool, bool) anyerror!?types.ProviderReplay = null,
/// The largest request body this provider's transport can carry, when it
/// has its own limit below any provider's.
max_request_bytes: ?usize = null,
/// Borrows unchanged payloads; projected payloads live in the caller's arena.
pub fn projectReplay(self: Provider, arena: Allocator, replay: ?types.ProviderReplay, calls: []const types.ToolCall, text: bool, reasoning: bool) !?types.ProviderReplay {
+1
View File
@@ -1366,6 +1366,7 @@ fn formatToolAction(
const presentation = tool_dispatch.presentationForArgs(spec.*, args);
const value = display_target orelse
call.resolved_display_target orelse
tool_presentation.resolvedSkillName(call, presentation) orelse
tool_dispatch.presentationLabelValue(presentation, args) orelse
presentation.label_arg_default;
+6 -2
View File
@@ -1770,7 +1770,8 @@ pub const State = struct {
var arena: std.heap.ArenaAllocator = .init(std.heap.c_allocator);
defer arena.deinit();
const a = arena.allocator();
const statuses = lease.host.runtime.statuses(a) catch return "";
// The status line never shows sign-in state, so it doesn't wait for it.
const statuses = lease.host.runtime.statusesNow(a) catch return "";
sink.mutex.lockUncancelable(io_mod.getIo());
const news: ?[]const u8 = if (sink.news) |text|
(if (now_ms - sink.news_at_ms < mcp_menu.news_ms) a.dupe(u8, text) catch null else null)
@@ -2045,7 +2046,10 @@ pub const State = struct {
try out.appendSlice(a, &verb_completions);
const lease = self.leaseHost() orelse return out.items;
defer lease.deinit();
const statuses = try lease.host.runtime.statuses(a);
// Rebuilt twice a second on the UI thread, so it doesn't wait for
// saved credentials: a description says "signed in" from the next
// rebuild after they are restored.
const statuses = try lease.host.runtime.statusesNow(a);
for (statuses, lease.host.runtime.servers()) |status, server| {
if (!mcp_command_provider.isValidServerName(status.name)) continue;
const description = try mcp_verbs.statusText(a, status);
+103 -6
View File
@@ -4530,6 +4530,9 @@ pub fn Runtime(comptime App: type) type {
workspace_root: []const u8,
map: std.StringHashMapUnmanaged(Label) = .empty,
/// Calls whose saved output later `read_tool_result` rows read, so
/// those rows name them as the live session did.
saved_output_sources: tool_presentation.ReplayedSavedOutputSources = .{},
fn deinit(self: *HistoricalSessionLabels, alloc: Allocator) void {
var it = self.map.iterator();
@@ -4539,6 +4542,7 @@ pub fn Runtime(comptime App: type) type {
if (entry.value_ptr.reflow) |value| alloc.free(value);
}
self.map.deinit(alloc);
self.saved_output_sources.deinit(alloc);
}
};
@@ -5066,6 +5070,13 @@ pub fn Runtime(comptime App: type) type {
result: types.PersistedToolResult,
labels: *HistoricalSessionLabels,
) !void {
try labels.saved_output_sources.record(
app.alloc,
app.toolAdvertisementSet().registry,
labels.workspace_root,
call,
result,
);
if (try writeAnsweredQuestionResult(app, sink, call, result)) return;
if (try writeCommittedFilePresentation(app, sink, call, result)) return;
@@ -5080,7 +5091,13 @@ pub fn Runtime(comptime App: type) type {
var action_arena = std.heap.ArenaAllocator.init(app.alloc);
defer mem_utils.deinit_arena(action_arena);
const session_target = historicalSessionTarget(action_arena.allocator(), labels, call);
const action_target = historicalSessionTarget(action_arena.allocator(), labels, call) orelse
try tool_presentation.readToolResultDisplayTarget(
action_arena.allocator(),
app.toolAdvertisementSet().registry,
call,
&labels.saved_output_sources,
);
const command_decision = if (is_command)
try tool_presentation.commandOutcomeDecision(
action_arena.allocator(),
@@ -5100,7 +5117,7 @@ pub fn Runtime(comptime App: type) type {
try app.describeToolActionDeniedWithAdvertised(
action_arena.allocator(),
call,
session_target,
action_target,
if (context_deferred)
types.context_deferred_tool_status_label
else
@@ -5111,7 +5128,7 @@ pub fn Runtime(comptime App: type) type {
try app.describeToolActionDeniedWithAdvertised(
action_arena.allocator(),
call,
session_target,
action_target,
tool_admission.permissionDeniedStatusLabel(reason),
&.{},
)
@@ -5119,7 +5136,7 @@ pub fn Runtime(comptime App: type) type {
try app.describeToolActionDeniedWithAdvertised(
action_arena.allocator(),
call,
session_target,
action_target,
decision.label,
&.{},
)
@@ -5143,13 +5160,13 @@ pub fn Runtime(comptime App: type) type {
break :success try app.describeToolActionCompletedWithAdvertised(
action_arena.allocator(),
call,
display_target orelse session_target,
display_target orelse action_target,
&.{},
);
} else try app.describeToolActionDeniedWithAdvertised(
action_arena.allocator(),
call,
session_target,
action_target,
try tooling_presentation.subagentFailureLabel(action_arena.allocator(), call, result.output),
&.{},
);
@@ -8259,6 +8276,86 @@ test "execution replay records session labels for tty-launched sessions" {
);
}
test "execution replay names the call whose saved output a later read opens" {
const alloc = std.testing.allocator;
var app = try TestApp.init(alloc, "/workspace");
defer app.deinit();
const replay = "fx-command-replay-0123.bin";
const run_output = "{\"exit_code\":0,\"full_output_handle\":\"" ++ replay ++ "\"}";
const file_output = "<path>src/main.zig</path>";
var source_calls = [_]types.ToolCall{
.{ .id = "call_run", .name = "shell", .arguments_json = "{\"action\":\"run\",\"command\":\"zig build test\"}" },
.{ .id = "call_file", .name = "read_file", .arguments_json = "{\"path\":\"src/main.zig\"}" },
};
var source_results = [_]types.PersistedToolResult{
.{
.tool_call_id = @constCast("call_run"),
.tool_name = @constCast("shell"),
.status = .success,
.output = @constCast(run_output),
.output_bytes = run_output.len,
.stored_output_bytes = run_output.len,
.command_output_replay = .{ .available = .{ .handle = replay, .framed_bytes = 64 } },
},
.{
.tool_call_id = @constCast("call_file"),
.tool_name = @constCast("read_file"),
.status = .success,
.output = @constCast(file_output),
.output_handle = @constCast("result-read_file-1.txt"),
.output_bytes = file_output.len,
.stored_output_bytes = file_output.len,
},
};
var source_steps = [_]types.ToolExecutionStep{.{
.tool_calls = source_calls[0..],
.tool_results = source_results[0..],
}};
var read_calls = [_]types.ToolCall{
.{ .id = "page", .name = "read_tool_result", .arguments_json = "{\"request\":{\"handle\":\"" ++ replay ++ "\",\"start_byte\":9}}" },
.{ .id = "query", .name = "read_tool_result", .arguments_json = "{\"request\":{\"handle\":\"" ++ replay ++ "\",\"query\":\"error:\"}}" },
.{ .id = "file", .name = "read_tool_result", .arguments_json = "{\"request\":{\"handle\":\"result-read_file-1\"}}" },
.{ .id = "unknown", .name = "read_tool_result", .arguments_json = "{\"request\":{\"handle\":\"result-gone.txt\"}}" },
};
var read_results: [read_calls.len]types.PersistedToolResult = undefined;
for (&read_results, read_calls) |*result, call| {
result.* = .{
.tool_call_id = @constCast(call.id),
.tool_name = @constCast(call.name),
.status = .success,
.output = @constCast("page"),
.output_bytes = 4,
.stored_output_bytes = 4,
};
}
var read_steps = [_]types.ToolExecutionStep{.{
.tool_calls = read_calls[0..],
.tool_results = read_results[0..],
}};
const history = [_]types.HistoryTurn{
.{ .assistant = .{
.user = .{ .text = @constCast("run the tests") },
.assistant = @constCast(""),
.execution = .{ .tool_steps = source_steps[0..] },
} },
.{ .assistant = .{
.user = .{ .text = @constCast("find the failure") },
.assistant = @constCast(""),
.execution = .{ .tool_steps = read_steps[0..] },
} },
};
try Runtime(TestApp).replayHistory(&app, &history);
const statuses = app.completed_tool_statuses.items;
try std.testing.expectEqual(@as(usize, 6), statuses.len);
try std.testing.expectEqualStrings("● Completed read_tool_result more output from zig build test\n", statuses[2]);
try std.testing.expectEqualStrings("● Completed read_tool_result zig build test output for \"error:\"\n", statuses[3]);
try std.testing.expectEqualStrings("● Completed read_tool_result more output from src/main.zig\n", statuses[4]);
try std.testing.expectEqualStrings("● Completed read_tool_result\n", statuses[5]);
}
test "resume projection stores reflow metadata for session action rows" {
const alloc = std.testing.allocator;
var app = try TestApp.init(alloc, "/workspace");
+105
View File
@@ -14,6 +14,11 @@ const pathing = @import("../workspace/pathing.zig");
const text_utils = @import("../shared/text_utils.zig");
pub const max_image_bytes: usize = 20 * 1024 * 1024;
/// The most images, and image bytes together, one libfx prompt holds: the
/// limits `sdk/fx-sdk.js` takes a prompt within (`maxPromptImages` and
/// `maxPromptImagesBytes`), which the core holds a prompt to as well.
pub const max_libfx_prompt_images: usize = 8;
pub const max_libfx_prompt_images_bytes: usize = 6 * 1024 * 1024;
const max_encoded_image_bytes: usize = 5 * 1024 * 1024;
pub const image_too_large_notice = "image exceeds the 20 MiB limit";
pub const image_preparation_failed_notice = "Unable to prepare this image for upload. Use a smaller image.";
@@ -848,6 +853,62 @@ pub fn attachmentEncodedBytes(attachment: types.ImageAttachment) ?usize {
return image_data.encodedImageBytes(size);
}
/// Leaves the images the session dropped out of a request, each with a note
/// in its place, so no later projection or provider sees an image without
/// bytes. Returns `messages` itself when none were dropped, otherwise a copy
/// allocated in `arena`. History is not modified.
pub fn withholdDroppedImages(arena: std.mem.Allocator, messages: []const types.ChatMessage) ![]const types.ChatMessage {
var result: ?[]types.ChatMessage = null;
for (messages, 0..) |message, index| {
if (message.tool_result_memory) |memory| {
var dropped_tools: usize = 0;
for (memory.tool_images) |image| {
if (image.droppedFromSession()) dropped_tools += 1;
}
if (dropped_tools > 0) {
const projected = result orelse try arena.dupe(types.ChatMessage, messages);
result = projected;
const kept = try arena.alloc(types.ToolImage, memory.tool_images.len - dropped_tools);
var next: usize = 0;
for (memory.tool_images) |image| {
if (image.droppedFromSession()) continue;
kept[next] = image;
next += 1;
}
var withheld = memory;
withheld.tool_images = kept;
projected[index].tool_result_memory = withheld;
const notice = if (dropped_tools == 1)
"[1 image from this result is no longer in this session, which keeps only its newest images.]\n"
else
try arena.print("[{d} images from this result are no longer in this session, which keeps only its newest images.]\n", .{dropped_tools});
projected[index].content = try std.mem.concat(arena, u8, &.{ notice, message.content orelse "" });
}
}
var dropped: usize = 0;
for (message.images) |image| {
if (image.droppedFromSession()) dropped += 1;
}
if (dropped == 0) continue;
const projected = result orelse try arena.dupe(types.ChatMessage, messages);
result = projected;
const kept = try arena.alloc(types.ImageAttachment, message.images.len - dropped);
var notice: std.Io.Writer.Allocating = .init(arena);
var next: usize = 0;
for (message.images) |image| {
if (image.droppedFromSession()) {
notice.writer.print("[Image #{d} is no longer in this session, which keeps only its newest images.]\n", .{image.id}) catch return error.OutOfMemory;
} else {
kept[next] = image;
next += 1;
}
}
projected[index].images = kept;
projected[index].content = try std.mem.concat(arena, u8, &.{ notice.written(), projected[index].content orelse "" });
}
return result orelse messages;
}
fn writeWithheldAttachmentNotice(
writer: *std.Io.Writer,
attachment: types.ImageAttachment,
@@ -3058,6 +3119,50 @@ test "capture keeps a byte-oversized original for request recovery" {
try std.testing.expectEqual(@as(usize, 1), try countSnapshotFiles(snapshot_dir));
}
test "requests leave out images the session dropped, each with a note" {
var arena_state = std.heap.ArenaAllocator.init(std.testing.allocator);
defer arena_state.deinit();
const arena = arena_state.allocator();
const digest = text_utils.repeat("a", 64);
const images = [_]types.ImageAttachment{
.{ .id = 1, .path = @constCast("inline://image-1"), .media_type = @constCast("image/png"), .snapshot_sha256 = @constCast(digest) },
.{ .id = 2, .path = @constCast("inline://image-2"), .media_type = @constCast("image/png"), .snapshot_sha256 = @constCast(digest), .inline_data = @constCast("png") },
};
const messages = [_]types.ChatMessage{
.{ .role = .user, .content = "[Image #1] [Image #2]", .images = &images },
.{ .role = .assistant, .content = "ok" },
};
const projected = try withholdDroppedImages(arena, &messages);
try std.testing.expectEqual(@as(usize, 1), projected[0].images.len);
try std.testing.expectEqual(@as(usize, 2), projected[0].images[0].id);
try std.testing.expectEqualStrings("[Image #1 is no longer in this session, which keeps only its newest images.]\n[Image #1] [Image #2]", projected[0].content.?);
// A request without dropped images passes through untouched.
const kept = [_]types.ChatMessage{.{ .role = .user, .content = "[Image #2]", .images = images[1..] }};
try std.testing.expect((try withholdDroppedImages(arena, &kept)).ptr == (&kept).ptr);
}
test "requests leave out tool images the session dropped, with one note per result" {
var arena_state = std.heap.ArenaAllocator.init(std.testing.allocator);
defer arena_state.deinit();
const arena = arena_state.allocator();
const name = text_utils.repeat("b", 64)[0..64].*;
const tool_images = [_]types.ToolImage{
.{ .data = @constCast(&[_]u8{}), .mime_type = @constCast("image/png"), .name = name },
.{ .data = @constCast("aGVsbG8="), .mime_type = @constCast("image/png"), .name = name },
.{ .data = @constCast(&[_]u8{}), .mime_type = @constCast("image/png"), .name = name },
};
const messages = [_]types.ChatMessage{
.{ .role = .tool, .content = "took it", .tool_call_id = "call-1", .tool_name = "screenshot", .tool_result_memory = .{ .tool_images = &tool_images } },
};
const projected = try withholdDroppedImages(arena, &messages);
const memory = projected[0].tool_result_memory.?;
try std.testing.expectEqual(@as(usize, 1), memory.tool_images.len);
try std.testing.expectEqualStrings("aGVsbG8=", memory.tool_images[0].data);
try std.testing.expectEqualStrings("[2 images from this result are no longer in this session, which keeps only its newest images.]\ntook it", projected[0].content.?);
// The session's own messages keep their images.
try std.testing.expectEqual(@as(usize, 3), messages[0].tool_result_memory.?.tool_images.len);
}
test "requests leave out attachments over the model pixel limit and name their saved file" {
var arena_state = std.heap.ArenaAllocator.init(std.testing.allocator);
defer arena_state.deinit();
+10 -26
View File
@@ -45,12 +45,6 @@ pub const Server = struct {
missing: ?[]const u8 = null,
};
/// Looks up the credential saved for a server; owned by `alloc`.
pub const Saved = struct {
context: *anyopaque,
get: *const fn (context: *anyopaque, alloc: Allocator, name: []const u8, url: []const u8) Allocator.Error!?[]u8,
};
pub const Resolved = struct {
servers: []Server,
/// What the engine runs, in order: `engine[i]` is `servers[engine_server[i]]`.
@@ -58,10 +52,6 @@ pub const Resolved = struct {
engine_server: []usize,
};
/// Allocates everything with `arena`, which must outlive the engine: the
/// engine copies each config but keeps pointing at a stdio server's
/// environment. A saved credential is copied into `arena`; zero it once the
/// engine has its own copy.
/// The command and its arguments as one line, quoting what a shell would split.
fn commandLine(arena: Allocator, command: []const u8, args: []const []const u8) Allocator.Error![]const u8 {
var out: std.ArrayList(u8) = .empty;
@@ -80,11 +70,14 @@ fn commandLine(arena: Allocator, command: []const u8, args: []const []const u8)
return out.items;
}
/// Allocates everything with `arena`, which must outlive the engine: the
/// engine copies each config but keeps pointing at a stdio server's
/// environment. Saved sign-in credentials are not read here: the runtime
/// hands them to the engine from its own thread.
pub fn resolve(
arena: Allocator,
configs: []const McpServerConfig,
inherited: *const std.process.Environ.Map,
saved: ?Saved,
) Allocator.Error!Resolved {
const servers = try arena.alloc(Server, configs.len);
var engine: std.ArrayList(mcp.engine.Config) = .empty;
@@ -126,9 +119,6 @@ pub fn resolve(
if (s.bridge and auth.resource == null) if (s.url) |url| {
auth.resource = try arena.print("{s}/", .{origin(url)});
};
if (saved) |lookup| if (s.url) |url| {
auth.saved = try lookup.get(lookup.context, arena, s.name, url);
};
try engine.append(arena, .{ .name = s.name, .transport = transport, .auth = auth });
try engine_server.append(arena, i);
}
@@ -227,14 +217,7 @@ test "stdio and HTTP servers become engine configs; the rest are held back with
.{ .name = "keyless", .transport = .http, .url = "https://k.example/mcp", .header_env = &missing_header },
.{ .name = "carried", .transport = .http, .url = "http://acp.invalid/x", .source = .acp, .acp_server_id = @constCast("x") },
};
const Lookup = struct {
fn get(_: *anyopaque, alloc: Allocator, name: []const u8, url: []const u8) Allocator.Error!?[]u8 {
if (std.mem.eql(u8, name, "linear") and std.mem.eql(u8, url, "https://mcp.linear.app/mcp")) return try alloc.dupe(u8, "{\"access\":\"a\"}");
return null;
}
};
var context: u8 = 0;
const r = try resolve(arena, &configs, &inherited, .{ .context = &context, .get = Lookup.get });
const r = try resolve(arena, &configs, &inherited);
try testing.expectEqual(@as(usize, 3), r.engine.len);
try testing.expectEqualSlices(usize, &.{ 0, 1, 2 }, r.engine_server);
@@ -250,7 +233,8 @@ test "stdio and HTTP servers become engine configs; the rest are held back with
try testing.expectEqualStrings("t1", linear.transport.http.headers[0].value);
try testing.expectEqualStrings("c1", linear.auth.client_id.?);
try testing.expectEqualStrings("s1", linear.auth.client_secret.?);
try testing.expectEqualStrings("{\"access\":\"a\"}", linear.auth.saved.?);
// The runtime's thread hands over saved credentials later.
try testing.expect(linear.auth.saved == null);
try testing.expectEqual(@as(?u16, 33418), r.servers[2].callback_port);
try testing.expectEqual(@as(?Held, .disabled), r.servers[3].held);
@@ -272,7 +256,7 @@ test "Slack signs in as fx's Slack app through the bridge, unless the config nam
.{ .name = "mine", .transport = .http, .url = "https://mcp.slack.com/mcp", .auth = .{ .client_id = @constCast("my-app") } },
.{ .name = "linear", .transport = .http, .url = "https://mcp.linear.app/mcp" },
};
const r = try resolve(arena_state.allocator(), &configs, &inherited, null);
const r = try resolve(arena_state.allocator(), &configs, &inherited);
try testing.expectEqualStrings(slack.client_id, r.engine[0].auth.client_id.?);
try testing.expect(r.servers[0].bridge);
try testing.expectEqualStrings("https://mcp.slack.com/", r.engine[0].auth.resource.?);
@@ -292,7 +276,7 @@ test "a stdio server starts in the directory its config names" {
.{ .name = "proj", .command = "node", .args = &.{"s.js"}, .cwd = @constCast("/work/app") },
.{ .name = "here", .command = "node", .args = &.{"s.js"} },
};
const r = try resolve(arena_state.allocator(), &configs, &inherited, null);
const r = try resolve(arena_state.allocator(), &configs, &inherited);
try testing.expectEqualStrings("/work/app", r.engine[0].transport.stdio.cwd.?);
try testing.expect(r.engine[1].transport.stdio.cwd == null);
}
@@ -307,7 +291,7 @@ test "a token from the environment becomes the Authorization header" {
.{ .name = "plain", .transport = .http, .url = "https://mcp.plain.com/mcp", .bearer_token_env = @constCast("PLAIN_KEY") },
.{ .name = "unset", .transport = .http, .url = "https://u.example/mcp", .bearer_token_env = @constCast("UNSET_KEY") },
};
const r = try resolve(arena_state.allocator(), &configs, &inherited, null);
const r = try resolve(arena_state.allocator(), &configs, &inherited);
try testing.expectEqual(@as(usize, 1), r.engine.len);
try testing.expectEqualStrings("Authorization", r.engine[0].transport.http.headers[0].name);
try testing.expectEqualStrings("Bearer k1", r.engine[0].transport.http.headers[0].value);
+59 -16
View File
@@ -39,15 +39,13 @@ pub const Store = struct {
return .{ .home = home, .backend = if (keychain) .keychain else .file };
}
/// The credential saved for `name` at `url`, or null. The caller owns it
/// and frees it with `secret.zeroAndFree`.
pub fn get(s: Store, alloc: Allocator, name: []const u8, url: []const u8) !?[]u8 {
/// Every saved credential from one read of the store, so a host with
/// many servers reads the Keychain once, not once per server. The
/// caller frees it with `Snapshot.deinit`.
pub fn snapshot(s: Store, alloc: Allocator) !Snapshot {
var locked = try s.lock();
defer locked.release();
var doc = try s.read(alloc, &locked.dir);
defer doc.deinit(alloc);
const i = doc.find(name, url) orelse return null;
return try alloc.dupe(u8, doc.entries.items[i].credential);
return .{ .doc = try s.read(alloc, &locked.dir) };
}
/// Saves `credential` for `name` at `url`, replacing any earlier one.
@@ -162,6 +160,24 @@ pub const Store = struct {
}
};
/// The saved credentials as one read of the store found them.
pub const Snapshot = struct {
doc: Doc,
/// The credential saved for `name` at `url`, or null. Borrowed from the
/// snapshot until `deinit`.
pub fn get(s: *const Snapshot, name: []const u8, url: []const u8) ?[]const u8 {
const i = s.doc.find(name, url) orelse return null;
return s.doc.entries.items[i].credential;
}
/// Zeroes the credentials and frees them.
pub fn deinit(s: *Snapshot, alloc: Allocator) void {
s.doc.deinit(alloc);
s.* = undefined;
}
};
fn readFile(alloc: Allocator, dir: *io_mod.VerifiedDir) !?[]u8 {
var file = dir.dir.openFile(io_mod.getIo(), file_name, .{
.mode = .read_only,
@@ -278,6 +294,33 @@ fn testHome(tmp: *testing.TmpDir) ![]u8 {
return io_mod.dirRealpathAlloc(testing.allocator, tmp.dir, ".");
}
/// The credential saved for `name` at `url` from a fresh read, or null.
/// The caller frees it with `secret.zeroAndFree`.
fn savedFor(store: Store, name: []const u8, url: []const u8) !?[]u8 {
var saved = try store.snapshot(testing.allocator);
defer saved.deinit(testing.allocator);
const credential = saved.get(name, url) orelse return null;
return try testing.allocator.dupe(u8, credential);
}
test "one snapshot holds every server's credential" {
var tmp = testing.tmpDir(.{});
defer tmp.cleanup();
const home = try testHome(&tmp);
defer testing.allocator.free(home);
const store: Store = .{ .home = home, .backend = .file };
try store.put(testing.allocator, "linear", "https://mcp.linear.app/mcp", "{\"access\":\"l1\"}");
try store.put(testing.allocator, "plain", "https://mcp.plain.com/mcp", "{\"access\":\"p1\"}");
var saved = try store.snapshot(testing.allocator);
defer saved.deinit(testing.allocator);
try testing.expectEqualStrings("{\"access\":\"l1\"}", saved.get("linear", "https://mcp.linear.app/mcp").?);
try testing.expectEqualStrings("{\"access\":\"p1\"}", saved.get("plain", "https://mcp.plain.com/mcp").?);
// A credential is kept per name and URL.
try testing.expectEqual(@as(?[]const u8, null), saved.get("plain", "https://mcp.linear.app/mcp"));
try testing.expectEqual(@as(?[]const u8, null), saved.get("slack", "https://mcp.slack.com/mcp"));
}
test "a credential is kept per server name and URL, replaced, and removed" {
var tmp = testing.tmpDir(.{});
defer tmp.cleanup();
@@ -285,21 +328,21 @@ test "a credential is kept per server name and URL, replaced, and removed" {
defer testing.allocator.free(home);
const store: Store = .{ .home = home, .backend = .file };
try testing.expectEqual(@as(?[]u8, null), try store.get(testing.allocator, "linear", "https://mcp.linear.app/mcp"));
try testing.expectEqual(@as(?[]u8, null), try savedFor(store, "linear", "https://mcp.linear.app/mcp"));
try store.put(testing.allocator, "linear", "https://mcp.linear.app/mcp", "{\"access\":\"a1\"}");
try store.put(testing.allocator, "linear", "https://other.example/mcp", "{\"access\":\"b1\"}");
try store.put(testing.allocator, "linear", "https://mcp.linear.app/mcp", "{\"access\":\"a2\"}");
const first = (try store.get(testing.allocator, "linear", "https://mcp.linear.app/mcp")).?;
const first = (try savedFor(store, "linear", "https://mcp.linear.app/mcp")).?;
defer secret.zeroAndFree(testing.allocator, first);
try testing.expectEqualStrings("{\"access\":\"a2\"}", first);
const other = (try store.get(testing.allocator, "linear", "https://other.example/mcp")).?;
const other = (try savedFor(store, "linear", "https://other.example/mcp")).?;
defer secret.zeroAndFree(testing.allocator, other);
try testing.expectEqualStrings("{\"access\":\"b1\"}", other);
try testing.expect(try store.remove(testing.allocator, "linear", "https://mcp.linear.app/mcp"));
try testing.expect(!try store.remove(testing.allocator, "linear", "https://mcp.linear.app/mcp"));
try testing.expectEqual(@as(?[]u8, null), try store.get(testing.allocator, "linear", "https://mcp.linear.app/mcp"));
try testing.expectEqual(@as(?[]u8, null), try savedFor(store, "linear", "https://mcp.linear.app/mcp"));
}
test "v1's logins are taken over once, and a sign-out isn't undone" {
@@ -325,16 +368,16 @@ test "v1's logins are taken over once, and a sign-out isn't undone" {
"{\"server_identity\":\"bound\",\"token_type\":\"DPoP\"," ++ entry ++ "}]}");
}
const store: Store = .{ .home = home, .backend = .file };
const got = (try store.get(testing.allocator, "linear", "https://mcp.linear.app/mcp")).?;
const got = (try savedFor(store, "linear", "https://mcp.linear.app/mcp")).?;
defer secret.zeroAndFree(testing.allocator, got);
try testing.expectEqualStrings(
\\{"access":"v1-access","refresh":"v1-refresh","expires_at_ms":null,"token_endpoint":"https://linear.app/oauth/token","revocation_endpoint":null,"resource":"https://mcp.linear.app/mcp","client_id":"c1","client_secret":null,"method":"none"}
, got);
// A token bound to a key isn't one MCP-v2 can send.
try testing.expectEqual(@as(?[]u8, null), try store.get(testing.allocator, "bound", "https://mcp.linear.app/mcp"));
try testing.expectEqual(@as(?[]u8, null), try savedFor(store, "bound", "https://mcp.linear.app/mcp"));
// Signed out, it stays signed out: v1's copy isn't taken again.
try testing.expect(try store.remove(testing.allocator, "linear", "https://mcp.linear.app/mcp"));
try testing.expectEqual(@as(?[]u8, null), try store.get(testing.allocator, "linear", "https://mcp.linear.app/mcp"));
try testing.expectEqual(@as(?[]u8, null), try savedFor(store, "linear", "https://mcp.linear.app/mcp"));
}
test "the credential file is private, and an unreadable one is replaced on the next save" {
@@ -350,9 +393,9 @@ test "the credential file is private, and an unreadable one is replaced on the n
try testing.expectEqual(@as(std.posix.mode_t, 0o600), stat.permissions.toMode() & 0o777);
try tmp.dir.writeFile(testing.io, .{ .sub_path = path, .data = "not json" });
try testing.expectEqual(@as(?[]u8, null), try store.get(testing.allocator, "plain", "https://mcp.plain.com/mcp"));
try testing.expectEqual(@as(?[]u8, null), try savedFor(store, "plain", "https://mcp.plain.com/mcp"));
try store.put(testing.allocator, "plain", "https://mcp.plain.com/mcp", "{\"access\":\"p2\"}");
const saved = (try store.get(testing.allocator, "plain", "https://mcp.plain.com/mcp")).?;
const saved = (try savedFor(store, "plain", "https://mcp.plain.com/mcp")).?;
defer secret.zeroAndFree(testing.allocator, saved);
try testing.expectEqualStrings("{\"access\":\"p2\"}", saved);
}
+1 -1
View File
@@ -48,6 +48,6 @@ pub const Fixture = struct {
}
pub fn options(f: *const Fixture) Options {
return .{ .client_version = "test", .credentials = .{ .home = f.home, .backend = .file } };
return .{ .client_version = "test", .credentials = .{ .store = .{ .home = f.home, .backend = .file } } };
}
};
+83 -23
View File
@@ -18,7 +18,6 @@ const host_target = @import("../hosts/target.zig");
const debug_trace = @import("../shared/debug_trace.zig");
const text_utils = @import("../shared/text_utils.zig");
const url_opener = @import("../hosts/url_opener.zig");
const secret = @import("../auth/secret.zig");
const config = @import("config.zig");
const credentials = @import("credentials.zig");
const loopback = @import("login.zig");
@@ -132,7 +131,11 @@ pub const Notify = struct {
pub const Options = struct {
client_version: []const u8,
credentials: credentials.Store,
/// Where sign-in credentials are kept. `.detect` names the home whose
/// store the owner thread picks with `credentials.Store.detect`, so
/// `start` never waits for the Keychain; the owner replaces it with
/// that store before it handles any request.
credentials: union(enum) { detect: []const u8, store: credentials.Store },
notify: ?Notify = null,
/// The questions fx's UI on this surface can ask, declared to servers so
/// they only ask those.
@@ -260,6 +263,9 @@ pub const Runtime = struct {
/// Guarded by `mutex`: the latest statuses, in `snapshot_arena`.
snapshot: []Status = &.{},
snapshot_arena: std.heap.ArenaAllocator,
/// Set once the owner gave the servers their saved credentials and
/// published that, so `statuses` never reports a premature signed_in.
credentials_restored: std.Io.Event = .unset,
/// Owner thread only.
calls: std.AutoHashMapUnmanaged(engine.Handle, *Request) = .empty,
@@ -280,7 +286,8 @@ pub const Runtime = struct {
/// owner thread. `configs` and `inherited` are read only during the call.
/// `gpa` must be thread-safe, and `r` must not move until `deinit`: the
/// engine and login tasks point at it. Nothing connects until a server
/// is needed.
/// is needed, and saved credentials are read on the owner thread, so
/// this never waits for the Keychain.
pub fn start(
r: *Runtime,
gpa: Allocator,
@@ -302,17 +309,15 @@ pub const Runtime = struct {
errdefer r.arena.deinit();
errdefer r.snapshot_arena.deinit();
const arena = r.arena.allocator();
r.resolved = try config.resolve(arena, configs, inherited, .{ .context = r, .get = savedCredential });
r.resolved = try config.resolve(arena, configs, inherited);
r.flags = try arena.alloc(Flags, r.resolved.servers.len);
for (r.flags) |*f| f.* = .{};
for (r.resolved.engine, r.resolved.engine_server) |c, i| r.flags[i].signed_in = c.auth.saved != null;
try r.engine.init(io, gpa, .{ .context = r, .allow = allow, .log = log }, .{
.client = .{ .info = .{ .name = "fx", .version = options.client_version }, .capabilities = declaredCapabilities(options.questions) },
}, r.resolved.engine);
errdefer r.engine.deinit();
// The engine has its own copies now.
for (r.resolved.engine) |c| if (c.auth.saved) |s| std.crypto.secureZero(u8, @constCast(s));
// The rest may point into `configs`, which the caller may free now.
// The engine has its own copies now, and these may point into
// `configs`, which the caller may free now.
r.resolved.engine = &.{};
try r.publish();
r.thread = try std.Thread.spawn(.{}, run, .{r});
@@ -340,8 +345,19 @@ pub const Runtime = struct {
r.* = undefined;
}
/// Every configured server's status, copied into `alloc`.
/// Every configured server's status, copied into `alloc`. Waits for the
/// owner to restore saved credentials, so `signed_in` is settled.
pub fn statuses(r: *Runtime, alloc: Allocator) Allocator.Error![]Status {
_ = r.active.fetchAdd(1, .acq_rel);
defer _ = r.active.fetchSub(1, .acq_rel);
r.credentials_restored.waitUncancelable(r.io);
return r.statusesNow(alloc);
}
/// `statuses` without the wait, for the UI thread's periodic refreshes,
/// which must not hold up a frame: until the owner restored saved
/// credentials, every server reads as not signed in.
pub fn statusesNow(r: *Runtime, alloc: Allocator) Allocator.Error![]Status {
_ = r.active.fetchAdd(1, .acq_rel);
defer _ = r.active.fetchSub(1, .acq_rel);
r.mutex.lockUncancelable(r.io);
@@ -514,6 +530,7 @@ pub const Runtime = struct {
// ---- the owner thread ----
fn run(r: *Runtime) void {
r.restoreCredentials();
while (true) {
const stopping = r.drain();
r.flush();
@@ -650,7 +667,7 @@ pub const Runtime = struct {
const s = r.resolved.servers[i];
if (s.transport != .http) return r.finish(req, .{ .failed = error.NotHttp });
try r.engine.signOut(s.name);
_ = r.options.credentials.remove(r.gpa, s.name, s.url.?) catch |err| {
_ = r.options.credentials.store.remove(r.gpa, s.name, s.url.?) catch |err| {
debug_trace.logf("mcp", "MCP credential not removed server={s} err={s}", .{ s.name, @errorName(err) });
};
r.flags[i] = .{};
@@ -931,11 +948,11 @@ pub const Runtime = struct {
out.deinit();
}
if (try r.engine.saveCredential(s.name, &out.writer)) {
r.options.credentials.put(r.gpa, s.name, url, out.written()) catch |err| {
r.options.credentials.store.put(r.gpa, s.name, url, out.written()) catch |err| {
debug_trace.logf("mcp", "MCP credential not saved server={s} err={s}", .{ s.name, @errorName(err) });
};
} else {
_ = r.options.credentials.remove(r.gpa, s.name, url) catch |err| {
_ = r.options.credentials.store.remove(r.gpa, s.name, url) catch |err| {
debug_trace.logf("mcp", "MCP credential not removed server={s} err={s}", .{ s.name, @errorName(err) });
};
}
@@ -1157,17 +1174,32 @@ pub const Runtime = struct {
return out;
}
fn savedCredential(context: *anyopaque, alloc: Allocator, name: []const u8, url: []const u8) Allocator.Error!?[]u8 {
const r: *Runtime = @ptrCast(@alignCast(context));
const found = r.options.credentials.get(r.gpa, name, url) catch |err| switch (err) {
error.OutOfMemory => return error.OutOfMemory,
else => {
debug_trace.logf("mcp", "MCP credential not loaded server={s} err={s}", .{ name, @errorName(err) });
return null;
},
} orelse return null;
defer secret.zeroAndFree(r.gpa, found);
return try alloc.dupe(u8, found);
/// Before any request: picks the store when `start` left that to the
/// owner, reads it once, and gives each HTTP server its saved
/// credential, so no server starts without it. A store that can't be
/// read leaves every server signed out, as one without credentials.
fn restoreCredentials(r: *Runtime) void {
defer r.credentials_restored.set(r.io);
const store: credentials.Store = switch (r.options.credentials) {
.store => |s| s,
.detect => |home| .detect(r.gpa, home),
};
r.options.credentials = .{ .store = store };
var saved = store.snapshot(r.gpa) catch |err| {
debug_trace.logf("mcp", "MCP credentials not loaded err={s}", .{@errorName(err)});
return;
};
defer saved.deinit(r.gpa);
for (r.resolved.engine_server) |i| {
const s = r.resolved.servers[i];
const credential = saved.get(s.name, s.url orelse continue) orelse continue;
r.engine.restoreCredential(s.name, credential) catch |err| {
debug_trace.logf("mcp", "MCP credential not loaded server={s} err={s}", .{ s.name, @errorName(err) });
continue;
};
r.flags[i].signed_in = true;
}
r.publish() catch |err| debug_trace.logf("mcp", "MCP status not published err={s}", .{@errorName(err)});
}
/// Permission is fx's tool admission, before the call reaches the
@@ -1373,6 +1405,34 @@ test "a cancelled call ends, and stopping ends a call that never answers" {
try testing.expectEqual(@as(?Error, error.Stopped), got);
}
test "saved credentials load on the owner thread, and statuses wait for them" {
var f: Fixture = undefined;
try f.init();
defer f.deinit();
const store: credentials.Store = .{ .home = f.home, .backend = .file };
try store.put(testing.allocator, "web", "https://web.example.invalid/mcp", "{\"access\":\"w1\"}");
try store.put(testing.allocator, "moved", "https://old.example.invalid/mcp", "{\"access\":\"m1\"}");
const configs = [_]mcp_contract.McpServerConfig{
.{ .name = "web", .transport = .http, .url = "https://web.example.invalid/mcp" },
// A credential saved for another URL is not this server's.
.{ .name = "moved", .transport = .http, .url = "https://new.example.invalid/mcp" },
.{ .name = "none", .transport = .http, .url = "https://none.example.invalid/mcp" },
.{ .name = "fake", .command = f.server },
};
var r: Runtime = undefined;
try r.start(testing.allocator, testing.io, &configs, &f.inherited, f.options());
defer r.deinit();
var arena: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena.deinit();
const got = try r.statuses(arena.allocator());
try testing.expect(got[0].signed_in);
try testing.expect(!got[1].signed_in);
try testing.expect(!got[2].signed_in);
try testing.expect(!got[3].signed_in);
try testing.expectEqual(State.idle, got[0].state);
}
test "held servers say why, and unknown names and stdio logins are refused" {
var f: Fixture = undefined;
try f.init();
+77 -1
View File
@@ -598,11 +598,13 @@ fn serializeRecord(
}
fn parseRecord(alloc: Allocator, line: []const u8) !ParsedRecord {
// Strings may point into `line`, which outlives the parse; everything
// returned is copied before `parsed` is freed.
var parsed = try std.json.parseFromSlice(
std.json.Value,
alloc,
line,
.{ .allocate = .alloc_always },
.{ .allocate = .alloc_if_needed },
);
defer parsed.deinit();
if (parsed.value != .object or parsed.value.object.count() != 4) {
@@ -662,6 +664,9 @@ fn processCompleteLine(
entries: *std.ArrayList(LoadedPromptHistoryEntry),
) !void {
if (line.len + 1 > max_record_bytes) return;
// Most lines belong to other workspaces; skip them before the full parse,
// which costs far more and runs before the first frame.
if (!mayBelongToWorkspace(line, workspace_root)) return;
var record = parseRecord(alloc, line) catch |err| {
debug_trace.logf(
"prompt_history",
@@ -682,6 +687,30 @@ fn processCompleteLine(
};
}
/// False only when `line` can't be a record for `workspace_root`: every
/// `"workspace_root":"` in it has a value with no escape that differs from
/// it, and a value with no escape decodes to its own bytes. That key can't
/// occur inside a string, where each quote is escaped. Anything else, such
/// as an escaped value or a key written with spaces, goes to the full parse,
/// which decides the line as before.
fn mayBelongToWorkspace(line: []const u8, workspace_root: []const u8) bool {
const key = "\"workspace_root\":\"";
var found = false;
var i: usize = 0;
// Hopping between quotes is a vector scan; a substring search costs
// more than the lines it skips.
while (std.mem.findScalarPos(u8, line, i, '"')) |quote| {
i = quote + 1;
if (!std.mem.startsWith(u8, line[quote..], key)) continue;
found = true;
const start = quote + key.len;
const end = std.mem.findAnyPos(u8, line, start, "\"\\") orelse return true;
if (line[end] == '\\' or std.mem.eql(u8, line[start..end], workspace_root)) return true;
i = end + 1;
}
return !found;
}
fn repairIncompleteTail(file: std.Io.File) !void {
const length = try file.length(io_mod.getIo());
if (length == 0) return;
@@ -951,6 +980,53 @@ test "reverse load filters workspace bounds results and preserves chronology" {
try std.testing.expectEqualStrings("other", other[0].text);
}
test "the workspace prefilter skips only lines that can't be the workspace's" {
const root = "/tmp/workspace-a";
try std.testing.expect(mayBelongToWorkspace("{\"schema_version\":1,\"timestamp_ms\":1,\"workspace_root\":\"/tmp/workspace-a\",\"text\":\"x\"}", root));
try std.testing.expect(!mayBelongToWorkspace("{\"schema_version\":1,\"timestamp_ms\":1,\"workspace_root\":\"/tmp/workspace-b\",\"text\":\"x\"}", root));
// A prefix of the root, or the root inside a longer path, is another workspace.
try std.testing.expect(!mayBelongToWorkspace("{\"workspace_root\":\"/tmp/workspace\",\"text\":\"x\"}", root));
try std.testing.expect(!mayBelongToWorkspace("{\"workspace_root\":\"/tmp/workspace-a/sub\",\"text\":\"x\"}", root));
// Text that quotes a matching key doesn't count: its quotes are escaped.
try std.testing.expect(!mayBelongToWorkspace("{\"workspace_root\":\"/tmp/workspace-b\",\"text\":\"\\\"workspace_root\\\":\\\"/tmp/workspace-a\\\"\"}", root));
// Otherwise the full parse decides: an escaped value, a key with spaces,
// a second key, a value with no end, or a line with no key at all.
try std.testing.expect(mayBelongToWorkspace("{\"workspace_root\":\"\\/tmp\\/workspace-a\"}", root));
try std.testing.expect(mayBelongToWorkspace("{\"workspace_root\" : \"/tmp/workspace-b\"}", root));
try std.testing.expect(mayBelongToWorkspace("{\"workspace_root\":\"/tmp/workspace-b\",\"workspace_root\":\"/tmp/workspace-a\"}", root));
try std.testing.expect(mayBelongToWorkspace("{\"workspace_root\":\"/tmp/workspace-a", root));
try std.testing.expect(mayBelongToWorkspace("{malformed}", root));
}
test "reverse load keeps a record whose workspace is written with escapes or spaces" {
const alloc = std.testing.allocator;
var tmp = std.testing.tmpDir(.{});
defer tmp.cleanup();
const home = try io_mod.dirRealpathAlloc(alloc, tmp.dir, ".");
defer alloc.free(home);
const plain = try fixtureLine(alloc, 1, "/tmp/workspace-a", "plain");
defer alloc.free(plain);
const escaped = try fixtureLine(alloc, 2, "\\/tmp\\/workspace-a", "escaped");
defer alloc.free(escaped);
const other = try fixtureLine(alloc, 3, "/tmp/workspace-b", "other");
defer alloc.free(other);
const bytes = try std.mem.concat(alloc, u8, &.{
plain,
escaped,
"{\"schema_version\":1, \"timestamp_ms\":4, \"workspace_root\" : \"/tmp/workspace-a\", \"text\":\"spaced\"}\n",
other,
});
defer alloc.free(bytes);
try writeFixture(home, bytes);
var store = try Store.initFromHome(alloc, home);
defer store.deinit(alloc);
const entries = try store.loadRecentForWorkspace(alloc, "/tmp/workspace-a", 100);
defer freeLoadedEntries(alloc, entries);
try expectTexts(entries, &.{ "plain", "escaped", "spaced" });
}
test "reverse load scans beyond one mebibyte of newer interleaved workspace records" {
const alloc = std.testing.allocator;
var tmp = std.testing.tmpDir(.{});
+523 -55
View File
@@ -561,6 +561,196 @@ pub const ImageBlobReader = struct {
}
};
/// An image's name in a libfx host's store: the lowercase hex SHA-256 of its
/// bytes, as `ImageAttachment.snapshot_sha256` holds it for an inline image.
const image_name_len = 2 * Sha256.digest_length;
pub const ImageName = [image_name_len]u8;
/// `text` as an image name, or null when it is not 64 lowercase hex digits.
pub fn imageName(text: []const u8) ?ImageName {
if (text.len != image_name_len) return null;
for (text) |char| switch (char) {
'0'...'9', 'a'...'f' => {},
else => return null,
};
return text[0..image_name_len].*;
}
pub fn imageNameOf(bytes: []const u8) ImageName {
var digest: [Sha256.digest_length]u8 = undefined;
Sha256.hash(bytes, &digest, .{});
return std.fmt.bytesToHex(digest, .lower);
}
/// The images a libfx host's store holds, by name.
pub const StoredImages = struct {
names: std.array_hash_map.Auto(ImageName, void) = .empty,
pub fn deinit(self: *StoredImages, alloc: Allocator) void {
self.names.deinit(alloc);
self.* = undefined;
}
pub fn add(self: *StoredImages, alloc: Allocator, name: ImageName) Allocator.Error!void {
try self.names.put(alloc, name, {});
}
/// Whether `image` has bytes the store holds under its name.
pub fn holds(self: *const StoredImages, image: types.ImageAttachment) bool {
if (image.inline_data == null) return false;
const name = imageName(image.snapshot_sha256 orelse return false) orelse return false;
return self.names.contains(name);
}
};
/// Whether writing `user` names an image instead of holding its bytes, which
/// only the naming format can: one `stored` holds, or one the session
/// dropped, which keeps only its name with or without a store.
pub fn namesImages(user: session.UserTurn, stored: ?*const StoredImages) bool {
for (user.images) |image| {
if (image.droppedFromSession()) return true;
if (stored) |names| if (names.holds(image)) return true;
}
return false;
}
pub fn turnNamesImages(turn: session.HistoryTurn, stored: ?*const StoredImages) bool {
return switch (turn) {
.assistant => |entry| namesImages(entry.user, stored) or executionNamesImages(entry.execution, .{}, stored),
.interrupted => |entry| namesImages(entry.user, stored) or executionNamesImages(entry.execution, .{}, stored),
.compacted_summary => false,
};
}
/// Whether writing the items of `execution` after `from` names a tool image:
/// one the host stored, when `stored` is given, or one the session dropped.
pub fn executionNamesImages(execution: session.ExecutionMemory, from: ExecutionCounts, stored: ?*const StoredImages) bool {
for (execution.tool_steps[from.tool_steps..]) |step| for (step.tool_results) |result| {
if (toolImagesNameImages(result.tool_images, stored)) return true;
};
return false;
}
/// Whether writing `images` names one: one the host stored, when `stored` is
/// given, or one the session dropped.
pub fn toolImagesNameImages(images: []const types.ToolImage, stored: ?*const StoredImages) bool {
for (images) |image| {
if (image.droppedFromSession() or (stored != null and image.name != null)) return true;
}
return false;
}
/// Whether writing `checkpoint` names an image; with `delta`, only the
/// execution items after it, which is all a delta progress writes.
pub fn checkpointNamesImages(checkpoint: RecoveryCheckpoint, delta: ?ExecutionCounts, stored: ?*const StoredImages) bool {
if (delta == null and namesImages(checkpoint.user, stored)) return true;
return executionNamesImages(checkpoint.execution, delta orelse .{}, stored);
}
/// The images a libfx session with an image store keeps: its newest, at most
/// `max_images` of them holding at most `max_bytes`. An open stages these from
/// the host's store, with the open turn's, which the window leaves out. Older
/// images leave the session and keep only their names, so a session never
/// grows past what it can reopen.
pub const ImageWindow = struct {
max_images: usize = 4096,
max_bytes: usize = 512 * 1024 * 1024,
/// How many of the oldest images leave the window. `sizes` holds each
/// image's bytes, oldest first, or null for one that already left, which
/// keeps it and every older one out. Once an image does not fit, it and
/// every older one leave.
pub fn oldestOutside(self: ImageWindow, sizes: []const ?usize) usize {
var kept: usize = 0;
var bytes: usize = 0;
var start = sizes.len;
while (start > 0) : (start -= 1) {
const size = sizes[start - 1] orelse break;
if (kept == self.max_images or size > self.max_bytes - bytes) break;
kept += 1;
bytes += size;
}
return start;
}
};
/// Writes images the host stored as `inline_ref` names. Every other image
/// keeps its bytes: in `blobs`, a libfx kernel checkpoint's blob section,
/// when given, else inline.
const ImageNames = struct {
stored: *const StoredImages,
blobs: ?ImageBlobs = null,
};
/// Images a libfx host staged from its store for an open, by name. Owns the
/// bytes.
pub const StagedImages = struct {
images: std.array_hash_map.Auto(ImageName, []u8) = .empty,
bytes: usize = 0,
pub fn deinit(self: *StagedImages, alloc: Allocator) void {
for (self.images.values()) |bytes| alloc.free(bytes);
self.images.deinit(alloc);
self.* = undefined;
}
/// Stages `bytes`, `alloc`-owned, as `name`. A staged image must be
/// exactly the bytes its name is the hash of. On success it owns `bytes`,
/// freeing them when it already holds the name; on error the caller
/// still owns them.
pub fn add(self: *StagedImages, alloc: Allocator, name: ImageName, bytes: []u8) error{ OutOfMemory, ImageNameMismatch }!void {
if (!std.mem.eql(u8, &imageNameOf(bytes), &name)) return error.ImageNameMismatch;
if (self.images.contains(name)) {
alloc.free(bytes);
return;
}
try self.images.put(alloc, name, bytes);
self.bytes += bytes.len;
}
pub fn clear(self: *StagedImages, alloc: Allocator) void {
for (self.images.values()) |bytes| alloc.free(bytes);
self.images.clearRetainingCapacity();
self.bytes = 0;
}
};
/// Resolves `inline_ref` names to staged bytes while a libfx journal or
/// checkpoint is read. A name it cannot resolve is recorded in `missing` with
/// the size its record gives, and parses as an image without bytes, so one
/// read names every image an open needs; a caller that needs any of them
/// discards what it read.
pub const ImageResolver = struct {
alloc: Allocator,
staged: *const StagedImages,
missing: std.array_hash_map.Auto(ImageName, usize) = .empty,
/// The names it resolved, which the store therefore holds.
resolved: std.array_hash_map.Auto(ImageName, void) = .empty,
pub fn deinit(self: *ImageResolver) void {
self.missing.deinit(self.alloc);
self.resolved.deinit(self.alloc);
self.* = undefined;
}
/// A copy of the bytes named `name`, owned by `alloc`, or null when none
/// were staged. `size` is the image's size as its record gives it.
fn resolve(self: *ImageResolver, alloc: Allocator, name: ImageName, size: usize) Allocator.Error!?[]u8 {
const bytes = self.staged.images.get(name) orelse {
try self.missing.put(self.alloc, name, size);
return null;
};
try self.resolved.put(self.alloc, name, {});
return try alloc.dupe(u8, bytes);
}
};
/// Where a parse finds image bytes kept outside the image object.
pub const ImageSources = struct {
blobs: ?*ImageBlobReader = null,
names: ?*ImageResolver = null,
};
pub fn writeHistoryTurn(writer: *std.Io.Writer, turn: session.HistoryTurn) !void {
return writeHistoryTurnTo(writer, turn, {});
}
@@ -575,8 +765,15 @@ pub fn writeHistoryTurnWithImageBlobs(
return writeHistoryTurnTo(writer, turn, image_blobs);
}
/// `image_blobs` is `void` to embed inline image bytes or `ImageBlobs` to
/// collect them. The choice is comptime so durable writers keep their error set.
/// Like `writeHistoryTurn`, but names the images `names.stored` holds
/// instead of writing their bytes.
pub fn writeHistoryTurnNamingImages(writer: *std.Io.Writer, turn: session.HistoryTurn, names: ImageNames) !void {
return writeHistoryTurnTo(writer, turn, names);
}
/// `image_blobs` is `void` to embed inline image bytes, `ImageBlobs` to
/// collect them, or `ImageNames` to name stored ones. The choice is comptime
/// so durable writers keep their error set.
fn writeHistoryTurnTo(writer: *std.Io.Writer, turn: session.HistoryTurn, image_blobs: anytype) !void {
switch (turn) {
.compacted_summary => |entry| {
@@ -596,7 +793,7 @@ fn writeHistoryTurnTo(writer: *std.Io.Writer, turn: session.HistoryTurn, image_b
try writer.writeAll(",\"provider_replay\":");
try std.json.Stringify.value(entry.provider_replay, .{}, writer);
try writer.writeAll(",\"execution\":");
try writeExecutionMemoryJson(writer, entry.execution, .durable);
try writeExecutionMemoryJsonFrom(writer, entry.execution, .durable, .{}, image_blobs);
try writer.writeByte('}');
},
.interrupted => |entry| {
@@ -623,7 +820,7 @@ fn writeHistoryTurnTo(writer: *std.Io.Writer, turn: session.HistoryTurn, image_b
}
if (hasDurableExecutionMemory(entry.execution)) {
try writer.writeAll(",\"execution\":");
try writeExecutionMemoryJson(writer, entry.execution, .durable);
try writeExecutionMemoryJsonFrom(writer, entry.execution, .durable, .{}, image_blobs);
}
if (entry.cancelled_command) |presentation| {
try writer.writeAll(",\"cancelled_command\":");
@@ -667,6 +864,16 @@ pub fn parseHistoryTurnWithImageBlobs(
alloc: Allocator,
value: std.json.Value,
image_blobs: ?*ImageBlobReader,
) !session.HistoryTurn {
return parseHistoryTurnWithImages(alloc, value, .{ .blobs = image_blobs });
}
/// Like `parseHistoryTurn`, but reads the image bytes a libfx journal or
/// checkpoint keeps outside each image from `sources`.
pub fn parseHistoryTurnWithImages(
alloc: Allocator,
value: std.json.Value,
sources: ImageSources,
) !session.HistoryTurn {
const kind = try requireString(try requireObject(value), "kind");
if (std.mem.eql(u8, kind, "compacted_summary")) {
@@ -724,11 +931,11 @@ pub fn parseHistoryTurnWithImageBlobs(
if (std.mem.eql(u8, kind, "assistant")) {
const shape = try exactVariantObject(value, &.{ "kind", "user", "assistant", "execution" }, &.{ "kind", "user", "assistant", "execution", "provider_replay" });
const object = shape.object;
const user = try parseUserTurn(alloc, object.get("user") orelse return error.InvalidSessionFormat, image_blobs);
const user = try parseUserTurn(alloc, object.get("user") orelse return error.InvalidSessionFormat, sources);
errdefer session.freeUserTurn(alloc, user);
const assistant = try parseRequiredDurableBytes(alloc, object, "assistant");
errdefer mem_utils.free(alloc, assistant);
const execution = try parseExecutionMemory(alloc, object.get("execution") orelse return error.InvalidSessionFormat);
const execution = try parseExecutionMemoryWith(alloc, object.get("execution") orelse return error.InvalidSessionFormat, sources);
errdefer session.freeExecutionMemory(alloc, execution);
const replay = try parseProviderReplay(alloc, object.get("provider_replay") orelse .null);
return .{ .assistant = .{
@@ -745,7 +952,7 @@ pub fn parseHistoryTurnWithImageBlobs(
&.{ "kind", "user", "log_path", "expect_url", "url", "background_record_id", "assistant", "execution" },
);
const object = shape.object;
const user = try parseUserTurn(alloc, object.get("user") orelse return error.InvalidSessionFormat, image_blobs);
const user = try parseUserTurn(alloc, object.get("user") orelse return error.InvalidSessionFormat, sources);
errdefer session.freeUserTurn(alloc, user);
const legacy_assistant = if (shape.extended)
try parseOptionalDurableBytes(alloc, object.get("assistant") orelse return error.InvalidSessionFormat)
@@ -753,7 +960,7 @@ pub fn parseHistoryTurnWithImageBlobs(
null;
defer if (legacy_assistant) |owned| mem_utils.free(alloc, owned);
const execution = if (shape.extended)
try parseExecutionMemory(alloc, object.get("execution") orelse return error.InvalidSessionFormat)
try parseExecutionMemoryWith(alloc, object.get("execution") orelse return error.InvalidSessionFormat, sources)
else
session.ExecutionMemory{};
errdefer session.freeExecutionMemory(alloc, execution);
@@ -788,7 +995,7 @@ pub fn parseHistoryTurnWithImageBlobs(
has_terminal_reason,
has_cancellation_origin,
);
const user = try parseUserTurn(alloc, object.get("user") orelse return error.InvalidSessionFormat, image_blobs);
const user = try parseUserTurn(alloc, object.get("user") orelse return error.InvalidSessionFormat, sources);
errdefer session.freeUserTurn(alloc, user);
const assistant = try parseOptionalDurableBytes(alloc, object.get("assistant") orelse return error.InvalidSessionFormat);
errdefer if (assistant) |owned| mem_utils.free(alloc, owned);
@@ -809,7 +1016,7 @@ pub fn parseHistoryTurnWithImageBlobs(
else
.turn;
const execution = if (has_execution)
try parseExecutionMemory(alloc, object.get("execution") orelse return error.InvalidSessionFormat)
try parseExecutionMemoryWith(alloc, object.get("execution") orelse return error.InvalidSessionFormat, sources)
else
session.ExecutionMemory{};
errdefer session.freeExecutionMemory(alloc, execution);
@@ -1068,7 +1275,13 @@ fn parsePermissionState(
}
pub fn writeRecoveryCheckpoint(writer: *std.Io.Writer, checkpoint: RecoveryCheckpoint) !void {
return writeRecoveryCheckpointFrom(writer, checkpoint, null);
return writeRecoveryCheckpointFrom(writer, checkpoint, null, {});
}
/// Like `writeRecoveryCheckpoint`, but names the images `names.stored` holds
/// instead of writing their bytes.
pub fn writeRecoveryCheckpointNamingImages(writer: *std.Io.Writer, checkpoint: RecoveryCheckpoint, names: ImageNames) !void {
return writeRecoveryCheckpointFrom(writer, checkpoint, null, names);
}
/// How many items of each list in an open turn's execution an earlier
@@ -1083,10 +1296,16 @@ pub const ExecutionCounts = struct {
/// execution items after `from`: what a progress adds to the one before it,
/// which holds the rest.
pub fn writeRecoveryCheckpointDelta(writer: *std.Io.Writer, checkpoint: RecoveryCheckpoint, from: ExecutionCounts) !void {
return writeRecoveryCheckpointFrom(writer, checkpoint, from);
return writeRecoveryCheckpointFrom(writer, checkpoint, from, {});
}
fn writeRecoveryCheckpointFrom(writer: *std.Io.Writer, checkpoint: RecoveryCheckpoint, delta: ?ExecutionCounts) !void {
/// Like `writeRecoveryCheckpointDelta`, but names the tool images the host
/// stored instead of writing their bytes.
pub fn writeRecoveryCheckpointDeltaNamingImages(writer: *std.Io.Writer, checkpoint: RecoveryCheckpoint, from: ExecutionCounts, names: ImageNames) !void {
return writeRecoveryCheckpointFrom(writer, checkpoint, from, names);
}
fn writeRecoveryCheckpointFrom(writer: *std.Io.Writer, checkpoint: RecoveryCheckpoint, delta: ?ExecutionCounts, image_mode: anytype) !void {
if (checkpoint.disposition == .history_only) return error.InvalidDurableField;
if (delta) |from| {
const execution = checkpoint.execution;
@@ -1100,12 +1319,12 @@ fn writeRecoveryCheckpointFrom(writer: *std.Io.Writer, checkpoint: RecoveryCheck
});
if (delta == null) {
try writer.writeAll(",\"user\":");
try writeUserTurn(writer, checkpoint.user, {});
try writeUserTurn(writer, checkpoint.user, image_mode);
}
try writer.writeAll(",\"assistant_source\":");
try writeDurableBytes(writer, checkpoint.assistant_source);
try writer.writeAll(",\"execution\":");
try writeExecutionMemoryJsonFrom(writer, checkpoint.execution, .durable, delta orelse .{});
try writeExecutionMemoryJsonFrom(writer, checkpoint.execution, .durable, delta orelse .{}, image_mode);
try writer.writeAll(",\"cause\":");
try writeJsonString(writer, @tagName(checkpoint.cause));
try writer.writeAll(",\"action\":");
@@ -1346,6 +1565,12 @@ fn decodeStateImpl(alloc: Allocator, source: *std.Io.Reader, limits: DecodeLimit
}
pub fn parseRecoveryCheckpoint(alloc: Allocator, value: std.json.Value) !RecoveryCheckpoint {
return parseRecoveryCheckpointWithImages(alloc, value, .{});
}
/// Like `parseRecoveryCheckpoint`, but reads the image bytes a libfx journal
/// keeps outside each image from `sources`.
pub fn parseRecoveryCheckpointWithImages(alloc: Allocator, value: std.json.Value, sources: ImageSources) !RecoveryCheckpoint {
const raw_object = try requireObject(value);
const version = try requireU64(raw_object, "version");
const legacy_route = raw_object.contains("route_identity");
@@ -1384,11 +1609,11 @@ pub fn parseRecoveryCheckpoint(alloc: Allocator, value: std.json.Value) !Recover
}),
else => return error.InvalidDurableField,
};
const user = try parseUserTurn(alloc, object.get("user") orelse return error.InvalidSessionFormat, null);
const user = try parseUserTurn(alloc, object.get("user") orelse return error.InvalidSessionFormat, sources);
errdefer session.freeUserTurn(alloc, user);
const assistant_source = try parseDurableBytes(alloc, object.get("assistant_source") orelse return error.InvalidSessionFormat);
errdefer alloc.free(assistant_source);
const execution = try parseExecutionMemory(alloc, object.get("execution") orelse return error.InvalidSessionFormat);
const execution = try parseExecutionMemoryWith(alloc, object.get("execution") orelse return error.InvalidSessionFormat, sources);
errdefer session.freeExecutionMemory(alloc, execution);
const authority = if (legacy_route) TurnAuthority{
.provider = .gateway,
@@ -1544,10 +1769,15 @@ fn parseTurnAuthority(alloc: Allocator, value: std.json.Value) !TurnAuthority {
}
fn writeUserTurn(writer: *std.Io.Writer, user: session.UserTurn, image_blobs: anytype) !void {
const collect_blobs = comptime switch (@TypeOf(image_blobs)) {
void => false,
ImageBlobs => true,
else => @compileError("image_blobs must be void or ImageBlobs"),
const Mode = @TypeOf(image_blobs);
comptime switch (Mode) {
void, ImageBlobs, ImageNames => {},
else => @compileError("image_blobs must be void, ImageBlobs or ImageNames"),
};
const blobs: ?ImageBlobs = switch (Mode) {
ImageBlobs => image_blobs,
ImageNames => image_blobs.blobs,
else => null,
};
try writer.writeAll("{\"text\":");
try writeDurableBytes(writer, user.text);
@@ -1563,15 +1793,24 @@ fn writeUserTurn(writer: *std.Io.Writer, user: session.UserTurn, image_blobs: an
try writer.writeAll(",\"snapshot_sha256\":");
try writeOptionalDurableBytes(writer, image.snapshot_sha256);
// Inline image bytes appear only for sessions without a filesystem
// snapshot backend, so durable sessions keep their exact format.
// snapshot backend, so durable sessions keep their exact format. An
// image the host stored is named instead; its name is the digest,
// and its size lets an open choose the images it keeps unread.
if (image.inline_data) |inline_data| {
if (collect_blobs) {
try writer.print(",\"inline_blob\":{d}", .{image_blobs.items.items.len});
try image_blobs.items.append(image_blobs.alloc, inline_data);
if (Mode == ImageNames and image_blobs.stored.holds(image)) {
try writer.writeAll(",\"inline_ref\":");
try writeJsonString(writer, image.snapshot_sha256.?);
try writer.print(",\"inline_size\":{d}", .{inline_data.len});
} else if (blobs) |collected| {
try writer.print(",\"inline_blob\":{d}", .{collected.items.items.len});
try collected.items.append(collected.alloc, inline_data);
} else {
try writer.writeAll(",\"inline_data\":");
try writeDurableBytes(writer, inline_data);
}
} else if (image.droppedFromSession()) {
// An image the session dropped keeps only its digest.
try writer.writeAll(",\"dropped\":true");
}
if (image.source_ref) |source_ref| {
if (!image_data.validSourceRef(source_ref)) return error.InvalidSessionFormat;
@@ -1608,16 +1847,19 @@ pub noinline fn writeExecutionMemoryJson(
execution: session.ExecutionMemory,
format: ExecutionMemoryJsonFormat,
) !void {
return writeExecutionMemoryJsonFrom(writer, execution, format, .{});
return writeExecutionMemoryJsonFrom(writer, execution, format, .{}, {});
}
/// `writeExecutionMemoryJson` with each list starting after `from` items.
/// The caller checks that every list holds at least that many.
/// The caller checks that every list holds at least that many. `image_mode`
/// is as for `writeHistoryTurnTo`; only `ImageNames` changes how tool images
/// are written.
fn writeExecutionMemoryJsonFrom(
writer: *std.Io.Writer,
execution: session.ExecutionMemory,
format: ExecutionMemoryJsonFormat,
from: ExecutionCounts,
image_mode: anytype,
) !void {
std.debug.assert(from.tool_steps <= execution.tool_steps.len and from.files <= execution.files.len and from.steering <= execution.steering.len);
try writer.writeAll(if (format == .durable)
@@ -1626,7 +1868,7 @@ fn writeExecutionMemoryJsonFrom(
"{\"schema_version\":3,\"tool_steps\":[");
for (execution.tool_steps[from.tool_steps..], 0..) |step, i| {
if (i > 0) try writer.writeByte(',');
try writeExecutionToolStep(writer, step, format);
try writeExecutionToolStep(writer, step, format, image_mode);
}
try writer.writeAll("],\"files\":[");
for (execution.files[from.files..], 0..) |file, i| {
@@ -1651,7 +1893,7 @@ fn writeExecutionMemoryJsonFrom(
try writer.writeByte('}');
}
fn writeExecutionToolStep(writer: *std.Io.Writer, step: session.ToolExecutionStep, format: ExecutionMemoryJsonFormat) !void {
fn writeExecutionToolStep(writer: *std.Io.Writer, step: session.ToolExecutionStep, format: ExecutionMemoryJsonFormat, image_mode: anytype) !void {
try writer.writeAll("{\"assistant\":");
try writeOptionalExecutionBytes(writer, step.assistant, format);
if (format == .durable) {
@@ -1666,7 +1908,7 @@ fn writeExecutionToolStep(writer: *std.Io.Writer, step: session.ToolExecutionSte
try writer.writeAll("],\"tool_results\":[");
for (step.tool_results, 0..) |result, result_index| {
if (result_index > 0) try writer.writeByte(',');
try writeExecutionToolResult(writer, result, format);
try writeExecutionToolResult(writer, result, format, image_mode);
}
try writer.writeAll("]}");
}
@@ -1740,7 +1982,7 @@ pub fn executionHashes(alloc: Allocator, checkpoint: RecoveryCheckpoint) !Execut
for (results) |*result| result.created_at_ms = 0;
var stable = step;
stable.tool_results = results;
try writeExecutionToolStep(&out.writer, stable, .durable);
try writeExecutionToolStep(&out.writer, stable, .durable, {});
hash.* = std.hash.Wyhash.hash(0, out.written());
}
const files = try alloc.alloc(u64, execution.files.len);
@@ -1824,6 +2066,7 @@ fn writeExecutionToolResult(
writer: *std.Io.Writer,
result: session.PersistedToolResult,
format: ExecutionMemoryJsonFormat,
image_mode: anytype,
) !void {
if (format == .durable and
result.review_feedback and
@@ -1891,7 +2134,7 @@ fn writeExecutionToolResult(
try writeDurableBytes(writer, handle);
} else if (result.tool_images.len > 0) {
try writer.writeAll(",\"tool_images\":");
writePersistedToolImages(writer, result.tool_images) catch |err| switch (err) {
writeToolImagesTo(writer, result.tool_images, @TypeOf(image_mode) == ImageNames) catch |err| switch (err) {
error.InvalidSourceRef, error.InvalidImage => return error.InvalidSessionFormat,
else => |write_err| return write_err,
};
@@ -2095,7 +2338,7 @@ fn writeExecutionFileEvidence(
try writer.writeByte('}');
}
fn parseUserTurn(alloc: Allocator, value: std.json.Value, image_blobs: ?*ImageBlobReader) !session.UserTurn {
fn parseUserTurn(alloc: Allocator, value: std.json.Value, sources: ImageSources) !session.UserTurn {
const source = try requireObject(value);
const object = if (source.count() == 2)
try exactObject(value, &.{ "text", "images" })
@@ -2137,7 +2380,8 @@ fn parseUserTurn(alloc: Allocator, value: std.json.Value, image_blobs: ?*ImageBl
image.get("snapshot_sha256") orelse .null,
);
errdefer if (snapshot_sha256) |sha256_bytes| mem_utils.free(alloc, sha256_bytes);
const inline_data = try parseInlineImageBytes(alloc, image, image_blobs);
const inline_image = try parseInlineImageBytes(alloc, image, sources);
const inline_data = inline_image.bytes;
errdefer if (inline_data) |inline_bytes| mem_utils.free(alloc, inline_bytes);
const source_ref = if (image.get("source_ref")) |_| source: {
const source_bytes = try requireString(image, "source_ref");
@@ -2145,7 +2389,22 @@ fn parseUserTurn(alloc: Allocator, value: std.json.Value, image_blobs: ?*ImageBl
break :source try alloc.dupe(u8, source_bytes);
} else null;
errdefer if (source_ref) |source_bytes| alloc.free(source_bytes);
if (inline_data) |inline_bytes| {
const dropped = if (image.get("dropped")) |dropped_value| switch (dropped_value) {
.bool => |flag| if (flag) true else return error.InvalidSessionFormat,
else => return error.InvalidSessionFormat,
} else false;
if (dropped) {
// An image the session dropped keeps only the digest naming it.
if (inline_image.name != null or inline_data != null or snapshot_path != null) return error.InvalidSessionFormat;
_ = imageName(snapshot_sha256 orelse return error.InvalidSessionFormat) orelse return error.InvalidSessionFormat;
} else if (inline_image.name) |name| {
// A named image is the host's stored copy of exactly the bytes
// its digest names; with none staged, `sources` records it.
if (snapshot_path != null) return error.InvalidSessionFormat;
const digest = snapshot_sha256 orelse return error.InvalidSessionFormat;
if (!std.mem.eql(u8, digest, &name)) return error.InvalidSessionFormat;
if (inline_data) |inline_bytes| try validateInlineImageBytes(inline_bytes, media_type);
} else if (inline_data) |inline_bytes| {
// Inline images carry their own bytes: no snapshot file, but the
// digest stays so request-time verification is unchanged.
if (snapshot_path != null or snapshot_sha256 == null) return error.InvalidSessionFormat;
@@ -2165,19 +2424,43 @@ fn parseUserTurn(alloc: Allocator, value: std.json.Value, image_blobs: ?*ImageBl
return .{ .text = text, .images = images, .work_id = work_id };
}
/// Reads an inline image's bytes from `inline_data` or, inside a libfx kernel
/// checkpoint, from the raw blob named by `inline_blob`. Caller owns the result.
/// An image's bytes as a parse found them, and its name when the object
/// named it. A named image with no bytes was not staged.
const InlineImage = struct {
bytes: ?[]u8 = null,
name: ?ImageName = null,
};
/// Reads an inline image's bytes from `inline_data`, from the raw blob of a
/// libfx kernel checkpoint named by `inline_blob`, or from the host's staged
/// copy named by `inline_ref`. Caller owns the bytes.
fn parseInlineImageBytes(
alloc: Allocator,
image: std.json.ObjectMap,
image_blobs: ?*ImageBlobReader,
) !?[]u8 {
sources: ImageSources,
) !InlineImage {
if (image.get("inline_ref")) |ref_value| {
const resolver = sources.names orelse return error.InvalidSessionFormat;
if (image.get("inline_data") != null or image.get("inline_blob") != null or ref_value != .string) {
return error.InvalidSessionFormat;
}
const name = imageName(ref_value.string) orelse return error.InvalidSessionFormat;
const size = try requireUsize(image, "inline_size");
if (size == 0 or size > image_attachments.max_image_bytes) return error.InvalidSessionFormat;
const bytes = try resolver.resolve(alloc, name, size);
if (bytes) |staged| if (staged.len != size) {
alloc.free(staged);
return error.InvalidSessionFormat;
};
return .{ .bytes = bytes, .name = name };
}
if (image.get("inline_size") != null) return error.InvalidSessionFormat;
const blob_value = image.get("inline_blob") orelse
return parseOptionalDurableBytes(alloc, image.get("inline_data") orelse .null);
const blobs = image_blobs orelse return error.InvalidSessionFormat;
return .{ .bytes = try parseOptionalDurableBytes(alloc, image.get("inline_data") orelse .null) };
const blobs = sources.blobs orelse return error.InvalidSessionFormat;
if (image.get("inline_data") != null or blob_value != .integer) return error.InvalidSessionFormat;
const index = std.math.cast(usize, blob_value.integer) orelse return error.InvalidSessionFormat;
return try alloc.dupe(u8, try blobs.take(index));
return .{ .bytes = try alloc.dupe(u8, try blobs.take(index)) };
}
fn imageAttachmentObject(value: std.json.Value) !std.json.ObjectMap {
@@ -2197,6 +2480,9 @@ fn imageAttachmentObject(value: std.json.Value) !std.json.ObjectMap {
std.mem.eql(u8, entry.key_ptr.*, "snapshot_sha256") or
std.mem.eql(u8, entry.key_ptr.*, "inline_data") or
std.mem.eql(u8, entry.key_ptr.*, "inline_blob") or
std.mem.eql(u8, entry.key_ptr.*, "inline_ref") or
std.mem.eql(u8, entry.key_ptr.*, "inline_size") or
std.mem.eql(u8, entry.key_ptr.*, "dropped") or
std.mem.eql(u8, entry.key_ptr.*, "source_ref"))
{
continue;
@@ -2222,6 +2508,11 @@ fn validateInlineImageBytes(bytes: []const u8, media_type: []const u8) !void {
}
fn parseExecutionMemory(alloc: Allocator, value: std.json.Value) !session.ExecutionMemory {
return parseExecutionMemoryWith(alloc, value, .{});
}
/// `parseExecutionMemory`, reading named tool images from `sources`.
fn parseExecutionMemoryWith(alloc: Allocator, value: std.json.Value, sources: ImageSources) !session.ExecutionMemory {
const source = try requireObject(value);
const schema_version = try requireU64(source, "schema_version");
const object = switch (schema_version) {
@@ -2235,6 +2526,7 @@ fn parseExecutionMemory(alloc: Allocator, value: std.json.Value) !session.Execut
alloc,
object.get("tool_steps") orelse return error.InvalidSessionFormat,
schema_version,
sources,
);
errdefer session.freeExecutionMemory(alloc, .{ .tool_steps = tool_steps });
const files = try parseFiles(alloc, object.get("files") orelse return error.InvalidSessionFormat);
@@ -2352,6 +2644,7 @@ fn parseToolSteps(
alloc: Allocator,
value: std.json.Value,
schema_version: u64,
sources: ImageSources,
) ![]session.ToolExecutionStep {
if (value != .array) return error.InvalidSessionFormat;
if (value.array.items.len == 0) return &.{};
@@ -2377,6 +2670,7 @@ fn parseToolSteps(
alloc,
object.get("tool_results") orelse return error.InvalidSessionFormat,
schema_version,
sources,
);
errdefer session.freePersistedToolResults(alloc, tool_results);
try session.repairPersistedToolArguments(alloc, tool_calls, tool_results, .schema_v3);
@@ -2480,6 +2774,7 @@ fn parseToolResults(
alloc: Allocator,
value: std.json.Value,
schema_version: u64,
sources: ImageSources,
) ![]session.PersistedToolResult {
if (value != .array) return error.InvalidSessionFormat;
if (value.array.items.len == 0) return &.{};
@@ -2503,7 +2798,7 @@ fn parseToolResults(
}
};
for (value.array.items, 0..) |result_value, i| {
results[i] = try parseToolResult(alloc, result_value, schema_version);
results[i] = try parseToolResultWith(alloc, result_value, schema_version, sources);
parsed_count += 1;
}
return results;
@@ -2513,6 +2808,15 @@ fn parseToolResult(
alloc: Allocator,
value: std.json.Value,
schema_version: u64,
) !session.PersistedToolResult {
return parseToolResultWith(alloc, value, schema_version, .{});
}
fn parseToolResultWith(
alloc: Allocator,
value: std.json.Value,
schema_version: u64,
sources: ImageSources,
) !session.PersistedToolResult {
const v1_keys = &.{
"tool_call_id",
@@ -2630,7 +2934,7 @@ fn parseToolResult(
result_shape.extended,
);
errdefer presentations.deinit(alloc);
var images = try parseToolResultImages(alloc, object, &text.output);
var images = try parseToolResultImages(alloc, object, &text.output, sources);
errdefer images.deinit(alloc);
return .{
.tool_images = images.items,
@@ -2775,16 +3079,41 @@ fn parseToolResultPresentations(
/// Writes typed image blocks with persistence-owned source_ref spelling.
pub fn writePersistedToolImages(writer: *std.Io.Writer, images: []const types.ToolImage) !void {
return writeToolImagesTo(writer, images, false);
}
/// `writePersistedToolImages`, but with `naming`, an image the host stored is
/// written as its name (`inline_ref`) and decoded size instead of its bytes.
/// An image the session dropped is written as its name alone either way.
/// Writes a tool result's `images` as a list: with `naming`, each the host
/// stored by its name, and every other with its bytes; one the session
/// dropped by its name alone. `parseToolImages` reads it back.
pub fn writeToolImages(writer: *std.Io.Writer, images: []const types.ToolImage, naming: bool) !void {
return writeToolImagesTo(writer, images, naming);
}
fn writeToolImagesTo(writer: *std.Io.Writer, images: []const types.ToolImage, naming: bool) !void {
try writer.writeByte('[');
for (images, 0..) |image, index| {
if (index > 0) try writer.writeByte(',');
if (image.source_ref) |source_ref| {
if (!image_data.validSourceRef(source_ref)) return error.InvalidSourceRef;
} else if (image.data.len == 0) return error.InvalidImage;
} else if (image.data.len == 0 and image.name == null) return error.InvalidImage;
try writer.writeAll("{\"type\":\"image\",\"mimeType\":");
try writeJsonString(writer, image.mime_type);
try writer.writeAll(",\"data\":");
try writeJsonString(writer, image.data);
if (image.droppedFromSession()) {
try writer.writeAll(",\"inline_ref\":");
try writeJsonString(writer, &image.name.?);
try writer.writeAll(",\"dropped\":true");
} else if (naming and image.name != null) {
const size = std.base64.standard.Decoder.calcSizeForSlice(image.data) catch return error.InvalidImage;
try writer.writeAll(",\"inline_ref\":");
try writeJsonString(writer, &image.name.?);
try writer.print(",\"inline_size\":{d}", .{size});
} else {
try writer.writeAll(",\"data\":");
try writeJsonString(writer, image.data);
}
if (image.source_ref) |source_ref| {
try writer.writeAll(",\"source_ref\":");
try writeJsonString(writer, source_ref);
@@ -2825,6 +3154,106 @@ pub fn parsePersistedToolImages(alloc: Allocator, content: []const std.json.Valu
return image_data.parseToolImages(alloc, normalized[0..initialized]);
}
/// `parsePersistedToolImages`, but an image written by name (`inline_ref`)
/// reads its bytes from the host's staged copy in `sources.names`. One the
/// session dropped, or one an open has yet to stage, parses with its name and
/// without bytes; the open's `missing` tells the two apart. Caller owns the
/// images.
/// Reads a list `writeToolImages` wrote. Caller owns the images. A name
/// `sources.names` has no bytes for reads as an image without bytes, and the
/// resolver records it as missing.
pub fn parseToolImages(alloc: Allocator, content: []const std.json.Value, sources: ImageSources) ![]types.ToolImage {
return parseToolImagesWith(alloc, content, sources);
}
fn parseToolImagesWith(alloc: Allocator, content: []const std.json.Value, sources: ImageSources) ![]types.ToolImage {
for (content) |item| {
if (item == .object and item.object.contains("inline_ref")) break;
} else return parsePersistedToolImages(alloc, content);
if (content.len > image_data.max_tool_images) return error.ImageLimitExceeded;
var images: std.ArrayList(types.ToolImage) = .empty;
defer {
for (images.items) |image| freeToolImage(alloc, image);
images.deinit(alloc);
}
for (content) |item| {
if (item != .object) return error.InvalidImage;
if (!item.object.contains("inline_ref")) {
// An image written inline keeps its own admission; an unsupported
// one parses as none, which the caller notes.
const inline_images = try parsePersistedToolImages(alloc, &[_]std.json.Value{item});
images.ensureUnusedCapacity(alloc, inline_images.len) catch |err| {
types.freeToolImages(alloc, inline_images);
return err;
};
images.appendSliceAssumeCapacity(inline_images);
alloc.free(inline_images);
continue;
}
const image = try parseNamedToolImage(alloc, item.object, sources);
images.append(alloc, image) catch |err| {
freeToolImage(alloc, image);
return err;
};
}
return images.toOwnedSlice(alloc);
}
fn parseNamedToolImage(alloc: Allocator, object: std.json.ObjectMap, sources: ImageSources) !types.ToolImage {
var keys = object.iterator();
while (keys.next()) |entry| {
for ([_][]const u8{ "type", "mimeType", "inline_ref", "inline_size", "dropped", "source_ref" }) |known| {
if (std.mem.eql(u8, entry.key_ptr.*, known)) break;
} else return error.InvalidSessionFormat;
}
if (!std.mem.eql(u8, try requireString(object, "type"), "image")) return error.InvalidSessionFormat;
const mime_type = try requireString(object, "mimeType");
if (!image_data.supportedMediaType(mime_type)) return error.InvalidSessionFormat;
const name = imageName(try requireString(object, "inline_ref")) orelse return error.InvalidSessionFormat;
const source_ref: ?[]const u8 = if (object.get("source_ref")) |value| ref: {
if (value != .string or !image_data.validSourceRef(value.string)) return error.InvalidSourceRef;
break :ref value.string;
} else null;
if (object.get("dropped")) |value| {
if (value != .bool or !value.bool or object.contains("inline_size")) return error.InvalidSessionFormat;
return ownedToolImage(alloc, try alloc.dupe(u8, ""), mime_type, source_ref, name);
}
const resolver = sources.names orelse return error.InvalidSessionFormat;
const size = try requireUsize(object, "inline_size");
if (size == 0 or size > image_data.max_encoded_image_bytes) return error.InvalidSessionFormat;
const resolved = try resolver.resolve(alloc, name, size);
const bytes = resolved orelse return ownedToolImage(alloc, try alloc.dupe(u8, ""), mime_type, source_ref, name);
defer alloc.free(bytes);
if (bytes.len != size) return error.InvalidSessionFormat;
const encoded = try alloc.alloc(u8, std.base64.standard.Encoder.calcSize(bytes.len));
_ = std.base64.standard.Encoder.encode(encoded, bytes);
// The same admission as an image written inline.
image_data.validateImage(alloc, encoded, mime_type) catch |err| {
alloc.free(encoded);
return err;
};
return ownedToolImage(alloc, encoded, mime_type, source_ref, name);
}
/// A tool image owning `data`, which it takes, and copies of the rest.
fn ownedToolImage(alloc: Allocator, data: []u8, mime_type: []const u8, source_ref: ?[]const u8, name: ImageName) Allocator.Error!types.ToolImage {
errdefer alloc.free(data);
const owned_type = try alloc.dupe(u8, mime_type);
errdefer alloc.free(owned_type);
return .{
.data = data,
.mime_type = owned_type,
.source_ref = if (source_ref) |value| try alloc.dupe(u8, value) else null,
.name = name,
};
}
fn freeToolImage(alloc: Allocator, image: types.ToolImage) void {
alloc.free(image.data);
alloc.free(image.mime_type);
if (image.source_ref) |source_ref| alloc.free(source_ref);
}
const ParsedToolResultImages = struct {
handle: ?[]u8,
items: []types.ToolImage,
@@ -2840,6 +3269,7 @@ fn parseToolResultImages(
alloc: Allocator,
object: std.json.ObjectMap,
output: *[]u8,
sources: ImageSources,
) !ParsedToolResultImages {
const handle = if (object.get("tool_image_handle")) |value|
try parseOptionalDurableBytes(alloc, value)
@@ -2848,9 +3278,11 @@ fn parseToolResultImages(
errdefer if (handle) |owned| mem_utils.free(alloc, owned);
const items = if (object.get("tool_images")) |images| items: {
if (images != .array) return error.InvalidSessionFormat;
const parsed = parsePersistedToolImages(alloc, images.array.items) catch |err| {
const parsed = parseToolImagesWith(alloc, images.array.items, sources) catch |err| {
if (err == error.OutOfMemory) return error.OutOfMemory;
if (err == error.InvalidSourceRef) return error.InvalidSessionFormat;
// A named image that cannot be read is a format error, not an
// image to leave out: the record says the store holds it.
if (err == error.InvalidSourceRef or err == error.InvalidSessionFormat) return error.InvalidSessionFormat;
const notice = try alloc.print(
"{s}\n[Saved tool image unavailable: {s}]",
.{ output.*, @errorName(err) },
@@ -3552,7 +3984,7 @@ test "durable tool image validation preserves the session format error contract"
.provider_native = false,
.created_at_ms = 0,
.tool_images = &images,
}, .durable));
}, .durable, {}));
}
}
@@ -4054,7 +4486,7 @@ test "review feedback durable codec rejects invalid provenance and unknown field
result.review_feedback = true;
var encoded: std.Io.Writer.Allocating = .init(alloc);
defer encoded.deinit();
try writeExecutionToolResult(&encoded.writer, result, .durable);
try writeExecutionToolResult(&encoded.writer, result, .durable, {});
var parsed = try std.json.parseFromSlice(std.json.Value, alloc, encoded.written(), .{});
defer parsed.deinit();
const cases = [_]struct { status: []const u8, provider_native: bool, marker: std.json.Value }{
@@ -4080,7 +4512,7 @@ test "review feedback durable codec rejects invalid provenance and unknown field
invalid.provider_native = native;
try std.testing.expectError(
error.InvalidSessionFormat,
writeExecutionToolResult(&encoded.writer, invalid, .durable),
writeExecutionToolResult(&encoded.writer, invalid, .durable, {}),
);
}
}
@@ -4092,7 +4524,7 @@ test "review feedback defaults old durable results without inspecting output" {
result.output = @constCast("Security review held this action.");
var encoded: std.Io.Writer.Allocating = .init(alloc);
defer encoded.deinit();
try writeExecutionToolResult(&encoded.writer, result, .durable);
try writeExecutionToolResult(&encoded.writer, result, .durable, {});
var parsed = try std.json.parseFromSlice(std.json.Value, alloc, encoded.written(), .{});
defer parsed.deinit();
try std.testing.expect(parsed.value.object.swapRemove("review_feedback"));
@@ -5088,6 +5520,42 @@ test "inline image bytes round trip through the history turn codec" {
}
}
test "the image window keeps the newest images that fit, and drops every older one" {
const window: ImageWindow = .{ .max_images = 3, .max_bytes = 100 };
try std.testing.expectEqual(@as(usize, 0), window.oldestOutside(&.{}));
try std.testing.expectEqual(@as(usize, 0), window.oldestOutside(&.{ 10, 20, 30 }));
// Three images at most: the oldest of four leaves.
try std.testing.expectEqual(@as(usize, 1), window.oldestOutside(&.{ 10, 20, 30, 40 }));
// 100 bytes at most: 60 does not fit beside 40 and 30, so it and the
// 5 before it leave, though 5 alone would fit.
try std.testing.expectEqual(@as(usize, 2), window.oldestOutside(&.{ 5, 60, 30, 40 }));
// An image already dropped keeps every older one out.
try std.testing.expectEqual(@as(usize, 2), window.oldestOutside(&.{ 10, null, 30, 40 }));
// An image larger than the whole window leaves too.
try std.testing.expectEqual(@as(usize, 1), window.oldestOutside(&.{101}));
}
test "a dropped image parses only as a name without bytes" {
const alloc = std.testing.allocator;
const digest = "\"" ++ text_utils.repeat("a", image_name_len) ++ "\"";
const prefix = "{\"text\":\"[Image #1]\",\"images\":[{\"id\":1,\"path\":\"inline://image-1\",\"media_type\":\"image/png\",\"snapshot_path\":null,\"snapshot_sha256\":";
var parsed = try std.json.parseFromSlice(std.json.Value, alloc, prefix ++ digest ++ ",\"dropped\":true}]}", .{});
defer parsed.deinit();
const user = try parseUserTurn(alloc, parsed.value, .{});
defer types.freeUserTurn(alloc, user);
try std.testing.expect(user.images[0].droppedFromSession());
for ([_][]const u8{
prefix ++ digest ++ ",\"dropped\":false}]}",
prefix ++ "\"not-a-digest\",\"dropped\":true}]}",
prefix ++ "null,\"dropped\":true}]}",
prefix ++ digest ++ ",\"dropped\":true,\"inline_data\":\"iVBORw0KGgo=\"}]}",
}) |json| {
var invalid = try std.json.parseFromSlice(std.json.Value, alloc, json, .{});
defer invalid.deinit();
try std.testing.expectError(error.InvalidSessionFormat, parseUserTurn(alloc, invalid.value, .{}));
}
}
test "inline image parsing rejects missing digest and mismatched bytes" {
const alloc = std.testing.allocator;
const png = "\x89PNG\r\n\x1a\ninline-bytes";
@@ -5646,13 +6114,13 @@ test "persisted image source refs reject malformed metadata and preserve the byt
for (invalid) |value| {
user_ref.* = value;
tool_ref.* = value;
try std.testing.expectError(error.InvalidSessionFormat, parseUserTurn(alloc, user.value, null));
try std.testing.expectError(error.InvalidSessionFormat, parseUserTurn(alloc, user.value, .{}));
try std.testing.expectError(error.InvalidSourceRef, parsePersistedToolImages(alloc, tools.value.array.items));
}
const boundary = text_utils.repeat("é", 256);
user_ref.* = .{ .string = boundary };
tool_ref.* = .{ .string = boundary };
const decoded_user = try parseUserTurn(alloc, user.value, null);
const decoded_user = try parseUserTurn(alloc, user.value, .{});
defer session.freeUserTurn(alloc, decoded_user);
const decoded_tools = try parsePersistedToolImages(alloc, tools.value.array.items);
defer types.freeToolImages(alloc, decoded_tools);
+38 -3
View File
@@ -246,6 +246,10 @@ pub const ToolLifecycleEvent = union(enum) {
activity_kind: ToolActivityKind,
arguments_json: ?[]const u8 = null,
place_after_current_transcript: bool = false,
/// The call's `ToolCall.resolved_display_target`, for hosts that
/// title a call when it starts. Borrowed for this push only; copies
/// omit it.
resolved_display_target: ?[]const u8 = null,
},
progress: struct {
id: ToolLifecycleId,
@@ -957,6 +961,10 @@ pub const ToolCall = struct {
provenance: ToolExecutionProvenance = .fx_local,
/// Borrowed only for this action. Copies and durable/provider encodings omit it.
resolved_skill: ?*const skill_contract.PreparedSkill = null,
/// What this action works on, resolved from earlier results, such as the
/// command whose saved output a `read_tool_result` call reads. Borrowed
/// only for this action. Copies and durable/provider encodings omit it.
resolved_display_target: ?[]const u8 = null,
};
pub const WebSearchProgress = union(enum) {
@@ -1310,11 +1318,23 @@ pub const ExecutionMemory = struct {
/// Image bytes returned by a tool; never a grant to read a user file.
pub const ToolImage = struct {
/// Owned encoded bytes; empty only when a valid source_ref is present.
/// Owned encoded bytes; empty only when a valid source_ref is present, or
/// when the session dropped the image (`droppedFromSession`).
data: []u8,
mime_type: []u8,
/// Owned opaque host reference, never filesystem or network authority.
source_ref: ?[]u8 = null,
/// The image's name in a libfx host's store, the lowercase hex SHA-256
/// of its decoded bytes, when the host stored them before it sent the
/// result. Writers that name stored images write this instead of `data`.
name: ?[64]u8 = null,
/// Whether the session no longer holds this image: a libfx session with
/// an image store keeps only its newest images, and an older one keeps
/// its name, without its bytes.
pub fn droppedFromSession(self: ToolImage) bool {
return self.data.len == 0 and self.name != null;
}
};
pub fn freeToolImages(alloc: std.mem.Allocator, images: []const ToolImage) void {
@@ -1346,6 +1366,7 @@ pub fn dupeToolImages(alloc: std.mem.Allocator, images: []const ToolImage) ![]To
.data = data,
.mime_type = mime_type,
.source_ref = if (image.source_ref) |source_ref| try alloc.dupe(u8, source_ref) else null,
.name = image.name,
};
initialized += 1;
}
@@ -1498,6 +1519,13 @@ pub const ImageAttachment = struct {
inline_data: ?[]u8 = null,
/// Owned opaque host reference, never filesystem or network authority.
source_ref: ?[]u8 = null,
/// Whether the session no longer holds this image: a libfx session with
/// an image store keeps only its newest images, and an older one keeps
/// its digest, which names it in the host's store, without its bytes.
pub fn droppedFromSession(self: ImageAttachment) bool {
return self.inline_data == null and self.snapshot_path == null and self.snapshot_sha256 != null;
}
};
pub const UserTurn = struct {
@@ -3084,12 +3112,19 @@ test "dupeToolCall preserves argument integrity" {
try std.testing.expectEqual(ToolArgumentIntegrity.malformed_json, copy.argument_integrity);
}
test "dupeToolCall drops action scoped skill bindings" {
test "dupeToolCall drops action scoped bindings" {
const skill: skill_contract.PreparedSkill = .{ .skill = .{ .name = "workflow", .description = "", .path = "/skills/workflow", .source = .global_fx } };
const source: ToolCall = .{ .id = "skill", .name = "skill", .arguments_json = "{\"location\":\"/skills/workflow\"}", .resolved_skill = &skill };
const source: ToolCall = .{
.id = "skill",
.name = "skill",
.arguments_json = "{\"location\":\"/skills/workflow\"}",
.resolved_skill = &skill,
.resolved_display_target = "more output from zig build test",
};
const copy = try dupeToolCall(std.testing.allocator, source);
defer freeToolCall(std.testing.allocator, copy);
try std.testing.expect(copy.resolved_skill == null);
try std.testing.expect(copy.resolved_display_target == null);
try std.testing.expectEqualStrings(source.arguments_json, copy.arguments_json);
}
@@ -0,0 +1,241 @@
const std = @import("std");
const result_store = @import("../session/result_store.zig");
const command_replay_store = @import("../session/command_replay_store.zig");
const compactor = @import("../compactor/compactor.zig");
const text_utils = @import("../shared/text_utils.zig");
const tool_dispatch = @import("tool_dispatch.zig");
const Allocator = std.mem.Allocator;
/// Longest search text a row repeats, after terminal-safe encoding.
const max_query_label_bytes = 60;
/// A handle as `validate` resolves it: without surrounding whitespace, plus
/// the `.txt` a stored result or image handle may omit.
const HandleNormalization = struct {
trimmed: []const u8,
suffix: []const u8,
};
pub fn classifyHandleNormalization(handle: []const u8) HandleNormalization {
const trimmed = std.mem.trim(u8, handle, " \t\r\n");
const suffix = if ((std.mem.startsWith(u8, trimmed, "result-") or result_store.isImageHandle(trimmed)) and
std.mem.findScalar(u8, trimmed, '.') == null)
".txt"
else
"";
return .{ .trimmed = trimmed, .suffix = suffix };
}
/// What a call asks for, read the way the tool's `decode` and `validate` read it,
/// so its row can say what the agent is reading.
const RequestView = struct {
kind: Kind,
/// The handle without surrounding whitespace. Empty for a history search.
handle: []const u8 = "",
query: []const u8 = "",
const Kind = enum { range, query, images, record, history_search };
};
fn requestView(args: std.json.ObjectMap) ?RequestView {
const object = if (args.get("request")) |request| switch (request) {
.object => |value| value,
else => return null,
} else args;
if (object.get("search") != null) return .{ .kind = .history_search };
const handle = switch (object.get("handle") orelse return null) {
.string => |value| std.mem.trim(u8, value, " \t\r\n"),
else => return null,
};
if (handle.len == 0) return null;
var record_buffer: compactor.RecordFileBuffer = undefined;
if (compactor.recordFile(&record_buffer, handle) != null) return .{ .kind = .record, .handle = handle };
if (result_store.isImageHandle(handle)) return .{ .kind = .images, .handle = handle };
if (object.get("query")) |query| {
if (query == .string and query.string.len > 0) {
return .{ .kind = .query, .handle = handle, .query = query.string };
}
}
return .{ .kind = .range, .handle = handle };
}
fn parseArguments(alloc: Allocator, arguments_json: []const u8) Allocator.Error!?std.json.Parsed(std.json.Value) {
const parsed = std.json.parseFromSlice(std.json.Value, alloc, arguments_json, .{}) catch |err| return switch (err) {
error.OutOfMemory => error.OutOfMemory,
else => null,
};
if (parsed.value != .object) {
parsed.deinit();
return null;
}
return parsed;
}
fn outputNoun(handle: []const u8, command: []const u8, other: []const u8) []const u8 {
return if (command_replay_store.isReplayHandle(handle)) command else other;
}
fn recordNoun(kind: u8) []const u8 {
return switch (std.ascii.toUpper(kind)) {
'M' => "turn",
'T' => "tool call",
else => "compaction",
};
}
fn readPresentation(
action_label: []const u8,
completed_action_label: []const u8,
label_arg_default: []const u8,
) tool_dispatch.CallPresentation {
return .{
.activity_kind = .read,
.action_label = action_label,
.completed_action_label = completed_action_label,
.label_arg_kind = .none,
.label_arg_default = label_arg_default,
};
}
/// Labels a call by what it reads instead of one shared "tool result" label.
/// The resolved row text from `displayTarget` replaces the default object.
pub fn presentation(args: std.json.ObjectMap) ?tool_dispatch.CallPresentation {
const view = requestView(args) orelse return null;
return switch (view.kind) {
.range => readPresentation("Reading", "Read", outputNoun(view.handle, "more command output", "more output")),
.query => readPresentation("Searching", "Searched", outputNoun(view.handle, "command output", "saved output")),
.images => readPresentation("Reloading", "Reloaded", "saved images"),
.record => readPresentation("Opening", "Opened", "earlier conversation"),
.history_search => readPresentation("Searching", "Searched", "earlier conversation"),
};
}
/// The saved output a call reads, with the suffix `validate` restores, or null
/// for a history search, a saved record, or arguments `decode` rejects. The
/// caller owns the returned handle.
pub fn requestedHandle(alloc: Allocator, arguments_json: []const u8) Allocator.Error!?[]u8 {
var parsed = (try parseArguments(alloc, arguments_json)) orelse return null;
defer parsed.deinit();
const view = requestView(parsed.value.object) orelse return null;
switch (view.kind) {
.record, .history_search => return null,
.range, .query, .images => {},
}
const normalization = classifyHandleNormalization(view.handle);
return try std.mem.concat(alloc, u8, &.{ normalization.trimmed, normalization.suffix });
}
/// The row text for what a call reads: the output of the earlier call that
/// `source` names, the text it searches for, or the saved record it opens.
/// Returns null when the default object from `presentation` already says it.
/// `source` must be terminal-safe. The caller owns the returned text.
pub fn displayTarget(
alloc: Allocator,
arguments_json: []const u8,
source: ?[]const u8,
) Allocator.Error!?[]u8 {
var parsed = (try parseArguments(alloc, arguments_json)) orelse return null;
defer parsed.deinit();
const view = requestView(parsed.value.object) orelse return null;
switch (view.kind) {
.range => {
const name = source orelse return null;
return try alloc.print("more output from {s}", .{name});
},
.query => {
var query = try text_utils.encodeTerminalSafe(alloc, view.query, max_query_label_bytes);
defer query.deinit(alloc);
if (source) |name| return try alloc.print("{s} output for \"{s}\"", .{ name, query.bytes });
return try alloc.print("{s} for \"{s}\"", .{
outputNoun(view.handle, "command output", "saved output"),
query.bytes,
});
},
// `requestView` admits only a record ID: one letter, then digits.
.record => return try alloc.print("earlier {s} {c}{s}", .{
recordNoun(view.handle[0]),
std.ascii.toUpper(view.handle[0]),
view.handle[1..],
}),
.images, .history_search => return null,
}
}
fn expectPresentation(arguments_json: []const u8, action_label: []const u8, completed_action_label: []const u8, label_arg_default: []const u8) !void {
var parsed = try std.json.parseFromSlice(std.json.Value, std.testing.allocator, arguments_json, .{});
defer parsed.deinit();
const resolved = presentation(parsed.value.object) orelse return error.TestExpectedPresentation;
try std.testing.expectEqual(tool_dispatch.LabelArgKind.none, resolved.label_arg_kind);
try std.testing.expectEqualStrings(action_label, resolved.action_label);
try std.testing.expectEqualStrings(completed_action_label, resolved.completed_action_label);
try std.testing.expectEqualStrings(label_arg_default, resolved.label_arg_default);
}
fn expectDisplayTarget(arguments_json: []const u8, source: ?[]const u8, expected: ?[]const u8) !void {
const alloc = std.testing.allocator;
const target = try displayTarget(alloc, arguments_json, source);
defer if (target) |value| alloc.free(value);
if (expected) |text| {
try std.testing.expectEqualStrings(text, target orelse return error.TestExpectedDisplayTarget);
} else {
try std.testing.expect(target == null);
}
}
test "read_tool_result rows name what each request reads" {
const replay = "fx-command-replay-0123.bin";
try expectPresentation("{\"request\":{\"handle\":\"" ++ replay ++ "\"}}", "Reading", "Read", "more command output");
try expectPresentation("{\"handle\":\"result-x.txt\",\"start_byte\":9}", "Reading", "Read", "more output");
try expectPresentation("{\"request\":{\"handle\":\"" ++ replay ++ "\",\"query\":\"error:\"}}", "Searching", "Searched", "command output");
try expectPresentation("{\"handle\":\"result-x\",\"query\":\"error:\"}", "Searching", "Searched", "saved output");
// An empty query reads a range, as `decode` treats it.
try expectPresentation("{\"handle\":\"result-x\",\"query\":\"\"}", "Reading", "Read", "more output");
try expectPresentation("{\"request\":{\"handle\":\"image-result-abc\"}}", "Reloading", "Reloaded", "saved images");
try expectPresentation("{\"request\":{\"handle\":\" t5 \"}}", "Opening", "Opened", "earlier conversation");
try expectPresentation("{\"request\":{\"search\":[\"retry budget\"]}}", "Searching", "Searched", "earlier conversation");
var parsed = try std.json.parseFromSlice(std.json.Value, std.testing.allocator, "{\"request\":\"handle\"}", .{});
defer parsed.deinit();
try std.testing.expect(presentation(parsed.value.object) == null);
}
test "read_tool_result display targets name the source, query, and record" {
const replay = "fx-command-replay-0123.bin";
try expectDisplayTarget("{\"request\":{\"handle\":\"" ++ replay ++ "\"}}", "zig build test", "more output from zig build test");
try expectDisplayTarget("{\"request\":{\"handle\":\"" ++ replay ++ "\"}}", null, null);
try expectDisplayTarget(
"{\"request\":{\"handle\":\"" ++ replay ++ "\",\"query\":\"error:\"}}",
"zig build test",
"zig build test output for \"error:\"",
);
try expectDisplayTarget("{\"request\":{\"handle\":\"" ++ replay ++ "\",\"query\":\"error:\"}}", null, "command output for \"error:\"");
try expectDisplayTarget("{\"handle\":\"result-x\",\"query\":\"needle\"}", null, "saved output for \"needle\"");
try expectDisplayTarget("{\"request\":{\"handle\":\"t5\"}}", "ignored", "earlier tool call T5");
try expectDisplayTarget("{\"request\":{\"handle\":\"M12\"}}", null, "earlier turn M12");
try expectDisplayTarget("{\"request\":{\"handle\":\"l2\"}}", null, "earlier compaction L2");
try expectDisplayTarget("{\"request\":{\"handle\":\"image-result-abc\"}}", "vision", null);
try expectDisplayTarget("{\"request\":{\"search\":\"retry\"}}", null, null);
try expectDisplayTarget("not json", "zig build test", null);
}
test "read_tool_result display targets keep search text terminal-safe and short" {
const alloc = std.testing.allocator;
const target = (try displayTarget(alloc, "{\"handle\":\"result-x\",\"query\":\"\\u001b[31mred" ++ text_utils.repeat("x", 120) ++ "\"}", null)).?;
defer alloc.free(target);
try std.testing.expect(std.mem.findScalar(u8, target, 0x1b) == null);
try std.testing.expect(target.len < "saved output for \"\"".len + max_query_label_bytes + 1);
}
test "read_tool_result requested handles match what validate opens" {
const alloc = std.testing.allocator;
const restored = (try requestedHandle(alloc, "{\"request\":{\"handle\":\" result-shell-abc \"}}")).?;
defer alloc.free(restored);
try std.testing.expectEqualStrings("result-shell-abc.txt", restored);
const replay = (try requestedHandle(alloc, "{\"handle\":\"fx-command-replay-0123.bin\",\"query\":\"x\"}")).?;
defer alloc.free(replay);
try std.testing.expectEqualStrings("fx-command-replay-0123.bin", replay);
try std.testing.expect(try requestedHandle(alloc, "{\"request\":{\"handle\":\"T5\"}}") == null);
try std.testing.expect(try requestedHandle(alloc, "{\"request\":{\"search\":[\"x\"]}}") == null);
try std.testing.expect(try requestedHandle(alloc, "[]") == null);
}
+62
View File
@@ -1,4 +1,5 @@
const std = @import("std");
const debug_trace = @import("../shared/debug_trace.zig");
const image_data = @import("../images/image_data.zig");
const types = @import("../shared/types.zig");
const tool_dispatch = @import("tool_dispatch.zig");
@@ -21,6 +22,7 @@ pub fn parseRichResult(alloc: Allocator, bytes: []const u8, text_limit: usize, i
else => return .{ .failure = try alloc.print("Host tool image could not be used: {s}", .{@errorName(err)}) },
};
errdefer types.freeToolImages(alloc, images);
try nameStoredImages(alloc, images, values.array.items);
const omitted = images.len != values.array.items.len;
const source = if (omitted) try alloc.print("[Unsupported tool images were omitted.]\n{s}", .{text.string}) else text.string;
defer if (omitted) alloc.free(source);
@@ -28,6 +30,38 @@ pub fn parseRichResult(alloc: Allocator, bytes: []const u8, text_limit: usize, i
return .{ .rich = .{ .text = body, .images = images, .is_error = is_error } };
}
/// Names each image a libfx host stored before it sent the result: one whose
/// decoded bytes hash to a `stored` mark the result carries. A mark that
/// matches no image leaves the images inline.
fn nameStoredImages(alloc: Allocator, images: []types.ToolImage, values: []const std.json.Value) Allocator.Error!void {
var marks: usize = 0;
for (values) |item| {
if (item == .object and item.object.get("stored") != null) marks += 1;
}
if (marks == 0) return;
var named: usize = 0;
for (images) |*image| {
if (image.data.len == 0) continue;
const size = std.base64.standard.Decoder.calcSizeForSlice(image.data) catch continue;
const bytes = try alloc.alloc(u8, size);
defer alloc.free(bytes);
std.base64.standard.Decoder.decode(bytes, image.data) catch continue;
var digest: [std.crypto.hash.sha2.Sha256.digest_length]u8 = undefined;
std.crypto.hash.sha2.Sha256.hash(bytes, &digest, .{});
const name = std.fmt.bytesToHex(digest, .lower);
for (values) |item| {
if (item != .object) continue;
const mark = item.object.get("stored") orelse continue;
if (mark == .string and std.mem.eql(u8, mark.string, &name)) {
image.name = name;
named += 1;
break;
}
}
}
if (named < marks) debug_trace.logf("tools", "event=host_tool_images_not_stored marks={d} named={d} reason=name_mismatch", .{ marks, named });
}
/// Replaces binary fields in a caller-owned parsed MCP block with delivery notes.
/// Images themselves travel through the typed result, never JSON text.
pub fn projectMediaForText(alloc: std.mem.Allocator, content: *std.json.Value) std.mem.Allocator.Error!void {
@@ -55,3 +89,31 @@ fn projectMediaBlock(alloc: std.mem.Allocator, item: *std.json.Value) std.mem.Al
}
}
}
test "a host tool image is named only when its stored mark is the digest of its bytes" {
const alloc = std.testing.allocator;
const png = "\x89PNG\r\n\x1a\nhost-tool-shot";
var encoded: [64]u8 = undefined;
const data = std.base64.standard.Encoder.encode(&encoded, png);
var digest: [std.crypto.hash.sha2.Sha256.digest_length]u8 = undefined;
std.crypto.hash.sha2.Sha256.hash(png, &digest, .{});
const name = std.fmt.bytesToHex(digest, .lower);
for ([_]struct { mark: []const u8, named: bool }{
.{ .mark = &name, .named = true },
.{ .mark = "0000000000000000000000000000000000000000000000000000000000000000", .named = false },
}) |case| {
const json = try alloc.print("{{\"text\":\"shot\",\"images\":[{{\"type\":\"image\",\"data\":\"{s}\",\"mimeType\":\"image/png\",\"stored\":\"{s}\"}}]}}", .{ data, case.mark });
defer alloc.free(json);
const result = try parseRichResult(alloc, json, 1024, false);
const rich = result.rich;
defer {
alloc.free(rich.text);
types.freeToolImages(alloc, rich.images);
}
try std.testing.expectEqual(@as(usize, 1), rich.images.len);
// A mark that does not match leaves the image inline, never lost.
try std.testing.expectEqualStrings(data, rich.images[0].data);
try std.testing.expectEqual(case.named, rich.images[0].name != null);
if (case.named) try std.testing.expectEqualStrings(&name, &rich.images[0].name.?);
}
}
+82
View File
@@ -672,12 +672,53 @@ pub fn formatPlainAction(alloc: Allocator, input: ToolActionInput) ![]const u8 {
return alloc.print("{s} {s}", .{ presentation.action_label, try formatWebSearchActionDetail(scratch, args) });
}
const value = input.display_target orelse
call.resolved_display_target orelse
resolvedSkillName(call, presentation) orelse
tool_dispatch.presentationLabelValue(presentation, args) orelse
presentation.label_arg_default;
return alloc.print("{s} {s}", .{ presentation.action_label, value });
}
/// Names an earlier call in the row of a later call that reads its saved
/// output: the command it ran, the path or URL it read, or the name of an MCP
/// tool. Returns null when the call has none of these, or when its command
/// cannot be described, so a label never fails the turn that shows it. The
/// text is terminal-safe. The caller owns the returned allocation.
pub fn savedOutputSourceLabel(
alloc: Allocator,
registry: tool_dispatch.Registry,
workspace_root: []const u8,
call: ToolCall,
) Allocator.Error!?[]const u8 {
const command_activity = formatRunCommandActivity(alloc, registry, workspace_root, call) catch |err| switch (err) {
error.OutOfMemory => return error.OutOfMemory,
else => return null,
};
if (command_activity) |activity| return activity.detail;
const spec = registry.lookup(call.name) orelse {
if (isProviderSearchAlias(call.name)) return null;
return try encodeSourceLabel(alloc, call.name);
};
var scratch_state = std.heap.ArenaAllocator.init(alloc);
defer mem_utils.deinit_arena(scratch_state);
const args = tool_args.parseToolArgsObject(scratch_state.allocator(), call.arguments_json) catch |err| switch (err) {
error.OutOfMemory => return error.OutOfMemory,
else => return null,
};
const presentation = tool_dispatch.presentationForArgs(spec.*, args);
switch (presentation.label_arg_kind) {
.path, .url => {},
else => return null,
}
const value = tool_dispatch.presentationLabelValue(presentation, args) orelse return null;
return try encodeSourceLabel(alloc, value);
}
fn encodeSourceLabel(alloc: Allocator, raw: []const u8) ![]const u8 {
const encoded = try text_utils.encodeTerminalSafe(alloc, raw, max_run_command_activity_bytes);
return encoded.bytes;
}
/// The caller owns the returned allocation and must free it with `alloc`.
pub fn formatPermissionLabel(alloc: Allocator, registry: tool_dispatch.Registry, call: ToolCall) ![]const u8 {
var scratch_state = std.heap.ArenaAllocator.init(alloc);
@@ -1502,3 +1543,44 @@ test "tool presentation frees all formatted output with a normal allocator" {
fn expectContains(text: []const u8, needle: []const u8) !void {
try std.testing.expect(std.mem.find(u8, text, needle) != null);
}
test "read_tool_result rows use the resolved source and the request kind" {
const alloc = std.testing.allocator;
const registry = test_builtin_tools.advertisement_set.registry;
const page: ToolCall = .{
.id = "page",
.name = "read_tool_result",
.arguments_json = "{\"request\":{\"handle\":\"fx-command-replay-0123.bin\"}}",
};
const generic = try formatPlainAction(alloc, .{ .tool_registry = registry, .call = page });
defer alloc.free(generic);
try std.testing.expectEqualStrings("Reading more command output", generic);
var resolved = page;
resolved.resolved_display_target = "more output from zig build test";
const named = try formatPlainAction(alloc, .{ .tool_registry = registry, .call = resolved });
defer alloc.free(named);
try std.testing.expectEqualStrings("Reading more output from zig build test", named);
}
test "saved output sources name commands, paths, URLs, and MCP tools" {
const alloc = std.testing.allocator;
const registry = test_builtin_tools.advertisement_set.registry;
const cases = [_]struct { call: ToolCall, expected: ?[]const u8 }{
.{ .call = .{ .id = "run", .name = "shell", .arguments_json = "{\"action\":\"run\",\"command\":\"zig build test\"}" }, .expected = "zig build test" },
.{ .call = .{ .id = "file", .name = "read_file", .arguments_json = "{\"path\":\"src/\\u001b[2Jmain.zig\"}" }, .expected = "src/\\x1b[2Jmain.zig" },
.{ .call = .{ .id = "fetch", .name = "web_fetch", .arguments_json = "{\"url\":\"https://fx.sh/docs\"}" }, .expected = "https://fx.sh/docs" },
.{ .call = .{ .id = "mcp", .name = "mcp_linear_list_issues", .arguments_json = "{}" }, .expected = "mcp_linear_list_issues" },
.{ .call = .{ .id = "grep", .name = "grep_files", .arguments_json = "{\"pattern\":\"needle\"}" }, .expected = null },
.{ .call = .{ .id = "bad", .name = "read_file", .arguments_json = "not json" }, .expected = null },
};
for (cases) |case| {
const label = try savedOutputSourceLabel(alloc, registry, "/workspace", case.call);
defer if (label) |value| alloc.free(value);
if (case.expected) |expected| {
try std.testing.expectEqualStrings(expected, label orelse return error.TestExpectedSourceLabel);
} else {
try std.testing.expect(label == null);
}
}
}
+2
View File
@@ -4833,6 +4833,8 @@ test "semantic code block preserves indentation on wrapped continuation rows" {
}
test {
_ = @import("core/agent/runtime/journal.zig");
_ = @import("acp/libfx_journal.zig");
_ = @import("mcp/mcp.zig");
_ = @import("core/mcp_host/runtime.zig");
_ = @import("napi_fetch_state.zig");
+33
View File
@@ -297,6 +297,16 @@ pub const Engine = struct {
return s.saveCredential(w);
}
/// Gives server `name` the credential `saveCredential` wrote, as
/// `Config.auth.saved` does, for a host that reads it after `init`.
/// `saved` is copied. Call it before anything starts the server: a
/// running server keeps the sign-in it has.
pub fn restoreCredential(e: *Engine, name: []const u8, saved: []const u8) Error!void {
const slot = try e.find(name);
std.debug.assert(slot.server == null);
slot.config.auth.saved = try slot.arena.allocator().dupe(u8, saved);
}
/// Wakes a `next` that waits, from any thread: it returns null early, so
/// the host can act on what it was woken for.
pub fn wake(e: *Engine) void {
@@ -637,6 +647,29 @@ test "wake from another thread ends a waiting next early" {
try std.testing.expect(e.now() - started < 5_000);
}
test "a credential restored after init is the one the server starts with" {
const Fake = struct {
fn allow(_: *anyopaque, _: []const u8, _: []const u8, _: ?[]const u8) bool {
return true;
}
fn log(_: *anyopaque, _: []const u8, _: []const u8) void {}
};
var context: u8 = 0;
var e: Engine = undefined;
const servers = [_]Config{.{ .name = "web", .transport = .{ .http = .{ .url = "https://web.example.invalid/mcp" } } }};
try e.init(std.testing.io, std.testing.allocator, .{ .context = &context, .allow = Fake.allow, .log = Fake.log }, .{
.client = .{ .info = .{ .name = "test", .version = "0" } },
}, &servers);
defer e.deinit();
var saved = "{\"access\":\"w1\"}".*;
try e.restoreCredential("web", &saved);
// The engine keeps its own copy.
@memset(&saved, 0);
try std.testing.expectEqualStrings("{\"access\":\"w1\"}", e.slots.items[0].config.auth.saved.?);
try std.testing.expectError(error.UnknownServer, e.restoreCredential("nope", "{}"));
}
test "a stdio server's directory is part of its config, so a new one restarts it" {
const argv = [_][]const u8{ "node", "s.js" };
const a: Config = .{ .name = "x", .transport = .{ .stdio = .{ .argv = &argv, .cwd = "/a" } } };
+23 -4
View File
@@ -4,6 +4,7 @@ const acp_server = @import("acp/server.zig");
const jsonrpc = @import("acp/jsonrpc.zig");
const gateway_provider = @import("core/gateway/gateway_provider.zig");
const provider_set = @import("core/gateway/provider_set.zig");
const stream_provider = @import("core/agent/stream_provider.zig");
const agent_steps = @import("core/config/agent_steps.zig");
const context_contract = @import("core/workspace/context_contract.zig");
const host = @import("core/hosts/host.zig");
@@ -31,6 +32,24 @@ const max_input_bytes = 8 * 1024 * 1024;
const max_output_bytes = 8 * 1024 * 1024;
const max_output_message_bytes = 64 * 1024 * 1024;
const max_fetch_request_bytes = 8 * 1024 * 1024;
/// The largest request body a provider sends over the fetch bridge: its
/// budget less room for the request's method, URL, and headers, which hold
/// at most the URL and API key this addon accepts and the other headers.
const max_bridge_request_body_bytes = max_fetch_request_bytes - (max_url_bytes + max_api_key_bytes + 64 * 1024);
/// `provider` with its requests fit to the fetch bridge, so they leave out
/// their oldest images instead of failing.
fn bridgeBound(provider: stream_provider.Provider) stream_provider.Provider {
var bound = provider;
bound.max_request_bytes = max_bridge_request_body_bytes;
return bound;
}
fn configuredBridgeBundle(raw: ?*anyopaque, definition: *const configured_provider.Definition) provider_set.Bundle {
var bundle = host_chat_completions.bundle(raw, definition);
if (bundle.agent_stream) |stream| bundle.agent_stream = bridgeBound(stream);
return bundle;
}
const max_fetch_response_bytes = 8 * 1024 * 1024;
// One attachment holds one prompt image or one kernel checkpoint, the larger.
const max_attachment_bytes = agent_checkpoint.max_checkpoint_bytes;
@@ -686,23 +705,23 @@ const Runtime = struct {
.presentation = builtin_gateway.provider_bundle.presentation,
.auth_strategy = .vercel,
.fallback_model_capabilities_fn = builtin_gateway.provider_bundle.fallback_model_capabilities_fn,
.agent_stream = host_stream_provider.provider(&self.stream_context),
.agent_stream = bridgeBound(host_stream_provider.provider(&self.stream_context)),
.model_catalog = @import("gateway/host_model_catalog.zig").provider(&self.stream_context.transport),
});
providers.codex = .{
.presentation = provider_catalog.find(.codex),
.auth_strategy = .chatgpt,
.agent_stream = host_openai_codex.provider(&self.codex_context),
.agent_stream = bridgeBound(host_openai_codex.provider(&self.codex_context)),
.model_catalog = host_openai_codex.modelCatalogProvider(&self.codex_context),
};
providers.grok = .{
.presentation = provider_catalog.find(.grok),
.auth_strategy = .grok,
.agent_stream = host_xai_grok.provider(&self.grok_context),
.agent_stream = bridgeBound(host_xai_grok.provider(&self.grok_context)),
.model_catalog = host_xai_grok.modelCatalogProvider(&self.grok_context),
};
if (self.custom_context) |*context| {
providers.configured_fn = host_chat_completions.bundle;
providers.configured_fn = configuredBridgeBundle;
providers.configured_context = context;
}
acp_server.runWithTransport(
+5 -16
View File
@@ -5,14 +5,10 @@ const session_child_store = @import("../../core/session/session_child_store.zig"
const io_mod = @import("../../core/shared/io.zig");
const tool_dispatch = @import("../../core/tooling/tool_dispatch.zig");
const compactor = @import("../../core/compactor/compactor.zig");
const read_result_presentation = @import("../../core/tooling/read_tool_result_presentation.zig");
const Allocator = std.mem.Allocator;
const HandleNormalization = struct {
trimmed: []const u8,
suffix: []const u8,
};
pub const Input = struct {
handle: []u8,
selector: union(enum) {
@@ -130,20 +126,10 @@ fn inputDeinit(ptr: *anyopaque, alloc: Allocator) void {
alloc.destroy(input);
}
fn classifyHandleNormalization(handle: []const u8) HandleNormalization {
const trimmed = std.mem.trim(u8, handle, " \t\r\n");
const suffix = if ((std.mem.startsWith(u8, trimmed, "result-") or result_store.isImageHandle(trimmed)) and
std.mem.findScalar(u8, trimmed, '.') == null)
".txt"
else
"";
return .{ .trimmed = trimmed, .suffix = suffix };
}
pub fn validate(ctx: tool_dispatch.DispatchContext, erased: tool_dispatch.ToolInput) tool_dispatch.DispatchError!?[]u8 {
const input = erased.as(Input);
if (input.selector == .search) return null;
const normalization = classifyHandleNormalization(input.handle);
const normalization = read_result_presentation.classifyHandleNormalization(input.handle);
if (normalization.trimmed.len == 0) return try ctx.allocator.dupe(u8, "read_tool_result field \"handle\" must not be empty");
var record_buffer: compactor.RecordFileBuffer = undefined;
if (compactor.recordFile(&record_buffer, normalization.trimmed)) |file| {
@@ -279,6 +265,9 @@ pub fn isIrreversible(_: tool_dispatch.ToolInput) bool {
return false;
}
/// Labels each call by what it reads. See `read_tool_result_presentation.zig`.
pub const presentation = read_result_presentation.presentation;
test "read_tool_result decodes range and query inputs" {
const alloc = std.testing.allocator;
const decoded_range = try decode(.{ .allocator = alloc }, "{\"handle\":\"h.txt\",\"start_byte\":2,\"byte_count\":9}");
+5 -1
View File
@@ -6356,6 +6356,8 @@ printf '%s' ${JSON.stringify(trailingMarker)} > ${JSON.stringify(effectPath)}
const pane = await tui.waitForPane((text) => hasEmptyComposer(text) && (text.includes("RETRIEVAL_TURN_COMPLETE") || text.includes("request failed:")), 30000);
expect(pane).not.toContain("request failed:");
expect(pane).toContain("RETRIEVAL_TURN_COMPLETE");
// The read names the command whose saved output it searches.
expect(pane).toContain("cat source.txt output for \"RETRIEVAL_MATCH\"");
if (trigger === "manual") {
await compactAndWait(tui, root, 20000);
}
@@ -6391,7 +6393,9 @@ printf '%s' ${JSON.stringify(trailingMarker)} > ${JSON.stringify(effectPath)}
expect(readFileSync(join(toolResults, "compacted-T1.txt"), "utf8")).toContain(sourceHandle);
const resumed = await runFx(["ask", "--json", "--resume-id", sessionId, "Recover the clipped tail from the saved retrieval without rerunning the command."], { cwd: root.workspace, env, timeoutMs: 30000 });
expect(resumed.code).toBe(0);
expect(resumed.stderr).toBe("Reading tool result\n");
// Compaction folded the command into T1, so the read has no call
// left to name.
expect(resumed.stderr).toBe("Reading more command output\n");
expect(JSON.parse(resumed.stdout).final_output).toBe("RETRIEVAL_RESTART_COMPLETE");
expect(gateway.requests).toHaveLength(8);
expect(readFileSync(join(root.workspace, "effects.txt"), "utf8")).toBe("once\n");
+6 -1
View File
@@ -2881,7 +2881,12 @@ test.skipIf(!tmuxAvailable())("text typed while the app answers steers that turn
await resumed.session.sendText("Continue after the steered turn.");
await resumed.session.waitForText("AFTER_STEERING_RESUME", TIMEOUT);
await quitApp(resumed);
expect(gateway.requests.at(-1)!.body).toContain(steering);
// The resumed session sends the steering as the live update the model
// saw, not as a plain user message.
const continued = gateway.requests.at(-1)!.body;
expect(continued).toContain(steering);
expect(continued).toContain("<user_steering>");
expect(continued.indexOf("<user_steering>")).toBeLessThan(continued.indexOf(steering));
expectWholeLog(fixture, id);
} finally {
hold.release?.();
+99 -1
View File
@@ -3178,7 +3178,7 @@ describe.skipIf(!tmuxAvailable())("TUI gateway stream lifecycle", () => {
const savedSession = readSavedSession();
expect(pane).toContain(finalText);
expect(scrollback).toContain("Failed tool result");
expect(scrollback).toContain("Failed more output");
expect(continuation).toContain(expectedFailure);
expect(savedSession).toContain('"status":"failure"');
expect(savedSession).toContain(expectedFailure);
@@ -5186,6 +5186,104 @@ describe.skipIf(!tmuxAvailable())("TUI gateway stream lifecycle", () => {
TIMEOUT,
);
test(
"parallel saved-output reads name the command they read",
async () => {
root = realpathSync(mkdtempSync(join(tmpdir(), "fx-parallel-saved-reads-")));
const home = join(root, "home");
const workspace = join(root, "workspace");
const stderrPath = join(root, "stderr.log");
const tracePath = join(root, "trace.log");
mkdirSync(join(home, ".fx"), { recursive: true });
mkdirSync(workspace);
writeFileSync(join(home, ".fx", "settings.json"), "{}");
// Large enough that fx saves the command output behind a handle.
const lines = Array.from({ length: 4000 }, (_, index) => `SAVED_LINE_${String(index).padStart(5, "0")}`);
lines[2500] = "PARALLEL_READ_MATCH";
writeFileSync(join(workspace, "source.txt"), `${lines.join("\n")}\n`);
const readGateway = startDynamicFakeGateway((body) => {
const request = JSON.parse(body) as {
prompt: Array<{ role: string; content: unknown }>;
};
const parts = request.prompt.flatMap((message) =>
Array.isArray(message.content) ? message.content : [],
) as Array<Record<string, unknown>>;
const output = (callId: string): string | null => {
const part = parts.find((candidate) =>
candidate.type === "tool-result" && candidate.toolCallId === callId,
);
return part ? contentText(part.output) : null;
};
if (output("parallel_read_query") !== null && output("parallel_read_range") !== null) {
return fakeGatewayFinalText("PARALLEL_READS_FINISHED");
}
const source = output("parallel_read_source");
if (source === null) {
return fakeGatewayToolCall("parallel_read_source", "shell", {
request: { action: "run", command: "cat source.txt", profile: "clean", yield_time_ms: 30000 },
});
}
const handle = (JSON.parse(source) as { full_output_handle: string | null }).full_output_handle;
if (!handle) return new Response("missing saved output handle", { status: 500 });
return fakeGatewaySse([
{
type: "tool-call",
toolCallId: "parallel_read_query",
toolName: "read_tool_result",
input: { request: { handle, query: "PARALLEL_READ_MATCH" } },
},
{
type: "tool-call",
toolCallId: "parallel_read_range",
toolName: "read_tool_result",
input: { request: { handle, start_byte: 20000, byte_count: 1024 } },
},
{
type: "finish",
finishReason: { unified: "tool-calls", raw: "tool-calls" },
},
]);
}, { classifierDecision: "clear" });
gateway = readGateway;
session = await TmuxSession.create({
cwd: workspace,
width: 110,
height: 35,
stderrPath,
env: {
HOME: home,
AI_GATEWAY_API_KEY: "fake-parallel-saved-reads-key",
VERCEL_OIDC_TOKEN: undefined,
FX_DISABLE_KEYCHAIN: "1",
FX_SOUND: "0",
FX_AUTO_UPGRADE: "0",
FX_PERMISSION_MODE: "auto",
FX_GATEWAY_BASE_URL: readGateway.baseUrl,
FX_GATEWAY_CHAT_URL: readGateway.chatUrl,
FX_E2E_GATEWAY_CHAT_URL: readGateway.chatUrl,
FX_MODEL: MODEL,
FX_TRACE_LOG: tracePath,
FX_TRACE_SCOPES: "tool",
},
});
await session.waitForComposer(TIMEOUT);
await session.sendText("Run the fixture, then search and page its saved output.");
await session.waitForText("PARALLEL_READS_FINISHED", TIMEOUT);
const scrollback = await session.captureFullScrollback();
// Both reads ran as one parallel group and still name their source.
expect(readFileSync(tracePath, "utf8")).toMatch(/parallel_tool_group_start[^\n]*count=2/);
expect(scrollback).toContain("Searched cat source.txt output for \"PARALLEL_READ_MATCH\"");
expect(scrollback).toContain("Read more output from cat source.txt");
expect(readFileSync(stderrPath, "utf8")).toBe("");
await session.sendText("/quit");
expect(await session.waitForSessionEnd(TIMEOUT)).toBe(true);
},
TIMEOUT * 2,
);
test(
"separate provider messages render as separate paragraphs",
async () => {
+42 -2
View File
@@ -3397,7 +3397,13 @@ describe("request size budget", () => {
type SizeRequest = { body: string; images: string[]; tools: boolean; status: number };
function startSizeCappedGateway(rejection: { status: number; body: string }, images: Map<string, string>) {
// Refuses every request over `cap`. By default each answer asks for the next
// image with read_file; `describe` answers at once instead.
function startSizeCappedGateway(
rejection: { status: number; body: string },
images: Map<string, string>,
{ cap = SIZE_CAP_BYTES, answer = "read" }: { cap?: number; answer?: "read" | "describe" } = {},
) {
const byData = new Map([...images].map(([name, data]) => [data, name]));
const requests: SizeRequest[] = [];
let issued = 0;
@@ -3415,12 +3421,13 @@ describe("request size budget", () => {
const parsed = JSON.parse(body) as { tools?: unknown[] };
const tools = Array.isArray(parsed.tools) && parsed.tools.length > 0;
const carried = nativeFileParts(body).map((part) => byData.get(part.data.data) ?? "unknown");
const status = body.length > SIZE_CAP_BYTES ? rejection.status : 200;
const status = body.length > cap ? rejection.status : 200;
requests.push({ body, images: carried, tools, status });
if (status !== 200) {
return new Response(rejection.body, { status, headers: { "content-type": "application/json" } });
}
if (!tools) return sseText("Summary: the user asked fx to read six images.");
if (answer === "describe") return sseText("done: described the images");
if (issued < SIZE_IMAGE_COUNT) {
issued += 1;
return sseToolCall("read_file", { path: `img0${issued}.png` }, `call_img_${issued}`);
@@ -3498,4 +3505,37 @@ describe("request size budget", () => {
}
}, TIMEOUT * 3);
}
test("a size limit two retries can't reach still leaves out images instead of compacting", async () => {
const root = createIsolatedRoot();
const images = writeSizeImages(root.workspace);
// Six images of about 1 MB in one prompt against a 1.5 MB limit: each
// refusal lowers fx's limit by a quarter, so the request fits only after
// a third refusal leaves out all but the newest image.
const cap = 1_500_000;
const gateway = startSizeCappedGateway({ status: 413, body: payloadTooLargeBody }, images, { cap, answer: "describe" });
try {
const imageArgs = [...images.keys()].flatMap((name) => ["--image", join(root.workspace, name)]);
const result = await runFx(["ask", "--json", "--auto", "--no-color", ...imageArgs, "Describe these images."], {
cwd: root.workspace,
env: gateway.env(root),
timeoutMs: TIMEOUT * 2,
});
const json = parseFxJson(result);
expect(json.output).toContain("done: described the images");
const rejected = gateway.requests.filter((request) => request.status !== 200);
expect(rejected.length).toBeGreaterThan(2);
// No summary request: the conversation was not compacted.
expect(gateway.requests.every((request) => request.tools)).toBe(true);
const accepted = gateway.requests.at(-1)!;
expect(accepted.status).toBe(200);
expect(accepted.body.length).toBeLessThanOrEqual(cap);
expect(accepted.images).toEqual(["img06.png"]);
expect(accepted.body).toContain(SIZE_NOTICE);
} finally {
gateway.stop();
rmSync(root.root, { recursive: true, force: true });
}
}, TIMEOUT * 3);
});
+2 -2
View File
@@ -10,12 +10,12 @@ TLA+ specifications for fx, checked with TLC, together with the scripts and Bun
| `indexer/` | When a scanned root's listing may be reused, merging parallel grep results, the file picker's loading row, and listing snapshots (`src/core/indexer/`). | `indexer/check.sh` |
| `mcp/` | The MCP-v2 engine (`src/mcp/`): wire format, requests, stdio, version detection, the tool catalog, both HTTP transports, input-required rounds, authorization, subscriptions, elicitation, and the engine. `rules.md` lists the MCP specification rules the models cite by ID. | `mcp/check.sh`, then the drivers in `drivers/mcp/` |
| `mcp_host/` | fx's MCP host (`src/core/mcp_host/`): host lifecycle, saved credentials, logins shared between processes, sign-in, and elicitation routing. | `mcp_host/check.sh`, `bun test drivers/mcp_host/` |
| `sdk/` | libfx durable sessions (`DurableSession.tla`), agent names and routing (`AgentRouting.tla`), tenants whose agents share one process (`TenantIsolation.tla`), steering (`DurableSteer.tla`), and the provider credential edge (`ProviderAuth.tla`). `check.sh` also writes the TLC state graphs the drivers walk. | `sdk/check.sh`, then the drivers |
| `sdk/` | libfx durable sessions (`DurableSession.tla`), how a session object takes prompts in (`PromptIntake.tla`), images saved once by name (`ImageStore.tla`), agent names and routing (`AgentRouting.tla`), tenants whose agents share one process (`TenantIsolation.tla`), steering (`DurableSteer.tla`), and the provider credential edge (`ProviderAuth.tla`). `check.sh` also writes the TLC state graphs the drivers walk. | `sdk/check.sh`, then the drivers |
| `session_manager/` | The session log, turn lifecycle, writer lock, snapshots, fork, subagents, catalog, host wiring, the side-file move, and v1 conversion. `rollout/` holds the models of the changes that moved fx onto the session manager. `system-design.md` is the design record the specs check and defines the `D1` to `D62` decision numbers that code comments cite. | `session_manager/check.sh`, `session_manager/rollout/run.py`, `bun test drivers/session_manager/` |
| `shell_command/` | How an agent git command is admitted and run (`src/core/shell_command/safe_git.zig`), and one model turn through the shell tool, for the current design and a redesign. | `shell_command/check.sh`, `bun test drivers/shell_command/` |
| `terminal/` | The in-memory shell snapshot owner (`src/core/terminal/shell_snapshot.zig`) and the lifecycle of hosted terminals. | `terminal/check.sh` |
| `usage/` | `/usage` refreshes and the shared usage cache (`src/usage/`): `LeanUsage.tla` and `UsageModel.tla`. | `usage/check.sh`, then the drivers in `drivers/usage/` |
| `drivers/` | The Bun drivers. They run the real fx binary or SDK code, record what happens, and have TLC check each run against its model. `drive.mjs`, `drive-agents.mjs`, `drive-tenants.mjs`, and `drive-steer.mjs` walk the SDK code along the `sdk/` state graphs. | See each folder's `check.sh` |
| `drivers/` | The Bun drivers. They run the real fx binary or SDK code, record what happens, and have TLC check each run against its model. `drive.mjs`, `drive-intake.mjs`, `drive-images.mjs`, `drive-agents.mjs`, `drive-tenants.mjs`, and `drive-steer.mjs` walk the SDK code along the `sdk/` state graphs. | See each folder's `check.sh` |
| `lib/` | `tlc.sh`, which finds TLC and Java and runs a table of configs with their expected verdicts, `tlc.py`, which asks `tlc.sh` for TLC and Java on behalf of the Python checkers, and `run-tlc.sh`, which runs TLC once for the drivers. | |
## Requirements
+407
View File
@@ -0,0 +1,407 @@
#!/usr/bin/env bun
// Drives real libfx along paths of the TLC state graph of ImageStore.tla,
// and checks after every step that the code is in a state the model allows.
//
// bun tests/tla/drivers/drive-images.mjs [--walks N] [--seed S] [--backend native|wasm]
//
// Each walk keeps one session in a memory persistence with an image store,
// first written by an engine without one, so its first image is saved
// inline as an older libfx saved it. Then the walk opens an engine on it and
// makes the prompts it chooses, each with image a or b; a turn's tool may
// return image a or b as well, which the engine stores before the core gets
// the result. Every image write
// waits until the driver lets it land or fail; every model request waits
// until the driver ends its turn. A crash leaves the engine behind at once:
// its writes in flight never land, and nothing it does later reaches the
// store. The driver reopens the session, resuming a turn the crash left
// open, and may delete an image from the store. After each step the driver
// waits until the code is quiet and moves to the model states that show the
// same prompts, store, open state and held request. On every step, nothing
// saved may name an image the store does not hold (until one is deleted),
// only the old image, or a tool's image the store refused, may be saved
// inline, and a model request must carry
// exactly the history the model holds.
//
// `tests/tla/sdk/check.sh` writes graphs/images.dot from ImageStore-drive.cfg.
import { strict as assert } from "node:assert";
import { readFile } from "node:fs/promises";
import { resolve } from "node:path";
import { fileURLToPath } from "node:url";
import { createFxEngine, createMemoryPersistence } from "../../../sdk/node.js";
import { loadGraph, walkGraph } from "./graph.mjs";
const args = process.argv.slice(2);
const option = (name, fallback) => {
const index = args.indexOf(`--${name}`);
return index >= 0 ? args[index + 1] : fallback;
};
const walks = Number(option("walks", 40));
let seed = Number(option("seed", 1));
const backend = option("backend", "native");
if (!["native", "wasm"].includes(backend)) throw new Error("--backend is native or wasm");
const verbose = args.includes("--verbose");
const here = fileURLToPath(new URL(".", import.meta.url));
const addon = resolve(here, "../../../zig-out/lib/libfx.node");
const wasm = backend === "wasm" ? await readFile(resolve(here, "../../../zig-out/bin/fx-core.wasm")) : undefined;
const quietMs = 150;
const stepTimeoutMs = 10_000;
// The model's window, which every engine the walk opens keeps too.
const windowImages = Number(/^\s*Window = (\d+)$/m.exec(await readFile(resolve(here, "../sdk/ImageStore-drive.cfg"), "utf8"))[1]);
const imageWindow = Symbol.for("libfx.imageWindow");
const decoder = new TextDecoder();
const sleep = (ms) => new Promise((done) => setTimeout(done, ms));
const deferred = () => {
let resolveIt;
let rejectIt;
const promise = new Promise((resolveP, rejectP) => { resolveIt = resolveP; rejectIt = rejectP; });
promise.catch(() => {});
return { promise, resolve: resolveIt, reject: rejectIt };
};
const random = () => {
seed = (seed * 1103515245 + 12345) % 2147483648;
return seed / 2147483648;
};
// The model's images, as PNG bytes the kernel accepts, each its own.
const png = Buffer.from("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+jP0cAAAAASUVORK5CYII=", "base64");
const labels = ["o", "a", "b"];
const bytesOf = Object.fromEntries(labels.map((label) => [label, new Uint8Array(Buffer.concat([png, Buffer.from(`image-${label}-`.repeat(400))]))]));
const base64Of = Object.fromEntries(labels.map((label) => [label, Buffer.from(bytesOf[label]).toString("base64")]));
const nameOf = Object.fromEntries(await Promise.all(labels.map(async (label) => [label, Buffer.from(await crypto.subtle.digest("SHA-256", bytesOf[label])).toString("hex")])));
const labelOfName = Object.fromEntries(labels.map((label) => [nameOf[label], label]));
const labelOfBase64 = Object.fromEntries(labels.map((label) => [base64Of[label], label]));
const labelOfBytes = (bytes) => labelOfBase64[Buffer.from(bytes).toString("base64")] ?? "?";
// What a session saved, decoded: the names its records and checkpoint give,
// and the images they hold inline.
function durableBytes(value) {
if (typeof value === "string") return Buffer.from(value, "utf8");
if (value?.encoding === "base64") return Buffer.from(value.data, "base64");
return null;
}
function savedImages(store) {
const named = new Set();
const inline = [];
const visit = (value, blobs) => {
if (Array.isArray(value)) {
for (const item of value) visit(item, blobs);
return;
}
if (!value || typeof value !== "object") return;
// One the session dropped keeps its name alone, which needs no bytes: an
// image held inline takes the name of its bytes when it leaves.
if (typeof value.inline_ref === "string" && value.dropped !== true) named.add(value.inline_ref);
if (value.inline_data !== undefined) inline.push(durableBytes(value.inline_data));
// A tool's image held inline: its base64 data.
if (value.type === "image" && typeof value.data === "string" && typeof value.mimeType === "string") inline.push(Buffer.from(value.data, "base64"));
if (Number.isInteger(value.inline_blob)) inline.push(blobs[value.inline_blob]);
for (const item of Object.values(value)) visit(item, blobs);
};
for (const record of store.records) visit(JSON.parse(decoder.decode(record.data)), []);
if (store.checkpoint) {
// FXSN: magic, version, seq, turn, id length and id, an open turn from
// version 2 on, then the FXCP checkpoint.
const bytes = Buffer.from(store.checkpoint.data);
const version = bytes[4];
let at = 4 + 1 + 8 + 8;
at += 1 + bytes[at];
if (version >= 2) {
const length = bytes.readUInt32LE(at);
visit(JSON.parse(bytes.subarray(at + 4, at + 4 + length).toString("utf8")), []);
at += 4 + length;
}
const checkpoint = bytes.subarray(at);
const payload = checkpoint.subarray(4 + 2 + 2 + 4 + 32);
if (checkpoint.readUInt16LE(4) === 1) {
visit(JSON.parse(payload.toString("utf8")), []);
} else {
const jsonLength = payload.readUInt32LE(0);
const json = JSON.parse(payload.subarray(4, 4 + jsonLength).toString("utf8"));
const blobs = [];
let offset = 4 + jsonLength;
const count = payload.readUInt32LE(offset);
offset += 4;
for (let index = 0; index < count; index += 1) {
const length = payload.readUInt32LE(offset);
blobs.push(payload.subarray(offset + 4, offset + 4 + length));
offset += 4 + length;
}
visit(json, blobs);
}
}
return { named, inline };
}
const vision = "images/vision";
const usage = { inputTokens: { total: 1 }, outputTokens: { total: 1 } };
// The model calls the turn's tool for image `label`.
const callShot = (p, label) => new Response(
[{ type: "tool-call", toolCallId: `call-${p}`, toolName: "shot", input: { label } }, { type: "finish", finishReason: { unified: "tool-calls", raw: "tool-calls" }, usage }, "[DONE]"]
.map((frame) => `data: ${typeof frame === "string" ? frame : JSON.stringify(frame)}\n\n`).join(""),
{ headers: { "content-type": "text/event-stream" } },
);
// The tool returns the image it is asked for.
const shot = {
name: "shot",
description: "Returns an image",
inputSchema: { type: "object", properties: { label: { type: "string" } }, required: ["label"] },
execute: async ({ label }) => ({ type: "libfx.tool-result", text: `image ${label}`, images: [{ type: "image", data: base64Of[label], mimeType: "image/png" }] }),
};
const answer = () => new Response(
[{ type: "text-delta", id: "a", delta: "seen" }, { type: "finish", finishReason: { unified: "stop", raw: "stop" }, usage }, "[DONE]"]
.map((frame) => `data: ${typeof frame === "string" ? frame : JSON.stringify(frame)}\n\n`).join(""),
{ headers: { "content-type": "text/event-stream" } },
);
// libfx's own key for a persistence's image store, which durable sessions
// give the engines they run.
const persistenceImages = Symbol.for("libfx.persistenceImages");
async function harness(n) {
const store = createMemoryPersistence();
// The image store: each image once, by name.
const storedImages = new Map();
const problems = [];
// Held image writes by name, and the model request held now.
const imageGates = new Map();
let held = null;
let answerNow = false;
// Each engine's own fetch: a crashed engine reaches the model no more.
const fetchFor = (current) => async (input, init) => {
if (current && !current.alive) throw new Error("crashed");
const request = input instanceof Request ? input : new Request(input, init);
if (request.method === "GET") return Response.json({ object: "list", data: [{ id: vision, type: "language", tags: ["vision", "file-input"] }] });
const prompt = JSON.parse(await request.text()).prompt;
const users = prompt.filter((message) => message.role === "user");
const text = users.map((message) => (Array.isArray(message.content) ? message.content : []).filter((part) => part.type === "text").map((part) => part.text).join("")).join("\n");
const numbers = [...text.matchAll(/prompt-(\d+)/g)].map((match) => Number(match[1]));
const images = users.flatMap((message) => (Array.isArray(message.content) ? message.content : []).filter((part) => part.type === "file"))
.map((part) => labelOfBase64[part.data?.data] ?? "?");
if (answerNow) return answer();
const response = deferred();
held = { p: numbers.at(-1) ?? 0, images, response, life: current };
const signal = init?.signal ?? request.signal;
signal?.addEventListener("abort", () => response.reject(signal.reason ?? new Error("aborted")), { once: true });
return response.promise;
};
// Each engine's view of the store, until it crashes.
let life = null;
const persistenceFor = (current) => ({
load: () => (current.alive ? store.load() : Promise.reject(new Error("crashed"))),
append: (input) => (current.alive ? store.append(input) : Promise.reject(new Error("crashed"))),
saveCheckpoint: (input) => (current.alive ? store.saveCheckpoint(input) : Promise.reject(new Error("crashed"))),
[persistenceImages]: {
async put(name, bytes) {
if (!current.alive) throw new Error("crashed");
const gate = imageGates.get(name);
if (gate) {
imageGates.delete(name);
const lands = await gate.promise;
if (!lands) throw new Error("injected image write failure");
// A write the crash caught in flight never lands.
if (!current.alive) return new Promise(() => {});
}
if (!storedImages.has(name)) storedImages.set(name, bytes.slice());
},
get: async (name) => {
if (!current.alive) throw new Error("crashed");
return storedImages.get(name)?.slice();
},
},
});
const engineOptions = (persistence, current = null) => ({
backend,
nativeAddon: addon,
...(wasm ? { wasm } : {}),
apiKey: "images-key",
model: vision,
fetch: fetchFor(current),
persistence,
checkpointAfterBytes: 0,
tools: [shot],
[imageWindow]: { images: windowImages },
});
// The session an older libfx saved: its image inline, in a record and in a
// checkpoint.
{
answerNow = true;
const old = await createFxEngine(engineOptions({ load: () => store.load(), append: (input) => store.append(input), saveCheckpoint: (input) => store.saveCheckpoint(input) }));
const turn = old.prompt([{ type: "text", text: "old" }, { type: "image", data: bytesOf.o, mimeType: "image/png" }]);
for await (const _ of turn) {}
assert.equal((await turn.result).stopReason, "end_turn");
await old.close();
answerNow = false;
}
let engine = null;
let up = "down";
let lost = false;
let made = 0;
const ph = Array(n).fill("unmade");
const gates = {};
// Each turn's tool image write, held until the driver lets it land or fail,
// and the images the store refused, which the session holds inline.
const toolGates = {};
const refused = new Set();
const watch = (p, turn) => {
turn.result.then(
(result) => { if (result.stopReason === "end_turn") ph[p - 1] = "ended"; },
(error) => { if (error?.message === "injected image write failure") ph[p - 1] = "failed"; },
);
};
async function perform(act) {
const [name, p, image] = act;
switch (name) {
case "Make": {
made = p;
ph[p - 1] = "putting";
gates[p] = deferred();
imageGates.set(nameOf[image], { promise: gates[p].promise });
const turn = engine.prompt([{ type: "text", text: `prompt-${p}` }, { type: "image", data: bytesOf[image].slice(), mimeType: "image/png" }]);
void turn.result.catch(() => {});
watch(p, turn);
return;
}
case "PutLand": gates[p].resolve(true); return;
case "PutFail": gates[p].resolve(false); return;
case "ToolCall": {
if (!held || held.p !== p) throw new Error(`no model request is held for prompt ${p}`);
toolGates[p] = { gate: deferred(), image };
imageGates.set(nameOf[image], { promise: toolGates[p].gate.promise });
const { response } = held;
held = null;
response.resolve(callShot(p, image));
return;
}
case "ToolPutLand": toolGates[p].gate.resolve(true); return;
case "ToolPutFail": {
refused.add(toolGates[p].image);
toolGates[p].gate.resolve(false);
return;
}
case "TurnEnd": {
if (!held || held.p !== p) throw new Error(`no model request is held for prompt ${p}`);
const { response } = held;
held = null;
response.resolve(answer());
return;
}
case "Crash": {
life.alive = false;
// The crash ends the engine's held request and image writes with it.
if (held) {
held.response.reject(new Error("crashed"));
held = null;
}
for (let q = 1; q <= n; q += 1) if (ph[q - 1] === "putting") ph[q - 1] = "lost";
for (const gate of Object.values(gates)) gate.resolve(true);
for (const { gate } of Object.values(toolGates)) gate.resolve(true);
const left = engine;
engine = null;
up = "down";
void left.close().catch(() => {});
return;
}
case "OpenBegin": {
life = { alive: true };
try {
engine = await createFxEngine(engineOptions(persistenceFor(life), life));
} catch (error) {
if (error?.code !== "FX_JOURNAL_IMAGE_MISSING") throw error;
up = "failed";
return;
}
up = "open";
// A turn the crash left open continues.
const resumed = engine.resume();
if (resumed) {
const open = ph.findIndex((phase) => phase === "started") + 1;
if (open === 0) problems.push("the engine resumed a turn the model has no started prompt for");
void resumed.result.catch(() => {});
watch(open, resumed);
}
return;
}
case "Lose": {
lost = true;
storedImages.delete(nameOf[p]);
return;
}
default: throw new Error(`the driver cannot take ${name}`);
}
}
function observe() {
// A prompt whose model request is held has started.
if (held && held.life === life && ph[held.p - 1] !== "ended") ph[held.p - 1] = "started";
const { named, inline } = savedImages(store);
if (!lost) for (const name of named) if (!storedImages.has(name)) problems.push(`a record or checkpoint names ${labelOfName[name] ?? name}, which the store does not hold`);
for (const bytes of inline) {
const label = labelOfBytes(bytes);
if (label !== "o" && !refused.has(label)) problems.push(`image ${label} is saved inline`);
}
return {
made,
ph: [...ph],
up,
store: [...storedImages.keys()].map((name) => labelOfName[name] ?? "?").sort(),
held: held && held.life === life ? held.images : null,
};
}
async function settle() {
let last = JSON.stringify(observe());
let since = Date.now();
const begun = Date.now();
while (Date.now() - since < quietMs) {
await sleep(15);
const now = JSON.stringify(observe());
if (now !== last) { last = now; since = Date.now(); }
if (Date.now() - begun > stepTimeoutMs) break;
}
return observe();
}
async function close() {
if (held) held.response.resolve(answer());
for (const gate of Object.values(gates)) gate.resolve(true);
for (const { gate } of Object.values(toolGates)) gate.resolve(true);
if (engine) await Promise.race([engine.close().catch(() => {}), sleep(3000)]);
}
return { perform, settle, close, problems };
}
// ---------------------------------------------------------------------------
// Walks.
const graph = await loadGraph(resolve(here, "../sdk/graphs", "images.dot"));
const n = graph.states.get(graph.init).ph.length;
const autoActs = new Set(["Send", "TurnStart", "ToolResult", "Checkpoint", "Stage", "OpenMissing", "OpenOk"]);
// While a turn's tool stores its image or hands it on, no request is held.
const toolRunning = (state) => state.tph.some((phase) => ["called", "put", "refused"].includes(phase));
const view = (state) => JSON.stringify({
made: state.made,
ph: state.ph,
up: state.up,
store: [...state.store].sort(),
// A model request carries the images the core keeps; a dropped one is a note.
held: state.up === "open" && state.ph.includes("started") && !toolRunning(state) ? state.mem.filter((entry) => !entry.dropped).map((entry) => entry.i) : null,
});
const { steps, covered, totalEdges } = await walkGraph({
graph,
walks,
random,
verbose,
autoActs,
view,
nameOf: (act) => act[0],
sameAct: (a, b) => JSON.stringify(a) === JSON.stringify(b),
label: (act) => `${act[0]}(${act.slice(1).join(",")})`,
open: () => harness(n),
});
if (!process.exitCode) console.log(`ok images: ${walks} walks, ${steps} steps matched the model; ${covered} of ${totalEdges} edges covered (${backend})`);
process.exit(process.exitCode ?? 0);
+466
View File
@@ -0,0 +1,466 @@
#!/usr/bin/env bun
// Drives real libfx along paths of the TLC state graph of PromptIntake.tla,
// and checks after every step that the code is in a state the model allows.
//
// bun tests/tla/drivers/drive-intake.mjs [--walks N] [--seed S] [--durability memory|world]
// [--harness fx|scripted] [--backend native|wasm]
//
// Each walk opens one agent and one session object, which makes the prompts
// the walk chooses: text, or a prompt with an image to prepare: base64 data
// whose write to the session's image store waits until the driver lets it
// land or fail. Every log write
// of a prompt, or of a prompt's cancel, waits until the driver lets it land
// or fail. With the fx harness each turn runs on a real fx engine, whose
// model request the driver holds until it ends the turn; with the scripted
// harness a turn is a stand-in the driver ends. After each step the driver
// waits until the code is quiet, reads back what it shows, and moves to the
// model states that show the same. Every stored prompt must name its image,
// already in the image store with the caller's bytes; and every model
// request must carry its prompt's image unchanged.
//
// `tests/tla/sdk/check.sh` writes graphs/intake.dot from PromptIntake-drive.cfg.
import { mkdtemp, readFile, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import { isDeepStrictEqual } from "node:util";
import { createFxAgent, memory } from "../../../sdk/node.js";
import { createDurableAgentFactory } from "../../../sdk/durable.js";
import { loadGraph, walkGraph } from "./graph.mjs";
const args = process.argv.slice(2);
const option = (name, fallback) => {
const index = args.indexOf(`--${name}`);
return index >= 0 ? args[index + 1] : fallback;
};
const walks = Number(option("walks", 40));
let seed = Number(option("seed", 1));
const durabilityKind = option("durability", "memory");
const harnessKind = option("harness", "fx");
const backend = option("backend", "native");
if (!["memory", "world"].includes(durabilityKind)) throw new Error("--durability is memory or world");
if (!["fx", "scripted"].includes(harnessKind)) throw new Error("--harness is fx or scripted");
const verbose = args.includes("--verbose");
const here = fileURLToPath(new URL(".", import.meta.url));
const addon = resolve(here, "../../../zig-out/lib/libfx.node");
const wasm = harnessKind === "fx" && backend === "wasm" ? await readFile(resolve(here, "../../../zig-out/bin/fx-core.wasm")) : undefined;
const { appWorldAt, closeAppWorlds } = durabilityKind === "world" ? await import("../../../sdk/tests/app-world.mjs") : {};
// How long the code must show nothing new before a step has settled. An fx
// engine takes a moment to open, and a World's stream a moment to read.
const quietMs = (harnessKind === "fx" ? 200 : 60) + (durabilityKind === "world" ? 350 : 0);
const stepTimeoutMs = 10_000;
const encoder = new TextEncoder();
const decoder = new TextDecoder();
const sleep = (ms) => new Promise((done) => setTimeout(done, ms));
const until = async (check, ms = stepTimeoutMs) => {
const started = Date.now();
while (!check()) {
if (Date.now() - started > ms) return false;
await sleep(5);
}
return true;
};
const base64 = (bytes) => Buffer.from(bytes).toString("base64");
const deferred = () => {
let resolveIt;
let rejectIt;
const promise = new Promise((resolveP, rejectP) => { resolveIt = resolveP; rejectIt = rejectP; });
promise.catch(() => {});
return { promise, resolve: resolveIt, reject: rejectIt };
};
// A PNG the kernel accepts, followed by bytes that name the walk and prompt,
// so each prompt's image is its own.
const png = Buffer.from("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+jP0cAAAAASUVORK5CYII=", "base64");
const imageOf = (walk, i) => new Uint8Array(Buffer.concat([png, Buffer.from(`walk-${walk}-prompt-${i}`.padEnd(24, "."))]));
const nameOf = async (bytes) => Buffer.from(await crypto.subtle.digest("SHA-256", bytes)).toString("hex");
// An image's stream on a World, as libfx names it.
const imageStreamSuffix = (name) => `_user_${Buffer.from(`libfx-image-${name}`).toString("base64url")}`;
// A durability whose log writes of prompts and their cancels wait for the
// driver, which lets each land or fail, as do the image writes it is told to
// hold (`holdImage`). `images` has the bytes of every image write that
// landed, by name. Works on memory() and on a World.
function heldInputs(durability) {
const held = new Map();
const landed = [];
const images = new Map();
const heldImages = new Map();
let open = false;
const throughImage = async (name, bytes, write) => {
const hold = heldImages.get(name);
if (hold && !open) {
heldImages.delete(name);
if (!(await hold.promise)) throw Object.assign(new Error("injected image write failure"), { code: "FX_TEST_FAULT" });
}
const result = await write();
images.set(name, Buffer.from(bytes).toString("base64"));
return result;
};
const keyOf = (entry) => {
if (entry?.k !== "input") return null;
if (entry.type === "prompt") return `prompt:${entry.messageId}`;
if (entry.type === "cancel" && typeof entry.target === "string") return `cancel:${entry.target}`;
return null;
};
const through = async (entry, write) => {
const key = keyOf(entry);
if (key !== null && !open) {
const lands = await new Promise((go) => held.set(key, { go }));
held.delete(key);
if (!lands) throw Object.assign(new Error("injected log write failure"), { code: "FX_TEST_FAULT" });
}
const result = await write();
if (key !== null) landed.push(entry);
return result;
};
const control = {
held,
landed,
images,
// The next write of the image named `name` waits for `settleImage`.
holdImage(name) {
const gate = {};
gate.promise = new Promise((go) => { gate.go = go; });
heldImages.set(name, gate);
return gate;
},
async settle(key, lands) {
if (!(await until(() => held.has(key)))) throw new Error(`no log write is held for ${key}`);
held.get(key).go(lands);
},
openAll() {
open = true;
for (const item of [...held.values()]) item.go(true);
for (const gate of [...heldImages.values()]) gate.go(true);
},
};
if (typeof durability.create === "function") {
const create = durability.create;
const wrapped = new WeakMap();
return [{
...durability,
create() {
const backend = create();
if (!wrapped.has(backend)) {
wrapped.set(backend, {
...backend,
async session(id) {
const log = await backend.session(id);
return {
...log,
append: (entry) => through(entry, () => log.append(entry)),
images: { put: (name, bytes) => throughImage(name, bytes, () => log.images.put(name, bytes)), get: (name) => log.images.get(name) },
};
},
});
}
return wrapped.get(backend);
},
}, control];
}
const bound = (target, key) => {
const value = Reflect.get(target, key);
return typeof value === "function" ? value.bind(target) : value;
};
return [{
...durability,
async world() {
const world = await durability.world();
// An image write is a write of its stream; the driver knows its name.
const streams = new Proxy(world.streams, {
get(target, key) {
if (key !== "write") return bound(target, key);
return async (runId, stream, chunk) => {
const name = [...pendingNames].find((candidate) => String(stream).endsWith(imageStreamSuffix(candidate)));
if (name === undefined) return target.write(runId, stream, chunk);
return throughImage(name, chunk, () => target.write(runId, stream, chunk));
};
},
});
const events = new Proxy(world.events, {
get(target, key) {
if (key !== "create") return bound(target, key);
return async (runId, event, params) => {
if (event?.eventType !== "step_created" || event.eventData?.stepName !== "fx.log") return target.create(runId, event, params);
// What the World stores, read back as the log reads it.
const entry = JSON.parse(decoder.decode(event.eventData.input));
return through(entry, () => target.create(runId, event, params));
};
},
});
return new Proxy(world, { get: (target, key) => (key === "events" ? events : key === "streams" ? streams : bound(target, key)) });
},
}, control];
}
// The names of the walk's images, so a World's stream writes can be told
// apart from its other streams'.
const pendingNames = new Set();
// ---------------------------------------------------------------------------
// The code under test: one agent, one session object.
const vision = "intake/vision";
const usage = { inputTokens: { total: 1 }, outputTokens: { total: 1 } };
const answer = (text) => new Response(
[{ type: "text-delta", id: "a", delta: text }, { type: "finish", finishReason: { unified: "stop", raw: "stop" }, usage }, "[DONE]"]
.map((frame) => `data: ${typeof frame === "string" ? frame : JSON.stringify(frame)}\n\n`).join(""),
{ headers: { "content-type": "text/event-stream" } },
);
const promptNumber = (text) => Number(/prompt-(\d+)/.exec(text ?? "")?.[1]);
async function harness(walk, n) {
const dir = await mkdtemp(join(tmpdir(), "libfx-intake-"));
const [durability, writes] = heldInputs(durabilityKind === "world" ? await appWorldAt(dir) : memory());
const expected = {};
const names = {};
const kinds = {};
const imageGates = {};
const controllers = {};
const turnsRunning = new Map();
const started = new Set();
const acc = {};
const accError = {};
const res = {};
const early = {};
const problems = [];
let made = 0;
let closed = false;
let closing = null;
// Each turn's image as the model or the stand-in turn received it; a text
// prompt's turn receives none.
const received = (i, image) => {
if (kinds[i] === "text") {
if (image !== undefined) problems.push(`text prompt ${i}'s turn received an image`);
return;
}
if (image?.data !== expected[i] || image?.mimeType !== "image/png") problems.push(`prompt ${i}'s turn received ${JSON.stringify(image).slice(0, 120)}`);
};
const turnStarted = (i, end, abortWith) => {
if (started.has(i)) problems.push(`prompt ${i}'s turn started twice`);
started.add(i);
turnsRunning.set(i, { end, abortWith });
};
let agent;
if (harnessKind === "fx") {
agent = createFxAgent({
backend,
nativeAddon: addon,
...(wasm ? { wasm } : {}),
apiKey: "intake-key",
model: vision,
durability,
async fetch(input, init) {
// The WebAssembly host passes a Request, the native one a URL and init.
const request = input instanceof Request ? input : new Request(input, init);
if (request.method === "GET") return Response.json({ object: "list", data: [{ id: vision, type: "language", tags: ["vision", "file-input"] }] });
const prompt = JSON.parse(await request.text()).prompt;
const last = prompt.filter((message) => message.role === "user").at(-1);
const parts = Array.isArray(last?.content) ? last.content : [];
const i = promptNumber(parts.filter((part) => part.type === "text").map((part) => part.text).join(""));
const images = parts.filter((part) => part.type === "file");
if (images.length !== (kinds[i] === "text" ? 0 : 1)) problems.push(`prompt ${i}'s model request carries ${images.length} images`);
received(i, images[0] && { data: images[0].data?.data, mimeType: images[0].mediaType });
const signal = init?.signal ?? request.signal;
return new Promise((resolveResponse, rejectResponse) => {
turnStarted(i, () => { turnsRunning.delete(i); resolveResponse(answer(`answered ${i}`)); });
signal?.addEventListener("abort", () => { turnsRunning.delete(i); rejectResponse(signal.reason ?? new Error("aborted")); }, { once: true });
});
},
});
} else {
agent = createDurableAgentFactory({
harness: () => ({
async open({ store }) {
await store.load();
let key = 0;
const append = (record, marks) => store.append({ idempotencyKey: `intake:${Date.now()}:${key++}`, data: encoder.encode(JSON.stringify(record)), marks });
return {
prompt(input, { turnId }) {
const i = promptNumber(input?.[0]?.text);
received(i, input?.[1]);
const ended = deferred();
const result = (async () => {
await append({ turnId, start: true }, [{ start: turnId }]);
turnStarted(i, () => { turnsRunning.delete(i); ended.resolve("end_turn"); });
const stopReason = await ended.promise;
turnsRunning.delete(i);
await append({ turnId, end: true }, [{ end: true }]);
return { stopReason, usage: {} };
})();
return { result, steer: async () => {}, cancel: () => ended.resolve("cancelled"), async *[Symbol.asyncIterator]() { await result; } };
},
resume() { throw new Error("no walk leaves a turn open"); },
get openTurn() { return null; },
settled: async () => {},
saveCheckpoint: async () => {},
exportCheckpoint: async () => new Uint8Array(),
close: async () => {},
};
},
}),
defaultDurability: async () => memory(),
})({ durability });
}
const session = agent.session();
async function perform(act) {
const i = act.p;
switch (act.name) {
case "Make": {
const controller = new AbortController();
if (act.early) controller.abort();
const image = imageOf(walk, i);
expected[i] = base64(image);
names[i] = await nameOf(image);
kinds[i] = act.kind;
early[i] = act.early;
const blocks = [{ type: "text", text: `prompt-${i}` }];
// An image prompt's image write is held until the driver lets it go.
if (act.kind === "image") {
blocks.push({ type: "image", data: expected[i], mimeType: "image/png" });
pendingNames.add(names[i]);
imageGates[i] = writes.holdImage(names[i]);
}
const turn = session.prompt(blocks, { messageId: `p${i}`, signal: controller.signal });
controllers[i] = controller;
made = i;
acc[i] = "pending";
res[i] = "pending";
turn.accepted.then(() => { acc[i] = "resolved"; }, (error) => { acc[i] = "rejected"; accError[i] = error; });
turn.result.then(
(result) => { res[i] = result.stopReason === "end_turn" ? "end" : result.stopReason === "cancelled" ? "cancelled" : `other:${result.stopReason}:${result.error?.message ?? ""}`; },
() => { res[i] = "rejected"; },
);
return;
}
case "Prepared":
imageGates[i].go(true);
return;
case "PrepareFail":
imageGates[i].go(false);
return;
case "Store": return writes.settle(`prompt:p${i}`, true);
case "StoreFail": return writes.settle(`prompt:p${i}`, false);
case "Abort": controllers[i].abort(); return;
case "CancelStore": return writes.settle(`cancel:p${i}`, true);
case "Close": {
closed = true;
closing = agent.close();
// A closed agent's session object takes no more prompts.
try {
session.prompt("after close");
problems.push("prompt() after close did not throw");
} catch (error) {
if (!/is closed/.test(error.message)) problems.push(`prompt() after close threw ${error.message}`);
}
return;
}
case "End": {
if (!(await until(() => turnsRunning.has(i)))) throw new Error(`prompt ${i} has no turn to end`);
turnsRunning.get(i).end();
return;
}
default: throw new Error(`the driver cannot take ${act.name}`);
}
}
function observe() {
const stored = writes.landed.filter((entry) => entry.type === "prompt");
const cancels = new Set(writes.landed.filter((entry) => entry.type === "cancel").map((entry) => entry.target));
for (const entry of stored) {
const i = promptNumber(entry.input?.[0]?.text);
const image = entry.input?.[1];
// What the log holds is JSON. An image prompt names its image, which
// the image store already holds, with the caller's original bytes.
if (!isDeepStrictEqual(JSON.parse(JSON.stringify(entry)), entry)) problems.push(`prompt ${i}'s entry is not JSON`);
if (kinds[i] === "text") {
if (entry.input?.length !== 1) problems.push(`text prompt ${i} stored ${JSON.stringify(entry.input).slice(0, 120)}`);
continue;
}
if (image?.data?.sha256 !== names[i] || Object.keys(image.data).length !== 1 || image?.mimeType !== "image/png" || image?.type !== "image") problems.push(`prompt ${i} stored ${JSON.stringify(image).slice(0, 120)}`);
if (writes.images.get(names[i]) !== expected[i]) problems.push(`prompt ${i} was stored before its image, or with other bytes`);
}
const order = stored.map((entry) => promptNumber(entry.input?.[0]?.text));
const ph = [];
for (let i = 1; i <= n; i += 1) {
if (i > made) ph.push("unmade");
else if (early[i]) ph.push("early");
else if (order.includes(i)) ph.push("stored");
else if (writes.held.has(`prompt:p${i}`)) ph.push("sending");
else if (acc[i] === "rejected") ph.push(/is closed/.test(accError[i]?.message ?? "") ? "refused" : "failed");
// Accepted, yet neither stored nor on its way: an abort withdrew it.
else if (acc[i] === "resolved") ph.push("withdrawn");
else ph.push("preparing");
}
const each = (fn) => Array.from({ length: n }, (_, index) => fn(index + 1));
return {
made,
closed,
log: order,
ph,
acc: each((i) => acc[i] ?? "pending"),
res: each((i) => res[i] ?? "pending"),
cancel: each((i) => (cancels.has(`p${i}`) ? "stored" : writes.held.has(`cancel:p${i}`) ? "held" : "none")),
ran: each((i) => started.has(i)),
};
}
// Waits until the code shows nothing new for quietMs.
async function settle() {
let last = JSON.stringify(observe());
let since = Date.now();
const begun = Date.now();
while (Date.now() - since < quietMs) {
await sleep(15);
const now = JSON.stringify(observe());
if (now !== last) { last = now; since = Date.now(); }
if (Date.now() - begun > stepTimeoutMs) break;
}
return observe();
}
async function close() {
writes.openAll();
pendingNames.clear();
for (const turn of [...turnsRunning.values()]) turn.end();
await Promise.race([(closing ?? agent.close()).catch(() => {}), sleep(3000)]);
await closeAppWorlds?.();
await rm(dir, { recursive: true, force: true });
}
return { perform, settle, close, problems };
}
// ---------------------------------------------------------------------------
// Walks.
const random = () => {
seed = (seed * 1103515245 + 12345) % 2147483648;
return seed / 2147483648;
};
const graph = await loadGraph(resolve(here, "../sdk/graphs", "intake.dot"));
const n = graph.states.get(graph.init).ph.length;
const autoActs = new Set(["Released", "Start", "Skip", "CancelRun"]);
const view = (state) => JSON.stringify({
made: state.made, closed: state.closed, log: state.log, ph: state.ph, acc: state.acc, res: state.res, cancel: state.cancel, ran: state.ran,
});
const { steps, covered, totalEdges } = await walkGraph({
graph,
walks,
random,
verbose,
autoActs,
view,
nameOf: (act) => act.name,
sameAct: (a, b) => a.name === b.name && a.p === b.p && a.kind === b.kind && a.early === b.early,
label: (act) => `${act.name}(${act.p}${act.name === "Make" ? `,${act.kind}${act.early ? ",early" : ""}` : ""})`,
// Closing ends what a walk can do, so it comes late more often than not.
choose: (pool) => (pool.length > 1 && random() < 0.75 ? pool.filter((item) => item.act.name !== "Close") : pool),
open: (walk) => harness(walk, n),
});
if (!process.exitCode) console.log(`ok intake: ${walks} walks, ${steps} steps matched the model; ${covered} of ${totalEdges} edges covered (${harnessKind}, ${durabilityKind}${harnessKind === "fx" ? `, ${backend}` : ""})`);
// A walk that ended with prompts stored after close leaves the memory
// queue retrying with nobody listening; nothing more runs.
process.exit(process.exitCode ?? 0);
+3 -2
View File
@@ -231,8 +231,9 @@ async function harness() {
for (const [messageId, owner] of ownerOf) if (owner) stateOf.set(owner, ended.get(messageId) ?? "running");
// The turns the model was sent, from the newest request: a message that
// started a turn on the stream opens it, and every other message joins
// the turn before it. A steer a resumed turn replays reaches the model as
// a plain message, so the stream, not the text, says who owns a turn.
// the turn before it. A steer that joined a turn reaches the model as a
// live update, a resumed turn's too, but the stream, not the text, says
// who owns a turn.
const turns = [];
const counts = new Map();
for (const text of requests.at(-1)?.users ?? []) {
+100 -2
View File
@@ -1,4 +1,5 @@
// The state graphs tests/tla/sdk/check.sh has TLC write, as the drivers read them.
// The state graphs tests/tla/sdk/check.sh has TLC write, as the drivers read
// them, and the walk drive-intake.mjs and drive-images.mjs take along one.
import { strict as assert } from "node:assert";
import { readFile } from "node:fs/promises";
@@ -91,9 +92,106 @@ export async function loadGraph(path) {
if (match) state[match[1]] = parseValue(match[2].trim());
}
states.set(id, state);
if (state.act?.name === "Init") init = id;
// `act` is a record with a name, or a tuple whose first item is one.
if (state.act?.name === "Init" || (Array.isArray(state.act) && state.act[0] === "Init")) init = id;
}
}
assert.ok(init, "the graph has an initial state");
return { states, next, init };
}
// Walks the code along `graph`, `walks` times. Each walk opens the code with
// `open(walk)`, which returns `{ perform(act), settle(), close(), problems }`.
// Each step is one the model allows from where the walk stands, chosen at
// random among those that cross an edge no walk has crossed yet, if any;
// `choose` may narrow that pool. After it the code must show, through
// `view`, a state the model reaches once the code has taken every step in
// `autoActs` it can on its own; `nameOf` gives a step's action name. A walk
// the model does not allow, or whose code reports a problem, prints its path
// and sets `process.exitCode`, and no more walks run. Returns the steps taken
// and the edges covered.
export async function walkGraph({ graph, walks, random, autoActs, nameOf, sameAct, label, view, open, choose = (pool) => pool, verbose = false }) {
const actOf = (id) => graph.states.get(id).act;
const successors = (id) => [...(graph.next.get(id) ?? [])].filter((to) => to !== id);
// The states the code settles in after a step: every path of the steps it
// takes on its own, to where it can take none.
const settledFrom = (id, edges) => {
const seen = new Set([id]);
const queue = [id];
const quiet = [];
while (queue.length > 0) {
const from = queue.shift();
const autos = successors(from).filter((to) => autoActs.has(nameOf(actOf(to))));
if (autos.length === 0) quiet.push(from);
for (const to of autos) {
edges.push(`${from}>${to}`);
if (!seen.has(to)) { seen.add(to); queue.push(to); }
}
}
return quiet;
};
const covered = new Set();
const totalEdges = [...graph.next.values()].reduce((sum, targets) => sum + [...targets].length, 0);
let steps = 0;
for (let walk = 1; walk <= walks; walk += 1) {
const code = await open(walk);
const path = [];
let current = new Set(settledFrom(graph.init, []));
try {
for (;;) {
const options = [];
for (const from of current) {
for (const to of successors(from)) {
const act = actOf(to);
if (autoActs.has(nameOf(act))) continue;
let item = options.find((entry) => sameAct(entry.act, act));
if (!item) options.push(item = { act, fresh: false });
if (!covered.has(`${from}>${to}`)) item.fresh = true;
}
}
if (options.length === 0) break;
const fresh = options.filter((item) => item.fresh);
const pool = choose(fresh.length > 0 ? fresh : options);
const { act } = pool[Math.floor(random() * pool.length)];
path.push(label(act));
if (verbose) console.log(`walk ${walk}: ${label(act)}`);
await code.perform(act);
const shown = await code.settle();
if (code.problems.length > 0) throw new Error([...new Set(code.problems)].join("; "));
const seen = JSON.stringify(shown);
if (verbose) console.log(` shows ${seen}`);
steps += 1;
const next = new Set();
const allowed = new Set();
for (const from of current) {
for (const to of successors(from).filter((id) => sameAct(actOf(id), act))) {
const edges = [`${from}>${to}`];
for (const end of settledFrom(to, edges)) {
allowed.add(view(graph.states.get(end)));
if (view(graph.states.get(end)) !== seen) continue;
next.add(end);
for (const edge of edges) covered.add(edge);
}
}
}
if (next.size === 0) {
console.error(`FAIL walk ${walk} after ${path.join(" ")}`);
console.error(` the code shows ${seen}`);
for (const item of allowed) console.error(` the model allows ${item}`);
process.exitCode = 1;
break;
}
current = next;
}
} catch (error) {
console.error(`FAIL walk ${walk} after ${path.join(" ")}: ${error.message}`);
process.exitCode = 1;
} finally {
await code.close();
}
if (process.exitCode) break;
if (walk % 25 === 0 || walk === walks) console.log(`${walk} walks, ${steps} steps, ${covered.size} of ${totalEdges} edges`);
}
assert.ok(steps > 0 || process.exitCode, "the walks took steps");
return { steps, covered: covered.size, totalEdges };
}
+5 -4
View File
@@ -16,8 +16,9 @@
\* document lacks), and the document and the note are written at once,
\* even by `get`. A document with the flag but no note, saved before the
\* note existed, gets the note.
\* Operations: get (config.resolve -> Runtime.savedCredential, for every
\* HTTP server, whenever a host starts: `fx mcp list`, `fx ask`, the shell),
\* Operations: get (Store.snapshot from Runtime.restoreCredentials, one read
\* for every HTTP server, on the owner thread whenever a host starts:
\* `fx mcp list`, `fx ask`, the shell),
\* put (a `credential` event after a sign-in or refresh), remove (logout:
\* runtime.zig `.logout`; verbs.zig remove and logout).
\*
@@ -100,8 +101,8 @@ ReadEffect ==
ELSE IF ~marked THEN Mark
ELSE Same
\* Store.get, from config.resolve when a host starts. An unreadable document
\* gives no login and stays as it is.
\* Store.snapshot, from Runtime.restoreCredentials when a host starts. An
\* unreadable document gives no login and stays as it is.
Get ==
/\ ReadEffect
/\ UNCHANGED <<v1, loggedOut, kept>>
+28
View File
@@ -0,0 +1,28 @@
\* Witness: an open that drops an image the store lost breaks OpenExact.
CONSTANTS
N = 2
Imgs = {"a", "b"}
Old = TRUE
Durable = TRUE
MaxFaults = 1
MaxCrashes = 2
MaxLoss = 1
StoreFirst = TRUE
MarkOnlyStored = TRUE
FailOnMissing = FALSE
MaxRetries = 0
DropRepeats = TRUE
Window = 2
LiveWindow = TRUE
WindowAtOpen = TRUE
CountInline = TRUE
Tools = FALSE
ToolStoreFirst = TRUE
SPECIFICATION Spec
INVARIANTS
TypeOK
NoDanglingRef
OneCopy
OpenExact
KnownStoredIsStored
FailsOnlyOnLoss
@@ -0,0 +1,33 @@
\* Witness: a window that never counts a tool image the store refused, held
\* inline, lets the session hold more than Window images, breaking
\* WindowBound.
CONSTANTS
N = 2
Imgs = {"a", "b"}
Old = TRUE
Durable = FALSE
MaxFaults = 1
MaxCrashes = 2
MaxLoss = 1
StoreFirst = TRUE
MarkOnlyStored = TRUE
FailOnMissing = TRUE
MaxRetries = 0
DropRepeats = TRUE
Window = 2
LiveWindow = TRUE
WindowAtOpen = TRUE
CountInline = FALSE
Tools = TRUE
ToolStoreFirst = TRUE
SPECIFICATION Spec
INVARIANTS
TypeOK
NoDanglingRef
OneCopy
OpenExact
KnownStoredIsStored
FailsOnlyOnLoss
FailsOnlyOnLossInWindow
WindowHeld
WindowBound
+24
View File
@@ -0,0 +1,24 @@
\* Witness: a core that keeps every image, past its window, breaks WindowHeld.
CONSTANTS
N = 2
Imgs = {"a", "b"}
Old = TRUE
Durable = FALSE
MaxFaults = 1
MaxCrashes = 2
MaxLoss = 1
StoreFirst = TRUE
MarkOnlyStored = TRUE
FailOnMissing = TRUE
MaxRetries = 0
DropRepeats = TRUE
Window = 1
LiveWindow = FALSE
WindowAtOpen = TRUE
CountInline = TRUE
Tools = FALSE
ToolStoreFirst = TRUE
SPECIFICATION Spec
INVARIANTS
TypeOK
WindowHeld
+28
View File
@@ -0,0 +1,28 @@
\* Witness: a checkpoint naming an image the store never held breaks NoDanglingRef.
CONSTANTS
N = 2
Imgs = {"a", "b"}
Old = TRUE
Durable = TRUE
MaxFaults = 1
MaxCrashes = 2
MaxLoss = 1
StoreFirst = TRUE
MarkOnlyStored = FALSE
FailOnMissing = TRUE
MaxRetries = 0
DropRepeats = TRUE
Window = 2
LiveWindow = TRUE
WindowAtOpen = TRUE
CountInline = TRUE
Tools = FALSE
ToolStoreFirst = TRUE
SPECIFICATION Spec
INVARIANTS
TypeOK
NoDanglingRef
OneCopy
OpenExact
KnownStoredIsStored
FailsOnlyOnLoss
+25
View File
@@ -0,0 +1,25 @@
\* Witness: an open that stages every named image fails when the store lost
\* one outside the window, and breaks FailsOnlyOnLossInWindow.
CONSTANTS
N = 2
Imgs = {"a", "b"}
Old = TRUE
Durable = FALSE
MaxFaults = 1
MaxCrashes = 2
MaxLoss = 1
StoreFirst = TRUE
MarkOnlyStored = TRUE
FailOnMissing = TRUE
MaxRetries = 0
DropRepeats = TRUE
Window = 1
LiveWindow = TRUE
WindowAtOpen = FALSE
CountInline = TRUE
Tools = FALSE
ToolStoreFirst = TRUE
SPECIFICATION Spec
INVARIANTS
TypeOK
FailsOnlyOnLossInWindow
+29
View File
@@ -0,0 +1,29 @@
\* Witness: a retried prompt's input that lands names an image the store may
\* never have got, and breaks NoDanglingRef.
CONSTANTS
N = 2
Imgs = {"a", "b"}
Old = TRUE
Durable = TRUE
MaxFaults = 1
MaxCrashes = 2
MaxLoss = 1
StoreFirst = TRUE
MarkOnlyStored = TRUE
FailOnMissing = TRUE
MaxRetries = 1
DropRepeats = FALSE
Window = 2
LiveWindow = TRUE
WindowAtOpen = TRUE
CountInline = TRUE
Tools = FALSE
ToolStoreFirst = TRUE
SPECIFICATION Spec
INVARIANTS
TypeOK
NoDanglingRef
OneCopy
OpenExact
KnownStoredIsStored
FailsOnlyOnLoss
@@ -0,0 +1,28 @@
\* Witness: a prompt sent before its image is stored breaks NoDanglingRef.
CONSTANTS
N = 2
Imgs = {"a", "b"}
Old = TRUE
Durable = FALSE
MaxFaults = 1
MaxCrashes = 2
MaxLoss = 1
StoreFirst = FALSE
MarkOnlyStored = TRUE
FailOnMissing = TRUE
MaxRetries = 0
DropRepeats = TRUE
Window = 2
LiveWindow = TRUE
WindowAtOpen = TRUE
CountInline = TRUE
Tools = FALSE
ToolStoreFirst = TRUE
SPECIFICATION Spec
INVARIANTS
TypeOK
NoDanglingRef
OneCopy
OpenExact
KnownStoredIsStored
FailsOnlyOnLoss
@@ -0,0 +1,31 @@
\* Witness: a tool result handed to the core before its image is stored breaks
\* NoDanglingRef.
CONSTANTS
N = 2
Imgs = {"a", "b"}
Old = TRUE
Durable = FALSE
MaxFaults = 1
MaxCrashes = 2
MaxLoss = 1
StoreFirst = TRUE
MarkOnlyStored = TRUE
FailOnMissing = TRUE
MaxRetries = 0
DropRepeats = TRUE
Window = 2
LiveWindow = TRUE
WindowAtOpen = TRUE
CountInline = TRUE
Tools = TRUE
ToolStoreFirst = FALSE
SPECIFICATION Spec
INVARIANTS
TypeOK
NoDanglingRef
OneCopy
OpenExact
KnownStoredIsStored
FailsOnlyOnLoss
FailsOnlyOnLossInWindow
WindowHeld
+34
View File
@@ -0,0 +1,34 @@
\* The graph drive-images.mjs walks: a plain engine with an image store, two
\* prompts, an image an older libfx saved inline, a window of one image, so
\* each prompt drops the image before it, and driver steps only once
\* the code has settled. ImageStore.cfg checks the durable session.
CONSTANTS
N = 2
Imgs = {"a", "b"}
Old = TRUE
Durable = FALSE
MaxFaults = 1
MaxCrashes = 2
MaxLoss = 1
StoreFirst = TRUE
MarkOnlyStored = TRUE
FailOnMissing = TRUE
MaxRetries = 0
DropRepeats = TRUE
Window = 1
LiveWindow = TRUE
WindowAtOpen = TRUE
CountInline = TRUE
Tools = TRUE
ToolStoreFirst = TRUE
SPECIFICATION DriveSpec
INVARIANTS
TypeOK
NoDanglingRef
OneCopy
OpenExact
KnownStoredIsStored
FailsOnlyOnLoss
FailsOnlyOnLossInWindow
WindowHeld
WindowBound
+31
View File
@@ -0,0 +1,31 @@
\* A plain engine with an image store: prompts go straight to the core.
CONSTANTS
N = 2
Imgs = {"a", "b"}
Old = TRUE
Durable = FALSE
MaxFaults = 1
MaxCrashes = 2
MaxLoss = 1
StoreFirst = TRUE
MarkOnlyStored = TRUE
FailOnMissing = TRUE
MaxRetries = 0
DropRepeats = TRUE
Window = 2
LiveWindow = TRUE
WindowAtOpen = TRUE
CountInline = TRUE
Tools = TRUE
ToolStoreFirst = TRUE
SPECIFICATION Spec
INVARIANTS
TypeOK
NoDanglingRef
OneCopy
OpenExact
KnownStoredIsStored
FailsOnlyOnLoss
FailsOnlyOnLossInWindow
WindowHeld
WindowBound
+31
View File
@@ -0,0 +1,31 @@
\* A durable session: prompts pass through the log, which names their images.
CONSTANTS
N = 2
Imgs = {"a", "b"}
Old = TRUE
Durable = TRUE
MaxFaults = 1
MaxCrashes = 2
MaxLoss = 1
StoreFirst = TRUE
MarkOnlyStored = TRUE
FailOnMissing = TRUE
MaxRetries = 1
DropRepeats = TRUE
Window = 2
LiveWindow = TRUE
WindowAtOpen = TRUE
CountInline = TRUE
Tools = TRUE
ToolStoreFirst = TRUE
SPECIFICATION Spec
INVARIANTS
TypeOK
NoDanglingRef
OneCopy
OpenExact
KnownStoredIsStored
FailsOnlyOnLoss
FailsOnlyOnLossInWindow
WindowHeld
WindowBound
+528
View File
@@ -0,0 +1,528 @@
------------------------------ MODULE ImageStore ------------------------------
(***************************************************************************)
(* How a libfx session saves its images: each image once, in the host's *)
(* store, named by the hash of its bytes, while the session's records and *)
(* checkpoints hold only the name. *)
(* *)
(* A prompt's image goes to the store first and the prompt is sent only *)
(* once the write landed; with StoreFirst = FALSE it is sent while the *)
(* write is in flight (ImageStore-SendBeforeStore.cfg). With Durable the *)
(* prompt is stored in the session's log after its image, naming it, and *)
(* the worker resolves the name before it runs the turn. *)
(* *)
(* A turn's first record names its image. A checkpoint writes the whole *)
(* history again: it names the images the core knows are stored and keeps *)
(* the rest inline. With Old the session starts with an image an older *)
(* libfx saved inline, which the store never held; with MarkOnlyStored = *)
(* FALSE a checkpoint names it anyway (ImageStore-NameUnstored.cfg). *)
(* *)
(* A durable caller may retry a prompt the log already holds, under its *)
(* messageId, carrying any image. The retry stores nothing: the log keeps *)
(* the input it holds and drops the retry's. With DropRepeats = FALSE the *)
(* retry's input lands too, naming an image the store may never have got *)
(* (ImageStore-RetryLands.cfg). *)
(* *)
(* The session keeps only its newest Window images. When a turn enters the *)
(* history, the core drops each image older than the window: it keeps the *)
(* name and gives up the bytes, and a checkpoint saves it by name alone. *)
(* An open counts the history alone, as that turn's end did, and stages *)
(* the named images inside the window and the open turn's. With LiveWindow *)
(* = FALSE the core keeps every image (ImageStore-KeepAll.cfg); with *)
(* WindowAtOpen = FALSE an open stages every named image, so one the store *)
(* lost outside the window fails it (ImageStore-NeedAll.cfg). *)
(* *)
(* With Tools a turn's host tool may return an image. The engine stores it *)
(* before it hands the result to the core, whose next record names it; a *)
(* store that refuses it leaves it in the result, held inline. The window *)
(* counts it as any image, and once it leaves it keeps the name of its *)
(* bytes; with CountInline = FALSE the window never counts it, so the *)
(* session holds more than Window images (ImageStore-InlineUncounted.cfg). *)
(* With ToolStoreFirst = FALSE the result reaches the core while the write *)
(* is in flight (ImageStore-ToolBeforeStore.cfg). A tool image joins the *)
(* window with its turn, as the prompt's does, and an open stages every *)
(* image of the open turn. *)
(* *)
(* The core may crash, losing what it held and the image writes in flight. *)
(* It opens again in two phases: it names the images it needs, the host *)
(* stages each from its store, then the core opens. An image the store *)
(* lost fails the open; with FailOnMissing = FALSE the open drops it *)
(* (ImageStore-DropMissing.cfg). *)
(* *)
(* `act` names each step and its arguments, so a driver can replay any *)
(* path of the state graph against the real code. *)
(***************************************************************************)
EXTENDS Naturals, Sequences, FiniteSets
CONSTANTS
N, \* prompts the caller makes, at most
Imgs, \* images a prompt may carry; equal images share a name
Old, \* whether the session starts with an image saved inline
Durable, \* whether prompts pass through a durable session's log
MaxFaults, \* bound on failed image writes
MaxCrashes, \* bound on crashes of the core
MaxLoss, \* bound on images the store loses
StoreFirst, \* whether a prompt is sent only after its image is stored
MarkOnlyStored, \* whether the core names only images it knows are stored
FailOnMissing, \* whether an open that misses an image fails
MaxRetries, \* bound on retries of prompts the log holds
DropRepeats, \* whether the log drops a retried prompt's input
Window, \* the newest images the session keeps
LiveWindow, \* whether the core drops images older than the window
WindowAtOpen, \* whether an open stages only the window's images
CountInline, \* whether the window counts a tool image held inline
Tools, \* whether a turn's tool may return an image
ToolStoreFirst \* whether a tool's result reaches the core only after its image is stored
ASSUME N \in Nat /\ MaxFaults \in Nat /\ MaxCrashes \in Nat /\ MaxLoss \in Nat
ASSUME Old \in BOOLEAN /\ Durable \in BOOLEAN /\ StoreFirst \in BOOLEAN
ASSUME MarkOnlyStored \in BOOLEAN /\ FailOnMissing \in BOOLEAN
ASSUME MaxRetries \in Nat /\ DropRepeats \in BOOLEAN
ASSUME Window \in Nat \ {0} /\ LiveWindow \in BOOLEAN /\ WindowAtOpen \in BOOLEAN
ASSUME CountInline \in BOOLEAN
ASSUME Tools \in BOOLEAN /\ ToolStoreFirst \in BOOLEAN
ASSUME "o" \notin Imgs
Prompts == 1..N
None == "none"
OldImg == "o"
AllImgs == Imgs \cup {OldImg}
Phases == {"unmade", "putting", "put", "queued", "sent", "started", "ended", "failed", "lost"}
CoreStates == {"open", "down", "opening", "failed"}
\* Where each turn's tool image is: none yet; called, its write in flight;
\* put or refused by the store; given to the core; or lost to a crash.
ToolPhases == {"none", "called", "put", "refused", "given", "lost"}
VARIABLES
made, \* how many prompts the caller has made
img, \* each prompt's image
ph, \* where each prompt is
putting, \* the prompts whose image write is in flight
store, \* the images the host's store holds
lostImgs, \* the images the store lost
saved, \* the saved history: each turn's image, named, inline or dropped
inputs, \* with Durable: the prompts the log holds, in the order they landed
mem, \* the core's history while it runs: each image, whether it is known stored, and whether it was dropped
up, \* the core: open, down, opening, or failed to open
staged, \* the images staged for the open in progress
dirty, \* whether records landed since the last checkpoint
faults, \* failed image writes so far
crashes, \* crashes so far
retries, \* retries so far
repeats, \* the images retried inputs that landed in the log name
tool, \* the image each turn's tool returned
tph, \* where each turn's tool image is
tputting, \* the turns whose tool image write is in flight
turnAt, \* where the open turn's images start in the saved history
act \* the last step and its arguments
tools == <<tool, tph, tputting, turnAt>>
vars == <<made, img, ph, putting, store, lostImgs, saved, inputs, mem, up, staged, dirty, faults, crashes, retries, repeats, tools, act>>
TypeOK ==
/\ made \in 0..N
/\ img \in [Prompts -> Imgs \cup {None}]
/\ ph \in [Prompts -> Phases]
/\ putting \subseteq Prompts
/\ store \subseteq AllImgs
/\ lostImgs \subseteq AllImgs
/\ saved \in Seq([i : AllImgs, ref : BOOLEAN, dropped : BOOLEAN, named : BOOLEAN])
/\ inputs \in Seq(Prompts)
/\ mem \in Seq([i : AllImgs, stored : BOOLEAN, dropped : BOOLEAN, named : BOOLEAN])
/\ up \in CoreStates
/\ staged \subseteq AllImgs
/\ dirty \in BOOLEAN
/\ faults \in 0..MaxFaults
/\ crashes \in 0..MaxCrashes
/\ retries \in 0..MaxRetries
/\ repeats \subseteq Imgs
/\ tool \in [Prompts -> Imgs \cup {None}]
/\ tph \in [Prompts -> ToolPhases]
/\ tputting \subseteq Prompts
/\ turnAt \in Nat
Init ==
/\ made = 0
/\ img = [p \in Prompts |-> None]
/\ ph = [p \in Prompts |-> "unmade"]
/\ putting = {}
/\ store = {}
/\ lostImgs = {}
/\ saved = IF Old THEN << [i |-> OldImg, ref |-> FALSE, dropped |-> FALSE, named |-> TRUE] >> ELSE << >>
/\ inputs = << >>
/\ mem = << >>
/\ up = "down"
/\ staged = {}
/\ dirty = FALSE
/\ faults = 0
/\ crashes = 0
/\ retries = 0
/\ repeats = {}
/\ tool = [p \in Prompts |-> None]
/\ tph = [p \in Prompts |-> "none"]
/\ tputting = {}
/\ turnAt = 0
/\ act = << "Init" >>
\* The images the saved history names.
Named == {saved[k].i : k \in {j \in 1..Len(saved) : saved[j].ref}}
\* Whether image k of s is one the window keeps when it counts the first n:
\* it counts every image, a tool image the store refused, held inline,
\* too, and keeps every image past the first n. With CountInline = FALSE it
\* counts only images with a name, which a refused tool image lacks.
WindowKeeps(s, k, n) ==
IF CountInline THEN Cardinality(k..n) <= Window
ELSE ~s[k].named \/ Cardinality({j \in k..n : s[j].named}) <= Window
\* A turn has started and not ended: its images are the last saved, from
\* turnAt on, and it has not entered the history the window counts.
OpenTurn == \E p \in Prompts : ph[p] = "started"
Counted == IF OpenTurn THEN turnAt - 1 ELSE Len(saved)
\* The named images an open stages: those inside the window, and the open
\* turn's, which are past every counted one.
Needed == {saved[k].i : k \in {j \in 1..Len(saved) : saved[j].ref /\ (WindowKeeps(saved, j, Counted) \/ ~WindowAtOpen)}}
\* A core history whose images older than the window are dropped: each
\* keeps its name, and the core no longer counts it as stored.
Windowed(s) ==
IF ~LiveWindow THEN s
ELSE [k \in 1..Len(s) |-> IF WindowKeeps(s, k, Len(s)) THEN s[k] ELSE [i |-> s[k].i, stored |-> FALSE, dropped |-> TRUE, named |-> TRUE]]
Images(s) == [k \in 1..Len(s) |-> s[k].i]
Running == \E p \in Prompts : ph[p] \in {"sent", "started"}
Queued(p) == \E k \in 1..Len(inputs) : inputs[k] = p
\* The prompt the worker takes next: the first in the log it has not taken.
NextInput(p) ==
/\ ph[p] = "queued"
/\ \E k \in 1..Len(inputs) :
/\ inputs[k] = p
/\ \A j \in 1..(k - 1) : ph[inputs[j]] # "queued"
Set(f, p, v) == [f EXCEPT ![p] = v]
----------------------------------------------------------------------------
\* The caller
\* A prompt carrying image i. A plain engine takes prompts only while open.
Make(p, i) ==
/\ made + 1 = p
/\ Durable \/ up = "open"
/\ made' = p
/\ img' = Set(img, p, i)
/\ ph' = Set(ph, p, "putting")
/\ putting' = putting \cup {p}
/\ act' = << "Make", p, i >>
/\ UNCHANGED <<store, lostImgs, saved, inputs, mem, up, staged, dirty, faults, crashes, retries, repeats, tools>>
\* The prompt's image write lands in the store.
PutLand(p) ==
/\ p \in putting
/\ store' = store \cup {img[p]}
/\ putting' = putting \ {p}
/\ ph' = IF ph[p] = "putting" THEN Set(ph, p, "put") ELSE ph
/\ act' = << "PutLand", p >>
/\ UNCHANGED <<made, img, lostImgs, saved, inputs, mem, up, staged, dirty, faults, crashes, retries, repeats, tools>>
\* The write fails. A prompt not yet sent fails with it; one sent early
\* (StoreFirst = FALSE) goes on naming an image the store never got.
PutFail(p) ==
/\ p \in putting
/\ faults < MaxFaults
/\ faults' = faults + 1
/\ putting' = putting \ {p}
/\ ph' = IF ph[p] = "putting" THEN Set(ph, p, "failed") ELSE ph
/\ act' = << "PutFail", p >>
/\ UNCHANGED <<made, img, store, lostImgs, saved, inputs, mem, up, staged, dirty, crashes, retries, repeats, tools>>
----------------------------------------------------------------------------
\* The host and the core
\* A plain engine sends a prompt once its image is stored.
Send(p) ==
/\ ~Durable
/\ up = "open"
/\ ~Running
/\ IF StoreFirst THEN ph[p] = "put" ELSE ph[p] \in {"putting", "put"}
/\ ph' = Set(ph, p, "sent")
/\ act' = << "Send", p >>
/\ UNCHANGED <<made, img, putting, store, lostImgs, saved, inputs, mem, up, staged, dirty, faults, crashes, retries, repeats, tools>>
\* A durable prompt lands in the log after its image, naming it.
Enqueue(p) ==
/\ Durable
/\ ph[p] = "put"
/\ inputs' = Append(inputs, p)
/\ ph' = Set(ph, p, "queued")
/\ act' = << "Enqueue", p >>
/\ UNCHANGED <<made, img, putting, store, lostImgs, saved, mem, up, staged, dirty, faults, crashes, retries, repeats, tools>>
\* The worker takes the next prompt and resolves its image's name. An
\* image the store lost fails the turn.
Take(p) ==
/\ Durable
/\ up = "open"
/\ ~Running
/\ NextInput(p)
/\ ph' = Set(ph, p, IF img[p] \in store THEN "sent" ELSE "failed")
/\ act' = << "Take", p >>
/\ UNCHANGED <<made, img, putting, store, lostImgs, saved, inputs, mem, up, staged, dirty, faults, crashes, retries, repeats, tools>>
\* The turn's first record lands, naming the image the host stored.
TurnStart(p) ==
/\ ph[p] = "sent"
/\ up = "open"
/\ mem' = Append(mem, [i |-> img[p], stored |-> TRUE, dropped |-> FALSE, named |-> TRUE])
/\ saved' = Append(saved, [i |-> img[p], ref |-> TRUE, dropped |-> FALSE, named |-> TRUE])
/\ turnAt' = Len(saved) + 1
/\ dirty' = TRUE
/\ ph' = Set(ph, p, "started")
/\ act' = << "TurnStart", p >>
/\ UNCHANGED <<made, img, putting, store, lostImgs, inputs, up, staged, faults, crashes, retries, repeats, tool, tph, tputting>>
\* The model answers and the turn enters the history. Its image is the
\* newest; the core drops any older than the window.
TurnEnd(p) ==
/\ ph[p] = "started"
/\ tph[p] \in {"none", "given", "lost"}
/\ up = "open"
/\ mem' = Windowed(mem)
/\ ph' = Set(ph, p, "ended")
/\ dirty' = TRUE
/\ act' = << "TurnEnd", p >>
/\ UNCHANGED <<made, img, putting, store, lostImgs, saved, inputs, up, staged, faults, crashes, retries, repeats, tools>>
\* A checkpoint between turns writes the history again from the core. A
\* dropped image is saved by its name alone, neither named nor inline.
Checkpoint ==
/\ up = "open"
/\ dirty
/\ ~Running
/\ saved' = [k \in 1..Len(mem) |->
[i |-> mem[k].i,
ref |-> ~mem[k].dropped /\ (mem[k].stored \/ ~MarkOnlyStored),
dropped |-> mem[k].dropped,
named |-> mem[k].named]]
/\ dirty' = FALSE
/\ act' = << "Checkpoint" >>
/\ UNCHANGED <<made, img, ph, putting, store, lostImgs, inputs, mem, up, staged, faults, crashes, retries, repeats, tools>>
\* The core crashes: it loses what it held and every write in flight. A
\* prompt not yet in the log or not yet started is lost to a plain engine;
\* a durable one stays in the log and runs again.
Crash ==
/\ crashes < MaxCrashes
/\ up \in {"open", "opening"}
/\ crashes' = crashes + 1
/\ up' = "down"
/\ mem' = << >>
/\ staged' = {}
/\ putting' = {}
/\ ph' = [p \in Prompts |->
CASE ph[p] \in {"putting", "put"} -> "lost"
[] ph[p] = "sent" -> IF Durable THEN "queued" ELSE "lost"
[] OTHER -> ph[p]]
/\ tputting' = {}
/\ tph' = [p \in Prompts |-> IF tph[p] \in {"called", "put", "refused"} THEN "lost" ELSE tph[p]]
/\ act' = << "Crash" >>
/\ UNCHANGED <<made, img, store, lostImgs, saved, inputs, dirty, faults, retries, repeats, tool, turnAt>>
\* Opening, first phase: the core reads the saved history and names the
\* images it does not hold.
OpenBegin ==
/\ up = "down"
/\ up' = "opening"
/\ staged' = {}
/\ act' = << "OpenBegin" >>
/\ UNCHANGED <<made, img, ph, putting, store, lostImgs, saved, inputs, mem, dirty, faults, crashes, retries, repeats, tools>>
\* The host stages one image the open needs from its store.
Stage(i) ==
/\ up = "opening"
/\ i \in Needed \ staged
/\ i \in store
/\ staged' = staged \cup {i}
/\ act' = << "Stage", i >>
/\ UNCHANGED <<made, img, ph, putting, store, lostImgs, saved, inputs, mem, up, dirty, faults, crashes, retries, repeats, tools>>
\* An image the open needs that the store does not hold fails the open.
OpenMissing ==
/\ up = "opening"
/\ FailOnMissing
/\ \E i \in Needed \ staged : i \notin store
/\ up' = "failed"
/\ act' = << "OpenMissing" >>
/\ UNCHANGED <<made, img, ph, putting, store, lostImgs, saved, inputs, mem, staged, dirty, faults, crashes, retries, repeats, tools>>
\* Opening, second phase: every image the open needs is staged, and the
\* core holds the saved history again; a named image outside the window
\* stays dropped, unread. With FailOnMissing = FALSE an image the store lost
\* is left out.
OpenOk ==
/\ up = "opening"
/\ \/ Needed \subseteq staged
\/ ~FailOnMissing /\ \A i \in Needed \ staged : i \notin store
/\ LET marked == [k \in 1..Len(saved) |->
[saved[k] EXCEPT !.dropped = @ \/ (saved[k].ref /\ saved[k].i \notin staged /\ ~WindowKeeps(saved, k, Counted))]]
Kept(s) == SelectSeq(s, LAMBDA e : e.dropped \/ ~e.ref \/ e.i \in staged)
AsCore(s) == [k \in 1..Len(s) |-> [i |-> s[k].i, stored |-> s[k].ref /\ ~s[k].dropped, dropped |-> s[k].dropped, named |-> s[k].named]]
IN mem' = Windowed(AsCore(Kept(SubSeq(marked, 1, Counted)))) \o AsCore(Kept(SubSeq(marked, Counted + 1, Len(saved))))
/\ up' = "open"
/\ staged' = {}
/\ act' = << "OpenOk" >>
/\ UNCHANGED <<made, img, ph, putting, store, lostImgs, saved, inputs, dirty, faults, crashes, retries, repeats, tools>>
\* A retry of prompt p, which the log holds, under its messageId, carrying
\* image i. It stores nothing; the log drops its input, or with DropRepeats
\* = FALSE keeps it, naming i.
Retry(p, i) ==
/\ Durable
/\ retries < MaxRetries
/\ Queued(p)
/\ retries' = retries + 1
/\ repeats' = IF DropRepeats THEN repeats ELSE repeats \cup {i}
/\ act' = << "Retry", p, i >>
/\ UNCHANGED <<made, img, ph, putting, store, lostImgs, saved, inputs, mem, up, staged, dirty, faults, crashes, tools>>
\* The model calls the turn's tool, which returns image i once. The engine
\* writes it to the store before it hands the result on.
ToolCall(p, i) ==
/\ Tools
/\ up = "open"
/\ ph[p] = "started"
/\ tph[p] = "none"
/\ tool' = Set(tool, p, i)
/\ tph' = Set(tph, p, "called")
/\ tputting' = tputting \cup {p}
/\ act' = << "ToolCall", p, i >>
/\ UNCHANGED <<made, img, ph, putting, store, lostImgs, saved, inputs, mem, up, staged, dirty, faults, crashes, retries, repeats, turnAt>>
ToolPutLand(p) ==
/\ p \in tputting
/\ store' = store \cup {tool[p]}
/\ tputting' = tputting \ {p}
/\ tph' = IF tph[p] = "called" THEN Set(tph, p, "put") ELSE tph
/\ act' = << "ToolPutLand", p >>
/\ UNCHANGED <<made, img, ph, putting, lostImgs, saved, inputs, mem, up, staged, dirty, faults, crashes, retries, repeats, tool, turnAt>>
\* The store refuses the image. The tool has run, so its result keeps it.
ToolPutFail(p) ==
/\ p \in tputting
/\ faults < MaxFaults
/\ faults' = faults + 1
/\ tputting' = tputting \ {p}
/\ tph' = IF tph[p] = "called" THEN Set(tph, p, "refused") ELSE tph
/\ act' = << "ToolPutFail", p >>
/\ UNCHANGED <<made, img, ph, putting, store, lostImgs, saved, inputs, mem, up, staged, dirty, crashes, retries, repeats, tool, turnAt>>
\* The result reaches the core, and the turn's next record names the image
\* the store holds; a refused one is held inline, without a name.
ToolResult(p) ==
/\ up = "open"
/\ ph[p] = "started"
/\ tph[p] \in IF ToolStoreFirst THEN {"put", "refused"} ELSE {"called", "put", "refused"}
/\ LET named == tph[p] # "refused"
IN /\ mem' = Append(mem, [i |-> tool[p], stored |-> named, dropped |-> FALSE, named |-> named])
/\ saved' = Append(saved, [i |-> tool[p], ref |-> named, dropped |-> FALSE, named |-> named])
/\ tph' = Set(tph, p, "given")
/\ dirty' = TRUE
/\ act' = << "ToolResult", p >>
/\ UNCHANGED <<made, img, ph, putting, store, lostImgs, inputs, up, staged, faults, crashes, retries, repeats, tool, tputting, turnAt>>
\* The store loses an image: corruption, or an operator deleting it.
Lose(i) ==
/\ Cardinality(lostImgs) < MaxLoss
/\ i \in store
/\ store' = store \ {i}
/\ lostImgs' = lostImgs \cup {i}
/\ act' = << "Lose", i >>
/\ UNCHANGED <<made, img, ph, putting, saved, inputs, mem, up, staged, dirty, faults, crashes, retries, repeats, tools>>
\* Nothing is left to do: every prompt settled, or the open failed.
Finished ==
\/ up = "failed"
\/ /\ up = "open"
/\ made = N
/\ putting = {}
/\ tputting = {}
/\ ~dirty
/\ \A p \in Prompts : ph[p] \in {"ended", "failed", "lost"}
Done == Finished /\ UNCHANGED vars
Next ==
\/ \E p \in Prompts, i \in Imgs : Make(p, i)
\/ \E p \in Prompts : PutLand(p) \/ PutFail(p) \/ Send(p) \/ Enqueue(p) \/ Take(p) \/ TurnStart(p) \/ TurnEnd(p)
\/ Checkpoint
\/ Crash
\/ OpenBegin
\/ \E i \in AllImgs : Stage(i)
\/ OpenMissing
\/ OpenOk
\/ \E i \in AllImgs : Lose(i)
\/ \E p \in Prompts, i \in Imgs : Retry(p, i)
\/ \E p \in Prompts, i \in Imgs : ToolCall(p, i)
\/ \E p \in Prompts : ToolPutLand(p) \/ ToolPutFail(p) \/ ToolResult(p)
\/ Done
Spec == Init /\ [][Next]_vars
\* The paths tests/tla/drivers/drive-images.mjs takes against a plain engine
\* (Durable = FALSE). The engine runs one prompt at a time, so a prompt is
\* made only when none is in flight. The code takes the steps in Auto on its
\* own as soon as it can, so the driver moves only when none of them can.
Idle == \A p \in Prompts : ph[p] \notin {"putting", "put", "sent", "started"}
Auto ==
\/ \E p \in Prompts : Send(p) \/ TurnStart(p) \/ ToolResult(p)
\/ Checkpoint
\/ \E i \in AllImgs : Stage(i)
\/ OpenMissing
\/ OpenOk
Quiet == ~ENABLED Auto
DriverStep ==
\/ \E p \in Prompts, i \in Imgs : Idle /\ Make(p, i)
\/ \E p \in Prompts : PutLand(p) \/ PutFail(p) \/ TurnEnd(p)
\/ \E p \in Prompts, i \in Imgs : ToolCall(p, i)
\/ \E p \in Prompts : ToolPutLand(p) \/ ToolPutFail(p)
\/ Crash
\/ OpenBegin
\/ \E i \in AllImgs : Lose(i)
DriveSpec == Init /\ [][(Quiet /\ DriverStep) \/ Auto \/ Done]_vars
----------------------------------------------------------------------------
\* What must hold
\* Nothing saved names an image the store does not hold, unless the store
\* lost it.
NoDanglingRef ==
lostImgs = {} =>
/\ Named \subseteq store
/\ Durable => \A k \in 1..Len(inputs) : img[inputs[k]] \in store
/\ repeats \subseteq store
\* Only the image an older libfx saved inline is ever saved inline, but for
\* a tool image the store refused, which has no name.
OneCopy == \A k \in 1..Len(saved) : ~saved[k].ref /\ ~saved[k].dropped /\ saved[k].named => saved[k].i = OldImg
\* An open core holds exactly the saved history's images, in order.
OpenExact == up = "open" => Images(mem) = Images(saved)
\* What the core counts as stored is stored, unless the store lost it.
KnownStoredIsStored ==
(lostImgs = {} /\ up = "open") => \A k \in 1..Len(mem) : mem[k].stored => mem[k].i \in store
\* An open fails only when the store lost an image.
FailsOnlyOnLoss == up = "failed" => lostImgs # {}
\* An open fails only when the store lost an image the window keeps.
FailsOnlyOnLossInWindow ==
up = "failed" =>
\E k \in 1..Len(saved) : saved[k].ref /\ WindowKeeps(saved, k, Counted) /\ saved[k].i \in lostImgs
\* An open core holds at most Window images between turns, counting each it
\* holds inline, as a tool image the store refused.
WindowBound ==
(up = "open" /\ ~OpenTurn) => Cardinality({k \in 1..Len(mem) : ~mem[k].dropped}) <= Window
\* An open core keeps exactly its newest Window named images between turns:
\* each older one is dropped, keeping its name. A running turn's images join
\* the window once the turn enters the history.
WindowHeld ==
(up = "open" /\ ~OpenTurn) => \A k \in 1..Len(mem) : mem[k].dropped = ~WindowKeeps(mem, k, Len(mem))
=============================================================================
@@ -0,0 +1,12 @@
\* Expected to DEADLOCK: a failed prompt that never lets the prompts behind
\* it go leaves them waiting forever, as long as no abort withdraws them.
CONSTANTS
N = 2
MaxFaults = 1
Gate = TRUE
CloseCheck = TRUE
GateReleasesOnFailure = FALSE
AbortWithdraws = FALSE
SPECIFICATION Spec
INVARIANTS
TypeOK
@@ -0,0 +1,12 @@
\* Expected to FAIL ClosedStopsPreparing: without the check a prompt
\* still preparing its images when the agent closes is stored anyway.
CONSTANTS
N = 1
MaxFaults = 0
Gate = TRUE
CloseCheck = FALSE
GateReleasesOnFailure = TRUE
AbortWithdraws = TRUE
SPECIFICATION Spec
INVARIANTS
ClosedStopsPreparing
+12
View File
@@ -0,0 +1,12 @@
\* Expected to FAIL ImageOrder: without the gate a later prompt is stored
\* before an earlier one still preparing its images.
CONSTANTS
N = 2
MaxFaults = 0
Gate = FALSE
CloseCheck = TRUE
GateReleasesOnFailure = TRUE
AbortWithdraws = TRUE
SPECIFICATION Spec
INVARIANTS
ImageOrder
+13
View File
@@ -0,0 +1,13 @@
\* Expected to FAIL CancelOnlyWhenSent: a prompt that goes on after an
\* abort before its send began is stored and then cancelled, and its cancel
\* goes out before the prompt does.
CONSTANTS
N = 1
MaxFaults = 0
Gate = TRUE
CloseCheck = TRUE
GateReleasesOnFailure = TRUE
AbortWithdraws = FALSE
SPECIFICATION Spec
INVARIANTS
CancelOnlyWhenSent
+21
View File
@@ -0,0 +1,21 @@
\* The graph drive-intake.mjs walks: two prompts, driver steps only once the
\* code has settled. PromptIntake.cfg checks three prompts on every path.
CONSTANTS
N = 2
MaxFaults = 2
Gate = TRUE
CloseCheck = TRUE
GateReleasesOnFailure = TRUE
AbortWithdraws = TRUE
SPECIFICATION DriveSpec
INVARIANTS
TypeOK
ImageOrder
StoredIffAccepted
CancelOnlyWhenSent
WithdrawnNeverStored
NeverStoredTwice
ClosedStopsPreparing
CancelledOnlyOwn
NoRunAfterRejected
RunsStored
+19
View File
@@ -0,0 +1,19 @@
CONSTANTS
N = 3
MaxFaults = 2
Gate = TRUE
CloseCheck = TRUE
GateReleasesOnFailure = TRUE
AbortWithdraws = TRUE
SPECIFICATION Spec
INVARIANTS
TypeOK
ImageOrder
StoredIffAccepted
CancelOnlyWhenSent
WithdrawnNeverStored
NeverStoredTwice
ClosedStopsPreparing
CancelledOnlyOwn
NoRunAfterRejected
RunsStored
+332
View File
@@ -0,0 +1,332 @@
----------------------------- MODULE PromptIntake -----------------------------
(***************************************************************************)
(* How one durable session object takes prompts in, from `prompt()` to the *)
(* session log, and how the worker runs what the log holds. *)
(* *)
(* A prompt is text, which needs nothing before it is stored, or carries *)
(* images, which are prepared first: each image goes to the session's *)
(* image store, which the log then names. A prompt made while an earlier *)
(* image prompt from the same session object is not yet stored waits for *)
(* it, then prepares its own images; with Gate = FALSE it does not wait *)
(* (PromptIntake-NoGate.cfg). Once ready, a prompt whose agent closed *)
(* meanwhile is refused; with CloseCheck = FALSE it is stored anyway *)
(* (PromptIntake-NoCloseCheck.cfg). The wait ends when the earlier prompt *)
(* is stored or fails; with GateReleasesOnFailure = FALSE a failure holds *)
(* every prompt behind it forever (PromptIntake-GateHeldOnFailure.cfg, a *)
(* deadlock). *)
(* *)
(* A prompt's signal may have aborted before the call, and then nothing is *)
(* stored, or abort later. Until the prompt's send begins, an abort *)
(* withdraws it: nothing is stored and no cancel goes out; with *)
(* AbortWithdraws = FALSE the prompt goes on, and a cancel naming it goes *)
(* out at once (PromptIntake-NoWithdraw.cfg). Once its send has begun, an *)
(* abort sends a cancel naming the prompt, and the cancel and the prompt *)
(* land in the log in either order. Preparing (an image write) or a log *)
(* write may fail, up to MaxFaults times. *)
(* *)
(* The worker runs the log's prompts one at a time, in log order. It ends *)
(* a prompt whose cancel landed without running it, and cancels a running *)
(* one when its cancel lands. After the agent closes, a worker still *)
(* running goes on until the log holds nothing more for it; no new one *)
(* starts. *)
(* *)
(* `act` names each step and its prompt, so *)
(* tests/tla/drivers/drive-intake.mjs can replay any path of the state graph *)
(* against the real code. *)
(***************************************************************************)
EXTENDS Naturals, Sequences, FiniteSets
CONSTANTS
N, \* prompts the caller makes, at most
MaxFaults, \* bound on failed preparations and failed log writes
Gate, \* whether a prompt waits for earlier image prompts
CloseCheck, \* whether a prompt ready after close is refused
GateReleasesOnFailure, \* whether a failed prompt lets later ones go
AbortWithdraws \* whether an abort before the send withdraws the prompt
ASSUME N \in Nat /\ MaxFaults \in Nat
ASSUME Gate \in BOOLEAN /\ CloseCheck \in BOOLEAN /\ GateReleasesOnFailure \in BOOLEAN /\ AbortWithdraws \in BOOLEAN
Prompts == 1..N
Kinds == {"text", "image"}
Phases == {"unmade", "early", "preparing", "sending", "stored", "failed", "refused", "withdrawn"}
VARIABLES
made, \* how many prompts the caller has made
kind, \* each prompt: text, or images to prepare
early, \* whether its signal had aborted before the call
before, \* the earlier image prompt it waits for, or 0
ph, \* where it is on its way into the log
acc, \* its `accepted`: pending, resolved or rejected
res, \* its `result`: pending, end, cancelled or rejected
aborted, \* whether its signal aborted after the call
cancel, \* its cancel: none, held on its way, or stored
log, \* the prompts the log holds, in the order they landed
ran, \* whether the worker started its turn
open, \* the prompt whose turn runs now, or 0
done, \* the prompts whose turns ended
active, \* whether a worker runs the session now
closed, \* whether the agent closed
faults, \* failures so far
prepAtClose, \* the prompts still preparing when the agent closed
act \* the step that led here
vars == <<made, kind, early, before, ph, acc, res, aborted, cancel, log, ran, open, done,
active, closed, faults, prepAtClose, act>>
None == "none"
Act(name, p) == [name |-> name, p |-> p, kind |-> None, early |-> FALSE]
TypeOK ==
/\ made \in 0..N
/\ kind \in [Prompts -> Kinds \cup {None}]
/\ early \in [Prompts -> BOOLEAN]
/\ before \in [Prompts -> 0..N]
/\ ph \in [Prompts -> Phases]
/\ acc \in [Prompts -> {"pending", "resolved", "rejected"}]
/\ res \in [Prompts -> {"pending", "end", "cancelled", "rejected"}]
/\ aborted \in [Prompts -> BOOLEAN]
/\ cancel \in [Prompts -> {"none", "held", "stored"}]
/\ log \in Seq(Prompts)
/\ ran \in [Prompts -> BOOLEAN]
/\ open \in 0..N
/\ done \subseteq Prompts
/\ active \in BOOLEAN
/\ closed \in BOOLEAN
/\ faults \in 0..MaxFaults
/\ prepAtClose \subseteq Prompts
/\ act \in [name: {"Init", "Make", "Released", "Prepared", "PrepareFail", "Store", "StoreFail",
"Abort", "CancelStore", "Close", "Start", "Skip", "CancelRun", "End", "Finished"},
p: 0..N, kind: Kinds \cup {None}, early: BOOLEAN]
Init ==
/\ made = 0
/\ kind = [i \in Prompts |-> None]
/\ early = [i \in Prompts |-> FALSE]
/\ before = [i \in Prompts |-> 0]
/\ ph = [i \in Prompts |-> "unmade"]
/\ acc = [i \in Prompts |-> "pending"]
/\ res = [i \in Prompts |-> "pending"]
/\ aborted = [i \in Prompts |-> FALSE]
/\ cancel = [i \in Prompts |-> "none"]
/\ log = <<>>
/\ ran = [i \in Prompts |-> FALSE]
/\ open = 0
/\ done = {}
/\ active = FALSE
/\ closed = FALSE
/\ faults = 0
/\ prepAtClose = {}
/\ act = Act("Init", 0)
Stored == {log[k] : k \in 1..Len(log)}
Pending(d) == SelectSeq(log, LAMBDA i : i \notin d)
\* The latest image prompt not yet stored or failed: what a new prompt waits
\* for. Every earlier image prompt is settled once it is, since it waited.
Waiting == {j \in 1..made : kind[j] = "image" /\ ~early[j] /\ acc[j] = "pending"}
GateNow == IF Gate /\ Waiting # {} THEN CHOOSE j \in Waiting : \A k \in Waiting : k <= j ELSE 0
\* Whether the prompt i waits for lets it go: it and every image prompt
\* before it have settled, since one withdrawn while it waited settles
\* before the prompts it waited for.
Ready(i) ==
IF before[i] = 0 THEN TRUE
ELSE IF GateReleasesOnFailure
THEN \A j \in 1..before[i] : (kind[j] = "image" /\ ~early[j]) => acc[j] # "pending"
ELSE acc[before[i]] = "resolved"
\* A prompt that cannot be stored: both of its promises reject.
Reject(i, phase) ==
/\ ph' = [ph EXCEPT ![i] = phase]
/\ acc' = [acc EXCEPT ![i] = "rejected"]
/\ res' = [res EXCEPT ![i] = "rejected"]
Make(i, k, e) ==
/\ ~closed
/\ made = i - 1
/\ made' = i
/\ kind' = [kind EXCEPT ![i] = k]
/\ early' = [early EXCEPT ![i] = e]
/\ IF e
THEN /\ ph' = [ph EXCEPT ![i] = "early"]
/\ acc' = [acc EXCEPT ![i] = "resolved"]
/\ res' = [res EXCEPT ![i] = "cancelled"]
/\ UNCHANGED before
ELSE /\ before' = [before EXCEPT ![i] = GateNow]
/\ ph' = [ph EXCEPT ![i] = IF k = "text" /\ GateNow = 0 THEN "sending" ELSE "preparing"]
/\ UNCHANGED <<acc, res>>
/\ act' = [name |-> "Make", p |-> i, kind |-> k, early |-> e]
/\ UNCHANGED <<aborted, cancel, log, ran, open, done, active, closed, faults, prepAtClose>>
\* A text prompt that waited goes on once the earlier prompt lets it.
Released(i) ==
/\ ph[i] = "preparing" /\ kind[i] = "text" /\ Ready(i)
/\ IF closed /\ CloseCheck
THEN Reject(i, "refused")
ELSE /\ ph' = [ph EXCEPT ![i] = "sending"]
/\ UNCHANGED <<acc, res>>
/\ act' = Act("Released", i)
/\ UNCHANGED <<made, kind, early, before, aborted, cancel, log, ran, open, done, active, closed, faults, prepAtClose>>
Prepared(i) ==
/\ ph[i] = "preparing" /\ kind[i] = "image" /\ Ready(i)
/\ IF closed /\ CloseCheck
THEN Reject(i, "refused")
ELSE /\ ph' = [ph EXCEPT ![i] = "sending"]
/\ UNCHANGED <<acc, res>>
/\ act' = Act("Prepared", i)
/\ UNCHANGED <<made, kind, early, before, aborted, cancel, log, ran, open, done, active, closed, faults, prepAtClose>>
PrepareFail(i) ==
/\ ph[i] = "preparing" /\ kind[i] = "image" /\ Ready(i)
/\ faults < MaxFaults
/\ Reject(i, "failed")
/\ faults' = faults + 1
/\ act' = Act("PrepareFail", i)
/\ UNCHANGED <<made, kind, early, before, aborted, cancel, log, ran, open, done, active, closed, prepAtClose>>
Store(i) ==
/\ ph[i] = "sending"
/\ log' = Append(log, i)
/\ ph' = [ph EXCEPT ![i] = "stored"]
/\ acc' = [acc EXCEPT ![i] = "resolved"]
/\ act' = Act("Store", i)
/\ UNCHANGED <<made, kind, early, before, res, aborted, cancel, ran, open, done, active, closed, faults, prepAtClose>>
StoreFail(i) ==
/\ ph[i] = "sending"
/\ faults < MaxFaults
/\ Reject(i, "failed")
/\ faults' = faults + 1
/\ act' = Act("StoreFail", i)
/\ UNCHANGED <<made, kind, early, before, aborted, cancel, log, ran, open, done, active, closed, prepAtClose>>
\* The signal aborts while the prompt's result is still pending. Before the
\* prompt's send begins, it is withdrawn: never stored, its `accepted`
\* resolved and its result cancelled, and no cancel goes out. Once the send
\* has begun, or the prompt is stored, a cancel naming it goes out.
Abort(i) ==
/\ ph[i] \notin {"unmade", "early"}
/\ ~aborted[i] /\ res[i] = "pending"
/\ aborted' = [aborted EXCEPT ![i] = TRUE]
/\ IF ph[i] = "preparing" /\ AbortWithdraws
THEN /\ ph' = [ph EXCEPT ![i] = "withdrawn"]
/\ acc' = [acc EXCEPT ![i] = "resolved"]
/\ res' = [res EXCEPT ![i] = "cancelled"]
/\ UNCHANGED cancel
ELSE /\ cancel' = [cancel EXCEPT ![i] = "held"]
/\ UNCHANGED <<ph, acc, res>>
/\ act' = Act("Abort", i)
/\ UNCHANGED <<made, kind, early, before, log, ran, open, done, active, closed, faults, prepAtClose>>
CancelStore(i) ==
/\ cancel[i] = "held"
/\ cancel' = [cancel EXCEPT ![i] = "stored"]
/\ act' = Act("CancelStore", i)
/\ UNCHANGED <<made, kind, early, before, ph, acc, res, aborted, log, ran, open, done, active, closed, faults, prepAtClose>>
Close ==
/\ ~closed
/\ closed' = TRUE
/\ prepAtClose' = {i \in Prompts : ph[i] = "preparing"}
/\ act' = Act("Close", 0)
/\ UNCHANGED <<made, kind, early, before, ph, acc, res, aborted, cancel, log, ran, open, done, active, faults>>
\* The worker. A new one starts only while the agent is open.
CanRun == open = 0 /\ (~closed \/ active) /\ Pending(done) # <<>>
Start(i) ==
/\ CanRun /\ Head(Pending(done)) = i /\ cancel[i] # "stored"
/\ open' = i
/\ ran' = [ran EXCEPT ![i] = TRUE]
/\ active' = TRUE
/\ act' = Act("Start", i)
/\ UNCHANGED <<made, kind, early, before, ph, acc, res, aborted, cancel, log, done, closed, faults, prepAtClose>>
\* A turn ends; the worker goes on while the log holds more for it.
Ended(i, how) ==
/\ done' = done \cup {i}
/\ res' = [res EXCEPT ![i] = how]
/\ active' = (Pending(done \cup {i}) # <<>>)
Skip(i) ==
/\ CanRun /\ Head(Pending(done)) = i /\ cancel[i] = "stored"
/\ Ended(i, "cancelled")
/\ act' = Act("Skip", i)
/\ UNCHANGED <<made, kind, early, before, ph, acc, aborted, cancel, log, ran, open, closed, faults, prepAtClose>>
CancelRun(i) ==
/\ open = i /\ i # 0 /\ cancel[i] = "stored"
/\ open' = 0
/\ Ended(i, "cancelled")
/\ act' = Act("CancelRun", i)
/\ UNCHANGED <<made, kind, early, before, ph, acc, aborted, cancel, log, ran, closed, faults, prepAtClose>>
End(i) ==
/\ open = i /\ i # 0 /\ cancel[i] # "stored"
/\ open' = 0
/\ Ended(i, "end")
/\ act' = Act("End", i)
/\ UNCHANGED <<made, kind, early, before, ph, acc, aborted, cancel, log, ran, closed, faults, prepAtClose>>
\* Every prompt made has been stored or refused: the run may stop here.
Settled == \A i \in 1..made : acc[i] # "pending"
Finished == Settled /\ UNCHANGED vars
Next ==
\/ \E i \in Prompts, k \in Kinds, e \in BOOLEAN : Make(i, k, e)
\/ \E i \in Prompts :
\/ Released(i) \/ Prepared(i) \/ PrepareFail(i) \/ Store(i) \/ StoreFail(i)
\/ Abort(i) \/ CancelStore(i) \/ Start(i) \/ Skip(i) \/ CancelRun(i) \/ End(i)
\/ Close
\/ Finished
Spec == Init /\ [][Next]_vars
\* The paths a driver can take. The code makes each of these steps on its
\* own as soon as it can, and a driver acts only once the code has settled,
\* so here the caller and the faults move only when none of them can.
Auto == \E i \in Prompts : Released(i) \/ Start(i) \/ Skip(i) \/ CancelRun(i)
Quiet == ~ENABLED Auto
DriverStep ==
\/ \E i \in Prompts, k \in Kinds, e \in BOOLEAN : Make(i, k, e)
\/ \E i \in Prompts :
Prepared(i) \/ PrepareFail(i) \/ Store(i) \/ StoreFail(i) \/ Abort(i) \/ CancelStore(i) \/ End(i)
\/ Close
DriveSpec == Init /\ [][(Quiet /\ DriverStep) \/ Auto \/ Finished]_vars
(***************************************************************************)
(* Invariants. *)
(***************************************************************************)
\* A prompt made after an image prompt is never stored before it.
ImageOrder ==
\A a, b \in 1..Len(log) : a < b => ~(log[b] < log[a] /\ kind[log[b]] = "image")
\* `accepted` resolves exactly when the prompt is stored, or for a prompt
\* whose signal aborted before its send began, which is never stored.
StoredIffAccepted ==
\A i \in 1..made : (acc[i] = "resolved") <=> (i \in Stored \/ ph[i] \in {"early", "withdrawn"})
\* A cancel goes out only for a prompt whose send has begun: one that is
\* on its way, stored, or whose store failed.
CancelOnlyWhenSent == \A i \in Prompts : cancel[i] # "none" => ph[i] \in {"sending", "stored", "failed"}
\* A withdrawn prompt leaves nothing in the log.
WithdrawnNeverStored == \A i \in Prompts : ph[i] = "withdrawn" => i \notin Stored /\ cancel[i] = "none"
NeverStoredTwice == \A a, b \in 1..Len(log) : a # b => log[a] # log[b]
\* A prompt still preparing when the agent closed is never stored.
ClosedStopsPreparing == prepAtClose \cap Stored = {}
\* Only a prompt's own signal cancels it.
CancelledOnlyOwn == \A i \in Prompts : res[i] = "cancelled" => (aborted[i] \/ early[i])
\* A prompt that could not be stored never runs.
NoRunAfterRejected == \A i \in Prompts : ran[i] => acc[i] = "resolved"
\* One turn at a time, each for a prompt the log holds.
RunsStored == open # 0 => (open \in Stored /\ open \notin done)
=============================================================================
+77 -3
View File
@@ -1,7 +1,8 @@
#!/bin/sh
# Model-checks DurableSession.tla with TLC, confirms each witness is
# violated, and writes the state graphs ../drivers/drive.mjs walks against
# the code.
# Model-checks DurableSession.tla, PromptIntake.tla, ImageStore.tla,
# DurableSteer.tla and AgentRouting.tla with TLC, confirms each witness is
# violated, and writes the state graphs the drivers in ../drivers/ walk
# against the code.
#
# Needs Java 11+ and tla2tools.jar:
# brew install openjdk
@@ -14,6 +15,8 @@
# bun ../drivers/drive.mjs write && bun ../drivers/drive.mjs cutoff
# bun ../drivers/drive.mjs leased # the Heartbeat graph, on a World without liveness
# bun ../drivers/drive.mjs queue # the QueueHeld graph, on a World whose queue shows it
# bun ../drivers/drive-intake.mjs # prompts with images
# bun ../drivers/drive-images.mjs # images stored once, across crashes and reopens
# bun ../drivers/drive-agents.mjs # AgentRouting
# bun ../drivers/drive-steer.mjs ideal && bun ../drivers/drive-steer.mjs faults # DurableSteer
# bun ../drivers/drive-tenants.mjs # TenantIsolation
@@ -98,6 +101,77 @@ for pair in NoInterrupt:NoSteerWaitsOnReply NoHandBack:NothingLost \
fi
done
# PromptIntake.tla: how a session object takes prompts in. Deadlock checking
# stays on, since a prompt left waiting forever is a failure there; the
# drive graph is the one drive-intake.mjs walks.
for config in PromptIntake PromptIntake-drive; do
echo "== $config"
dump=""
[ "$config" = PromptIntake-drive ] && dump="-dump dot,actionlabels graphs/intake"
# shellcheck disable=SC2086 # $dump is two words or none
if ! "$java" -XX:+UseParallelGC -cp "$jar" tlc2.TLC -workers auto -cleanup \
-metadir "$work/$config" $dump -config "$config.cfg" PromptIntake.tla > "$work/$config.log" 2>&1; then
cat "$work/$config.log"; status=1; continue
fi
grep -E 'distinct states|depth of' "$work/$config.log"
done
# NoGate breaks ImageOrder: a prompt that does not wait for an earlier one
# still preparing its images is stored first. NoCloseCheck breaks
# ClosedStopsPreparing: a prompt still preparing when the agent closes is
# stored anyway. GateHeldOnFailure deadlocks: a failed prompt that never
# lets the prompts behind it go leaves them waiting forever. NoWithdraw
# breaks CancelOnlyWhenSent: a prompt that goes on after an abort before
# its send began has its cancel go out before it does.
for pair in NoGate:"Invariant ImageOrder is violated" NoCloseCheck:"Invariant ClosedStopsPreparing is violated" \
GateHeldOnFailure:"Deadlock reached" NoWithdraw:"Invariant CancelOnlyWhenSent is violated"; do
witness="${pair%%:*}" expected="${pair#*:}"
if "$java" -cp "$jar" tlc2.TLC -workers auto -cleanup -metadir "$work/intake-$witness" \
-config "PromptIntake-$witness.cfg" PromptIntake.tla 2>&1 | grep -q -F "$expected"; then
echo "== witness PromptIntake-$witness reached"
else
echo "== witness PromptIntake-$witness NOT reached" >&2; status=1
fi
done
# ImageStore.tla: how a session saves each image once and names it. The
# durable config sends prompts through the log; the engine config straight
# to the core; the drive graph is the one drive-images.mjs walks. Deadlock
# checking stays on: an open that can never finish is a failure.
for config in ImageStore ImageStore-engine ImageStore-drive; do
echo "== $config"
dump=""
[ "$config" = ImageStore-drive ] && dump="-dump dot,actionlabels graphs/images"
# shellcheck disable=SC2086 # $dump is two words or none
if ! "$java" -XX:+UseParallelGC -cp "$jar" tlc2.TLC -workers auto -cleanup \
-metadir "$work/$config" $dump -config "$config.cfg" ImageStore.tla > "$work/$config.log" 2>&1; then
cat "$work/$config.log"; status=1; continue
fi
grep -E 'distinct states|depth of' "$work/$config.log"
done
# SendBeforeStore breaks NoDanglingRef: a prompt sent while its image write
# is in flight leaves a record naming an image the store does not hold.
# NameUnstored breaks NoDanglingRef: a checkpoint names an image an older
# libfx saved inline, which the store never held. DropMissing breaks
# OpenExact: an open that drops an image the store lost runs on a history
# that is not the one saved. RetryLands breaks NoDanglingRef: a retry stores
# nothing, so a log that kept its input would name an image never stored.
# KeepAll breaks WindowHeld: a core that keeps every image outgrows the
# window an open can stage. NeedAll breaks FailsOnlyOnLossInWindow: an open
# that stages every named image fails on one the window had dropped.
# ToolBeforeStore breaks NoDanglingRef: a tool result handed to the core
# while its image write is in flight leaves a record naming an image the
# store does not hold. InlineUncounted breaks WindowBound: a window that
# never counts a tool image the store refused holds more than Window images.
for pair in SendBeforeStore:NoDanglingRef NameUnstored:NoDanglingRef DropMissing:OpenExact RetryLands:NoDanglingRef KeepAll:WindowHeld NeedAll:FailsOnlyOnLossInWindow ToolBeforeStore:NoDanglingRef InlineUncounted:WindowBound; do
witness="${pair%%:*}" invariant="${pair#*:}"
if "$java" -cp "$jar" tlc2.TLC -workers auto -cleanup -metadir "$work/images-$witness" \
-config "ImageStore-$witness.cfg" ImageStore.tla 2>&1 | grep -q -F "Invariant $invariant is violated"; then
echo "== witness ImageStore-$witness reached"
else
echo "== witness ImageStore-$witness NOT reached" >&2; status=1
fi
done
# AgentRouting.tla: which agent runs a session's turns when an app has
# several. Its graph is what ../drivers/drive-agents.mjs walks, with process p2
# lacking one agent; every sender stores its input before it queues it. Drains is checked under fairness, a simulation runs a larger