mirrors/fx
0
0 Fork 0
mirror of https://github.com/vercel-labs/fx.git synced 2026-10-28 10:15:17 +08:00
Code Activity

Remove obsolete local benchmark tooling

Browse Source 
Remove the retired local Harbor benchmark harness and redundant Linux Docker wrapper. Keep the Bun eval suite under tests/evals unchanged.
This commit is contained in:
Pranit committed 2026-08-12 18:42:31 -04:00
1 parent 7e2938c524
commit 0c4b736c4c
93 files changed
-6625
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.gitignore
-7
View File
@@ -24,13 +24,6 @@ sdk/dist/
benchmarks/results/*.json
benchmarks/baseline.json
# Local Harbor release eval artifacts
evals/.build/
evals/jobs-out/
evals/images/fx
evals/**/__pycache__/
evals/**/*.pyc
# Local agent/editor/tooling directories from this workspace
.adal/
.agents/
docker/Dockerfile
-6
View File
@@ -1,6 +0,0 @@
FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y curl xz-utils && rm -rf /var/lib/apt/lists/*
ARG ZIG_VERSION=0.16.0
RUN curl -sSfL https://ziglang.org/download/${ZIG_VERSION}/zig-x86_64-linux-${ZIG_VERSION}.tar.xz \
| tar -xJ -C /opt && ln -s /opt/zig-x86_64-linux-${ZIG_VERSION}/zig /usr/local/bin/zig
WORKDIR /fx
docker/run.sh
-9
View File
@@ -1,9 +0,0 @@
#!/bin/bash
set -e
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
ROOT="$(dirname "$SCRIPT_DIR")"
cd "$ROOT"
docker build --platform linux/amd64 -t fx-linux -f docker/Dockerfile .
docker run --rm --platform linux/amd64 -v "$ROOT:/fx" fx-linux bash docker/test.sh
docker/test.sh
-13
View File
@@ -1,13 +0,0 @@
#!/bin/bash
set -e
echo "=== zig build ==="
zig build
echo "=== zig build test ==="
zig build test
echo "=== Cleanup ==="
rm -rf .zig-cache zig-cache zig-out
echo "=== All Linux checks passed ==="
evals/CLOUD-SANDBOXES.md
-55
View File
@@ -1,55 +0,0 @@
# Harbor Cloud Sandbox Notes
Local Docker remains the default for `fx` release evals. Cloud sandboxes are optional for larger published-dataset sweeps when local container startup and teardown dominate runtime.
Docs used for this slice:
- https://www.harborframework.com/docs/run-jobs/cloud-sandboxes
- https://www.harborframework.com/docs/datasets/publishing
- https://www.harborframework.com/docs/tasks/publishing
- https://www.harborframework.com/docs/sharing/sharing
## Prerequisites
Cloud runs need:
- Harbor auth for any private package references.
- A cloud sandbox provider account. The Harbor docs call out Daytona, Modal, E2B, and Runloop as options.
- Provider credentials configured outside this repository.
- A published dataset reference such as `vercel-labs-fx/fx-release@vX.Y.Z`.
- Agent credentials for the model-backed agent being tested.
Do not require Daytona or other cloud credentials for normal release evals.
## Command Template
Start with a small published-dataset run:
```bash
harbor run \
-d "vercel-labs-fx/fx-release@vX.Y.Z" \
-m "anthropic/claude-sonnet-4.6" \
-a "<agent>" \
-e daytona \
-n 4
```
Increase `-n` only after the smoke passes and provider quotas are clear. The Harbor docs note that cloud sandboxes can parallelize well above local CPU count because command execution shifts to remote sandboxes, but release-team sweeps should still begin with low concurrency.
## Caveats
Daytona accounts may have internet restrictions by default. The Harbor docs mention the `HARBOR_NETWORK` coupon code for removing those restrictions. Treat that as an account setup step, not a repository prerequisite.
Daytona supports multi-container task deployments through `environment/docker-compose.yaml`. The Harbor docs state that Modal, E2B, and Runloop do not currently support multi-container environments, so use single-container tasks with those providers or keep multi-container tasks on Daytona or local Docker.
The current `fx-release` and `fx-multistep` tasks are single-container tasks. Local Docker is still the signoff path because it exercises the same local binary packaging path used by the release harness.
## What Not To Change
Do not edit the default local job YAMLs to require a cloud environment. Keep cloud sandbox commands as explicit operator commands so developers without provider credentials can still run:
```bash
./evals/scripts/run-release-evals.sh
./evals/scripts/run-release-evals.sh --multistep
./evals/scripts/run-terminal-bench.sh --smoke
```
evals/PUBLISHING.md
-119
View File
@@ -1,119 +0,0 @@
# Harbor Publishing Notes
These notes are for release-team publishing dry runs and future registry handoff. Normal release evals stay local and do not publish anything.
Docs used for this slice:
- https://www.harborframework.com/docs/datasets/publishing
- https://www.harborframework.com/docs/tasks/publishing
- https://www.harborframework.com/docs/sharing/sharing
- https://www.harborframework.com/docs/run-jobs/cloud-sandboxes
## Defaults
Harbor publishes tasks and datasets privately unless `--public` is passed. Keep `fx-release` and `fx-multistep` private by default. Do not change visibility or publish from automation unless the release owner explicitly asks for it.
Current local package names:
- `vercel-labs-fx/fx-release`
- `vercel-labs-fx/fx-multistep`
## Preflight
From the repository root:
```bash
./evals/scripts/check-publish-ready.sh
harbor auth status
```
If not signed in:
```bash
harbor auth login
harbor auth status
```
The readiness script only checks local structure. It validates task names, dataset manifests, metric scripts, job config shape, and that generated Harbor output is not tracked. It never calls `harbor publish`.
## Sync
Before publishing, inspect the dataset manifest and refresh Harbor digests intentionally:
```bash
cd evals/datasets/fx-release
harbor add "." --scan
harbor add metric.py
harbor sync
```
For the multi-step dataset:
```bash
cd evals/datasets/fx-multistep
harbor add "." --scan
harbor sync
```
`harbor publish` refreshes local task and metric digests during upload, but an explicit `harbor sync` gives reviewers a separate manifest diff to inspect first.
## Publish
Private dataset publish commands:
```bash
harbor publish "evals/datasets/fx-release" -t "vX.Y.Z"
harbor publish "evals/datasets/fx-multistep" -t "vX.Y.Z"
```
Task-only publishing uses the task path:
```bash
harbor publish "evals/datasets/fx-release/create-file" -t "vX.Y.Z"
```
Use `--public` only when the release owner explicitly wants public registry visibility. Use `--no-tasks` only when tasks have already been published and the dataset manifest should reference existing task revisions.
## Visibility
Visibility can be checked or changed after publishing through Harbor Hub or the CLI:
```bash
harbor dataset visibility "vercel-labs-fx/fx-release" --private
harbor dataset visibility "vercel-labs-fx/fx-multistep" --private
harbor task visibility "vercel-labs-fx/create-file" --private
```
Private packages are visible only to members of the publishing org. Public packages are visible and runnable by everyone.
## Download
Use downloads to inspect a published package without changing local source:
```bash
harbor download "vercel-labs-fx/fx-release@vX.Y.Z" --output-dir evals/.build/downloads
harbor download "vercel-labs-fx/fx-multistep@vX.Y.Z" --output-dir evals/.build/downloads
```
Without `--output-dir`, Harbor downloads to its cache under `~/.cache/harbor`.
## Run By Reference
After publishing, run by registry reference instead of local path:
```bash
harbor run -d "vercel-labs-fx/fx-release@vX.Y.Z" -a "<agent>" -m "<model>"
harbor run -d "vercel-labs-fx/fx-multistep@vX.Y.Z" -a "<agent>" -m "<model>"
```
Local release signoff still uses `evals/jobs/*.yaml` and local Docker by default. Registry references are for sharing and post-publish validation.
## Do Not Publish
Never publish generated run output or host-local binaries:
- `evals/.build/`
- `evals/jobs-out/`
- `evals/images/fx`
- `evals/**/__pycache__/`
- `evals/**/*.pyc`
evals/README.md
-319
View File
@@ -1,319 +0,0 @@
# fx Release Evals
This folder contains manual Harbor release evals for `fx`. It adds a release benchmarking layer above the deterministic suites in `tests/e2e/`, the existing Bun evals in `tests/evals/`, and the startup benchmarks in `benchmarks/`. It does not replace those suites.
The intended release flow is:
```text
cut release -> run local fx release evals -> run Terminal-Bench smoke -> optionally run Terminal-Bench full or compare -> inspect happy/sad result
```
CI is intentionally out of scope for v1. These evals run real agents, require credentials, and can take minutes.
For the full operator guide, including single comparison runs, all configured eval types, result inspection, viewer usage, reporting, artifacts, publishing, RewardKit, and troubleshooting, read `evals/RUNBOOK.md`.
## Harbor Docs
Harbor docs and benchmark pages used by this harness:
- https://www.harborframework.com/docs
- https://www.harborframework.com/docs/getting-started
- https://www.harborframework.com/docs/core-concepts
- https://www.harborframework.com/docs/migration
- https://www.harborframework.com/docs/run-jobs
- https://www.harborframework.com/docs/run-jobs/run-evals
- https://www.harborframework.com/docs/run-jobs/results-and-artifacts
- https://www.harborframework.com/docs/run-jobs/cloud-sandboxes
- https://www.harborframework.com/docs/tasks
- https://www.harborframework.com/docs/tasks/multi-step
- https://www.harborframework.com/docs/tasks/task-difference
- https://www.harborframework.com/docs/tasks/task-tutorial
- https://www.harborframework.com/docs/tasks/windows-container-support
- https://www.harborframework.com/docs/tasks/publishing
- https://www.harborframework.com/docs/datasets
- https://www.harborframework.com/docs/datasets/metrics
- https://www.harborframework.com/docs/datasets/adapters
- https://www.harborframework.com/docs/datasets/adapters-human
- https://www.harborframework.com/docs/datasets/publishing
- https://www.harborframework.com/docs/sharing/sharing
- https://www.harborframework.com/docs/agents
- https://www.harborframework.com/docs/agents/trajectory-format
- https://www.harborframework.com/docs/agents/terminus-2
- https://www.harborframework.com/docs/tutorials/running-terminal-bench
- https://www.harborframework.com/docs/tutorials/mcp-server-task
- https://www.harborframework.com/docs/tutorials/llm-as-a-judge
- https://www.harborframework.com/docs/rewardkit
- https://www.harborframework.com/docs/rewardkit/judge-criteria
- https://www.harborframework.com/docs/rewardkit/built-in-criteria
- https://www.harborframework.com/docs/rewardkit/motivation
- https://www.harborframework.com/docs/training-workflows/sft
- https://www.harborframework.com/docs/training-workflows/rl
- https://www.harborframework.com/docs/contributing/roadmap
- https://hub.harborframework.com/datasets/terminal-bench/terminal-bench-2/latest
## Prerequisites
- Zig 0.16 or newer
- Docker
- `uv`
- Harbor, either installed as `harbor` or runnable through `uvx harbor`
- `AI_GATEWAY_API_KEY` or `VERCEL_OIDC_TOKEN` for `fx`
Comparison runs also need:
- `AI_GATEWAY_API_KEY` for `fx`, Codex CLI, Claude Code, and OpenCode
Before a comparison run, check the environment and job config shape:
```bash
export AI_GATEWAY_API_KEY="<redacted>"
./evals/scripts/check-compare-ready.sh
```
The readiness script validates all eval job YAML files with Harbor `JobConfig`, checks the comparison agent import paths, and verifies `AI_GATEWAY_API_KEY` is present. It prints missing variable names, but it does not print secret values. To validate only YAML and Harbor config shape without credentials:
```bash
./evals/scripts/check-compare-ready.sh --config-only
```
Comparison runners do not modify local Claude Code, Codex, or OpenCode configuration. Codex CLI, Claude Code, and OpenCode receive container-scoped Vercel AI Gateway environment/config only for the Harbor process. The Codex comparison path does not read or upload the host Codex `auth.json`.
RewardKit is optional. The examples under `evals/rewardkit/` do not need RewardKit installed unless you want the local readiness script to run its tiny deterministic built-in criteria smoke. LLM and agent judges can cost money and must not run in normal release signoff unless explicitly enabled.
## Run
From the repository root:
```bash
./evals/scripts/check-release-ready.sh
./evals/scripts/run-release-signoff.sh
```
The signoff wrapper runs the required local `fx-release` evals, then the registered Terminal-Bench smoke, then writes a dated Markdown report under `evals/runs/` and updates `evals/runs/RUNS.md`.
Optional signoff flags:
```bash
./evals/scripts/run-release-signoff.sh --multistep
./evals/scripts/run-release-signoff.sh --compare
./evals/scripts/run-release-signoff.sh --full-terminal-bench
./evals/scripts/run-release-signoff.sh --compare --full-terminal-bench
./evals/scripts/run-release-signoff.sh --view
```
`--full-terminal-bench` is the explicit approval gate for the full Terminal-Bench dataset. `--compare --full-terminal-bench` is the explicit approval gate for the full multi-agent Terminal-Bench comparison. Both paths are slow and can cost money, so the wrapper prints warnings before running them.
Use the individual runners below for debugging a specific slice.
For only the local `fx-release` job:
```bash
./evals/scripts/run-release-evals.sh
```
For the comparison job:
```bash
./evals/scripts/check-release-ready.sh --compare
./evals/scripts/run-release-evals.sh --compare
```
For the separate multi-step smoke:
```bash
./evals/scripts/run-release-evals.sh --multistep
```
The runner builds a ReleaseSafe Linux `fx` binary, bakes it into the local `fx-evals:local` Docker image, runs Harbor, and prints a compact pass/fail summary.
After changing local task environments, solutions, or verifiers, run:
```bash
./evals/scripts/check-local-tasks.sh
```
This check expects `fx-evals:local` to exist. Run `./evals/scripts/build-image.sh` first after changing the binary or task environments.
For a non-publishing registry preflight:
```bash
./evals/scripts/check-publish-ready.sh
```
This validates local task names, dataset manifests, metric scripts, job config shape, and that generated Harbor output is not tracked. It does not call `harbor publish`.
For optional RewardKit example validation:
```bash
./evals/scripts/check-rewardkit-ready.sh
```
This parses RewardKit example TOML, checks the built-in criteria example shape, and runs a deterministic built-in criteria smoke only when `rewardkit` is importable locally. It skips LLM and agent judge execution unless `FX_REWARDKIT_RUN_JUDGES=1` is explicitly set.
For the registered Terminal-Bench smoke:
```bash
./evals/scripts/run-terminal-bench.sh --smoke
```
For the full Terminal-Bench run:
```bash
./evals/scripts/run-terminal-bench.sh --full
```
For the Terminal-Bench comparison smoke:
```bash
./evals/scripts/check-release-ready.sh --compare
./evals/scripts/run-terminal-bench.sh --smoke --compare
```
For the full comparison against `fx`, Codex CLI, Claude Code, and OpenCode, get explicit release-owner approval first:
```bash
./evals/scripts/run-terminal-bench.sh --full --compare
```
The Terminal-Bench runner builds the ReleaseSafe Linux `fx` binary and uploads it into each registered task container during `FxAgent.install()`. It does not require the `fx-evals:local` image for registered datasets.
## Dataset Modes
`fx-release` is a local dataset. The job YAML uses `path: evals/datasets/fx-release`, so Harbor runs the task directories from this checkout. This is the deterministic release gate for the local `fx` binary and the custom `fx-evals:local` image.
Terminal-Bench is a published dataset. The job YAML uses `name: terminal-bench/terminal-bench-2`, so Harbor resolves it from the registry and downloads tasks into `evals/.build/harbor-datasets`. `FxAgent` uploads the freshly built Linux `fx` binary into each registered task container during setup.
`fx-multistep` is a separate local dataset for Harbor step semantics. Its task uses `steps/<name>/instruction.md`, `steps/<name>/tests/test.sh`, and `steps/<name>/solution/solve.sh` so each verifier runs against state left by the previous step. It is not part of the default release gate.
Every job declares `evals/metrics/fx_metrics.py` as a Harbor `uv-script` metric. The metric emits `mean_reward`, `pass_rate`, `n_trials`, and `n_passed`, treating missing or null rewards as zero. The local dataset `metric.py` delegates to the shared script for direct local dataset runs.
RewardKit examples live in `evals/rewardkit/` and are documented in `evals/REWARDKIT.md`. They are not wired into `evals/jobs/fx-release.yaml`, Terminal-Bench jobs, or the default release runner.
## Coverage Boundaries
The release harness is configured for Harbor's local Linux Docker path, registered Terminal-Bench runs, metrics, artifacts, publishing preflight, cloud-sandbox command templates, optional multi-step coverage, optional RewardKit examples, and explicit multi-agent comparison.
The remaining Harbor docs describe capabilities that are intentionally not part of the default release gate:
- Windows tasks: not configured because `fx-release`, `fx-multistep`, and Terminal-Bench run in Linux containers. A Windows-targeted task would need `[environment].os = "windows"` plus `.bat` solution and verifier entrypoints.
- MCP sidecar tasks: not configured because none of the current release tasks need a sidecar service. Add `environment/docker-compose.yaml`, `environment.mcp_servers`, and health checks only when a task actually needs an MCP server or another service container.
- LLM-as-a-Judge and RewardKit judges: documented as opt-in examples only. The default release gate stays deterministic to avoid paid, subjective verifier drift.
- ATIF, SFT, and RL workflows: not configured as release gates. Harbor can consume ATIF trajectories for viewer, SFT, and RL workflows, and Codex or Claude Code may produce trajectories through Harbor's installed agents. The `fx` wrapper currently preserves a compact `fx` JSON result bundle rather than claiming first-class ATIF export.
- Terminus-2 and Harbor roadmap items: not in the default comparison matrix because the release comparison target set is `fx`, Codex CLI, Claude Code, and OpenCode. Add Terminus-2 only when a Harbor-native reference baseline is needed. Track roadmap-only items separately until they become released setup docs.
- Full Terminal-Bench: configured but explicitly gated by `--full-terminal-bench` because it is slow and can cost money.
## Publishing And Cloud Sandboxes
Publishing is private by default and is not part of normal release signoff. Use `evals/PUBLISHING.md` for the exact auth, sync, publish, tag, visibility, download, and run-by-reference commands. Do not publish or change package visibility unless the release owner explicitly asks for it.
Cloud sandboxes are optional for larger published-dataset sweeps. Local Docker remains the default for release evals. Use `evals/CLOUD-SANDBOXES.md` for Daytona-style command templates, concurrency guidance, internet restriction notes, and multi-container caveats.
## Artifacts
Raw Harbor output is written under `evals/jobs-out/<job-name>/`. Each trial has `result.json`, `trial.log`, and an `artifacts/manifest.json` that records collection status.
`FxAgent` writes a compact run bundle to `/logs/artifacts`, which Harbor collects by convention into the trial `artifacts/` directory:
- `fx-exit-code.txt`
- `fx-result.json`
- `fx-stderr.txt`
- `fx-wrapper.json`
The job configs also collect `/logs/agent` as `artifacts/agent`, `/logs/verifier` as `artifacts/verifier`, and `/app` as `artifacts/workspace`. The `fx`-only jobs additionally collect `/tmp/fx-home/.fx` as `artifacts/fx-home`; comparison jobs intentionally omit that `fx`-specific path so Codex CLI, Claude Code, and OpenCode runs do not produce expected artifact warnings.
The compact `/logs/artifacts` bundle is also collected as `artifacts/fx-run` by job config and may appear at the trial artifact root when Harbor copies installed-agent artifacts.
Workspace collection intentionally excludes bulky generated directories from `/app`: `node_modules`, `.git`, `.cache`, `__pycache__`, `.pytest_cache`, `dist`, `build`, `coverage`, `target`, `.venv`, and `venv`.
For multi-step tasks, Harbor collects artifacts once per step. Per-step manifests and files live under `evals/jobs-out/<job>/<trial>/steps/<step-name>/artifacts/`. The root `artifacts/manifest.json` remains the single-step location.
To inspect full results:
```bash
harbor view evals/jobs-out
```
Or use the runner shortcut:
```bash
./evals/scripts/run-terminal-bench.sh --view
```
Preserve human-readable run reports under `evals/runs/`. The signoff wrapper calls `evals/scripts/write-run-report.py`, which writes one dated Markdown file and updates `evals/runs/RUNS.md`. For manual reports:
```bash
./evals/scripts/write-run-report.py evals/jobs-out/<job-dir> [evals/jobs-out/<job-dir> ...]
```
Do not add top-level run log files in `evals/`.
## Release Signoff Flow
Default release signoff is one command:
```bash
./evals/scripts/run-release-signoff.sh
```
Decision criteria:
- `fx-release` must pass.
- Terminal-Bench smoke must complete without Harbor harness exceptions. The verifier score can be inspected separately because the smoke is a published benchmark sanity check, not the deterministic release gate.
- Comparison is recommended for major releases or before public benchmark claims.
- Full Terminal-Bench requires explicit release-owner approval through `--full-terminal-bench`.
Run these supporting checks before or during release signoff:
1. `./evals/scripts/check-release-ready.sh`
2. `./evals/scripts/run-release-signoff.sh`
3. `./evals/scripts/check-local-tasks.sh` after task or verifier edits
4. Optional multi-step coverage: `./evals/scripts/run-release-signoff.sh --multistep`
5. Recommended for major releases or before public benchmark claims: `./evals/scripts/run-release-signoff.sh --compare`
6. Full Terminal-Bench comparison requires explicit approval: `./evals/scripts/run-release-signoff.sh --compare --full-terminal-bench`
7. Optional publishing preflight only: `./evals/scripts/check-publish-ready.sh`
`fx-release` remains required for every release. Comparison evals are recommended for major releases or before public benchmark claims. Full Terminal-Bench comparison is slow and costly enough that it should only run after explicit approval.
RewardKit example validation is separate from release signoff:
```bash
./evals/scripts/check-rewardkit-ready.sh
```
Do not treat LLM or agent judge output as part of normal release signoff unless the release owner explicitly asks for that paid, slower workflow.
## Files
- `agents/fx_agent.py`: custom Harbor installed-agent wrapper that uploads the local Linux `fx` binary into each task container.
- `images/Dockerfile`: task base image with `/usr/local/bin/fx`.
- `datasets/fx-release/`: local Harbor dataset ported from the first release eval behaviors in `tests/evals/`.
- `datasets/fx-multistep/`: separate local Harbor dataset for sequential step coverage.
- `adapters/README.md`: manual adapter scaffold for porting future Bun evals into Harbor tasks.
- `PUBLISHING.md`: private-by-default Harbor publishing workflow.
- `CLOUD-SANDBOXES.md`: optional cloud sandbox command guide.
- `RUNBOOK.md`: full operator guide for running evals, comparisons, result inspection, reports, artifacts, and troubleshooting.
- `metrics/fx_metrics.py`: shared Harbor metric script used by every job.
- `jobs/fx-release.yaml`: local `fx` release job.
- `jobs/fx-multistep.yaml`: separate multi-step job for `fx`.
- `jobs/compare-key-agents.yaml`: local comparison job for `fx`, Codex CLI, Claude Code, and OpenCode.
- `jobs/fx-terminal-bench-smoke.yaml`: single-task registered Terminal-Bench smoke job for `fx`.
- `jobs/fx-terminal-bench.yaml`: full registered Terminal-Bench job for `fx`.
- `jobs/compare-terminal-bench-smoke.yaml`: single-task registered Terminal-Bench comparison job for `fx`, Codex CLI, Claude Code, and OpenCode.
- `jobs/compare-terminal-bench.yaml`: full registered Terminal-Bench comparison job for `fx`, Codex CLI, Claude Code, and OpenCode.
- `runs/`: dated, human-readable run reports and `RUNS.md` index.
- `REWARDKIT.md`: optional RewardKit policy and workflow for release evals.
- `rewardkit/`: example RewardKit criteria files, not connected to release jobs.
- `scripts/build-image.sh`: builds the Linux binary and Docker image.
- `scripts/check-local-tasks.sh`: runs checked-in local task solutions and verifiers in Docker.
- `scripts/check-publish-ready.sh`: non-publishing registry readiness wrapper.
- `scripts/check-publish-ready.py`: non-publishing registry readiness checks.
- `scripts/check-rewardkit-ready.sh`: optional RewardKit example readiness wrapper.
- `scripts/check-rewardkit-ready.py`: local RewardKit example shape checks.
- `scripts/check-compare-ready.sh`: comparison credential and Harbor job config readiness wrapper.
- `scripts/check-compare-ready.py`: safe comparison env, YAML, import-path, and `JobConfig` checks.
- `scripts/check-release-ready.sh`: consolidated release readiness wrapper for tools, Docker, credentials, job config shape, publishing, RewardKit, and optional comparison credentials.
- `scripts/run-release-evals.sh`: one-command local runner.
- `scripts/run-terminal-bench.sh`: one-command Terminal-Bench runner.
- `scripts/run-release-signoff.sh`: one-command release signoff wrapper for local release evals, Terminal-Bench, optional comparison, optional multi-step, reporting, and viewer launch.
- `scripts/summarize-results.py`: compact benchmark summary report.
- `scripts/write-run-report.py`: Markdown report generator for one or more Harbor job output directories.
evals/REWARDKIT.md
-75
View File
@@ -1,75 +0,0 @@
# RewardKit In fx Release Evals
RewardKit is optional for this eval slice. It can make some Harbor verifier criteria easier to share, but it does not replace the checked-in deterministic bash verifiers under `evals/datasets/*/*/tests/test.sh`.
Reference docs:
- https://www.harborframework.com/docs/rewardkit
- https://www.harborframework.com/docs/rewardkit/built-in-criteria
- https://www.harborframework.com/docs/rewardkit/judge-criteria
- https://www.harborframework.com/docs/rewardkit/motivation
- https://www.harborframework.com/docs/agents/trajectory-format
- https://www.harborframework.com/docs/tutorials/llm-as-a-judge
## Default Release Signoff
Normal fx release signoff must continue to use the existing deterministic release path:
1. `./evals/scripts/run-release-signoff.sh`
2. `./evals/scripts/check-local-tasks.sh` after task or verifier edits
Those checks use explicit task scripts and deterministic verifiers. They are the release signal because they are inspectable, cheap enough to repeat, and do not depend on a model judging its own output.
Do not add LLM or agent judges to normal release signoff unless the release owner explicitly asks for that specific run.
## Which Verifier To Use
| Verifier shape | Use it for | Release signoff default |
| --- | --- | --- |
| Deterministic bash verifier | Exact filesystem, command, JSON, CLI, and test outcomes. This is the right default when a task has objective pass/fail behavior. | Yes |
| RewardKit built-in criteria | Concise deterministic file, command, JSON, CSV, HTTP, image, or trajectory checks when RewardKit is already part of the task verifier. Prefer this for small future Harbor tasks, not as a silent rewrite of existing release verifiers. | No, unless the task has explicitly adopted it |
| RewardKit LLM judge | Qualitative review such as readability, risk quality, explanation quality, or release-note usefulness. Use only after objective checks pass. | No |
| RewardKit agent judge | Qualitative review that needs workspace exploration or command execution. This is slower and broader than an LLM judge. | No |
| Trajectory-oriented criteria | Future process review if `fx` or Harbor emits ATIF trajectory artifacts for a task. Use it to inspect approach, tool usage, or turn budget, not final correctness. | No |
## Cost And Gating
RewardKit built-in criteria can be deterministic. LLM judges and agent judges are different:
- They are optional.
- They are slower than bash verifiers.
- They can cost money.
- They can depend on network access and provider credentials.
- They must not run during normal release signoff unless explicitly enabled.
Use `FX_REWARDKIT_RUN_JUDGES=1` only in a deliberate paid eval workflow. The local readiness script validates judge TOML shape by default but does not execute judge examples by default.
## Examples
The example criteria files live under `evals/rewardkit/`:
- `built_in_file_command_json.py`: built-in file, command, and JSON criteria.
- `qualitative_review.toml`: LLM judge rubric for qualitative review.
- `trajectory_review.toml`: optional future trajectory-oriented judge shape for ATIF-style artifacts.
Harbor's ATIF docs list Codex CLI and Claude Code as agents with automatic trajectory support. The custom `fx` wrapper does not claim that yet; it keeps `fx-result.json`, `fx-stderr.txt`, `fx-exit-code.txt`, and wrapper metadata in `/logs/artifacts`. Add true ATIF export only when process-quality judging, SFT export, or RL rollout work needs it.
These files are examples. They are not referenced by `evals/jobs/fx-release.yaml`, `evals/jobs/fx-terminal-bench-smoke.yaml`, or the default release runner.
## Local Readiness Check
Run this from the repository root:
```bash
./evals/scripts/check-rewardkit-ready.sh
```
The check:
- Parses checked-in task, dataset, and RewardKit example TOML.
- Validates the judge TOML example shape.
- Validates that the built-in criteria example mentions file, command, and JSON checks.
- Runs a tiny deterministic built-in criteria smoke only when `rewardkit` is already importable by the local Python interpreter.
- Skips LLM and agent judge execution unless explicitly enabled.
If RewardKit is not installed locally, the deterministic smoke is skipped. That skip is expected for normal release signoff.
evals/RUNBOOK.md
-462
View File
@@ -1,462 +0,0 @@
# fx Harbor Eval Runbook
This is the operator guide for the Harbor eval harness under `evals/`. It covers how to run every configured eval path, how to run one comparison, how to find results, how to inspect artifacts, and how to decide whether a release passes.
Run every command from the repository root:
```bash
cd /Users/faxes/Developer/Fx/fx
```
Do not run bare `fx` for local verification. The eval scripts build and use the local binary from this checkout.
## What Is Configured
| Eval type | Job config | Runner | Agents | Default release gate |
| --- | --- | --- | --- | --- |
| Local release evals | `evals/jobs/fx-release.yaml` | `./evals/scripts/run-release-evals.sh` | `fx` | Yes |
| Local release comparison | `evals/jobs/compare-key-agents.yaml` | `./evals/scripts/run-release-evals.sh --compare` | `fx`, Codex CLI, Claude Code, OpenCode | Recommended before benchmark claims |
| Local multi-step smoke | `evals/jobs/fx-multistep.yaml` | `./evals/scripts/run-release-evals.sh --multistep` | `fx` | Optional |
| Terminal-Bench smoke | `evals/jobs/fx-terminal-bench-smoke.yaml` | `./evals/scripts/run-terminal-bench.sh --smoke` | `fx` | Yes, as harness sanity |
| Terminal-Bench full | `evals/jobs/fx-terminal-bench.yaml` | `./evals/scripts/run-terminal-bench.sh --full` | `fx` | Explicit approval only |
| Terminal-Bench comparison smoke | `evals/jobs/compare-terminal-bench-smoke.yaml` | `./evals/scripts/run-terminal-bench.sh --smoke --compare` | `fx`, Codex CLI, Claude Code, OpenCode | Recommended before benchmark claims |
| Terminal-Bench full comparison | `evals/jobs/compare-terminal-bench.yaml` | `./evals/scripts/run-terminal-bench.sh --full --compare` | `fx`, Codex CLI, Claude Code, OpenCode | Explicit approval only |
The one-command wrapper combines the release paths:
```bash
./evals/scripts/run-release-signoff.sh
```
By default it runs local `fx-release`, runs Terminal-Bench smoke, writes a dated report under `evals/runs/`, and updates `evals/runs/RUNS.md`.
## Prerequisites
Install or start:
- Zig 0.16 or newer
- Docker Desktop or Colima with a running Docker daemon
- `uv`
- Harbor, either as `harbor`, `uvx harbor`, or `uv tool run harbor`
Set credentials:
```bash
export AI_GATEWAY_API_KEY="<redacted>"
```
`fx`-only runs may also work with `VERCEL_OIDC_TOKEN`, but comparison runs require `AI_GATEWAY_API_KEY` because Codex CLI, Claude Code, and OpenCode are routed through Vercel AI Gateway inside task containers.
Comparison runs do not modify host Codex, Claude Code, or OpenCode config. The comparison wrappers create container-scoped config only for the Harbor task process.
## Preflight Checks
Run the normal release preflight:
```bash
./evals/scripts/check-release-ready.sh
```
Run the comparison preflight:
```bash
./evals/scripts/check-release-ready.sh --compare
```
Check only Harbor job config shape without credentials:
```bash
./evals/scripts/check-compare-ready.sh --config-only
```
After editing local task Dockerfiles, task TOML, oracle solutions, or verifier scripts, run:
```bash
./evals/scripts/build-image.sh
./evals/scripts/check-local-tasks.sh
```
## Release Signoff
Default release signoff:
```bash
./evals/scripts/check-release-ready.sh
./evals/scripts/run-release-signoff.sh
```
Default pass criteria:
- `fx-release` must pass.
- Terminal-Bench smoke must complete without Harbor harness exceptions.
- Artifact collection failures should be zero.
- The generated report should be added under `evals/runs/` if it is a meaningful release or comparison run.
Recommended major-release signoff with comparison:
```bash
./evals/scripts/check-release-ready.sh --compare
./evals/scripts/run-release-signoff.sh --compare
```
Optional multi-step coverage:
```bash
./evals/scripts/run-release-signoff.sh --multistep
```
Full Terminal-Bench release run, only after explicit approval:
```bash
./evals/scripts/run-release-signoff.sh --full-terminal-bench
```
Full Terminal-Bench comparison, only after explicit approval:
```bash
./evals/scripts/run-release-signoff.sh --compare --full-terminal-bench
```
Open the Harbor viewer after a signoff run:
```bash
./evals/scripts/run-release-signoff.sh --view
```
## Single Comparison
Use this when you want exactly one side-by-side comparison job on the local release dataset:
```bash
./evals/scripts/check-release-ready.sh --compare
./evals/scripts/run-release-evals.sh --compare
```
That runs `evals/jobs/compare-key-agents.yaml` against `evals/datasets/fx-release` with:
- `fx`
- Codex CLI
- Claude Code
- OpenCode
Use this when you want exactly one side-by-side Terminal-Bench smoke comparison:
```bash
./evals/scripts/check-release-ready.sh --compare
./evals/scripts/run-terminal-bench.sh --smoke --compare
```
Do not use the full Terminal-Bench comparison as a casual smoke:
```bash
./evals/scripts/run-terminal-bench.sh --full --compare
```
That path is intentionally slow and can cost money across all agents.
## Individual Eval Commands
Local `fx` release evals only:
```bash
./evals/scripts/run-release-evals.sh
```
Local comparison only:
```bash
./evals/scripts/run-release-evals.sh --compare
```
Local multi-step smoke only:
```bash
./evals/scripts/run-release-evals.sh --multistep
```
Terminal-Bench smoke for `fx` only:
```bash
./evals/scripts/run-terminal-bench.sh --smoke
```
Terminal-Bench full for `fx` only:
```bash
./evals/scripts/run-terminal-bench.sh --full
```
Terminal-Bench smoke comparison:
```bash
./evals/scripts/run-terminal-bench.sh --smoke --compare
```
Terminal-Bench full comparison:
```bash
./evals/scripts/run-terminal-bench.sh --full --compare
```
Open existing Harbor output without running a job:
```bash
./evals/scripts/run-terminal-bench.sh --view
```
## Results
Raw Harbor output is written to:
```text
evals/jobs-out/<timestamp>/
```
`evals/jobs-out/` is ignored by git because it can contain bulky local artifacts.
Each job directory normally contains:
- `config.json`: resolved Harbor job config.
- `job.log`: top-level Harbor run log.
- `result.json`: machine-readable job result and metrics.
- `lock.json`: Harbor lock metadata.
- one directory per trial.
Each trial normally contains:
- `result.json`: trial reward, exception, agent, model, and timing fields.
- `trial.log`: trial log.
- `artifacts/manifest.json`: artifact collection status.
- `artifacts/agent/`: agent logs and trajectories when available.
- `artifacts/verifier/`: verifier logs and `reward.txt`.
- `artifacts/workspace/`: collected `/app` workspace evidence.
- `artifacts/fx-run/`: compact `fx` wrapper bundle when produced.
- `artifacts/fx-home/`: only in `fx`-only jobs.
Multi-step jobs place step artifacts under:
```text
evals/jobs-out/<timestamp>/<trial>/steps/<step-name>/artifacts/
```
## Summaries
Print a compact summary for the latest job:
```bash
./evals/scripts/summarize-results.py evals/jobs-out
```
Print a compact summary for a specific job:
```bash
./evals/scripts/summarize-results.py evals/jobs-out/<timestamp>
```
The summary reports:
- `n_trials`
- `n_passed`
- `pass_rate`
- `mean_reward`
- failed tasks
- exceptions
- artifact collection failures
- per-agent comparison tables when the job has multiple agents
For comparison jobs, inspect both the summary and the generated report before making claims. A pass rate can hide timeouts or setup exceptions unless you check the exception and artifact sections.
## Viewer
Open the Harbor viewer:
```bash
harbor view evals/jobs-out
```
If Harbor is only available through `uvx`:
```bash
uvx harbor view evals/jobs-out
```
The runner also prints the correct viewer command after a run. Use:
```bash
./evals/scripts/run-terminal-bench.sh --view
```
The viewer is useful for drilling into trial logs, artifacts, rewards, exceptions, and trajectories.
## Reports
The signoff wrapper writes a dated Markdown report automatically:
```text
evals/runs/YYYY-MM-DD-release-signoff.md
```
It also updates:
```text
evals/runs/RUNS.md
```
Generate or regenerate a report manually:
```bash
./evals/scripts/write-run-report.py evals/jobs-out/<timestamp>
```
Generate one report from multiple job directories:
```bash
./evals/scripts/write-run-report.py \
evals/jobs-out/<release-job> \
evals/jobs-out/<terminal-bench-job>
```
For comparison signoff, include both the local comparison job and the Terminal-Bench comparison smoke job when generating a report.
## Artifacts To Check
Start with:
```text
evals/jobs-out/<timestamp>/result.json
evals/jobs-out/<timestamp>/job.log
```
Then inspect failed trials:
```text
evals/jobs-out/<timestamp>/<trial>/result.json
evals/jobs-out/<timestamp>/<trial>/trial.log
evals/jobs-out/<timestamp>/<trial>/artifacts/manifest.json
evals/jobs-out/<timestamp>/<trial>/artifacts/verifier/
evals/jobs-out/<timestamp>/<trial>/artifacts/agent/
evals/jobs-out/<timestamp>/<trial>/artifacts/workspace/
```
For `fx` trials, also check:
```text
artifacts/fx-run/fx-exit-code.txt
artifacts/fx-run/fx-result.json
artifacts/fx-run/fx-stderr.txt
artifacts/fx-run/fx-wrapper.json
```
Artifact collection failures should be treated as harness problems unless the path is explicitly optional and documented.
## Publishing And Cloud Sandboxes
Publishing is not part of default release signoff. Use:
```bash
./evals/scripts/check-publish-ready.sh
```
Then follow:
```text
evals/PUBLISHING.md
```
Cloud sandboxes are optional for larger published-dataset sweeps. Follow:
```text
evals/CLOUD-SANDBOXES.md
```
Local Docker remains the default environment for release signoff.
## RewardKit
RewardKit examples are optional and are not part of default release signoff.
Validate the examples:
```bash
./evals/scripts/check-rewardkit-ready.sh
```
The script parses example TOML and only runs deterministic RewardKit smoke checks when `rewardkit` is already importable locally. It does not run paid LLM or agent judges unless explicitly enabled:
```bash
FX_REWARDKIT_RUN_JUDGES=1 ./evals/scripts/check-rewardkit-ready.sh
```
Read:
```text
evals/REWARDKIT.md
```
## Troubleshooting
Docker daemon unavailable:
```bash
docker info
```
Start Docker Desktop or Colima, then rerun the preflight. On Colima, the scripts auto-detect `$HOME/.colima/default/docker.sock` when it exists.
Harbor command missing:
```bash
uvx harbor --help
```
The scripts use `harbor`, then `uvx harbor`, then `uv tool run harbor`.
Comparison preflight fails:
```bash
export AI_GATEWAY_API_KEY="<redacted>"
./evals/scripts/check-release-ready.sh --compare
```
The current comparison path does not require `CODEX_AUTH_JSON_PATH` or `CODEX_ALLOW_AUTH_UPLOAD`.
Local task verifier fails without a reward:
- Fix the verifier so every bad-solution path writes `/logs/verifier/reward.txt`.
- Rerun `./evals/scripts/check-local-tasks.sh`.
Host binary becomes the wrong architecture:
- This should not happen. `evals/scripts/build-image.sh` writes the Linux eval binary under `evals/.build/linux/bin/fx` and copies it to `evals/images/fx`.
- Rebuild the native binary with `zig build`.
- Confirm with `./zig-out/bin/fx help`.
Artifact warnings in comparison jobs:
- Comparison jobs intentionally do not collect `/tmp/fx-home/.fx`.
- If warnings appear, inspect `artifacts/manifest.json` and the job YAML artifact list.
Full Terminal-Bench takes too long or costs too much:
- Use smoke jobs by default.
- Require explicit release-owner approval before `--full` or `--full-terminal-bench`.
## Git Hygiene
Do not commit local Harbor output:
```text
evals/.build/
evals/jobs-out/
evals/images/fx
evals/**/__pycache__/
evals/**/*.pyc
```
Check before committing:
```bash
git status --short --ignored=matching .gitignore evals
```
Only tracked docs, scripts, job configs, task definitions, metrics, adapters, and preserved run reports should be committed.
evals/adapters/README.md
-56
View File
@@ -1,56 +0,0 @@
# fx Harbor Adapter Notes
This folder documents how future `tests/evals` Bun scenarios should be ported into Harbor task directories.
Docs used for this adapter scaffold:
- https://www.harborframework.com/docs/datasets/adapters
- https://www.harborframework.com/docs/datasets/adapters-human
- https://www.harborframework.com/docs/tasks
- https://www.harborframework.com/docs/tasks/task-difference
- https://www.harborframework.com/docs/tasks/task-tutorial
- https://www.harborframework.com/docs/migration
- https://www.harborframework.com/docs/tasks/windows-container-support
## Current Support
There is no automatic converter in this slice. The Bun eval files under `tests/evals` combine prompt setup, fixture setup, runner behavior, and judge assertions through TypeScript helpers, so the supported path is manual porting into explicit Harbor tasks.
Supported validation command for the manually ported local tasks:
```bash
./evals/scripts/check-local-tasks.sh
```
That command expects `fx-evals:local` to exist, builds each local Harbor task image, runs the checked-in oracle solution, then runs the verifier and requires `/logs/verifier/reward.txt` to contain `1`.
## Future Adapter CLI Shape
If a converter is added, its command line must match the Harbor adapter contract:
```bash
uv run python -m fx_bun_adapter.main --output-dir evals/datasets/fx-bun
uv run python -m fx_bun_adapter.main --output-dir evals/datasets/fx-bun --limit 5
uv run python -m fx_bun_adapter.main --output-dir evals/datasets/fx-bun --task-ids create-file,run-command
uv run python -m fx_bun_adapter.main --output-dir evals/datasets/fx-bun --overwrite
```
Flag semantics:
- `--output-dir`: required destination for generated Harbor task directories.
- `--limit`: optional maximum number of tasks to emit after `--task-ids` filtering.
- `--overwrite`: replace existing generated task directories. Without it, existing directories must be preserved and reported as an error.
- `--task-ids`: comma-separated upstream task ids, using the `tests/evals/<task-id>.test.ts` file stem. Generated Harbor task names should be `vercel-labs-fx/<task-id>`.
## Manual Porting Checklist
For each Bun eval chosen for Harbor:
1. Convert the user-facing prompt into `instruction.md`. Include the expected output paths and constraints, but not oracle answers.
2. Put deterministic environment setup in `environment/Dockerfile` or step `workdir/` files.
3. Put the oracle in `solution/solve.sh` for single-step tasks, or under each `steps/<name>/solution/solve.sh` for multi-step tasks.
4. Put verifier logic in `tests/test.sh` or `steps/<name>/tests/test.sh`.
5. Always write `/logs/verifier/reward.txt` with a numeric reward on bad-solution paths.
6. Prefer absolute paths in verifiers.
7. Keep generated tasks Linux-targeted unless a source eval explicitly needs Windows container behavior. Windows tasks require `[environment].os = "windows"` plus `.bat` entrypoints.
8. Keep adapter parity and comparison work manual until the TypeScript helpers have a stable metadata format that can be converted without guessing.
evals/agents/__init__.py
-1
View File
@@ -1 +0,0 @@
"""Custom Harbor agents for fx release evals."""
evals/agents/claude_gateway.py
-79
View File
@@ -1,79 +0,0 @@
from __future__ import annotations
import os
import shlex
from pathlib import Path
from typing import Any
from harbor.agents.installed.claude_code import ClaudeCode
class ClaudeCodeGatewayAgent(ClaudeCode):
"""Claude Code wrapper with container-scoped Vercel AI Gateway env."""
_GATEWAY_BASE_URL = "https://ai-gateway.vercel.sh"
_CONFIG_DIR = "/tmp/claude-code-config"
@staticmethod
def name() -> str:
return "claude-code"
def __init__(
self,
logs_dir: Path,
*args: Any,
extra_env: dict[str, str] | None = None,
**kwargs: Any,
) -> None:
gateway_key = os.environ.get("AI_GATEWAY_API_KEY", "")
if not gateway_key:
raise ValueError("AI_GATEWAY_API_KEY is required for Claude Code compare")
scoped_env = dict(extra_env) if extra_env else {}
scoped_env["ANTHROPIC_BASE_URL"] = self._GATEWAY_BASE_URL
scoped_env["ANTHROPIC_AUTH_TOKEN"] = gateway_key
scoped_env["ANTHROPIC_API_KEY"] = ""
scoped_env["CLAUDE_CONFIG_DIR"] = self._CONFIG_DIR
super().__init__(logs_dir, *args, extra_env=scoped_env, **kwargs)
async def _prepare_agent_log_dir(self, environment: Any) -> None:
user = str(getattr(environment, "default_user", None) or "agent")
await self.exec_as_root(
environment,
command=(
"mkdir -p /logs/agent /logs/artifacts && "
f"(chown -R {shlex.quote(user)} /logs/agent /logs/artifacts "
"2>/dev/null || true) && "
"chmod -R a+rwX /logs/agent /logs/artifacts"
),
)
async def _copy_sessions_to_logs(self, environment: Any) -> None:
user = str(getattr(environment, "default_user", None) or "agent")
await self.exec_as_root(
environment,
command=(
f"if [ -d {shlex.quote(self._CONFIG_DIR)} ]; then "
"rm -rf /logs/agent/sessions && "
"mkdir -p /logs/agent && "
f"cp -R {shlex.quote(self._CONFIG_DIR)} /logs/agent/sessions && "
f"(chown -R {shlex.quote(user)} /logs/agent/sessions "
"2>/dev/null || true) && "
"chmod -R a+rwX /logs/agent/sessions; "
"fi"
),
)
async def run(self, *args: Any, **kwargs: Any) -> None:
environment = kwargs.get("environment")
if environment is None and len(args) >= 2:
environment = args[1]
if environment is None:
raise ValueError("ClaudeCodeGatewayAgent.run requires an environment")
await self._prepare_agent_log_dir(environment)
try:
await super().run(*args, **kwargs)
finally:
await self._copy_sessions_to_logs(environment)
evals/agents/codex_gateway.py
-36
View File
@@ -1,36 +0,0 @@
from __future__ import annotations
import os
from pathlib import Path
from typing import Any
from harbor.agents.installed.codex import Codex
class CodexGatewayAgent(Codex):
"""Codex wrapper with container-scoped Vercel AI Gateway auth/config."""
_OPENAI_COMPAT_BASE_URL = "https://ai-gateway.vercel.sh/v1"
@staticmethod
def name() -> str:
return "codex"
def __init__(
self,
logs_dir: Path,
*args: Any,
extra_env: dict[str, str] | None = None,
**kwargs: Any,
) -> None:
gateway_key = os.environ.get("AI_GATEWAY_API_KEY", "")
if not gateway_key:
raise ValueError("AI_GATEWAY_API_KEY is required for Codex compare")
scoped_env = dict(extra_env) if extra_env else {}
scoped_env["OPENAI_API_KEY"] = gateway_key
scoped_env["OPENAI_BASE_URL"] = self._OPENAI_COMPAT_BASE_URL
scoped_env["CODEX_AUTH_JSON_PATH"] = ""
scoped_env["CODEX_FORCE_AUTH_JSON"] = "0"
super().__init__(logs_dir, *args, extra_env=scoped_env, **kwargs)
evals/agents/fx_agent.py
-241
View File
@@ -1,241 +0,0 @@
from __future__ import annotations
import hashlib
import json
import os
import shlex
from pathlib import Path
from typing import Any
from harbor.agents.installed.base import BaseInstalledAgent, with_prompt_template
from harbor.environments.base import BaseEnvironment
from harbor.models.agent.context import AgentContext
class FxAgent(BaseInstalledAgent):
"""Harbor installed-agent wrapper for the local Linux fx binary."""
_REMOTE_SETUP_BINARY = "/installed-agent/fx"
_REMOTE_VERSION_FILE = "/installed-agent/fx.version"
_REMOTE_BINARY = "/usr/local/bin/fx"
_EFFORT_VALUES = {"auto", "none", "minimal", "low", "medium", "high", "xhigh", "max"}
def __init__(
self,
logs_dir: Path,
*args: Any,
effort: str | None = None,
**kwargs: Any,
) -> None:
super().__init__(logs_dir, *args, **kwargs)
self.effort = self._parse_effort(effort)
@staticmethod
def name() -> str:
return "fx"
@staticmethod
def _repo_root() -> Path:
return Path(__file__).resolve().parents[2]
@classmethod
def _binary_path(cls) -> Path:
configured = os.environ.get("FX_AGENT_BINARY_PATH")
path = Path(configured) if configured else cls._repo_root() / "evals/images/fx"
return path.expanduser().resolve()
@classmethod
def _validate_binary(cls, path: Path) -> None:
if not path.exists():
raise FileNotFoundError(f"fx agent binary not found: {path}")
if not path.is_file():
raise ValueError(f"fx agent binary is not a file: {path}")
if not os.access(path, os.X_OK):
raise PermissionError(f"fx agent binary is not executable: {path}")
@classmethod
def _source_version(cls) -> str:
version_prefix = 'pub const version = "'
main_path = cls._repo_root() / "src/main.zig"
for line in main_path.read_text(encoding="utf-8").splitlines():
stripped = line.strip()
if stripped.startswith(version_prefix) and stripped.endswith('";'):
return stripped[len(version_prefix) : -2]
raise ValueError(f"could not find fx version in {main_path}")
async def install(self, environment: BaseEnvironment) -> None:
binary_path = self._binary_path()
self._validate_binary(binary_path)
version_path = self.logs_dir / "setup" / "fx-version.txt"
version_path.parent.mkdir(parents=True, exist_ok=True)
version_path.write_text(f"{self._source_version()}\n", encoding="utf-8")
await environment.upload_file(binary_path, self._REMOTE_SETUP_BINARY)
await environment.upload_file(version_path, self._REMOTE_VERSION_FILE)
await self.exec_as_root(
environment,
command=(
"mkdir -p /usr/local/bin && "
f"cp {shlex.quote(self._REMOTE_SETUP_BINARY)} "
f"{shlex.quote(self._REMOTE_BINARY)} && "
f"chmod 0755 {shlex.quote(self._REMOTE_BINARY)} && "
"(mkdir -p /home/agent/.fx && "
"chown -R agent:agent /home/agent/.fx || true) && "
"(mkdir -p /logs/artifacts && "
"chown -R agent:agent /logs/artifacts || true)"
),
)
def get_version_command(self) -> str | None:
return f"cat {shlex.quote(self._REMOTE_VERSION_FILE)}"
@staticmethod
def _parse_timeout_ms(raw: str | None) -> int:
if not raw:
return 300_000
try:
value = int(raw)
except ValueError as exc:
raise ValueError(f"FX_AGENT_TIMEOUT_MS must be an integer: {raw}") from exc
if value <= 0:
raise ValueError("FX_AGENT_TIMEOUT_MS must be positive")
return value
@staticmethod
def _read_json(path: Path) -> dict[str, Any] | None:
try:
with path.open("r", encoding="utf-8") as handle:
value = json.load(handle)
except (OSError, json.JSONDecodeError):
return None
return value if isinstance(value, dict) else None
@classmethod
def _parse_effort(cls, raw: str | None) -> str | None:
if raw is None:
return None
effort = raw.strip().lower()
if effort not in cls._EFFORT_VALUES:
expected = ", ".join(sorted(cls._EFFORT_VALUES))
raise ValueError(f"invalid fx effort {raw!r}; expected one of: {expected}")
return effort
@with_prompt_template
async def run(
self,
instruction: str,
environment: BaseEnvironment,
context: AgentContext,
) -> None:
model = self.model_name or self._get_env("FX_MODEL")
if not model:
raise ValueError("FxAgent requires a Harbor model name")
timeout_ms = self._parse_timeout_ms(self._get_env("FX_AGENT_TIMEOUT_MS"))
timeout_sec = (timeout_ms + 999) // 1000 + 30
binary_version = self._source_version()
instruction_bytes = instruction.encode("utf-8")
wrapper = {
"agent": self.name(),
"model": model,
"timeout_ms": timeout_ms,
"binary_version": binary_version,
"command": {
"cwd": "/app",
"argv": [
"/usr/local/bin/fx",
"ask",
"--auto",
"--json",
"--no-save",
"--timeout",
str(timeout_ms),
"<instruction>",
],
"stdout_path": "/logs/agent/fx-result.json",
"stderr_path": "/logs/agent/fx-stderr.txt",
"exit_code_path": "/logs/agent/fx-exit-code.txt",
},
"instruction": {
"sha256": hashlib.sha256(instruction_bytes).hexdigest(),
"bytes": len(instruction_bytes),
},
}
if self.effort is not None:
wrapper["effort"] = self.effort
wrapper_json = json.dumps(wrapper, sort_keys=True)
settings: dict[str, Any] = {}
if self.effort is not None:
settings["effort"] = self.effort
settings_json = json.dumps(settings, sort_keys=True)
env = {
"HOME": "/tmp/fx-home",
"NO_COLOR": "1",
"FX_MODEL": model,
}
for key in ("AI_GATEWAY_API_KEY", "VERCEL_OIDC_TOKEN"):
value = self._get_env(key)
if value:
env[key] = value
command = (
"mkdir -p /logs/agent /tmp/fx-home/.fx && "
f"printf '%s\\n' {shlex.quote(settings_json)} > /app/.fx.json && "
"cd /app && "
"/usr/local/bin/fx ask --auto --json --no-save "
f"--timeout {timeout_ms} {shlex.quote(instruction)} "
"> /logs/agent/fx-result.json 2> /logs/agent/fx-stderr.txt; "
'code="$?"; '
'echo "$code" > /logs/agent/fx-exit-code.txt; '
"exit 0"
)
await self.exec_as_agent(
environment,
command=command,
env=env,
cwd="/app",
timeout_sec=timeout_sec,
)
await self.exec_as_root(
environment,
command=(
"mkdir -p /logs/artifacts && "
"cp -f /logs/agent/fx-exit-code.txt /logs/artifacts/fx-exit-code.txt && "
"cp -f /logs/agent/fx-result.json /logs/artifacts/fx-result.json && "
"cp -f /logs/agent/fx-stderr.txt /logs/artifacts/fx-stderr.txt && "
f"printf '%s\\n' {shlex.quote(wrapper_json)} > /logs/artifacts/fx-wrapper.json"
),
)
def populate_context_post_run(self, context: AgentContext) -> None:
metadata: dict[str, Any] = {}
exit_code_path = self.logs_dir / "fx-exit-code.txt"
try:
metadata["exit_code"] = int(exit_code_path.read_text().strip())
except (OSError, ValueError):
pass
result = self._read_json(self.logs_dir / "fx-result.json")
if result:
if "exit_code" in result:
metadata["exit_code"] = result["exit_code"]
if "steps" in result:
metadata["steps"] = result["steps"]
if "model" in result:
metadata["model"] = result["model"]
if "session_id" in result:
metadata["session_id"] = result["session_id"]
tool_calls = result.get("tool_calls")
if isinstance(tool_calls, list):
metadata["tool_call_count"] = len(tool_calls)
if metadata:
existing = context.metadata or {}
existing.update(metadata)
context.metadata = existing
evals/agents/opencode_gateway.py
-108
View File
@@ -1,108 +0,0 @@
from __future__ import annotations
import copy
import json
import os
import shlex
from typing import Any
from harbor.agents.installed.base import with_prompt_template
from harbor.agents.installed.opencode import OpenCode
from harbor.environments.base import BaseEnvironment
from harbor.models.agent.context import AgentContext
class OpenCodeGatewayAgent(OpenCode):
"""OpenCode wrapper with container-local Vercel AI Gateway auth/config."""
_OPENAI_COMPAT_BASE_URL = "https://ai-gateway.vercel.sh/v1"
@staticmethod
def name() -> str:
return "opencode"
def _gateway_setup_command(self, model_id: str) -> str:
config: dict[str, Any] = {
"$schema": "https://opencode.ai/config.json",
"provider": {
"openai": {
"models": {
model_id: {},
},
"options": {
"baseURL": self._OPENAI_COMPAT_BASE_URL,
},
},
},
}
config = self._deep_merge(config, copy.deepcopy(self._opencode_config))
config_json = shlex.quote(json.dumps(config, indent=2))
return (
"mkdir -p ~/.config/opencode ~/.local/share/opencode && "
"cat > ~/.local/share/opencode/auth.json <<EOF\n"
"{\"openai\":{\"type\":\"api\",\"key\":\"${OPENAI_API_KEY}\"}}\n"
"EOF\n"
f"printf %s {config_json} > ~/.config/opencode/opencode.json"
)
def _gateway_model_id(self) -> str:
if not self.model_name or "/" not in self.model_name:
raise ValueError(
"OpenCode gateway model must use provider/<gateway-model> format"
)
provider, model_id = self.model_name.split("/", 1)
if provider not in {"openai", "vercel"}:
raise ValueError(
"OpenCode gateway model must use openai/<gateway-model> format"
)
return model_id
@with_prompt_template
async def run(
self,
instruction: str,
environment: BaseEnvironment,
context: AgentContext,
) -> None:
escaped_instruction = shlex.quote(instruction)
gateway_key = self._get_env("AI_GATEWAY_API_KEY") or os.environ.get(
"AI_GATEWAY_API_KEY"
)
if not gateway_key:
raise ValueError("AI_GATEWAY_API_KEY is required for OpenCode compare")
model_id = self._gateway_model_id()
env = {
"AI_GATEWAY_API_KEY": gateway_key,
"OPENAI_API_KEY": gateway_key,
"OPENAI_BASE_URL": self._OPENAI_COMPAT_BASE_URL,
"OPENCODE_FAKE_VCS": "git",
}
skills_command = self._build_register_skills_command()
if skills_command:
await self.exec_as_agent(environment, command=skills_command, env=env)
await self.exec_as_agent(
environment,
command=self._gateway_setup_command(model_id),
env=env,
)
cli_flags = self.build_cli_flags()
cli_flags_arg = (cli_flags + " ") if cli_flags else ""
model = shlex.quote(f"openai/{model_id}")
await self.exec_as_agent(
environment,
command=(
". ~/.nvm/nvm.sh; "
f"opencode --model={model} run --format=json "
f"{cli_flags_arg}--thinking --dangerously-skip-permissions -- "
f"{escaped_instruction} 2>&1 </dev/null | stdbuf -oL tee "
"/logs/agent/opencode.txt"
),
env=env,
)
evals/datasets/fx-multistep/dataset.toml
-7
View File
@@ -1,7 +0,0 @@
schema_version = "1.0"
[dataset]
name = "vercel-labs-fx/fx-multistep"
description = "Manual Harbor multi-step evals for fx."
authors = [{ name = "Vercel" }]
keywords = ["fx", "multi-step", "agent", "evals"]
evals/datasets/fx-multistep/stateful-report/environment/Dockerfile
-3
View File
@@ -1,3 +0,0 @@
FROM --platform=linux/amd64 fx-evals:local
WORKDIR /app
evals/datasets/fx-multistep/stateful-report/steps/extend-report/instruction.md
-17
View File
@@ -1,17 +0,0 @@
Continue from the files created in the previous step.
Update `tasks.json` by preserving the existing `draft` and `ship` tasks, then add exactly one new object:
```json
{"id": "review", "status": "blocked"}
```
Update `report.py` so running `python3 report.py` prints exactly:
```text
blocked: 1
closed: 1
open: 1
```
Run the report and write the same output to `report.txt`.
evals/datasets/fx-multistep/stateful-report/steps/extend-report/solution/solve.sh
-28
View File
@@ -1,28 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
path = Path("/app/tasks.json")
tasks = json.loads(path.read_text(encoding="utf-8"))
tasks.append({"id": "review", "status": "blocked"})
path.write_text(json.dumps(tasks, indent=2) + "\n", encoding="utf-8")
PY
cat > /app/report.py <<'PY'
#!/usr/bin/env python3
import json
from collections import Counter
from pathlib import Path
tasks = json.loads(Path("tasks.json").read_text(encoding="utf-8"))
counts = Counter(task["status"] for task in tasks)
for status in ("blocked", "closed", "open"):
print(f"{status}: {counts[status]}")
PY
chmod 0755 /app/report.py
python3 /app/report.py > /app/report.txt
chown agent:agent /app/tasks.json /app/report.py /app/report.txt
evals/datasets/fx-multistep/stateful-report/steps/extend-report/tests/test.sh
-54
View File
@@ -1,54 +0,0 @@
#!/usr/bin/env bash
set -u
mkdir -p /logs/verifier
python3 - <<'PY'
import json
import subprocess
from pathlib import Path
log_path = Path("/logs/verifier/extend-report.txt")
reward_path = Path("/logs/verifier/reward.txt")
ok = False
messages = []
try:
tasks = json.loads(Path("/app/tasks.json").read_text(encoding="utf-8"))
expected = [
{"id": "draft", "status": "open"},
{"id": "ship", "status": "closed"},
{"id": "review", "status": "blocked"},
]
if tasks != expected:
messages.append(f"unexpected tasks.json: {tasks!r}")
expected_output = "blocked: 1\nclosed: 1\nopen: 1\n"
result = subprocess.run(
["python3", "/app/report.py"],
cwd="/app",
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
check=False,
)
if result.stdout != expected_output:
messages.append(f"unexpected stdout: {result.stdout!r}")
if result.returncode != 0:
messages.append(f"report.py exited {result.returncode}: {result.stderr!r}")
try:
report_txt = Path("/app/report.txt").read_text(encoding="utf-8")
except OSError as exc:
report_txt = ""
messages.append(f"could not read report.txt: {exc}")
if report_txt != expected_output:
messages.append(f"unexpected report.txt: {report_txt!r}")
ok = not messages
except Exception as exc:
messages.append(f"{type(exc).__name__}: {exc}")
log_path.write_text("\n".join(messages) + ("\n" if messages else "ok\n"), encoding="utf-8")
reward_path.write_text("1\n" if ok else "0\n", encoding="utf-8")
PY
evals/datasets/fx-multistep/stateful-report/steps/seed-report/instruction.md
-13
View File
@@ -1,13 +0,0 @@
Create two files in the current directory.
First, create `tasks.json` as a JSON array with exactly these two objects:
- `{"id": "draft", "status": "open"}`
- `{"id": "ship", "status": "closed"}`
Second, create `report.py`. When run with `python3 report.py`, it must read `tasks.json` and print exactly:
```text
open: 1
closed: 1
```
evals/datasets/fx-multistep/stateful-report/steps/seed-report/solution/solve.sh
-30
View File
@@ -1,30 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
cat > /app/tasks.json <<'JSON'
[
{
"id": "draft",
"status": "open"
},
{
"id": "ship",
"status": "closed"
}
]
JSON
cat > /app/report.py <<'PY'
#!/usr/bin/env python3
import json
from collections import Counter
from pathlib import Path
tasks = json.loads(Path("tasks.json").read_text(encoding="utf-8"))
counts = Counter(task["status"] for task in tasks)
print(f"open: {counts['open']}")
print(f"closed: {counts['closed']}")
PY
chmod 0755 /app/report.py
chown agent:agent /app/tasks.json /app/report.py
evals/datasets/fx-multistep/stateful-report/steps/seed-report/tests/test.sh
-45
View File
@@ -1,45 +0,0 @@
#!/usr/bin/env bash
set -u
mkdir -p /logs/verifier
python3 - <<'PY'
import json
import subprocess
from pathlib import Path
log_path = Path("/logs/verifier/seed-report.txt")
reward_path = Path("/logs/verifier/reward.txt")
ok = False
messages = []
try:
tasks = json.loads(Path("/app/tasks.json").read_text(encoding="utf-8"))
expected = [
{"id": "draft", "status": "open"},
{"id": "ship", "status": "closed"},
]
if tasks != expected:
messages.append(f"unexpected tasks.json: {tasks!r}")
result = subprocess.run(
["python3", "/app/report.py"],
cwd="/app",
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
check=False,
)
expected_output = "open: 1\nclosed: 1\n"
if result.stdout != expected_output:
messages.append(f"unexpected stdout: {result.stdout!r}")
if result.returncode != 0:
messages.append(f"report.py exited {result.returncode}: {result.stderr!r}")
ok = not messages
except Exception as exc:
messages.append(f"{type(exc).__name__}: {exc}")
log_path.write_text("\n".join(messages) + ("\n" if messages else "ok\n"), encoding="utf-8")
reward_path.write_text("1\n" if ok else "0\n", encoding="utf-8")
PY
evals/datasets/fx-multistep/stateful-report/task.toml
-44
View File
@@ -1,44 +0,0 @@
schema_version = "1.1"
multi_step_reward_strategy = "final"
[task]
name = "vercel-labs-fx/stateful-report"
description = "Build a tiny status report across sequential Harbor steps."
keywords = ["fx", "multi-step", "state"]
[metadata]
category = "coding"
difficulty = "easy"
[environment]
build_timeout_sec = 600.0
os = "linux"
cpus = 1
memory_mb = 2048
storage_mb = 10240
allow_internet = true
[[steps]]
name = "seed-report"
min_reward = 1.0
artifacts = ["/app/tasks.json", "/app/report.py"]
[steps.agent]
timeout_sec = 180.0
user = "agent"
[steps.verifier]
timeout_sec = 45.0
user = "root"
[[steps]]
name = "extend-report"
artifacts = ["/app/tasks.json", "/app/report.py", "/app/report.txt"]
[steps.agent]
timeout_sec = 180.0
user = "agent"
[steps.verifier]
timeout_sec = 45.0
user = "root"
evals/datasets/fx-release/cli-with-tests/environment/Dockerfile
-3
View File
@@ -1,3 +0,0 @@
FROM --platform=linux/amd64 fx-evals:local
WORKDIR /app
evals/datasets/fx-release/cli-with-tests/instruction.md
-3
View File
@@ -1,3 +0,0 @@
Create a Node.js CLI tool called `wordcount`.
The CLI must read a file path and print the number of words, lines, and characters in that file. Include a test suite using Vitest. Make sure `npm test` passes.
evals/datasets/fx-release/cli-with-tests/solution/solve.sh
-52
View File
@@ -1,52 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
mkdir -p /app/src /app/test
cat > /app/package.json <<'JSON'
{
"type": "module",
"bin": {
"wordcount": "./src/index.js"
},
"scripts": {
"test": "vitest run"
},
"devDependencies": {
"vitest": "^3.0.0"
}
}
JSON
cat > /app/src/index.js <<'JS'
#!/usr/bin/env node
import { readFileSync } from "node:fs";
export function countText(text) {
const lines = text.length === 0 ? 0 : text.split(/\n/).length;
const words = text.trim() === "" ? 0 : text.trim().split(/\s+/).length;
return { words, lines, characters: text.length };
}
if (process.argv[1] && process.argv[1].endsWith("index.js")) {
const file = process.argv[2];
const counts = countText(readFileSync(file, "utf-8"));
console.log(`words: ${counts.words}`);
console.log(`lines: ${counts.lines}`);
console.log(`characters: ${counts.characters}`);
}
JS
cat > /app/test/wordcount.test.js <<'JS'
import { describe, expect, test } from "vitest";
import { countText } from "../src/index.js";
describe("wordcount", () => {
test("counts words lines and characters", () => {
expect(countText("hello world\nagain")).toEqual({
words: 3,
lines: 2,
characters: 17,
});
});
});
JS
chmod +x /app/src/index.js
chown -R agent:agent /app
evals/datasets/fx-release/cli-with-tests/task.toml
-26
View File
@@ -1,26 +0,0 @@
schema_version = "1.1"
[task]
name = "vercel-labs-fx/cli-with-tests"
description = "Create a small Node.js CLI tool with passing tests."
keywords = ["node", "cli", "tests"]
[metadata]
category = "coding"
difficulty = "medium"
[verifier]
timeout_sec = 180.0
user = "root"
[agent]
timeout_sec = 360.0
user = "agent"
[environment]
build_timeout_sec = 600.0
os = "linux"
cpus = 1
memory_mb = 2048
storage_mb = 10240
allow_internet = true
evals/datasets/fx-release/cli-with-tests/tests/test.sh
-47
View File
@@ -1,47 +0,0 @@
#!/usr/bin/env bash
set -u
mkdir -p /logs/verifier
project_dir=""
if [ -f /app/package.json ]; then
project_dir="/app"
else
candidates=()
for child in /app/*; do
if [ -d "$child" ] && [ -f "$child/package.json" ]; then
candidates+=("$child")
fi
done
if [ "${#candidates[@]}" -eq 1 ]; then
project_dir="${candidates[0]}"
fi
fi
if [ -z "$project_dir" ]; then
echo "no package.json at /app and no single child project with package.json" \
> /logs/verifier/project-dir.txt
echo 0 > /logs/verifier/reward.txt
exit 0
fi
printf '%s\n' "$project_dir" > /logs/verifier/project-dir.txt
if [ ! -f "$project_dir/package.json" ]; then
echo 0 > /logs/verifier/reward.txt
exit 0
fi
if ! find "$project_dir" -maxdepth 4 -type f | grep -Eq '(wordcount|cli|index)\.(js|ts)$'; then
echo 0 > /logs/verifier/reward.txt
exit 0
fi
if cd "$project_dir" \
&& npm install > /logs/verifier/npm-install.txt 2>&1 \
&& npm test > /logs/verifier/npm-test.txt 2>&1; then
echo 1 > /logs/verifier/reward.txt
else
echo 0 > /logs/verifier/reward.txt
fi
evals/datasets/fx-release/create-file/environment/Dockerfile
-3
View File
@@ -1,3 +0,0 @@
FROM --platform=linux/amd64 fx-evals:local
WORKDIR /app
evals/datasets/fx-release/create-file/instruction.md
-7
View File
@@ -1,7 +0,0 @@
Create a file named `hello.txt` in the current directory.
The file must contain exactly:
```text
Hello, World!
```
evals/datasets/fx-release/create-file/solution/solve.sh
-4
View File
@@ -1,4 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
printf 'Hello, World!\n' > /app/hello.txt
evals/datasets/fx-release/create-file/task.toml
-26
View File
@@ -1,26 +0,0 @@
schema_version = "1.1"
[task]
name = "vercel-labs-fx/create-file"
description = "Create a simple file with exact content."
keywords = ["files", "write"]
[metadata]
category = "filesystem"
difficulty = "easy"
[verifier]
timeout_sec = 30.0
user = "root"
[agent]
timeout_sec = 120.0
user = "agent"
[environment]
build_timeout_sec = 600.0
os = "linux"
cpus = 1
memory_mb = 2048
storage_mb = 10240
allow_internet = true
evals/datasets/fx-release/create-file/tests/test.sh
-10
View File
@@ -1,10 +0,0 @@
#!/usr/bin/env bash
set -u
mkdir -p /logs/verifier
if [ -f /app/hello.txt ] && [ "$(cat /app/hello.txt)" = "Hello, World!" ]; then
echo 1 > /logs/verifier/reward.txt
else
echo 0 > /logs/verifier/reward.txt
fi
evals/datasets/fx-release/crud-api/environment/Dockerfile
-3
View File
@@ -1,3 +0,0 @@
FROM --platform=linux/amd64 fx-evals:local
WORKDIR /app
evals/datasets/fx-release/crud-api/instruction.md
-8
View File
@@ -1,8 +0,0 @@
Create an Express.js REST API with CRUD endpoints for a `todos` resource:
- `GET /todos`
- `POST /todos`
- `PUT /todos/:id`
- `DELETE /todos/:id`
Use TypeScript. Store todos in memory. Include tests with Vitest and Supertest. Make sure `npm test` passes.
evals/datasets/fx-release/crud-api/solution/solve.sh
-97
View File
@@ -1,97 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
mkdir -p /app/src /app/test
cat > /app/package.json <<'JSON'
{
"type": "module",
"scripts": {
"test": "vitest run"
},
"dependencies": {
"express": "^4.18.3"
},
"devDependencies": {
"supertest": "^7.0.0",
"tsx": "^4.19.0",
"typescript": "^5.7.0",
"vitest": "^3.0.0"
}
}
JSON
cat > /app/src/app.ts <<'TS'
import express from "express";
export interface Todo {
id: number;
title: string;
completed: boolean;
}
export const app = express();
app.use(express.json());
const todos: Todo[] = [];
let nextId = 1;
app.get("/todos", (_req, res) => {
res.json(todos);
});
app.post("/todos", (req, res) => {
const todo = {
id: nextId++,
title: String(req.body.title ?? ""),
completed: Boolean(req.body.completed ?? false),
};
todos.push(todo);
res.status(201).json(todo);
});
app.put("/todos/:id", (req, res) => {
const id = Number(req.params.id);
const todo = todos.find((item) => item.id === id);
if (!todo) {
res.status(404).json({ error: "not found" });
return;
}
if (req.body.title !== undefined) todo.title = String(req.body.title);
if (req.body.completed !== undefined) todo.completed = Boolean(req.body.completed);
res.json(todo);
});
app.delete("/todos/:id", (req, res) => {
const id = Number(req.params.id);
const index = todos.findIndex((item) => item.id === id);
if (index === -1) {
res.status(404).json({ error: "not found" });
return;
}
todos.splice(index, 1);
res.status(204).end();
});
TS
cat > /app/test/todos.test.ts <<'TS'
import request from "supertest";
import { describe, expect, test } from "vitest";
import { app } from "../src/app";
describe("todos API", () => {
test("creates lists updates and deletes todos", async () => {
const created = await request(app)
.post("/todos")
.send({ title: "ship", completed: false })
.expect(201);
expect(created.body.title).toBe("ship");
await request(app).get("/todos").expect(200);
await request(app)
.put(`/todos/${created.body.id}`)
.send({ completed: true })
.expect(200);
await request(app).delete(`/todos/${created.body.id}`).expect(204);
});
});
TS
chown -R agent:agent /app
evals/datasets/fx-release/crud-api/task.toml
-26
View File
@@ -1,26 +0,0 @@
schema_version = "1.1"
[task]
name = "vercel-labs-fx/crud-api"
description = "Create a tested Express CRUD API for todos."
keywords = ["node", "api", "tests"]
[metadata]
category = "coding"
difficulty = "medium"
[verifier]
timeout_sec = 180.0
user = "root"
[agent]
timeout_sec = 420.0
user = "agent"
[environment]
build_timeout_sec = 600.0
os = "linux"
cpus = 1
memory_mb = 2048
storage_mb = 10240
allow_internet = true
evals/datasets/fx-release/crud-api/tests/test.sh
-30
View File
@@ -1,30 +0,0 @@
#!/usr/bin/env bash
set -u
mkdir -p /logs/verifier
cd /app || {
echo 0 > /logs/verifier/reward.txt
exit 0
}
if [ ! -f /app/package.json ]; then
echo 0 > /logs/verifier/reward.txt
exit 0
fi
if ! grep -R "express" /app --include='*.ts' --include='*.js' > /logs/verifier/express-grep.txt 2>&1; then
echo 0 > /logs/verifier/reward.txt
exit 0
fi
if ! grep -R "todos" /app --include='*.ts' --include='*.js' > /logs/verifier/todos-grep.txt 2>&1; then
echo 0 > /logs/verifier/reward.txt
exit 0
fi
if npm install > /logs/verifier/npm-install.txt 2>&1 \
&& npm test > /logs/verifier/npm-test.txt 2>&1; then
echo 1 > /logs/verifier/reward.txt
else
echo 0 > /logs/verifier/reward.txt
fi
evals/datasets/fx-release/dataset.toml
-10
View File
@@ -1,10 +0,0 @@
schema_version = "1.0"
[dataset]
name = "vercel-labs-fx/fx-release"
description = "Manual Harbor release evals for fx."
authors = [{ name = "Vercel" }]
keywords = ["fx", "release", "agent", "evals"]
[[files]]
path = "metric.py"
evals/datasets/fx-release/edit-file/environment/Dockerfile
-13
View File
@@ -1,13 +0,0 @@
FROM --platform=linux/amd64 fx-evals:local
WORKDIR /app
RUN printf '%s\n' \
'{' \
' "name": "my-app",' \
' "port": 3000,' \
' "debug": false,' \
' "logLevel": "info"' \
'}' \
> /app/config.json \
&& chown agent:agent /app/config.json
evals/datasets/fx-release/edit-file/instruction.md
-5
View File
@@ -1,5 +0,0 @@
Edit `config.json`.
Change the `port` value from `3000` to `8080` and change `debug` from `false` to `true`.
Preserve the existing `name` and `logLevel` values.
evals/datasets/fx-release/edit-file/solution/solve.sh
-13
View File
@@ -1,13 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
path = Path("/app/config.json")
data = json.loads(path.read_text(encoding="utf-8"))
data["port"] = 8080
data["debug"] = True
path.write_text(json.dumps(data, indent=2) + "\n", encoding="utf-8")
PY
evals/datasets/fx-release/edit-file/task.toml
-26
View File
@@ -1,26 +0,0 @@
schema_version = "1.1"
[task]
name = "vercel-labs-fx/edit-file"
description = "Edit an existing JSON config without destroying surrounding fields."
keywords = ["files", "edit", "json"]
[metadata]
category = "filesystem"
difficulty = "easy"
[verifier]
timeout_sec = 30.0
user = "root"
[agent]
timeout_sec = 120.0
user = "agent"
[environment]
build_timeout_sec = 600.0
os = "linux"
cpus = 1
memory_mb = 2048
storage_mb = 10240
allow_internet = true
evals/datasets/fx-release/edit-file/tests/test.sh
-24
View File
@@ -1,24 +0,0 @@
#!/usr/bin/env bash
set -u
mkdir -p /logs/verifier
python3 - <<'PY'
import json
from pathlib import Path
path = Path("/app/config.json")
ok = False
try:
data = json.loads(path.read_text(encoding="utf-8"))
ok = (
data.get("port") == 8080
and data.get("debug") is True
and data.get("name") == "my-app"
and data.get("logLevel") == "info"
)
except (OSError, json.JSONDecodeError):
ok = False
Path("/logs/verifier/reward.txt").write_text("1\n" if ok else "0\n", encoding="utf-8")
PY
evals/datasets/fx-release/grep-files/environment/Dockerfile
-10
View File
@@ -1,10 +0,0 @@
FROM --platform=linux/amd64 fx-evals:local
WORKDIR /app
RUN mkdir -p /app/src \
&& printf 'export const API_URL = "https://example.com";\n' > /app/src/config.ts \
&& printf 'export const token = "SECRET_TOKEN_42";\n' > /app/src/auth.ts \
&& printf 'export function helper() { return 1; }\n' > /app/src/utils.ts \
&& printf '# My Project\n' > /app/README.md \
&& chown -R agent:agent /app
evals/datasets/fx-release/grep-files/instruction.md
-3
View File
@@ -1,3 +0,0 @@
Find which file contains the string `SECRET_TOKEN_42`.
Create `found-file.txt` containing the relative path to that file.
evals/datasets/fx-release/grep-files/solution/solve.sh
-4
View File
@@ -1,4 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
printf 'src/auth.ts\n' > /app/found-file.txt
evals/datasets/fx-release/grep-files/task.toml
-26
View File
@@ -1,26 +0,0 @@
schema_version = "1.1"
[task]
name = "vercel-labs-fx/grep-files"
description = "Search files and identify the file containing a target string."
keywords = ["grep", "search"]
[metadata]
category = "search"
difficulty = "easy"
[verifier]
timeout_sec = 30.0
user = "root"
[agent]
timeout_sec = 120.0
user = "agent"
[environment]
build_timeout_sec = 600.0
os = "linux"
cpus = 1
memory_mb = 2048
storage_mb = 10240
allow_internet = true
evals/datasets/fx-release/grep-files/tests/test.sh
-10
View File
@@ -1,10 +0,0 @@
#!/usr/bin/env bash
set -u
mkdir -p /logs/verifier
if [ -f /app/found-file.txt ] && grep -Eq '(^|/)auth\.ts$' /app/found-file.txt; then
echo 1 > /logs/verifier/reward.txt
else
echo 0 > /logs/verifier/reward.txt
fi
evals/datasets/fx-release/metric.py
-15
View File
@@ -1,15 +0,0 @@
#!/usr/bin/env python3
# /// script
# dependencies = []
# ///
from __future__ import annotations
import runpy
from pathlib import Path
SHARED_METRIC = Path(__file__).resolve().parents[2] / "metrics" / "fx_metrics.py"
if not SHARED_METRIC.exists():
raise SystemExit(f"shared fx metric script not found: {SHARED_METRIC}")
runpy.run_path(str(SHARED_METRIC), run_name="__main__")
evals/datasets/fx-release/run-command/environment/Dockerfile
-3
View File
@@ -1,3 +0,0 @@
FROM --platform=linux/amd64 fx-evals:local
WORKDIR /app
evals/datasets/fx-release/run-command/instruction.md
-7
View File
@@ -1,7 +0,0 @@
Run this command:
```bash
echo HELLO_FROM_FX
```
Create `command-output.txt` containing exactly the command output and no other text.
evals/datasets/fx-release/run-command/solution/solve.sh
-4
View File
@@ -1,4 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
echo HELLO_FROM_FX > /app/command-output.txt
evals/datasets/fx-release/run-command/task.toml
-26
View File
@@ -1,26 +0,0 @@
schema_version = "1.1"
[task]
name = "vercel-labs-fx/run-command"
description = "Run a shell command and record its output."
keywords = ["shell", "command"]
[metadata]
category = "commands"
difficulty = "easy"
[verifier]
timeout_sec = 30.0
user = "root"
[agent]
timeout_sec = 120.0
user = "agent"
[environment]
build_timeout_sec = 600.0
os = "linux"
cpus = 1
memory_mb = 2048
storage_mb = 10240
allow_internet = true
evals/datasets/fx-release/run-command/tests/test.sh
-10
View File
@@ -1,10 +0,0 @@
#!/usr/bin/env bash
set -u
mkdir -p /logs/verifier
if [ -f /app/command-output.txt ] && [ "$(cat /app/command-output.txt)" = "HELLO_FROM_FX" ]; then
echo 1 > /logs/verifier/reward.txt
else
echo 0 > /logs/verifier/reward.txt
fi
evals/images/Dockerfile
-24
View File
@@ -1,24 +0,0 @@
FROM --platform=linux/amd64 debian:bookworm-slim
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
bash \
ca-certificates \
curl \
git \
nodejs \
npm \
python3 \
ripgrep \
&& rm -rf /var/lib/apt/lists/*
COPY fx /usr/local/bin/fx
RUN chmod 0755 /usr/local/bin/fx \
&& useradd --create-home --shell /bin/bash agent \
&& mkdir -p /app /logs/agent /logs/verifier /logs/artifacts /home/agent/.fx \
&& chown -R agent:agent /app /logs /home/agent
WORKDIR /app
evals/jobs/compare-key-agents.yaml
-66
View File
@@ -1,66 +0,0 @@
# Manual local comparison eval.
# Comparison models:
# fx: anthropic/claude-opus-4.7 with high effort through Vercel AI Gateway
# codex: openai/gpt-5.5 with high reasoning effort through Vercel AI Gateway
# claude-code: anthropic/claude-opus-4.7 with high effort through Vercel AI Gateway
# opencode: anthropic/claude-sonnet-4.6 through Vercel AI Gateway's OpenAI-compatible API
jobs_dir: evals/jobs-out
n_attempts: 1
timeout_multiplier: 1.0
n_concurrent_trials: 1
quiet: false
environment:
type: docker
force_build: true
delete: true
agents:
- name: fx
import_path: evals.agents.fx_agent:FxAgent
model_name: anthropic/claude-opus-4.7
kwargs:
effort: high
- name: codex-gateway
import_path: evals.agents.codex_gateway:CodexGatewayAgent
model_name: openai/gpt-5.5
kwargs:
reasoning_effort: high
- name: claude-code-gateway
import_path: evals.agents.claude_gateway:ClaudeCodeGatewayAgent
model_name: anthropic/claude-opus-4.7
kwargs:
reasoning_effort: high
- name: opencode-gateway
import_path: evals.agents.opencode_gateway:OpenCodeGatewayAgent
model_name: openai/anthropic/claude-sonnet-4.6
datasets:
- path: evals/datasets/fx-release
metrics:
- type: uv-script
kwargs:
script_path: evals/metrics/fx_metrics.py
artifacts:
- source: /logs/agent
destination: agent
- source: /logs/verifier
destination: verifier
- source: /logs/artifacts
destination: fx-run
- source: /app
destination: workspace
exclude:
- node_modules
- .git
- .cache
- __pycache__
- .pytest_cache
- dist
- build
- coverage
- target
- .venv
- venv
evals/jobs/compare-terminal-bench-smoke.yaml
-64
View File
@@ -1,64 +0,0 @@
# Single-task Terminal-Bench comparison eval.
jobs_dir: evals/jobs-out
n_attempts: 1
timeout_multiplier: 1.0
n_concurrent_trials: 1
quiet: false
environment:
type: docker
force_build: true
delete: true
agents:
- name: fx
import_path: evals.agents.fx_agent:FxAgent
model_name: anthropic/claude-opus-4.7
kwargs:
effort: high
- name: codex-gateway
import_path: evals.agents.codex_gateway:CodexGatewayAgent
model_name: openai/gpt-5.5
kwargs:
reasoning_effort: high
- name: claude-code-gateway
import_path: evals.agents.claude_gateway:ClaudeCodeGatewayAgent
model_name: anthropic/claude-opus-4.7
kwargs:
reasoning_effort: high
- name: opencode-gateway
import_path: evals.agents.opencode_gateway:OpenCodeGatewayAgent
model_name: openai/anthropic/claude-sonnet-4.6
datasets:
- name: terminal-bench/terminal-bench-2
task_names:
- terminal-bench/break-filter-js-from-html
download_dir: evals/.build/harbor-datasets
metrics:
- type: uv-script
kwargs:
script_path: evals/metrics/fx_metrics.py
artifacts:
- source: /logs/agent
destination: agent
- source: /logs/verifier
destination: verifier
- source: /logs/artifacts
destination: fx-run
- source: /app
destination: workspace
exclude:
- node_modules
- .git
- .cache
- __pycache__
- .pytest_cache
- dist
- build
- coverage
- target
- .venv
- venv
evals/jobs/compare-terminal-bench.yaml
-63
View File
@@ -1,63 +0,0 @@
# Full Terminal-Bench comparison eval.
# terminal-bench/terminal-bench-2 currently contains 89 registered tasks.
jobs_dir: evals/jobs-out
n_attempts: 1
timeout_multiplier: 1.0
n_concurrent_trials: 1
quiet: false
environment:
type: docker
force_build: true
delete: true
agents:
- name: fx
import_path: evals.agents.fx_agent:FxAgent
model_name: anthropic/claude-opus-4.7
kwargs:
effort: high
- name: codex-gateway
import_path: evals.agents.codex_gateway:CodexGatewayAgent
model_name: openai/gpt-5.5
kwargs:
reasoning_effort: high
- name: claude-code-gateway
import_path: evals.agents.claude_gateway:ClaudeCodeGatewayAgent
model_name: anthropic/claude-opus-4.7
kwargs:
reasoning_effort: high
- name: opencode-gateway
import_path: evals.agents.opencode_gateway:OpenCodeGatewayAgent
model_name: openai/anthropic/claude-sonnet-4.6
datasets:
- name: terminal-bench/terminal-bench-2
download_dir: evals/.build/harbor-datasets
metrics:
- type: uv-script
kwargs:
script_path: evals/metrics/fx_metrics.py
artifacts:
- source: /logs/agent
destination: agent
- source: /logs/verifier
destination: verifier
- source: /logs/artifacts
destination: fx-run
- source: /app
destination: workspace
exclude:
- node_modules
- .git
- .cache
- __pycache__
- .pytest_cache
- dist
- build
- coverage
- target
- .venv
- venv
evals/jobs/fx-multistep.yaml
-50
View File
@@ -1,50 +0,0 @@
# Small local multi-step eval for fx.
# Kept separate from the default release gate because it exercises sequential
# Harbor steps and collects per-step artifacts.
jobs_dir: evals/jobs-out
n_attempts: 1
timeout_multiplier: 1.0
n_concurrent_trials: 1
quiet: false
environment:
type: docker
force_build: true
delete: true
agents:
- name: fx
import_path: evals.agents.fx_agent:FxAgent
model_name: anthropic/claude-sonnet-4.6
datasets:
- path: evals/datasets/fx-multistep
metrics:
- type: uv-script
kwargs:
script_path: evals/metrics/fx_metrics.py
artifacts:
- source: /logs/agent
destination: agent
- source: /logs/verifier
destination: verifier
- source: /tmp/fx-home/.fx
destination: fx-home
- source: /logs/artifacts
destination: fx-run
- source: /app
destination: workspace
exclude:
- node_modules
- .git
- .cache
- __pycache__
- .pytest_cache
- dist
- build
- coverage
- target
- .venv
- venv
evals/jobs/fx-release.yaml
-49
View File
@@ -1,49 +0,0 @@
# Manual local release eval for fx.
# FxAgent uses the Harbor model name below as FX_MODEL inside the task container.
jobs_dir: evals/jobs-out
n_attempts: 1
timeout_multiplier: 1.0
n_concurrent_trials: 1
quiet: false
environment:
type: docker
force_build: true
delete: true
agents:
- name: fx
import_path: evals.agents.fx_agent:FxAgent
model_name: anthropic/claude-sonnet-4.6
datasets:
- path: evals/datasets/fx-release
metrics:
- type: uv-script
kwargs:
script_path: evals/metrics/fx_metrics.py
artifacts:
- source: /logs/agent
destination: agent
- source: /logs/verifier
destination: verifier
- source: /logs/artifacts
destination: fx-run
- source: /tmp/fx-home/.fx
destination: fx-home
- source: /app
destination: workspace
exclude:
- node_modules
- .git
- .cache
- __pycache__
- .pytest_cache
- dist
- build
- coverage
- target
- .venv
- venv
evals/jobs/fx-terminal-bench-smoke.yaml
-52
View File
@@ -1,52 +0,0 @@
# Single-task Terminal-Bench smoke eval for fx.
# FxAgent uploads evals/images/fx into each registered task container.
jobs_dir: evals/jobs-out
n_attempts: 1
timeout_multiplier: 1.0
n_concurrent_trials: 1
quiet: false
environment:
type: docker
force_build: true
delete: true
agents:
- name: fx
import_path: evals.agents.fx_agent:FxAgent
model_name: anthropic/claude-sonnet-4.6
datasets:
- name: terminal-bench/terminal-bench-2
task_names:
- terminal-bench/break-filter-js-from-html
download_dir: evals/.build/harbor-datasets
metrics:
- type: uv-script
kwargs:
script_path: evals/metrics/fx_metrics.py
artifacts:
- source: /logs/agent
destination: agent
- source: /logs/verifier
destination: verifier
- source: /logs/artifacts
destination: fx-run
- source: /tmp/fx-home/.fx
destination: fx-home
- source: /app
destination: workspace
exclude:
- node_modules
- .git
- .cache
- __pycache__
- .pytest_cache
- dist
- build
- coverage
- target
- .venv
- venv
evals/jobs/fx-terminal-bench.yaml
-50
View File
@@ -1,50 +0,0 @@
# Full Terminal-Bench eval for fx.
# terminal-bench/terminal-bench-2 currently contains 89 registered tasks.
jobs_dir: evals/jobs-out
n_attempts: 1
timeout_multiplier: 1.0
n_concurrent_trials: 1
quiet: false
environment:
type: docker
force_build: true
delete: true
agents:
- name: fx
import_path: evals.agents.fx_agent:FxAgent
model_name: anthropic/claude-sonnet-4.6
datasets:
- name: terminal-bench/terminal-bench-2
download_dir: evals/.build/harbor-datasets
metrics:
- type: uv-script
kwargs:
script_path: evals/metrics/fx_metrics.py
artifacts:
- source: /logs/agent
destination: agent
- source: /logs/verifier
destination: verifier
- source: /logs/artifacts
destination: fx-run
- source: /tmp/fx-home/.fx
destination: fx-home
- source: /app
destination: workspace
exclude:
- node_modules
- .git
- .cache
- __pycache__
- .pytest_cache
- dist
- build
- coverage
- target
- .venv
- venv
evals/metrics/fx_metrics.py
-72
View File
@@ -1,72 +0,0 @@
#!/usr/bin/env python3
# /// script
# dependencies = []
# ///
from __future__ import annotations
import argparse
import json
from pathlib import Path
from typing import Any
def reward_value(reward: Any) -> float:
if reward is None:
return 0.0
if isinstance(reward, bool):
return float(reward)
if isinstance(reward, (int, float)):
return float(reward)
if isinstance(reward, dict):
if not reward:
return 0.0
return max(reward_value(item) for item in reward.values())
return 0.0
def read_rewards(input_path: Path) -> list[float]:
rewards: list[float] = []
for line in input_path.read_text(encoding="utf-8").splitlines():
if not line.strip():
continue
rewards.append(reward_value(json.loads(line)))
return rewards
def metric_values(rewards: list[float]) -> dict[str, float | int]:
n_trials = len(rewards)
n_passed = sum(1 for reward in rewards if reward >= 1.0)
return {
"mean_reward": sum(rewards) / n_trials if n_trials else 0.0,
"pass_rate": n_passed / n_trials if n_trials else 0.0,
"n_trials": n_trials,
"n_passed": n_passed,
}
def main(input_path: Path, output_path: Path) -> None:
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_text(
json.dumps(metric_values(read_rewards(input_path)), sort_keys=True) + "\n",
encoding="utf-8",
)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"-i",
"--input-path",
type=Path,
required=True,
help="Path to a JSONL file containing one reward value per trial.",
)
parser.add_argument(
"-o",
"--output-path",
type=Path,
required=True,
help="Path to the JSON object Harbor should read as job metrics.",
)
args = parser.parse_args()
main(args.input_path, args.output_path)
evals/rewardkit/built_in_file_command_json.py
-23
View File
@@ -1,23 +0,0 @@
"""Example RewardKit built-in criteria for deterministic fx task checks.
This file is not wired into release signoff. It shows the shape a future Harbor
task could place under its tests directory if that task deliberately adopts
RewardKit built-in criteria.
"""
import rewardkit as rk
rk.file_exists("summary.json", weight=1.0)
rk.file_contains("summary.json", '"status": "ok"', weight=1.0)
rk.command_succeeds("python3 -m json.tool summary.json", timeout=10, weight=1.0)
rk.command_output_contains(
"python3 -c \"import json; print(json.load(open('summary.json'))['status'])\"",
"ok",
timeout=10,
weight=1.0,
)
rk.json_key_equals("summary.json", "status", "ok", weight=1.0)
rk.json_path_equals("summary.json", "checks.build", "pass", weight=1.0)
evals/rewardkit/qualitative_review.toml
-33
View File
@@ -1,33 +0,0 @@
# Optional qualitative RewardKit judge example.
# This is not part of normal fx release signoff.
[judge]
judge = "anthropic/claude-sonnet-4-6"
files = ["/app/evals/runs/latest.md", "/logs/artifacts/fx-result.json"]
timeout = 120
reasoning_effort = "medium"
weight = 1.0
[[criterion]]
name = "release_risk_signal"
description = "Does the review identify behavior-changing release risks and avoid treating style-only notes as blockers?"
type = "likert"
points = 5
weight = 2.0
[[criterion]]
name = "evidence_quality"
description = "Does the review cite concrete files, commands, or artifacts instead of relying on unsupported impressions?"
type = "likert"
points = 5
weight = 1.0
[[criterion]]
name = "deterministic_gate_respected"
description = "Does the review keep deterministic release verifiers as the required release signal?"
type = "binary"
weight = 2.0
[scoring]
aggregation = "threshold"
threshold = 0.8
evals/rewardkit/trajectory_review.toml
-34
View File
@@ -1,34 +0,0 @@
# Optional future trajectory-oriented RewardKit judge example.
# This assumes a future fx or Harbor artifact exports an ATIF trajectory file.
# It is not wired into any fx release job today.
[judge]
judge = "anthropic/claude-sonnet-4-6"
atif-trajectory = "/logs/trajectory.json"
files = ["/logs/artifacts/fx-result.json", "/logs/artifacts/fx-wrapper.json"]
timeout = 180
reasoning_effort = "medium"
weight = 1.0
[[criterion]]
name = "efficient_task_path"
description = "Did the agent use a direct approach for the task without unnecessary retries or unrelated exploration?"
type = "likert"
points = 5
weight = 1.0
[[criterion]]
name = "artifact_usefulness"
description = "Do the collected fx and Harbor artifacts make the trajectory understandable to a release reviewer?"
type = "likert"
points = 5
weight = 1.0
[[criterion]]
name = "unsafe_shortcuts_absent"
description = "Did the trajectory avoid bypassing deterministic task verifiers or masking verifier failures?"
type = "binary"
weight = 2.0
[scoring]
aggregation = "weighted_mean"
evals/runs/2026-05-04-comparison-signoff.md
-96
View File
@@ -1,96 +0,0 @@
# 2026-05-04 fx comparison signoff
Viewer command:
```bash
uvx harbor view evals/jobs-out
```
## Summary
| Dataset | Agent | Model | n_trials | n_passed | pass_rate | mean_reward | exceptions | artifact failures | Result path |
| --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |
| `fx-release` | `claude-code` | `anthropic/claude-sonnet-4.6` | 6 | 0 | 0.00 | 0.00 | 6 | 6 | `evals/jobs-out/2026-05-04__21-51-17` |
| `fx-release` | `codex` | `openai/gpt-5.1-codex` | 6 | 0 | 0.00 | 0.00 | 6 | 6 | `evals/jobs-out/2026-05-04__21-51-17` |
| `fx-release` | `fx` | `anthropic/claude-sonnet-4.6` | 6 | 6 | 1.00 | 1.00 | 0 | 0 | `evals/jobs-out/2026-05-04__21-51-17` |
| `fx-release` | `opencode` | `vercel/anthropic/claude-sonnet-4.6` | 6 | 0 | 0.00 | 0.00 | 6 | 6 | `evals/jobs-out/2026-05-04__21-51-17` |
| `terminal-bench/terminal-bench-2` | `claude-code` | `anthropic/claude-sonnet-4.6` | 1 | 1 | 1.00 | 1.00 | 0 | 1 | `evals/jobs-out/2026-05-04__22-43-22` |
| `terminal-bench/terminal-bench-2` | `codex` | `openai/gpt-5.1-codex` | 1 | 0 | 0.00 | 0.00 | 1 | 1 | `evals/jobs-out/2026-05-04__22-43-22` |
| `terminal-bench/terminal-bench-2` | `fx` | `anthropic/claude-sonnet-4.6` | 1 | 0 | 0.00 | 0.00 | 0 | 0 | `evals/jobs-out/2026-05-04__22-43-22` |
| `terminal-bench/terminal-bench-2` | `opencode` | `vercel/anthropic/claude-sonnet-4.6` | 1 | 0 | 0.00 | 0.00 | 1 | 1 | `evals/jobs-out/2026-05-04__22-43-22` |
## fx-release
- Result path: `evals/jobs-out/2026-05-04__21-51-17`
- Started: `2026-05-04T21:51:17.938360`
- Finished: `2026-05-04T22:43:08.942353`
| Agent | Model | n_trials | n_passed | pass_rate | mean_reward | exceptions | artifact failures |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
| `claude-code` | `anthropic/claude-sonnet-4.6` | 6 | 0 | 0.00 | 0.00 | 6 | 6 |
| `codex` | `openai/gpt-5.1-codex` | 6 | 0 | 0.00 | 0.00 | 6 | 6 |
| `fx` | `anthropic/claude-sonnet-4.6` | 6 | 6 | 1.00 | 1.00 | 0 | 0 |
| `opencode` | `vercel/anthropic/claude-sonnet-4.6` | 6 | 0 | 0.00 | 0.00 | 6 | 6 |
Exceptions:
- claude-code/cli-with-tests: Traceback (most recent call last):
- codex/cli-with-tests: Traceback (most recent call last):
- opencode/cli-with-tests: Traceback (most recent call last):
- codex/create-file: Traceback (most recent call last):
- claude-code/create-file: Traceback (most recent call last):
- opencode/create-file: Traceback (most recent call last):
- codex/crud-api: Traceback (most recent call last):
- opencode/crud-api: Traceback (most recent call last):
- claude-code/crud-api: Traceback (most recent call last):
- codex/edit-file: Traceback (most recent call last):
- claude-code/edit-file: Traceback (most recent call last):
- opencode/edit-file: Traceback (most recent call last):
- codex/grep-files: Traceback (most recent call last):
- opencode/grep-files: Traceback (most recent call last):
- claude-code/grep-files: Traceback (most recent call last):
- claude-code/run-command: Traceback (most recent call last):
- opencode/run-command: Traceback (most recent call last):
- codex/run-command: Traceback (most recent call last):
Artifact failures:
- claude-code/cli-with-tests: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- codex/cli-with-tests: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- opencode/cli-with-tests: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- codex/create-file: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- claude-code/create-file: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- opencode/create-file: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- codex/crud-api: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- opencode/crud-api: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- claude-code/crud-api: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- codex/edit-file: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- claude-code/edit-file: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- opencode/edit-file: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- codex/grep-files: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- opencode/grep-files: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- claude-code/grep-files: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- claude-code/run-command: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- opencode/run-command: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- codex/run-command: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
## terminal-bench/terminal-bench-2
- Result path: `evals/jobs-out/2026-05-04__22-43-22`
- Started: `2026-05-04T22:43:24.470433`
- Finished: `2026-05-04T23:02:19.202187`
| Agent | Model | n_trials | n_passed | pass_rate | mean_reward | exceptions | artifact failures |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
| `claude-code` | `anthropic/claude-sonnet-4.6` | 1 | 1 | 1.00 | 1.00 | 0 | 1 |
| `codex` | `openai/gpt-5.1-codex` | 1 | 0 | 0.00 | 0.00 | 1 | 1 |
| `fx` | `anthropic/claude-sonnet-4.6` | 1 | 0 | 0.00 | 0.00 | 0 | 0 |
| `opencode` | `vercel/anthropic/claude-sonnet-4.6` | 1 | 0 | 0.00 | 0.00 | 1 | 1 |
Exceptions:
- codex/break-filter-js-from-html: Traceback (most recent call last):
- opencode/break-filter-js-from-html: Traceback (most recent call last):
Artifact failures:
- codex/break-filter-js-from-html: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- claude-code/break-filter-js-from-html: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- opencode/break-filter-js-from-html: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
evals/runs/2026-05-04-fx-multistep-smoke.md
-31
View File
@@ -1,31 +0,0 @@
# 2026-05-04 fx Multi-step Smoke
Docs used:
- https://www.harborframework.com/docs/tasks
- https://www.harborframework.com/docs/tasks/multi-step
- https://www.harborframework.com/docs/datasets/adapters
Command:
```bash
./evals/scripts/run-release-evals.sh --multistep
```
Result:
- Job output: `evals/jobs-out/2026-05-04__19-11-55`
- Dataset: `fx-multistep`
- Agent: `fx`
- Model: `claude-sonnet-4.6`
- Trials: 1
- Pass rate: `1.00`
- Mean reward: `1.00`
- Harbor metrics: `mean_reward=1.00 pass_rate=1.00 n_trials=1 n_passed=1`
- Artifact collection failures: `0`
Notes:
- Both steps ran: `seed-report`, then `extend-report`.
- Per-step artifacts are under `steps/<step-name>/artifacts/`.
- Both step `fx-exit-code.txt` files contained `0`.
evals/runs/2026-05-04-fx-release-smoke.md
-33
View File
@@ -1,33 +0,0 @@
# 2026-05-04 fx Release Smoke Run
## Summary
- Command: `./evals/scripts/run-release-evals.sh`
- Harbor job ID: `0edb8ddb-5da7-4529-816b-22d6030bb118`
- Harbor results: `evals/jobs-out/2026-05-04__17-47-44`
- Job config: `evals/jobs/fx-release.yaml`
- Dataset: `fx-release`
- Agent: `fx`
- Model: `anthropic/claude-sonnet-4.6`
- Runtime: 9m 56s
- Trials: 6
- Exceptions: 0
- Pass rate: 1.00
- Mean reward: 1.00
## Task Results
| Task | Reward |
| --- | ---: |
| `create-file` | 1 |
| `edit-file` | 1 |
| `run-command` | 1 |
| `grep-files` | 1 |
| `cli-with-tests` | 1 |
| `crud-api` | 1 |
## Interpretation
The freshly built `fx` binary completed all six deterministic Harbor release-smoke tasks. The run covered basic file creation, file editing, shell command execution, file search, a tested CLI project, and a tested CRUD API project.
Each captured `fx-exit-code.txt` was `0`, including the `crud-api` task.
evals/runs/2026-05-04-fx-release-task-slice.md
-30
View File
@@ -1,30 +0,0 @@
# 2026-05-04 fx Release Task Slice
Docs used:
- https://www.harborframework.com/docs/tasks
- https://www.harborframework.com/docs/tasks/multi-step
- https://www.harborframework.com/docs/datasets/adapters
Command:
```bash
./evals/scripts/run-release-evals.sh
```
Result:
- Job output: `evals/jobs-out/2026-05-04__19-15-20`
- Dataset: `fx-release`
- Agent: `fx`
- Model: `claude-sonnet-4.6`
- Trials: 6
- Pass rate: `1.00`
- Mean reward: `1.00`
- Harbor metrics: `mean_reward=1.00 pass_rate=1.00 n_trials=6 n_passed=6`
- Artifact collection failures: `0`
Notes:
- `cli-with-tests` passed and wrote `artifacts/verifier/project-dir.txt` with `/app`.
- `cli-with-tests` compact artifact bundle has `fx-exit-code.txt` in both the Harbor-installed artifact root and `artifacts/fx-run/`; both contained `0`.
evals/runs/2026-05-04-release-signoff.md
-46
View File
@@ -1,46 +0,0 @@
# 2026-05-04 fx release signoff
Viewer command:
```bash
uvx harbor view evals/jobs-out
```
## Summary
| Dataset | Agent | Model | n_trials | n_passed | pass_rate | mean_reward | exceptions | artifact failures | Result path |
| --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |
| `fx-release` | `fx` | `anthropic/claude-sonnet-4.6` | 6 | 6 | 1.00 | 1.00 | 0 | 0 | `evals/jobs-out/2026-05-04__21-29-30` |
| `terminal-bench/terminal-bench-2` | `fx` | `anthropic/claude-sonnet-4.6` | 1 | 0 | 0.00 | 0.00 | 0 | 0 | `evals/jobs-out/2026-05-04__21-38-19` |
## fx-release
- Result path: `evals/jobs-out/2026-05-04__21-29-30`
- Started: `2026-05-04T21:29:31.001721`
- Finished: `2026-05-04T21:38:17.888547`
| Agent | Model | n_trials | n_passed | pass_rate | mean_reward | exceptions | artifact failures |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
| `fx` | `anthropic/claude-sonnet-4.6` | 6 | 6 | 1.00 | 1.00 | 0 | 0 |
Exceptions:
- none
Artifact failures:
- none
## terminal-bench/terminal-bench-2
- Result path: `evals/jobs-out/2026-05-04__21-38-19`
- Started: `2026-05-04T21:38:21.307293`
- Finished: `2026-05-04T21:43:43.289617`
| Agent | Model | n_trials | n_passed | pass_rate | mean_reward | exceptions | artifact failures |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
| `fx` | `anthropic/claude-sonnet-4.6` | 1 | 0 | 0.00 | 0.00 | 0 | 0 |
Exceptions:
- none
Artifact failures:
- none
evals/runs/2026-05-04-terminal-bench-smoke.md
-28
View File
@@ -1,28 +0,0 @@
# Terminal-Bench Smoke
Command:
```bash
./evals/scripts/run-terminal-bench.sh --smoke
```
Result:
- Dataset: `terminal-bench/terminal-bench-2`
- Harbor output: `evals/jobs-out/2026-05-04__17-39-21`
- Agent: `fx`
- Version: `0.3.2`
- Model: `claude-sonnet-4.6`
- Trials: `1`
- Exceptions: `0`
- Mean reward: `0.000`
- Pass rate: `0.00`
- Captured fx exit code: `0`
The smoke path completed without the doom-loop permission abort. The verifier scored the task `0.0` because the generated `out.html` did not trigger the required alert after filtering.
Viewer command:
```bash
uvx harbor view evals/jobs-out
```
evals/runs/2026-05-05-comparison-signoff-fixed.md
-63
View File
@@ -1,63 +0,0 @@
# 2026-05-05 fx comparison signoff fixed
Viewer command:
```bash
uvx harbor view evals/jobs-out
```
## Summary
| Dataset | Agent | Model | n_trials | n_passed | pass_rate | mean_reward | exceptions | artifact failures | Result path |
| --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |
| `fx-release` | `claude-code` | `anthropic/claude-opus-4.7` | 6 | 6 | 1.00 | 1.00 | 0 | 0 | `evals/jobs-out/2026-05-05__02-16-55` |
| `fx-release` | `codex` | `openai/gpt-5.5` | 6 | 6 | 1.00 | 1.00 | 0 | 0 | `evals/jobs-out/2026-05-05__02-16-55` |
| `fx-release` | `fx` | `anthropic/claude-opus-4.7` | 6 | 6 | 1.00 | 1.00 | 0 | 0 | `evals/jobs-out/2026-05-05__02-16-55` |
| `fx-release` | `opencode` | `openai/anthropic/claude-sonnet-4.6` | 6 | 4 | 0.67 | 0.67 | 2 | 0 | `evals/jobs-out/2026-05-05__02-16-55` |
| `terminal-bench/terminal-bench-2` | `claude-code` | `anthropic/claude-opus-4.7` | 1 | 1 | 1.00 | 1.00 | 0 | 0 | `evals/jobs-out/2026-05-05__03-52-21` |
| `terminal-bench/terminal-bench-2` | `codex` | `openai/gpt-5.5` | 1 | 1 | 1.00 | 1.00 | 0 | 0 | `evals/jobs-out/2026-05-05__03-52-21` |
| `terminal-bench/terminal-bench-2` | `fx` | `anthropic/claude-opus-4.7` | 1 | 0 | 0.00 | 0.00 | 0 | 0 | `evals/jobs-out/2026-05-05__03-52-21` |
| `terminal-bench/terminal-bench-2` | `opencode` | `openai/anthropic/claude-sonnet-4.6` | 1 | 0 | 0.00 | 0.00 | 0 | 0 | `evals/jobs-out/2026-05-05__03-52-21` |
## fx-release
- Result path: `evals/jobs-out/2026-05-05__02-16-55`
- Started: `2026-05-05T02:16:55.309074`
- Finished: `2026-05-05T03:50:30.681902`
| Agent | Model | n_trials | n_passed | pass_rate | mean_reward | exceptions | artifact failures |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
| `claude-code` | `anthropic/claude-opus-4.7` | 6 | 6 | 1.00 | 1.00 | 0 | 0 |
| `codex` | `openai/gpt-5.5` | 6 | 6 | 1.00 | 1.00 | 0 | 0 |
| `fx` | `anthropic/claude-opus-4.7` | 6 | 6 | 1.00 | 1.00 | 0 | 0 |
| `opencode` | `openai/anthropic/claude-sonnet-4.6` | 6 | 4 | 0.67 | 0.67 | 2 | 0 |
Exceptions:
- opencode/cli-with-tests: harbor.trial.trial.AgentTimeoutError: Agent execution timed out after 360.0 seconds
- opencode/crud-api: harbor.trial.trial.AgentTimeoutError: Agent execution timed out after 420.0 seconds
Artifact failures:
- none
Auth and host config:
- Codex auth upload worked. Trial logs show the wrapper used `CODEX_AUTH_JSON_PATH` and linked the uploaded auth file inside container-local `CODEX_HOME`; no secret contents are printed in the report.
- No host Codex, Claude, or OpenCode config writes were made by the comparison wrappers. Claude used container-local `CLAUDE_CONFIG_DIR=/tmp/claude-code-config`, OpenCode wrote config/auth under the task container home, and Codex used container-local `CODEX_HOME` plus `/tmp/codex-secrets`.
## terminal-bench/terminal-bench-2
- Result path: `evals/jobs-out/2026-05-05__03-52-21`
- Started: `2026-05-05T03:52:23.365643`
- Finished: `2026-05-05T04:24:04.769849`
| Agent | Model | n_trials | n_passed | pass_rate | mean_reward | exceptions | artifact failures |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
| `claude-code` | `anthropic/claude-opus-4.7` | 1 | 1 | 1.00 | 1.00 | 0 | 0 |
| `codex` | `openai/gpt-5.5` | 1 | 1 | 1.00 | 1.00 | 0 | 0 |
| `fx` | `anthropic/claude-opus-4.7` | 1 | 0 | 0.00 | 0.00 | 0 | 0 |
| `opencode` | `openai/anthropic/claude-sonnet-4.6` | 1 | 0 | 0.00 | 0.00 | 0 | 0 |
Exceptions:
- none
Artifact failures:
- none
evals/runs/2026-05-05-comparison-signoff.md
-89
View File
@@ -1,89 +0,0 @@
# 2026-05-05 fx comparison signoff
Viewer command:
```bash
uvx harbor view evals/jobs-out
```
## Summary
| Dataset | Agent | Model | n_trials | n_passed | pass_rate | mean_reward | exceptions | artifact failures | Result path |
| --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |
| `fx-release` | `claude-code` | `anthropic/claude-opus-4.7` | 6 | 0 | 0.00 | 0.00 | 6 | 6 | `evals/jobs-out/2026-05-05__00-24-47` |
| `fx-release` | `codex` | `openai/gpt-5.5` | 6 | 6 | 1.00 | 1.00 | 0 | 6 | `evals/jobs-out/2026-05-05__00-24-47` |
| `fx-release` | `fx` | `anthropic/claude-opus-4.7` | 6 | 6 | 1.00 | 1.00 | 0 | 0 | `evals/jobs-out/2026-05-05__00-24-47` |
| `fx-release` | `opencode` | `vercel/anthropic/claude-sonnet-4.6` | 6 | 0 | 0.00 | 0.00 | 6 | 6 | `evals/jobs-out/2026-05-05__00-24-47` |
| `terminal-bench/terminal-bench-2` | `claude-code` | `anthropic/claude-opus-4.7` | 1 | 1 | 1.00 | 1.00 | 0 | 1 | `evals/jobs-out/2026-05-05__01-28-07` |
| `terminal-bench/terminal-bench-2` | `codex` | `openai/gpt-5.5` | 1 | 1 | 1.00 | 1.00 | 0 | 1 | `evals/jobs-out/2026-05-05__01-28-07` |
| `terminal-bench/terminal-bench-2` | `fx` | `anthropic/claude-opus-4.7` | 1 | 0 | 0.00 | 0.00 | 0 | 0 | `evals/jobs-out/2026-05-05__01-28-07` |
| `terminal-bench/terminal-bench-2` | `opencode` | `vercel/anthropic/claude-sonnet-4.6` | 1 | 0 | 0.00 | 0.00 | 1 | 1 | `evals/jobs-out/2026-05-05__01-28-07` |
## fx-release
- Result path: `evals/jobs-out/2026-05-05__00-24-47`
- Started: `2026-05-05T00:24:47.628170`
- Finished: `2026-05-05T01:27:50.170974`
| Agent | Model | n_trials | n_passed | pass_rate | mean_reward | exceptions | artifact failures |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
| `claude-code` | `anthropic/claude-opus-4.7` | 6 | 0 | 0.00 | 0.00 | 6 | 6 |
| `codex` | `openai/gpt-5.5` | 6 | 6 | 1.00 | 1.00 | 0 | 6 |
| `fx` | `anthropic/claude-opus-4.7` | 6 | 6 | 1.00 | 1.00 | 0 | 0 |
| `opencode` | `vercel/anthropic/claude-sonnet-4.6` | 6 | 0 | 0.00 | 0.00 | 6 | 6 |
Exceptions:
- claude-code/cli-with-tests: Traceback (most recent call last):
- opencode/cli-with-tests: Traceback (most recent call last):
- opencode/create-file: Traceback (most recent call last):
- claude-code/create-file: Traceback (most recent call last):
- opencode/crud-api: Traceback (most recent call last):
- claude-code/crud-api: Traceback (most recent call last):
- opencode/edit-file: Traceback (most recent call last):
- claude-code/edit-file: Traceback (most recent call last):
- claude-code/grep-files: Traceback (most recent call last):
- opencode/grep-files: Traceback (most recent call last):
- opencode/run-command: Traceback (most recent call last):
- claude-code/run-command: Traceback (most recent call last):
Artifact failures:
- claude-code/cli-with-tests: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- codex/cli-with-tests: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- opencode/cli-with-tests: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- opencode/create-file: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- codex/create-file: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- claude-code/create-file: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- codex/crud-api: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- opencode/crud-api: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- claude-code/crud-api: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- opencode/edit-file: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- codex/edit-file: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- claude-code/edit-file: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- claude-code/grep-files: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- codex/grep-files: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- opencode/grep-files: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- opencode/run-command: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- claude-code/run-command: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- codex/run-command: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
## terminal-bench/terminal-bench-2
- Result path: `evals/jobs-out/2026-05-05__01-28-07`
- Started: `2026-05-05T01:28:09.628623`
- Finished: `2026-05-05T01:52:51.155403`
| Agent | Model | n_trials | n_passed | pass_rate | mean_reward | exceptions | artifact failures |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
| `claude-code` | `anthropic/claude-opus-4.7` | 1 | 1 | 1.00 | 1.00 | 0 | 1 |
| `codex` | `openai/gpt-5.5` | 1 | 1 | 1.00 | 1.00 | 0 | 1 |
| `fx` | `anthropic/claude-opus-4.7` | 1 | 0 | 0.00 | 0.00 | 0 | 0 |
| `opencode` | `vercel/anthropic/claude-sonnet-4.6` | 1 | 0 | 0.00 | 0.00 | 1 | 1 |
Exceptions:
- opencode/break-filter-js-from-html: Traceback (most recent call last):
Artifact failures:
- codex/break-filter-js-from-html: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- opencode/break-filter-js-from-html: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
- claude-code/break-filter-js-from-html: artifacts/manifest.json: /tmp/fx-home/.fx -> artifacts/fx-home: failed
evals/runs/2026-05-05-website-results-handoff.md
-340
View File
@@ -1,340 +0,0 @@
# 2026-05-05 Website Results Handoff
This is the source-of-truth data from the live Harbor comparison runs for Fx, Codex, Claude Code, and OpenCode.
Use the values marked as available. Do not infer or publish the values marked as not captured.
## Source Runs
| Suite | Harbor output dir | Harbor job id | Started | Finished | Wall-clock runtime |
| --- | --- | --- | --- | --- | --- |
| `fx-release` | `/Users/faxes/Developer/Fx/fx/evals/jobs-out/2026-05-05__02-16-55` | `f557f698-22fc-4ae7-8c96-3ab3da2ac97a` | `2026-05-05T02:16:55.309074` | `2026-05-05T03:50:30.681902` | `1h 33m 35s` |
| `terminal-bench/terminal-bench-2` smoke | `/Users/faxes/Developer/Fx/fx/evals/jobs-out/2026-05-05__03-52-21` | `eef1690e-71f6-42fa-a273-41c033f4c7d7` | `2026-05-05T03:52:23.365643` | `2026-05-05T04:24:04.769849` | `31m 41s` |
Generated report in repo:
`/Users/faxes/Developer/Fx/fx/evals/runs/2026-05-05-comparison-signoff-fixed.md`
## Models And Effort
| Agent | Model used | Effort / thinking setting | Notes |
| --- | --- | --- | --- |
| Fx | `anthropic/claude-opus-4.7` | `high` | Run through the built `fx` binary wrapper. |
| Codex | `openai/gpt-5.5` | `high` | Run through Codex CLI with local auth upload into the Harbor container. |
| Claude Code | `anthropic/claude-opus-4.7` | `high` | Run through Claude Code with Vercel AI Gateway env and container-local config. |
| OpenCode | `openai/anthropic/claude-sonnet-4.6` | `--thinking` | OpenCode did not run Opus in this comparison; it ran Sonnet 4.6 through the OpenAI-compatible AI Gateway path. |
## Combined Leaderboard
Combined means 6 `fx-release` tasks plus 1 Terminal-Bench smoke task, 7 scored tasks per agent.
| Rank | Agent | Model | Record | Pass rate | Mean reward | Gap vs leaders | Agent execution runtime |
| ---: | --- | --- | ---: | ---: | ---: | --- | ---: |
| 1 | Codex | `openai/gpt-5.5` | `7-0` | `100%` | `1.00` | `tied` | `18m 55s` |
| 1 | Claude Code | `anthropic/claude-opus-4.7` | `7-0` | `100%` | `1.00` | `tied` | `8m 35s` |
| 3 | Fx | `anthropic/claude-opus-4.7` | `6-1` | `86%` | `0.86` | `-14 pp vs leaders` | `11m 07s` |
| 4 | OpenCode | `openai/anthropic/claude-sonnet-4.6` | `4-3` | `57%` | `0.57` | `-43 pp vs leaders` | `17m 43s` |
Runtime above is summed agent execution time from Harbor trial result JSON. It does not include agent install/setup, environment setup, verifier time, or artifact collection.
## Per-Suite Results
### fx-release
| Agent | Model | n_trials | n_passed | pass_rate | mean_reward | exceptions | artifact failures | Agent execution runtime |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| Fx | `anthropic/claude-opus-4.7` | 6 | 6 | `1.00` | `1.00` | 0 | 0 | `6m 35s` |
| Codex | `openai/gpt-5.5` | 6 | 6 | `1.00` | `1.00` | 0 | 0 | `10m 16s` |
| Claude Code | `anthropic/claude-opus-4.7` | 6 | 6 | `1.00` | `1.00` | 0 | 0 | `6m 32s` |
| OpenCode | `openai/anthropic/claude-sonnet-4.6` | 6 | 4 | `0.67` | `0.67` | 2 | 0 | `16m 14s` |
### Terminal-Bench Smoke
Only the smoke task was run. Full Terminal-Bench was not run.
| Agent | Model | n_trials | n_passed | pass_rate | mean_reward | exceptions | artifact failures | Agent execution runtime |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| Fx | `anthropic/claude-opus-4.7` | 1 | 0 | `0.00` | `0.00` | 0 | 0 | `4m 32s` |
| Codex | `openai/gpt-5.5` | 1 | 1 | `1.00` | `1.00` | 0 | 0 | `8m 39s` |
| Claude Code | `anthropic/claude-opus-4.7` | 1 | 1 | `1.00` | `1.00` | 0 | 0 | `2m 03s` |
| OpenCode | `openai/anthropic/claude-sonnet-4.6` | 1 | 0 | `0.00` | `0.00` | 0 | 0 | `1m 29s` |
## Task Outcome Matrix
### fx-release Tasks
| Task | Fx | Codex | Claude Code | OpenCode |
| --- | --- | --- | --- | --- |
| `create-file` | pass | pass | pass | pass |
| `cli-with-tests` | pass | pass | pass | fail, timeout after 360s |
| `run-command` | pass | pass | pass | pass |
| `crud-api` | pass | pass | pass | fail, timeout after 420s |
| `edit-file` | pass | pass | pass | pass |
| `grep-files` | pass | pass | pass | pass |
### Terminal-Bench Smoke Task
| Task | Fx | Codex | Claude Code | OpenCode |
| --- | --- | --- | --- | --- |
| `break-filter-js-from-html` | fail | pass | pass | fail |
## Exceptions
| Suite | Agent | Task | Exception |
| --- | --- | --- | --- |
| `fx-release` | OpenCode | `cli-with-tests` | `harbor.trial.trial.AgentTimeoutError: Agent execution timed out after 360.0 seconds` |
| `fx-release` | OpenCode | `crud-api` | `harbor.trial.trial.AgentTimeoutError: Agent execution timed out after 420.0 seconds` |
No exceptions occurred in the Terminal-Bench smoke run.
## Artifact Collection
| Suite | Artifact collection failures |
| --- | ---: |
| `fx-release` | 0 |
| Terminal-Bench smoke | 0 |
The earlier expected `/tmp/fx-home/.fx` artifact warnings were fixed before this run. These final runs had no artifact collection failures.
## Auth And Host Config Scope
| Item | Status |
| --- | --- |
| `AI_GATEWAY_API_KEY` | Required and used for gateway-backed agents. Secret value is not printed here. |
| Codex auth upload | Worked. Logs show `CODEX_AUTH_JSON_PATH` was used and auth was linked inside container-local `CODEX_HOME`. Secret contents were not printed. |
| Host Codex config writes | Not observed from the comparison wrappers. Codex used container-local `CODEX_HOME` plus `/tmp/codex-secrets`. |
| Host Claude config writes | Not observed from the comparison wrappers. Claude used container-local `CLAUDE_CONFIG_DIR=/tmp/claude-code-config`. |
| Host OpenCode config writes | Not observed from the comparison wrappers. OpenCode config/auth was written under the Harbor task container home. |
## Efficiency Data Availability
Use this section carefully. Token and cost capture was not uniform across agents.
| Agent | Agent execution runtime | Tokens in | Cached tokens | Tokens out | Captured total cost | Cost per pass |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Fx | `11m 07s` | not captured | not captured | not captured | not captured | not captured |
| Codex | `18m 55s` | `1,628,940` | `1,551,616` | `17,172` | partial only: `$0.721403` captured for smoke, release cost not captured | not reliable |
| Claude Code | `8m 35s` | `1,654,503` | `1,383,540` | `14,757` | `$8.262375` | `$1.18` |
| OpenCode | `17m 43s` | `712,433` | not captured | `10,318` | not captured | not captured |
Do not publish cost-per-pass as a cross-agent comparison from this run. Harbor only captured complete cost for Claude Code, partial cost for Codex, and no cost for Fx or OpenCode.
## Website Schema Mapping
### `meta`
Available:
```ts
meta: {
slug: "comparison-signoff-fixed",
version: "v0.3.2",
label: "comparison signoff fixed",
date: "2026-05-05",
runId: "f557f698 + eef1690e",
}
```
Do not set `current: true` unless this run is intended to become the active website result.
### `hero`
Available values:
```ts
hero: {
eyebrow: "Harbor live comparison / fx-release + Terminal-Bench smoke",
title: "Fx ties the leaders on release evals, then falls behind on the deeper smoke task.",
summary: "Across the six fx-release tasks, **Fx, Codex, and Claude Code all scored 6/6**. Adding the Terminal-Bench smoke task separates the field: **Codex and Claude Code finish 7-0**, **Fx finishes 6-1**, and **OpenCode finishes 4-3**.",
meta: [
{ label: "Run", value: "comparison-signoff-fixed" },
{ label: "Date", value: "2026-05-05" },
{ label: "Suite", value: "fx-release + Terminal-Bench smoke" },
{ label: "Tasks", value: "7 scored tasks per agent" },
{ label: "Status", value: "completed" },
],
}
```
### `kpi`
Available values:
```ts
kpi: [
{ label: "Fx release pass rate", value: "100%", delta: "6/6 on fx-release", deltaTone: "positive" },
{ label: "Fx combined pass rate", value: "86%", delta: "6/7 across release + smoke", deltaTone: "neutral" },
{ label: "Leaders", value: "100%", delta: "Codex and Claude Code both 7/7", deltaTone: "positive" },
{ label: "Artifact failures", value: "0", delta: "No collection failures in final runs", deltaTone: "positive" },
]
```
### `cliStanding.rows`
Available values:
```ts
[
{
name: "Codex",
initials: "C",
model: "gpt-5.5, high reasoning",
logo: "openai-logo",
passPct: 100,
record: "7-0",
gap: "tied",
runtime: "18m 55s agent execution",
costPerPass: "not reliable",
},
{
name: "Claude Code",
initials: "CC",
model: "claude-opus-4.7, high effort",
logo: "anthropic-logo",
passPct: 100,
record: "7-0",
gap: "tied",
runtime: "8m 35s agent execution",
costPerPass: "$1.18 captured",
},
{
name: "Fx",
initials: "FX",
model: "claude-opus-4.7, high effort",
logo: "vercel",
passPct: 86,
record: "6-1",
gap: "-14 pp vs leaders",
runtime: "11m 07s agent execution",
costPerPass: "not captured",
isFx: true,
},
{
name: "OpenCode",
initials: "OC",
model: "claude-sonnet-4.6 via AI Gateway",
passPct: 57,
record: "4-3",
gap: "-43 pp vs leaders",
runtime: "17m 43s agent execution",
costPerPass: "not captured",
},
]
```
Note: OpenCode has no logo in the current logo list, so use initials fallback unless a logo is added.
### `taskOutcomes`
Available task outcome cells:
```ts
agents: ["Fx", "Codex", "Claude Code", "OpenCode"],
rows: [
{ task: "fx-release/create-file", cells: [{ status: "pass" }, { status: "pass" }, { status: "pass" }, { status: "pass" }] },
{ task: "fx-release/cli-with-tests", cells: [{ status: "pass" }, { status: "pass" }, { status: "pass" }, { status: "fail", note: "AgentTimeoutError after 360s" }] },
{ task: "fx-release/run-command", cells: [{ status: "pass" }, { status: "pass" }, { status: "pass" }, { status: "pass" }] },
{ task: "fx-release/crud-api", cells: [{ status: "pass" }, { status: "pass" }, { status: "pass" }, { status: "fail", note: "AgentTimeoutError after 420s" }] },
{ task: "fx-release/edit-file", cells: [{ status: "pass" }, { status: "pass" }, { status: "pass" }, { status: "pass" }] },
{ task: "fx-release/grep-files", cells: [{ status: "pass" }, { status: "pass" }, { status: "pass" }, { status: "pass" }] },
{ task: "terminal-bench-smoke/break-filter-js-from-html", cells: [{ status: "fail" }, { status: "pass" }, { status: "pass" }, { status: "fail" }] },
]
```
Suggested summary callouts:
```ts
summary: [
{
label: "Release evals",
body: "Fx, Codex, and Claude Code all passed every fx-release task. OpenCode passed four and timed out on two larger tasks.",
},
{
label: "Deeper smoke task",
body: "Codex and Claude Code solved the Terminal-Bench smoke task. Fx and OpenCode did not.",
},
{
label: "Harness quality",
body: "Final runs had zero artifact collection failures and no Claude/OpenCode setup failures.",
},
]
```
### `efficiency`
Only use this if the UI can clearly show missing values.
```ts
efficiency: {
note: "Runtime is summed Harbor agent execution time only. Cost and token capture was not uniform across agents.",
rows: [
{
name: "Codex",
initials: "C",
model: "gpt-5.5, high reasoning",
logo: "openai-logo",
runtime: "18m 55s",
tokensIn: "1,628,940",
tokensOut: "17,172",
total: "partial only",
perPass: "not reliable",
},
{
name: "Claude Code",
initials: "CC",
model: "claude-opus-4.7, high effort",
logo: "anthropic-logo",
runtime: "8m 35s",
tokensIn: "1,654,503",
tokensOut: "14,757",
total: "$8.262375",
perPass: "$1.18",
},
{
name: "Fx",
initials: "FX",
model: "claude-opus-4.7, high effort",
logo: "vercel",
runtime: "11m 07s",
tokensIn: "not captured",
tokensOut: "not captured",
total: "not captured",
perPass: "not captured",
isFx: true,
},
{
name: "OpenCode",
initials: "OC",
model: "claude-sonnet-4.6 via AI Gateway",
runtime: "17m 43s",
tokensIn: "712,433",
tokensOut: "10,318",
total: "not captured",
perPass: "not captured",
},
],
}
```
## Information Not Available From These Runs
Do not publish or imply these:
- Full Terminal-Bench performance. Only `--smoke` ran, with one Terminal-Bench task.
- Multi-run variance, confidence intervals, or statistical significance. Each task was run once per agent.
- Complete cross-agent cost comparison. Cost capture is incomplete.
- Complete cross-agent token comparison. Fx token usage was not captured.
- OpenCode Opus 4.7 performance. OpenCode ran `anthropic/claude-sonnet-4.6`, not Opus 4.7.
- Any result for Anomaly or other agents not listed above.
- Runtime including setup/install/verifier/artifact collection per agent. The per-agent runtime values here are agent execution only.
- A claim that Fx beats Codex or Claude Code overall. It ties them on `fx-release`, but loses the Terminal-Bench smoke task.
- A claim that Fx failed release readiness. Fx passed all six `fx-release` comparison tasks.
## Plain-English Read
Fx is strong on the project-local release evals: it went 6/6, tied with Codex and Claude Code, and beat OpenCode.
The deeper smoke task changes the story. Codex and Claude Code solved `break-filter-js-from-html`; Fx and OpenCode did not. So the accurate public framing is:
> Fx is release-eval competitive with Codex and Claude Code on this suite, but the external Terminal-Bench smoke result shows there is still a gap on harder agentic tasks.
evals/runs/README.md
-5
View File
@@ -1,5 +0,0 @@
# fx Release Eval Runs
Manual Harbor eval run notes live in this folder.
Use `RUNS.md` as the run index. Do not create top-level run logs in `evals/`.
evals/runs/RUNS.md
-122
View File
@@ -1,122 +0,0 @@
# fx Eval Run History
This file indexes preserved Harbor eval run notes. Raw Harbor output is written to `evals/jobs-out/`, which is intentionally ignored because it contains bulky local artifacts.
Use one dated Markdown file per meaningful run:
```text
evals/runs/YYYY-MM-DD-<short-description>.md
```
Do not create top-level run logs in `evals/`; keep dated reports in this folder.
Reference docs and benchmark pages:
- https://www.harborframework.com/docs
- https://www.harborframework.com/docs/getting-started
- https://www.harborframework.com/docs/core-concepts
- https://www.harborframework.com/docs/migration
- https://www.harborframework.com/docs/run-jobs
- https://www.harborframework.com/docs/run-jobs/run-evals
- https://www.harborframework.com/docs/run-jobs/results-and-artifacts
- https://www.harborframework.com/docs/run-jobs/cloud-sandboxes
- https://www.harborframework.com/docs/tasks
- https://www.harborframework.com/docs/tasks/multi-step
- https://www.harborframework.com/docs/tasks/task-difference
- https://www.harborframework.com/docs/tasks/task-tutorial
- https://www.harborframework.com/docs/tasks/windows-container-support
- https://www.harborframework.com/docs/tasks/publishing
- https://www.harborframework.com/docs/datasets
- https://www.harborframework.com/docs/datasets/metrics
- https://www.harborframework.com/docs/datasets/adapters
- https://www.harborframework.com/docs/datasets/adapters-human
- https://www.harborframework.com/docs/datasets/publishing
- https://www.harborframework.com/docs/sharing/sharing
- https://www.harborframework.com/docs/agents
- https://www.harborframework.com/docs/agents/trajectory-format
- https://www.harborframework.com/docs/agents/terminus-2
- https://www.harborframework.com/docs/tutorials/running-terminal-bench
- https://www.harborframework.com/docs/tutorials/mcp-server-task
- https://www.harborframework.com/docs/tutorials/llm-as-a-judge
- https://www.harborframework.com/docs/rewardkit
- https://www.harborframework.com/docs/rewardkit/judge-criteria
- https://www.harborframework.com/docs/rewardkit/built-in-criteria
- https://www.harborframework.com/docs/rewardkit/motivation
- https://www.harborframework.com/docs/training-workflows/sft
- https://www.harborframework.com/docs/training-workflows/rl
- https://www.harborframework.com/docs/contributing/roadmap
- https://hub.harborframework.com/datasets/terminal-bench/terminal-bench-2/latest
## Dataset And Artifact Notes
Local release runs use `path: evals/datasets/fx-release`. Published Terminal-Bench runs use `name: terminal-bench/terminal-bench-2` and download registry tasks into `evals/.build/harbor-datasets`.
The separate multi-step run uses `path: evals/datasets/fx-multistep`. It exercises Harbor's `steps/` layout and keeps per-step instruction, solution, and verifier files under each step directory.
Every job uses `evals/metrics/fx_metrics.py` for explicit Harbor metrics: `mean_reward`, `pass_rate`, `n_trials`, and `n_passed`. Each trial collects a compact `/logs/artifacts` bundle plus agent logs, verifier logs, and workspace evidence. `fx`-only jobs also collect `fx` home state. Comparison jobs omit that `fx`-specific path so non-`fx` agents do not produce expected artifact warnings. Multi-step run artifacts appear under `steps/<step-name>/artifacts/` for each step.
RewardKit examples live under `evals/rewardkit/` and are documented in `evals/REWARDKIT.md`. They are optional examples only and are not connected to default release signoff jobs. LLM and agent judges are slower, can cost money, and require explicit opt-in.
Workspace artifacts exclude bulky generated directories from `/app`: `node_modules`, `.git`, `.cache`, `__pycache__`, `.pytest_cache`, `dist`, `build`, `coverage`, `target`, `.venv`, and `venv`. Check each trial's `artifacts/manifest.json` for collection status.
Publishing is private by default and remains an explicit release-owner action. Use `evals/PUBLISHING.md` for auth, sync, publish, tag, visibility, download, and run-by-reference commands. Use `./evals/scripts/check-publish-ready.sh` for a non-publishing local preflight.
Cloud sandboxes are optional for larger published-dataset sweeps. Use `evals/CLOUD-SANDBOXES.md` for Daytona-style commands and caveats. Local Docker remains the default release signoff environment.
Windows tasks, MCP sidecar tasks, LLM-as-a-Judge verifiers, ATIF export, SFT export, RL rollout integration, Terminus-2 baselines, and roadmap-only items are not part of the default release signoff. They are documented in `evals/README.md` as optional Harbor capabilities to add only when a release or benchmark claim needs them.
## Release Signoff Runs
The default release signoff sequence is now one wrapper:
```bash
./evals/scripts/run-release-signoff.sh
```
The wrapper runs local `fx-release`, runs Terminal-Bench smoke, writes a dated report in this folder, and updates this index. Use the older individual commands only when debugging a specific slice.
Decision criteria:
- `fx-release` must pass.
- Terminal-Bench smoke must complete without Harbor harness exceptions.
- Comparison is recommended for major releases or before public benchmark claims.
- Full Terminal-Bench requires explicit release-owner approval through `--full-terminal-bench`.
Supporting commands:
1. Consolidated preflight: `./evals/scripts/check-release-ready.sh`
2. Default signoff: `./evals/scripts/run-release-signoff.sh`
3. After task or verifier edits, local task oracle checks: `./evals/scripts/check-local-tasks.sh`
4. Optional multi-step coverage: `./evals/scripts/run-release-signoff.sh --multistep`
5. Recommended comparison path: `./evals/scripts/run-release-signoff.sh --compare`
6. Full Terminal-Bench comparison with explicit approval: `./evals/scripts/run-release-signoff.sh --compare --full-terminal-bench`
7. Optional publishing preflight only: `./evals/scripts/check-publish-ready.sh`
`fx-release` remains required for every release. Comparison evals are recommended for major releases and before public benchmark claims. Full Terminal-Bench comparison is not part of the default release path because it is slow and can cost money across all agents.
Optional RewardKit example validation is separate from release signoff:
```bash
./evals/scripts/check-rewardkit-ready.sh
```
This validates example TOML and deterministic built-in criteria shape without running paid judge evals by default.
Manual report generation for existing Harbor output:
```bash
./evals/scripts/write-run-report.py evals/jobs-out/<job-dir> [evals/jobs-out/<job-dir> ...]
```
## Runs
| Date | Run | Agent | Model | Pass Rate | Mean Reward | Notes |
| --- | --- | --- | --- | --- | --- | --- |
| 2026-05-05 | [fx comparison signoff fixed](2026-05-05-comparison-signoff-fixed.md) | `varies` | `varies` | `see report` | `see report` | fx-release: claude-code 1.00/codex 1.00/fx 1.00/opencode 0.67, terminal-bench/terminal-bench-2: claude-code 1.00/codex 1.00/fx 0.00/opencode 0.00 |
| 2026-05-05 | [fx comparison signoff](2026-05-05-comparison-signoff.md) | `varies` | `varies` | `see report` | `see report` | fx-release: claude-code 0.00/codex 1.00/fx 1.00/opencode 0.00, terminal-bench/terminal-bench-2: claude-code 1.00/codex 1.00/fx 0.00/opencode 0.00 |
| 2026-05-04 | [fx comparison signoff](2026-05-04-comparison-signoff.md) | `varies` | `varies` | `see report` | `see report` | fx-release: claude-code 0.00/codex 0.00/fx 1.00/opencode 0.00, terminal-bench/terminal-bench-2: claude-code 1.00/codex 0.00/fx 0.00/opencode 0.00 |
| 2026-05-04 | [fx release signoff](2026-05-04-release-signoff.md) | `varies` | `varies` | `see report` | `see report` | fx-release: fx 1.00, terminal-bench/terminal-bench-2: fx 0.00 |
| 2026-05-04 | [fx release task slice](2026-05-04-fx-release-task-slice.md) | `fx` | `claude-sonnet-4.6` | `1.00` | `1.00` | 6 trials, 8m 9s, fixed `cli-with-tests` path scored 1, 0 exceptions, 0 artifact collection failures. |
| 2026-05-04 | [fx multi-step smoke](2026-05-04-fx-multistep-smoke.md) | `fx` | `claude-sonnet-4.6` | `1.00` | `1.00` | 1 trial, 2 sequential steps, 1m 6s, explicit metrics, 0 artifact collection failures. |
| 2026-05-04 | [fx release smoke](2026-05-04-fx-release-smoke.md) | `fx` | `claude-sonnet-4.6` | `1.00` | `1.00` | 6 trials, 9m 56s, 0 exceptions, captured fx exit code 0 for each task. |
| 2026-05-04 | [Terminal-Bench smoke](2026-05-04-terminal-bench-smoke.md) | `fx` | `claude-sonnet-4.6` | `0.00` | `0.00` | 1 trial, 0 exceptions, captured fx exit code 0. |
evals/scripts/build-image.sh
-92
View File
@@ -1,92 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
binary_only=0
for arg in "$@"; do
case "$arg" in
--binary-only)
binary_only=1
;;
*)
echo "unknown argument: $arg" >&2
exit 1
;;
esac
done
if ! command -v zig >/dev/null 2>&1; then
echo "zig is required" >&2
exit 1
fi
repo_root="$(git rev-parse --show-toplevel 2>/dev/null || true)"
if [ -z "$repo_root" ]; then
echo "run this script from the fx repository" >&2
exit 1
fi
repo_root_real="$(cd "$repo_root" && pwd -P)"
current_dir_real="$(pwd -P)"
if [ "$current_dir_real" != "$repo_root_real" ]; then
echo "run this script from the fx repository root" >&2
exit 1
fi
cd "$repo_root_real"
if [ ! -f build.zig ] || [ ! -f src/main.zig ]; then
echo "run this script from the fx repository root" >&2
exit 1
fi
if [ "$binary_only" -eq 0 ]; then
if ! command -v docker >/dev/null 2>&1; then
echo "Docker is required" >&2
exit 1
fi
docker_plugin_dir="/opt/homebrew/lib/docker/cli-plugins"
if [ -d "$docker_plugin_dir" ] && [ -z "${DOCKER_CONFIG:-}" ]; then
mkdir -p evals/.build/docker-config
cat > evals/.build/docker-config/config.json <<JSON
{
"cliPluginsExtraDirs": [
"$docker_plugin_dir"
]
}
JSON
export DOCKER_CONFIG="$repo_root_real/evals/.build/docker-config"
fi
colima_socket="$HOME/.colima/default/docker.sock"
if [ -S "$colima_socket" ] && [ -z "${DOCKER_HOST:-}" ]; then
export DOCKER_HOST="unix://$colima_socket"
fi
if ! docker info >/dev/null 2>&1; then
echo "Docker daemon is not available; start Docker Desktop or Colima" >&2
exit 1
fi
fi
linux_prefix="$repo_root_real/evals/.build/linux"
linux_binary="$linux_prefix/bin/fx"
mkdir -p "$linux_prefix" evals/images
zig build \
--prefix "$linux_prefix" \
-Doptimize=ReleaseSafe \
-Dtarget=x86_64-linux
if [ ! -x "$linux_binary" ]; then
echo "freshly built Linux fx binary not found at $linux_binary" >&2
exit 1
fi
install -m 0755 "$linux_binary" evals/images/fx
if [ "$binary_only" -eq 1 ]; then
exit 0
fi
docker build --platform linux/amd64 -t fx-evals:local -f evals/images/Dockerfile evals/images
evals/scripts/check-compare-ready.py
-345
View File
@@ -1,345 +0,0 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import importlib
import os
import sys
from dataclasses import dataclass
from pathlib import Path
from typing import Any
REPO_ROOT = Path(__file__).resolve().parents[2]
JOBS_ROOT = REPO_ROOT / "evals" / "jobs"
COMPARISON_JOB_NAMES = {
"compare-key-agents.yaml",
"compare-terminal-bench-smoke.yaml",
"compare-terminal-bench.yaml",
}
COMPARISON_AGENTS = {"fx", "codex", "claude-code", "opencode"}
@dataclass
class YamlLine:
indent: int
text: str
lineno: int
def rel(path: Path) -> str:
return path.relative_to(REPO_ROOT).as_posix()
def fail(errors: list[str], path: Path | None, message: str) -> None:
if path is None:
errors.append(message)
else:
errors.append(f"{rel(path)}: {message}")
def strip_comment(line: str) -> str:
in_single = False
in_double = False
escaped = False
out: list[str] = []
for char in line:
if char == "\\" and in_double:
escaped = not escaped
out.append(char)
continue
if char == "'" and not in_double:
in_single = not in_single
elif char == '"' and not in_single and not escaped:
in_double = not in_double
if char == "#" and not in_single and not in_double:
break
out.append(char)
escaped = False
return "".join(out).rstrip()
def yaml_lines(path: Path) -> list[YamlLine]:
lines: list[YamlLine] = []
for lineno, raw in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
if "\t" in raw:
raise ValueError(f"line {lineno}: tabs are not supported in job YAML")
stripped = strip_comment(raw)
if not stripped.strip():
continue
indent = len(stripped) - len(stripped.lstrip(" "))
lines.append(YamlLine(indent=indent, text=stripped.strip(), lineno=lineno))
return lines
def split_key_value(text: str, lineno: int) -> tuple[str, str | None]:
if ":" not in text:
raise ValueError(f"line {lineno}: expected key: value")
key, value = text.split(":", 1)
key = key.strip()
if not key:
raise ValueError(f"line {lineno}: empty key")
value = value.strip()
return key, value if value else None
def scalar(value: str) -> Any:
if value in ("true", "false"):
return value == "true"
if value in ("null", "~"):
return None
if (
(value.startswith('"') and value.endswith('"'))
or (value.startswith("'") and value.endswith("'"))
):
return value[1:-1]
try:
if "." in value:
return float(value)
return int(value)
except ValueError:
return value
def parse_mapping(lines: list[YamlLine], index: int, indent: int) -> tuple[dict[str, Any], int]:
result: dict[str, Any] = {}
while index < len(lines):
line = lines[index]
if line.indent < indent:
break
if line.indent > indent:
raise ValueError(f"line {line.lineno}: unexpected indentation")
if line.text.startswith("- "):
break
key, value = split_key_value(line.text, line.lineno)
index += 1
if value is not None:
result[key] = scalar(value)
continue
if index >= len(lines) or lines[index].indent <= indent:
result[key] = {}
continue
child, index = parse_block(lines, index, lines[index].indent)
result[key] = child
return result, index
def parse_sequence(lines: list[YamlLine], index: int, indent: int) -> tuple[list[Any], int]:
result: list[Any] = []
while index < len(lines):
line = lines[index]
if line.indent < indent:
break
if line.indent > indent:
raise ValueError(f"line {line.lineno}: unexpected indentation")
if not line.text.startswith("- "):
break
item_text = line.text[2:].strip()
index += 1
if not item_text:
if index >= len(lines) or lines[index].indent <= indent:
result.append(None)
continue
item, index = parse_block(lines, index, lines[index].indent)
result.append(item)
continue
if ":" in item_text:
key, value = split_key_value(item_text, line.lineno)
item: dict[str, Any] = {}
if value is None:
if index >= len(lines) or lines[index].indent <= indent:
item[key] = {}
else:
item[key], index = parse_block(lines, index, lines[index].indent)
else:
item[key] = scalar(value)
if index < len(lines) and lines[index].indent > indent:
extra, index = parse_mapping(lines, index, lines[index].indent)
item.update(extra)
result.append(item)
else:
result.append(scalar(item_text))
return result, index
def parse_block(lines: list[YamlLine], index: int, indent: int) -> tuple[Any, int]:
if lines[index].text.startswith("- "):
return parse_sequence(lines, index, indent)
return parse_mapping(lines, index, indent)
def fallback_yaml_load(path: Path) -> dict[str, Any]:
lines = yaml_lines(path)
if not lines:
return {}
value, index = parse_block(lines, 0, lines[0].indent)
if index != len(lines):
line = lines[index]
raise ValueError(f"line {line.lineno}: could not parse remaining YAML")
if not isinstance(value, dict):
raise ValueError("top-level YAML value must be a mapping")
return value
def load_yaml(path: Path) -> dict[str, Any]:
try:
import yaml # type: ignore[import-not-found]
except ImportError:
return fallback_yaml_load(path)
value = yaml.safe_load(path.read_text(encoding="utf-8"))
if not isinstance(value, dict):
raise ValueError("top-level YAML value must be a mapping")
return value
def check_env(errors: list[str]) -> None:
if not os.environ.get("AI_GATEWAY_API_KEY"):
fail(
errors,
None,
"AI_GATEWAY_API_KEY is required for fx, Codex, Claude Code, and OpenCode comparison agents",
)
def job_paths() -> list[Path]:
return sorted(JOBS_ROOT.glob("*.yaml"))
def runtime_agent_name(
path: Path, item: dict[str, Any], errors: list[str]
) -> str | None:
import_path = item.get("import_path")
if isinstance(import_path, str):
agent_class = check_import_path(path, import_path, errors)
if agent_class is not None:
name_fn = getattr(agent_class, "name", None)
if callable(name_fn):
try:
name = name_fn()
except Exception as exc:
fail(errors, path, f"{import_path}.name() failed: {exc}")
else:
if isinstance(name, str) and name:
return name
name = item.get("name")
return name if isinstance(name, str) else None
def check_comparison_jobs(
parsed_by_path: dict[Path, dict[str, Any]], errors: list[str]
) -> None:
found_names = {
path.name for path in parsed_by_path if path.name in COMPARISON_JOB_NAMES
}
missing = sorted(COMPARISON_JOB_NAMES - found_names)
for name in missing:
fail(errors, JOBS_ROOT / name, "comparison job config is missing")
for path, parsed in parsed_by_path.items():
if path.name not in COMPARISON_JOB_NAMES:
continue
agents = parsed.get("agents")
if not isinstance(agents, list):
fail(errors, path, "agents must be a list")
continue
names = {
name
for item in agents
if isinstance(item, dict)
if (name := runtime_agent_name(path, item, errors)) is not None
}
missing_agents = sorted(COMPARISON_AGENTS - names)
if missing_agents:
fail(
errors,
path,
"missing comparison agents: " + ", ".join(missing_agents),
)
def check_import_path(path: Path, import_path: str, errors: list[str]) -> Any | None:
if ":" not in import_path:
fail(errors, path, f"import_path must be module:Class: {import_path}")
return None
module_name, class_name = import_path.split(":", 1)
try:
module = importlib.import_module(module_name)
except Exception as exc:
fail(errors, path, f"cannot import {module_name}: {exc}")
return None
if not hasattr(module, class_name):
fail(errors, path, f"{module_name} does not expose {class_name}")
return None
return getattr(module, class_name)
def check_harbor_job_config(path: Path, parsed: dict[str, Any], errors: list[str]) -> None:
try:
from harbor.models.job.config import JobConfig
except Exception as exc:
fail(errors, path, f"Harbor JobConfig is not importable: {exc}")
return
try:
JobConfig.model_validate(parsed)
except Exception as exc:
fail(errors, path, f"Harbor JobConfig validation failed: {exc}")
def check_job_configs(errors: list[str]) -> tuple[int, int]:
parsed_by_path: dict[Path, dict[str, Any]] = {}
for path in job_paths():
try:
parsed = load_yaml(path)
except Exception as exc:
fail(errors, path, f"YAML parse failed: {exc}")
continue
parsed_by_path[path] = parsed
check_harbor_job_config(path, parsed, errors)
check_comparison_jobs(parsed_by_path, errors)
return len(parsed_by_path), len(
[path for path in parsed_by_path if path.name in COMPARISON_JOB_NAMES]
)
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument(
"--config-only",
action="store_true",
help="Validate YAML and Harbor JobConfig shape without checking credentials.",
)
args = parser.parse_args()
sys.path.insert(0, str(REPO_ROOT))
os.chdir(REPO_ROOT)
errors: list[str] = []
if not args.config_only:
check_env(errors)
total_jobs, comparison_jobs = check_job_configs(errors)
if errors:
for error in errors:
print(error, file=sys.stderr)
return 1
print("comparison readiness checks passed")
if args.config_only:
print("credential checks skipped")
print(
f"validated {total_jobs} job YAML files and {comparison_jobs} comparison job configs"
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
evals/scripts/check-compare-ready.sh
-31
View File
@@ -1,31 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
repo_root="$(git rev-parse --show-toplevel 2>/dev/null || true)"
if [ -z "$repo_root" ]; then
echo "run this script from the fx repository" >&2
exit 1
fi
cd "$repo_root"
mkdir -p \
evals/.build/pycache \
evals/.build/uv-cache \
evals/.build/uv-tools \
evals/.build/uv-python
export PYTHONPYCACHEPREFIX="$repo_root/evals/.build/pycache"
export UV_CACHE_DIR="$repo_root/evals/.build/uv-cache"
export UV_TOOL_DIR="$repo_root/evals/.build/uv-tools"
export UV_PYTHON_INSTALL_DIR="$repo_root/evals/.build/uv-python"
if python3 -c 'import harbor.models.job.config' >/dev/null 2>&1; then
python3 evals/scripts/check-compare-ready.py "$@"
else
if ! command -v uv >/dev/null 2>&1; then
echo "uv is required when Harbor is not importable by python3" >&2
exit 1
fi
uv run --quiet --python 3.12 --with harbor python evals/scripts/check-compare-ready.py "$@"
fi
evals/scripts/check-local-tasks.sh
-154
View File
@@ -1,154 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
repo_root="$(git rev-parse --show-toplevel 2>/dev/null || true)"
if [ -z "$repo_root" ]; then
echo "run this script from the fx repository" >&2
exit 1
fi
cd "$repo_root"
if ! command -v docker >/dev/null 2>&1; then
echo "Docker is required" >&2
exit 1
fi
docker_plugin_dir="/opt/homebrew/lib/docker/cli-plugins"
if [ -d "$docker_plugin_dir" ] && [ -z "${DOCKER_CONFIG:-}" ]; then
mkdir -p evals/.build/docker-config
cat > evals/.build/docker-config/config.json <<JSON
{
"cliPluginsExtraDirs": [
"$docker_plugin_dir"
]
}
JSON
export DOCKER_CONFIG="$repo_root/evals/.build/docker-config"
fi
colima_socket="$HOME/.colima/default/docker.sock"
if [ -S "$colima_socket" ] && [ -z "${DOCKER_HOST:-}" ]; then
export DOCKER_HOST="unix://$colima_socket"
fi
if ! docker info >/dev/null 2>&1; then
echo "Docker daemon is not available; start Docker Desktop or Colima" >&2
exit 1
fi
if ! docker image inspect fx-evals:local >/dev/null 2>&1; then
echo "fx-evals:local image is required; run ./evals/scripts/build-image.sh first" >&2
exit 1
fi
safe_tag() {
local name="$1"
name="${name//[^a-zA-Z0-9_.-]/-}"
printf 'fx-task-check:%s\n' "$name"
}
build_task_image() {
local task_dir="$1"
local name="$2"
local tag
tag="$(safe_tag "$name")"
docker build \
--platform linux/amd64 \
-t "$tag" \
-f "$task_dir/environment/Dockerfile" \
"$task_dir/environment" \
>/dev/null
printf '%s\n' "$tag"
}
check_single_step_task() {
local task_dir="$1"
local task_name
local task_abs
local image
task_name="$(basename "$task_dir")"
task_abs="$(cd "$task_dir" && pwd -P)"
printf 'checking %s\n' "$task_name"
image="$(build_task_image "$task_abs" "release-$task_name")"
docker run --rm \
--platform linux/amd64 \
-v "$task_abs/solution:/solution:ro" \
-v "$task_abs/tests:/tests:ro" \
"$image" \
bash -lc '
set -u
rm -rf /logs/verifier
mkdir -p /logs/verifier
bash /solution/solve.sh
bash /tests/test.sh
reward=""
if [ -f /logs/verifier/reward.txt ]; then
reward="$(cat /logs/verifier/reward.txt)"
fi
case "$reward" in
1|1.0) exit 0 ;;
*)
echo "reward=$reward" >&2
find /logs/verifier -maxdepth 1 -type f -print >&2
exit 1
;;
esac
'
}
check_multistep_task() {
local task_dir="$1"
shift
local task_name
local task_abs
local image
task_name="$(basename "$task_dir")"
task_abs="$(cd "$task_dir" && pwd -P)"
printf 'checking %s\n' "$task_name"
image="$(build_task_image "$task_abs" "multistep-$task_name")"
docker run --rm \
--platform linux/amd64 \
-v "$task_abs/steps:/task/steps:ro" \
"$image" \
bash -lc '
set -u
for step in "$@"; do
rm -rf /logs/verifier
mkdir -p /logs/verifier
bash "/task/steps/$step/solution/solve.sh"
bash "/task/steps/$step/tests/test.sh"
reward=""
if [ -f /logs/verifier/reward.txt ]; then
reward="$(cat /logs/verifier/reward.txt)"
fi
case "$reward" in
1|1.0) ;;
*)
echo "$step reward=$reward" >&2
find /logs/verifier -maxdepth 1 -type f -print >&2
exit 1
;;
esac
done
' check-local-tasks "$@"
}
for task_dir in evals/datasets/fx-release/*; do
if [ -f "$task_dir/task.toml" ]; then
check_single_step_task "$task_dir"
fi
done
check_multistep_task \
evals/datasets/fx-multistep/stateful-report \
seed-report \
extend-report
echo "all local task solutions produced reward 1"
evals/scripts/check-publish-ready.py
-271
View File
@@ -1,271 +0,0 @@
#!/usr/bin/env python3
from __future__ import annotations
import re
import subprocess
import sys
from pathlib import Path
from typing import Any
try:
import yaml # type: ignore[import-not-found]
except ImportError:
yaml = None
REPO_ROOT = Path(__file__).resolve().parents[2]
EVALS_ROOT = REPO_ROOT / "evals"
DATASETS_ROOT = EVALS_ROOT / "datasets"
JOBS_ROOT = EVALS_ROOT / "jobs"
TASK_NAME_RE = re.compile(r"^[a-z0-9][a-z0-9-]*/[a-z0-9][a-z0-9-]*$")
GENERATED_PREFIXES = (
"evals/.build/",
"evals/jobs-out/",
"evals/images/fx",
)
def parse_value(raw: str) -> Any:
value = raw.strip()
if value.startswith('"') and value.endswith('"'):
return value[1:-1]
if value in ("true", "false"):
return value == "true"
if value.startswith("[") and value.endswith("]"):
inner = value[1:-1].strip()
if not inner:
return []
if inner.startswith("{"):
return [inner]
return [parse_value(item.strip()) for item in inner.split(",")]
try:
if "." in value:
return float(value)
return int(value)
except ValueError:
return value
def load_toml(path: Path) -> dict[str, Any]:
data: dict[str, Any] = {}
current: dict[str, Any] = data
current_step: dict[str, Any] | None = None
for line in path.read_text(encoding="utf-8").splitlines():
stripped = line.strip()
if not stripped or stripped.startswith("#"):
continue
if stripped == "[[steps]]":
steps = data.setdefault("steps", [])
if not isinstance(steps, list):
raise ValueError(f"steps was not a list in {path}")
current_step = {}
steps.append(current_step)
current = current_step
continue
if stripped.startswith("[") and stripped.endswith("]"):
section = stripped.strip("[]")
if section.startswith("steps."):
if current_step is None:
raise ValueError(f"{section} appeared before [[steps]] in {path}")
target = current_step
for part in section.split(".")[1:]:
child = target.setdefault(part, {})
if not isinstance(child, dict):
raise ValueError(f"{section} conflicted with an existing value in {path}")
target = child
current = target
else:
target = data
for part in section.split("."):
child = target.setdefault(part, {})
if not isinstance(child, dict):
raise ValueError(f"{section} conflicted with an existing value in {path}")
target = child
current = target
continue
if "=" not in stripped:
continue
key, raw_value = stripped.split("=", 1)
current[key.strip()] = parse_value(raw_value)
return data
def rel(path: Path) -> str:
return path.relative_to(REPO_ROOT).as_posix()
def fail(errors: list[str], path: Path, message: str) -> None:
errors.append(f"{rel(path)}: {message}")
def check_task(path: Path, errors: list[str]) -> None:
data = load_toml(path)
task = data.get("task")
if not isinstance(task, dict):
fail(errors, path, "missing [task] section")
return
name = task.get("name")
if not isinstance(name, str) or not TASK_NAME_RE.match(name):
fail(errors, path, "task.name must be org/name with lowercase slug segments")
elif name.rsplit("/", 1)[1] != path.parent.name:
fail(errors, path, "task.name suffix must match the task directory name")
environment = data.get("environment")
if not isinstance(environment, dict):
fail(errors, path, "missing [environment] section")
elif not (path.parent / "environment" / "Dockerfile").is_file():
fail(errors, path, "missing environment/Dockerfile")
steps = data.get("steps")
if steps is None:
for required in ("instruction.md", "solution/solve.sh", "tests/test.sh"):
if not (path.parent / required).is_file():
fail(errors, path, f"missing {required}")
return
if not isinstance(steps, list) or not steps:
fail(errors, path, "steps must be a non-empty list for multi-step tasks")
return
for step in steps:
if not isinstance(step, dict):
fail(errors, path, "each step entry must be a table")
continue
step_name = step.get("name")
if not isinstance(step_name, str) or not step_name:
fail(errors, path, "each step must have a name")
continue
step_dir = path.parent / "steps" / step_name
for required in ("instruction.md", "solution/solve.sh", "tests/test.sh"):
if not (step_dir / required).is_file():
fail(errors, path, f"missing steps/{step_name}/{required}")
def check_dataset(path: Path, errors: list[str]) -> None:
data = load_toml(path)
dataset = data.get("dataset")
if not isinstance(dataset, dict):
fail(errors, path, "missing [dataset] section")
return
name = dataset.get("name")
if not isinstance(name, str) or not TASK_NAME_RE.match(name):
fail(errors, path, "dataset.name must be org/name with lowercase slug segments")
authors = dataset.get("authors")
if not isinstance(authors, list) or not authors:
fail(errors, path, "dataset.authors must list at least one author")
task_paths = sorted(path.parent.glob("*/task.toml"))
if not task_paths:
fail(errors, path, "dataset directory must contain at least one task")
for task_path in task_paths:
check_task(task_path, errors)
def check_metric_scripts(errors: list[str]) -> None:
metric_path = EVALS_ROOT / "metrics" / "fx_metrics.py"
if not metric_path.is_file():
fail(errors, metric_path, "shared metric script is missing")
return
text = metric_path.read_text(encoding="utf-8")
for needle in ("# /// script", "dependencies = []", "--input-path", "--output-path"):
if needle not in text:
fail(errors, metric_path, f"missing metric marker {needle!r}")
local_metric = DATASETS_ROOT / "fx-release" / "metric.py"
if local_metric.is_file():
local_text = local_metric.read_text(encoding="utf-8")
if "metrics" not in local_text or "fx_metrics.py" not in local_text:
fail(errors, local_metric, "local dataset metric should delegate to fx_metrics.py")
def parse_job(path: Path) -> dict[str, Any] | None:
if yaml is None:
return None
value = yaml.safe_load(path.read_text(encoding="utf-8"))
return value if isinstance(value, dict) else None
def check_job_config(path: Path, errors: list[str]) -> None:
text = path.read_text(encoding="utf-8")
for key in ("jobs_dir:", "agents:", "datasets:", "metrics:", "artifacts:"):
if key not in text:
fail(errors, path, f"missing {key}")
if "script_path: evals/metrics/fx_metrics.py" not in text:
fail(errors, path, "missing shared metric script_path")
parsed = parse_job(path)
if parsed is None:
return
datasets = parsed.get("datasets")
if not isinstance(datasets, list) or not datasets:
fail(errors, path, "datasets must be a non-empty list")
else:
for dataset in datasets:
if not isinstance(dataset, dict):
fail(errors, path, "dataset entries must be maps")
continue
if "path" in dataset:
dataset_path = REPO_ROOT / str(dataset["path"])
if not dataset_path.is_dir():
fail(errors, path, f"local dataset path does not exist: {dataset['path']}")
elif "name" not in dataset:
fail(errors, path, "dataset entry must have path or name")
metrics = parsed.get("metrics")
if not isinstance(metrics, list) or not metrics:
fail(errors, path, "metrics must be a non-empty list")
def check_generated_outputs(errors: list[str]) -> None:
tracked = subprocess.run(
["git", "ls-files", "--", "evals"],
cwd=REPO_ROOT,
check=True,
text=True,
stdout=subprocess.PIPE,
).stdout.splitlines()
for path in tracked:
if path.endswith(".pyc") or "/__pycache__/" in path:
errors.append(f"{path}: generated Python cache must not be tracked")
if any(is_generated_path(path, prefix) for prefix in GENERATED_PREFIXES):
errors.append(f"{path}: generated Harbor output must not be tracked")
def is_generated_path(path: str, prefix: str) -> bool:
if prefix.endswith("/"):
return path.startswith(prefix)
return path == prefix
def main() -> int:
errors: list[str] = []
for dataset_path in sorted(DATASETS_ROOT.glob("*/dataset.toml")):
check_dataset(dataset_path, errors)
check_metric_scripts(errors)
for job_path in sorted(JOBS_ROOT.glob("*.yaml")):
check_job_config(job_path, errors)
check_generated_outputs(errors)
if errors:
for error in errors:
print(error, file=sys.stderr)
return 1
print("publish readiness checks passed")
print("checked local datasets, task names, metric scripts, job configs, and tracked generated outputs")
return 0
if __name__ == "__main__":
raise SystemExit(main())
evals/scripts/check-publish-ready.sh
-11
View File
@@ -1,11 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
repo_root="$(git rev-parse --show-toplevel 2>/dev/null || true)"
if [ -z "$repo_root" ]; then
echo "run this script from the fx repository" >&2
exit 1
fi
cd "$repo_root"
python3 evals/scripts/check-publish-ready.py
evals/scripts/check-release-ready.sh
-136
View File
@@ -1,136 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
compare=0
for arg in "$@"; do
case "$arg" in
--compare)
compare=1
;;
-h|--help)
cat <<'USAGE'
usage: ./evals/scripts/check-release-ready.sh [--compare]
Validates the local release eval environment without printing secret values.
Pass --compare to also validate comparison credentials for all Gateway-backed agents.
USAGE
exit 0
;;
*)
echo "unknown argument: $arg" >&2
exit 1
;;
esac
done
repo_root="$(git rev-parse --show-toplevel 2>/dev/null || true)"
if [ -z "$repo_root" ]; then
echo "run this script from the fx repository" >&2
exit 1
fi
cd "$repo_root"
mkdir -p \
evals/.build/pycache \
evals/.build/uv-cache \
evals/.build/uv-tools \
evals/.build/uv-python
export PYTHONPYCACHEPREFIX="$repo_root/evals/.build/pycache"
export UV_CACHE_DIR="$repo_root/evals/.build/uv-cache"
export UV_TOOL_DIR="$repo_root/evals/.build/uv-tools"
export UV_PYTHON_INSTALL_DIR="$repo_root/evals/.build/uv-python"
failures=0
tmp_dir="$(mktemp -d "${TMPDIR:-/tmp}/fx-release-ready.XXXXXX")"
trap 'rm -rf "$tmp_dir"' EXIT
check_step() {
local label="$1"
shift
printf 'checking %s... ' "$label"
if "$@" >"$tmp_dir/stdout" 2>"$tmp_dir/stderr"; then
printf 'ok\n'
else
printf 'failed\n' >&2
sed 's/^/ /' "$tmp_dir/stderr" >&2
failures=$((failures + 1))
fi
}
require_command() {
local name="$1"
if ! command -v "$name" >/dev/null 2>&1; then
echo "$name is required" >&2
return 1
fi
}
check_fx_credentials() {
if [ -n "${AI_GATEWAY_API_KEY:-}" ] || [ -n "${VERCEL_OIDC_TOKEN:-}" ]; then
return 0
fi
echo "AI_GATEWAY_API_KEY or VERCEL_OIDC_TOKEN is required for fx evals" >&2
return 1
}
check_docker_daemon() {
require_command docker
local docker_plugin_dir="/opt/homebrew/lib/docker/cli-plugins"
if [ -d "$docker_plugin_dir" ] && [ -z "${DOCKER_CONFIG:-}" ]; then
mkdir -p evals/.build/docker-config
cat > evals/.build/docker-config/config.json <<JSON
{
"cliPluginsExtraDirs": [
"$docker_plugin_dir"
]
}
JSON
export DOCKER_CONFIG="$repo_root/evals/.build/docker-config"
fi
local colima_socket="$HOME/.colima/default/docker.sock"
if [ -S "$colima_socket" ] && [ -z "${DOCKER_HOST:-}" ]; then
export DOCKER_HOST="unix://$colima_socket"
fi
docker info >/dev/null
}
check_harbor_cli() {
if command -v harbor >/dev/null 2>&1; then
harbor --help >/dev/null
return 0
fi
if command -v uvx >/dev/null 2>&1; then
uvx harbor --help >/dev/null
return 0
fi
require_command uv
uv tool run harbor --help >/dev/null
}
check_step "zig" require_command zig
check_step "python3" require_command python3
check_step "Docker daemon" check_docker_daemon
check_step "uv/Harbor" check_harbor_cli
check_step "fx credentials" check_fx_credentials
check_step "job YAML and Harbor JobConfig shape" ./evals/scripts/check-compare-ready.sh --config-only
check_step "publish readiness" ./evals/scripts/check-publish-ready.sh
check_step "RewardKit readiness" ./evals/scripts/check-rewardkit-ready.sh
if [ "$compare" -eq 1 ]; then
check_step "comparison readiness" ./evals/scripts/check-compare-ready.sh
else
echo "comparison readiness skipped; pass --compare to validate comparison credentials"
fi
if [ "$failures" -ne 0 ]; then
echo "release readiness checks failed ($failures failing check(s))" >&2
exit 1
fi
echo "release readiness checks passed"
evals/scripts/check-rewardkit-ready.py
-347
View File
@@ -1,347 +0,0 @@
#!/usr/bin/env python3
from __future__ import annotations
import ast
import importlib
import importlib.util
import os
import subprocess
import sys
import tempfile
from pathlib import Path
from typing import Any
REPO_ROOT = Path(__file__).resolve().parents[2]
EVALS_ROOT = REPO_ROOT / "evals"
DATASETS_ROOT = EVALS_ROOT / "datasets"
REWARDKIT_ROOT = EVALS_ROOT / "rewardkit"
JUDGE_ENV = "FX_REWARDKIT_RUN_JUDGES"
JUDGE_DIR_ENV = "FX_REWARDKIT_JUDGE_DIR"
def rel(path: Path) -> str:
return path.relative_to(REPO_ROOT).as_posix()
def fail(errors: list[str], path: Path, message: str) -> None:
errors.append(f"{rel(path)}: {message}")
def strip_comment(line: str) -> str:
in_string = False
escaped = False
out: list[str] = []
for char in line:
if char == "\\" and in_string:
escaped = not escaped
out.append(char)
continue
if char == '"' and not escaped:
in_string = not in_string
if char == "#" and not in_string:
break
out.append(char)
escaped = False
return "".join(out).strip()
def split_top_level(raw: str) -> list[str]:
parts: list[str] = []
current: list[str] = []
in_string = False
escaped = False
bracket_depth = 0
brace_depth = 0
for char in raw:
if char == "\\" and in_string:
escaped = not escaped
current.append(char)
continue
if char == '"' and not escaped:
in_string = not in_string
elif not in_string:
if char == "[":
bracket_depth += 1
elif char == "]":
bracket_depth -= 1
elif char == "{":
brace_depth += 1
elif char == "}":
brace_depth -= 1
elif char == "," and bracket_depth == 0 and brace_depth == 0:
parts.append("".join(current).strip())
current = []
escaped = False
continue
current.append(char)
escaped = False
if current:
parts.append("".join(current).strip())
return parts
def parse_value(raw: str) -> Any:
value = raw.strip()
if value.startswith('"') and value.endswith('"'):
return value[1:-1]
if value in ("true", "false"):
return value == "true"
if value.startswith("[") and value.endswith("]"):
inner = value[1:-1].strip()
if not inner:
return []
return [parse_value(item) for item in split_top_level(inner)]
if value.startswith("{") and value.endswith("}"):
return value
try:
if "." in value:
return float(value)
return int(value)
except ValueError:
return value
def table_target(data: dict[str, Any], section: str) -> dict[str, Any]:
parts = section.split(".")
if parts[0] in data and isinstance(data[parts[0]], list):
table_list = data[parts[0]]
if not table_list:
raise ValueError(f"{section} appeared before [[{parts[0]}]]")
target = table_list[-1]
if not isinstance(target, dict):
raise ValueError(f"{parts[0]} was not a table list")
parts = parts[1:]
else:
target = data
for part in parts:
child = target.setdefault(part, {})
if not isinstance(child, dict):
raise ValueError(f"{section} conflicted with an existing value")
target = child
return target
def parse_toml(path: Path) -> dict[str, Any]:
data: dict[str, Any] = {}
current: dict[str, Any] = data
for lineno, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
stripped = strip_comment(line)
if not stripped:
continue
if stripped.startswith("[[") and stripped.endswith("]]"):
section = stripped[2:-2].strip()
table_list = data.setdefault(section, [])
if not isinstance(table_list, list):
raise ValueError(f"{path}:{lineno}: [[{section}]] conflicted with an existing value")
current = {}
table_list.append(current)
continue
if stripped.startswith("[") and stripped.endswith("]"):
current = table_target(data, stripped[1:-1].strip())
continue
if "=" not in stripped:
raise ValueError(f"{path}:{lineno}: expected key = value")
key, raw_value = stripped.split("=", 1)
current[key.strip()] = parse_value(raw_value)
return data
def check_toml_parse(errors: list[str]) -> None:
paths = sorted(DATASETS_ROOT.rglob("*.toml")) + sorted(REWARDKIT_ROOT.rglob("*.toml"))
for path in paths:
try:
parse_toml(path)
except Exception as exc:
fail(errors, path, f"TOML parse failed: {exc}")
if paths:
print(f"parsed {len(paths)} task, dataset, and RewardKit TOML files")
def check_judge_toml(path: Path, errors: list[str]) -> None:
try:
data = parse_toml(path)
except Exception as exc:
fail(errors, path, f"TOML parse failed: {exc}")
return
judge = data.get("judge")
if not isinstance(judge, dict):
fail(errors, path, "missing [judge] section")
return
if not isinstance(judge.get("judge"), str) or not judge["judge"]:
fail(errors, path, "judge.judge must be a non-empty string")
files = judge.get("files")
if files is not None and not (
isinstance(files, list) and all(isinstance(item, str) for item in files)
):
fail(errors, path, "judge.files must be a list of strings when present")
trajectory = judge.get("atif-trajectory")
if trajectory is not None and not isinstance(trajectory, str):
fail(errors, path, "judge.atif-trajectory must be a string when present")
criteria = data.get("criterion")
if not isinstance(criteria, list) or not criteria:
fail(errors, path, "must contain at least one [[criterion]] entry")
return
for index, criterion in enumerate(criteria, 1):
if not isinstance(criterion, dict):
fail(errors, path, f"criterion {index} must be a table")
continue
if not isinstance(criterion.get("description"), str) or not criterion["description"]:
fail(errors, path, f"criterion {index} must have a description")
criterion_type = criterion.get("type")
if criterion_type not in ("binary", "likert", "numeric"):
fail(errors, path, f"criterion {index} has unsupported type {criterion_type!r}")
if criterion_type == "likert":
points = criterion.get("points")
if not isinstance(points, int) or points < 2:
fail(errors, path, f"criterion {index} likert points must be an integer >= 2")
if criterion_type == "numeric":
min_value = criterion.get("min")
max_value = criterion.get("max")
if not isinstance(min_value, (int, float)) or not isinstance(max_value, (int, float)):
fail(errors, path, f"criterion {index} numeric criteria need min and max")
elif min_value >= max_value:
fail(errors, path, f"criterion {index} numeric min must be less than max")
scoring = data.get("scoring")
if scoring is not None:
if not isinstance(scoring, dict):
fail(errors, path, "[scoring] must be a table")
return
aggregation = scoring.get("aggregation")
if aggregation not in ("weighted_mean", "all_pass", "any_pass", "threshold"):
fail(errors, path, f"unsupported scoring aggregation {aggregation!r}")
if aggregation == "threshold":
threshold = scoring.get("threshold")
if not isinstance(threshold, (int, float)) or not 0 <= threshold <= 1:
fail(errors, path, "threshold scoring requires threshold between 0 and 1")
def check_builtin_example(errors: list[str]) -> None:
path = REWARDKIT_ROOT / "built_in_file_command_json.py"
try:
tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
except Exception as exc:
fail(errors, path, f"Python parse failed: {exc}")
return
calls: set[str] = set()
for node in ast.walk(tree):
if not isinstance(node, ast.Call):
continue
func = node.func
if isinstance(func, ast.Attribute):
calls.add(func.attr)
elif isinstance(func, ast.Name):
calls.add(func.id)
required = {
"file_exists",
"file_contains",
"command_succeeds",
"command_output_contains",
"json_key_equals",
"json_path_equals",
}
missing = sorted(required - calls)
if missing:
fail(errors, path, "missing built-in criteria calls: " + ", ".join(missing))
def rewardkit_namespace() -> Any | None:
if importlib.util.find_spec("rewardkit") is None:
return None
rewardkit = importlib.import_module("rewardkit")
if hasattr(rewardkit, "file_exists"):
return rewardkit
if hasattr(rewardkit, "criteria"):
return rewardkit.criteria
try:
return importlib.import_module("rewardkit.criteria")
except ModuleNotFoundError:
return rewardkit
def run_builtin_smoke(errors: list[str]) -> None:
try:
namespace = rewardkit_namespace()
except Exception as exc:
errors.append(f"rewardkit deterministic built-in smoke failed during import: {exc}")
return
if namespace is None:
print("rewardkit deterministic built-in smoke skipped: rewardkit is not importable")
return
required = ("file_exists", "command_succeeds", "json_key_equals")
missing = [name for name in required if not hasattr(namespace, name)]
if missing:
errors.append("rewardkit import smoke: missing built-in criteria " + ", ".join(missing))
return
passed = True
with tempfile.TemporaryDirectory(prefix="fx-rewardkit-") as tmp:
workspace = Path(tmp)
(workspace / "summary.json").write_text(
'{"status": "ok", "checks": {"build": "pass"}}\n',
encoding="utf-8",
)
old_cwd = Path.cwd()
try:
os.chdir(str(workspace))
namespace.file_exists("summary.json")
namespace.command_succeeds("python3 -m json.tool summary.json", timeout=10)
namespace.json_key_equals("summary.json", "status", "ok")
except Exception as exc:
passed = False
errors.append(f"rewardkit deterministic built-in smoke failed: {exc}")
finally:
os.chdir(str(old_cwd))
if passed:
print("rewardkit deterministic built-in smoke passed")
def maybe_run_judges(errors: list[str]) -> None:
if os.environ.get(JUDGE_ENV) != "1":
print(f"judge execution skipped: set {JUDGE_ENV}=1 only for explicit paid eval workflows")
return
judge_dir = os.environ.get(JUDGE_DIR_ENV)
if not judge_dir:
print(f"{JUDGE_ENV}=1 set, but {JUDGE_DIR_ENV} is unset; no judge example executed")
return
if importlib.util.find_spec("rewardkit") is None:
errors.append(f"{JUDGE_ENV}=1 requested judge execution, but rewardkit is not importable")
return
subprocess.run([sys.executable, "-m", "rewardkit", judge_dir], cwd=REPO_ROOT, check=True)
print(f"executed RewardKit judge directory: {judge_dir}")
def main() -> int:
errors: list[str] = []
check_toml_parse(errors)
for path in sorted(REWARDKIT_ROOT.glob("*.toml")):
check_judge_toml(path, errors)
check_builtin_example(errors)
run_builtin_smoke(errors)
maybe_run_judges(errors)
if errors:
for error in errors:
print(error, file=sys.stderr)
return 1
print("RewardKit readiness checks passed")
return 0
if __name__ == "__main__":
raise SystemExit(main())
evals/scripts/check-rewardkit-ready.sh
-15
View File
@@ -1,15 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
repo_root="$(git rev-parse --show-toplevel 2>/dev/null || true)"
if [ -z "$repo_root" ]; then
echo "run this script from the fx repository" >&2
exit 1
fi
cd "$repo_root"
mkdir -p evals/.build/pycache
export PYTHONPYCACHEPREFIX="$repo_root/evals/.build/pycache"
python3 evals/scripts/check-rewardkit-ready.py
evals/scripts/run-release-evals.sh
-118
View File
@@ -1,118 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
compare=0
multistep=0
for arg in "$@"; do
case "$arg" in
--compare)
compare=1
;;
--multistep)
multistep=1
;;
*)
echo "unknown argument: $arg" >&2
exit 1
;;
esac
done
if [ "$compare" -eq 1 ] && [ "$multistep" -eq 1 ]; then
echo "--compare and --multistep cannot be combined" >&2
exit 1
fi
repo_root="$(git rev-parse --show-toplevel 2>/dev/null || true)"
if [ -z "$repo_root" ]; then
echo "run this script from the fx repository" >&2
exit 1
fi
cd "$repo_root"
mkdir -p \
evals/.build/pycache \
evals/.build/uv-cache \
evals/.build/uv-tools
export PYTHONPYCACHEPREFIX="$repo_root/evals/.build/pycache"
export UV_CACHE_DIR="$repo_root/evals/.build/uv-cache"
export UV_TOOL_DIR="$repo_root/evals/.build/uv-tools"
require_command() {
if ! command -v "$1" >/dev/null 2>&1; then
echo "$1 is required" >&2
exit 1
fi
}
require_command zig
require_command docker
require_command uv
require_command python3
docker_plugin_dir="/opt/homebrew/lib/docker/cli-plugins"
if [ -d "$docker_plugin_dir" ]; then
mkdir -p evals/.build/docker-config
cat > evals/.build/docker-config/config.json <<JSON
{
"cliPluginsExtraDirs": [
"$docker_plugin_dir"
]
}
JSON
export DOCKER_CONFIG="$repo_root/evals/.build/docker-config"
fi
colima_socket="$HOME/.colima/default/docker.sock"
if [ -S "$colima_socket" ]; then
export DOCKER_HOST="unix://$colima_socket"
fi
if [ -z "${AI_GATEWAY_API_KEY:-}" ] && [ -z "${VERCEL_OIDC_TOKEN:-}" ]; then
echo "AI_GATEWAY_API_KEY or VERCEL_OIDC_TOKEN is required for fx" >&2
exit 1
fi
export AI_GATEWAY_API_KEY="${AI_GATEWAY_API_KEY:-}"
export VERCEL_OIDC_TOKEN="${VERCEL_OIDC_TOKEN:-}"
if [ "$compare" -eq 1 ]; then
./evals/scripts/check-compare-ready.sh
export ANTHROPIC_BASE_URL="https://ai-gateway.vercel.sh"
export ANTHROPIC_AUTH_TOKEN="$AI_GATEWAY_API_KEY"
export ANTHROPIC_API_KEY=""
export OPENAI_API_KEY=""
export OPENAI_BASE_URL=""
fi
if command -v harbor >/dev/null 2>&1; then
harbor_cmd=(harbor)
elif command -v uvx >/dev/null 2>&1; then
harbor_cmd=(uvx harbor)
else
harbor_cmd=(uv tool run harbor)
fi
./evals/scripts/build-image.sh
mkdir -p evals/jobs-out
export PYTHONPATH="$repo_root${PYTHONPATH:+:$PYTHONPATH}"
if [ "$compare" -eq 1 ]; then
job_config="evals/jobs/compare-key-agents.yaml"
elif [ "$multistep" -eq 1 ]; then
job_config="evals/jobs/fx-multistep.yaml"
else
job_config="evals/jobs/fx-release.yaml"
fi
"${harbor_cmd[@]}" run -c "$job_config"
python3 evals/scripts/summarize-results.py evals/jobs-out
echo
echo "full Harbor viewer:"
echo "harbor view evals/jobs-out"
evals/scripts/run-release-signoff.sh
-161
View File
@@ -1,161 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
compare=0
multistep=0
full_terminal_bench=0
view=0
for arg in "$@"; do
case "$arg" in
--compare)
compare=1
;;
--multistep)
multistep=1
;;
--full-terminal-bench)
full_terminal_bench=1
;;
--view)
view=1
;;
-h|--help)
cat <<'USAGE'
usage: ./evals/scripts/run-release-signoff.sh [--multistep] [--compare] [--full-terminal-bench] [--view]
Default signoff:
1. local fx release evals
2. Terminal-Bench smoke
3. Markdown report under evals/runs/
Flags:
--multistep also run the local Harbor multi-step smoke
--compare also run local and Terminal-Bench comparison smoke jobs
--full-terminal-bench run full Terminal-Bench instead of smoke
--view open Harbor viewer after the report is written
USAGE
exit 0
;;
*)
echo "unknown argument: $arg" >&2
exit 1
;;
esac
done
repo_root="$(git rev-parse --show-toplevel 2>/dev/null || true)"
if [ -z "$repo_root" ]; then
echo "run this script from the fx repository" >&2
exit 1
fi
cd "$repo_root"
mkdir -p \
evals/.build/pycache \
evals/.build/uv-cache \
evals/.build/uv-tools
export PYTHONPYCACHEPREFIX="$repo_root/evals/.build/pycache"
export UV_CACHE_DIR="$repo_root/evals/.build/uv-cache"
export UV_TOOL_DIR="$repo_root/evals/.build/uv-tools"
if command -v harbor >/dev/null 2>&1; then
harbor_cmd=(harbor)
elif command -v uvx >/dev/null 2>&1; then
harbor_cmd=(uvx harbor)
else
harbor_cmd=(uv tool run harbor)
fi
latest_job_dir() {
find evals/jobs-out -mindepth 1 -maxdepth 1 -type d -print 2>/dev/null \
| sort \
| tail -n 1
}
run_and_record() {
local label="$1"
shift
echo
echo "==> $label"
"$@"
local job_dir
job_dir="$(latest_job_dir)"
if [ -z "$job_dir" ]; then
echo "no Harbor job directory found after $label" >&2
exit 1
fi
job_dirs+=("$job_dir")
}
print_warning() {
local message="$1"
echo "WARNING: $message" >&2
}
if [ "$full_terminal_bench" -eq 1 ]; then
print_warning "full Terminal-Bench is slow and can cost money. The explicit --full-terminal-bench flag is treated as approval for this run."
fi
if [ "$compare" -eq 1 ]; then
print_warning "comparison jobs run multiple agents through Vercel AI Gateway. Secrets are validated but not printed."
if [ "$full_terminal_bench" -eq 1 ]; then
print_warning "full Terminal-Bench comparison is the slowest, costliest path and requires both --compare and --full-terminal-bench."
fi
fi
if [ "$compare" -eq 1 ]; then
./evals/scripts/check-release-ready.sh --compare
else
./evals/scripts/check-release-ready.sh
fi
job_dirs=()
run_and_record "local fx release evals" ./evals/scripts/run-release-evals.sh
if [ "$multistep" -eq 1 ]; then
run_and_record "local multi-step smoke" ./evals/scripts/run-release-evals.sh --multistep
fi
terminal_args=(--smoke)
terminal_label="Terminal-Bench smoke"
if [ "$full_terminal_bench" -eq 1 ]; then
terminal_args=(--full)
terminal_label="full Terminal-Bench"
fi
run_and_record "$terminal_label" ./evals/scripts/run-terminal-bench.sh "${terminal_args[@]}"
if [ "$compare" -eq 1 ]; then
run_and_record "local comparison evals" ./evals/scripts/run-release-evals.sh --compare
compare_terminal_args=(--smoke --compare)
compare_terminal_label="Terminal-Bench comparison smoke"
if [ "$full_terminal_bench" -eq 1 ]; then
compare_terminal_args=(--full --compare)
compare_terminal_label="full Terminal-Bench comparison"
fi
run_and_record "$compare_terminal_label" ./evals/scripts/run-terminal-bench.sh "${compare_terminal_args[@]}"
fi
echo
report_path="$(
python3 evals/scripts/write-run-report.py \
--title "fx release signoff" \
--slug "release-signoff" \
"${job_dirs[@]}"
)"
echo "release signoff report: $report_path"
echo
printf "full Harbor viewer: "
printf "%q " "${harbor_cmd[@]}"
printf "view %q\n" "evals/jobs-out"
if [ "$view" -eq 1 ]; then
"${harbor_cmd[@]}" view evals/jobs-out
fi
evals/scripts/run-terminal-bench.sh
-158
View File
@@ -1,158 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
mode="smoke"
compare=0
view=0
non_view_args=0
for arg in "$@"; do
case "$arg" in
--smoke)
mode="smoke"
non_view_args=1
;;
--full)
mode="full"
non_view_args=1
;;
--compare)
compare=1
non_view_args=1
;;
--view)
view=1
;;
*)
echo "unknown argument: $arg" >&2
exit 1
;;
esac
done
repo_root="$(git rev-parse --show-toplevel 2>/dev/null || true)"
if [ -z "$repo_root" ]; then
echo "run this script from the fx repository" >&2
exit 1
fi
cd "$repo_root"
require_command() {
if ! command -v "$1" >/dev/null 2>&1; then
echo "$1 is required" >&2
exit 1
fi
}
mkdir -p \
evals/.build/harbor-datasets \
evals/.build/pycache \
evals/.build/uv-cache \
evals/.build/uv-tools
export PYTHONPYCACHEPREFIX="$repo_root/evals/.build/pycache"
export UV_CACHE_DIR="$repo_root/evals/.build/uv-cache"
export UV_TOOL_DIR="$repo_root/evals/.build/uv-tools"
if command -v harbor >/dev/null 2>&1; then
harbor_cmd=(harbor)
elif command -v uvx >/dev/null 2>&1; then
harbor_cmd=(uvx harbor)
else
require_command uv
harbor_cmd=(uv tool run harbor)
fi
print_viewer_command() {
printf "full Harbor viewer: "
printf "%q " "${harbor_cmd[@]}"
printf "view %q\n" "evals/jobs-out"
}
run_job=1
if [ "$view" -eq 1 ] && [ "$non_view_args" -eq 0 ]; then
run_job=0
fi
if [ "$run_job" -eq 1 ]; then
if [ "$mode" = "full" ]; then
echo "WARNING: full Terminal-Bench is slow and can cost money. The explicit --full flag is treated as approval for this run." >&2
if [ "$compare" -eq 1 ]; then
echo "WARNING: full Terminal-Bench comparison runs multiple agents and is the slowest, costliest path." >&2
fi
fi
require_command zig
require_command docker
require_command python3
docker_plugin_dir="/opt/homebrew/lib/docker/cli-plugins"
if [ -d "$docker_plugin_dir" ]; then
mkdir -p evals/.build/docker-config
cat > evals/.build/docker-config/config.json <<JSON
{
"cliPluginsExtraDirs": [
"$docker_plugin_dir"
]
}
JSON
export DOCKER_CONFIG="$repo_root/evals/.build/docker-config"
fi
colima_socket="$HOME/.colima/default/docker.sock"
if [ -S "$colima_socket" ]; then
export DOCKER_HOST="unix://$colima_socket"
fi
if ! docker info >/dev/null 2>&1; then
echo "Docker daemon is not available; start Docker Desktop or Colima" >&2
exit 1
fi
if [ -z "${AI_GATEWAY_API_KEY:-}" ] && [ -z "${VERCEL_OIDC_TOKEN:-}" ]; then
echo "AI_GATEWAY_API_KEY or VERCEL_OIDC_TOKEN is required for fx" >&2
exit 1
fi
export AI_GATEWAY_API_KEY="${AI_GATEWAY_API_KEY:-}"
export VERCEL_OIDC_TOKEN="${VERCEL_OIDC_TOKEN:-}"
if [ "$compare" -eq 1 ]; then
./evals/scripts/check-compare-ready.sh
export ANTHROPIC_BASE_URL="https://ai-gateway.vercel.sh"
export ANTHROPIC_AUTH_TOKEN="$AI_GATEWAY_API_KEY"
export ANTHROPIC_API_KEY=""
export OPENAI_API_KEY=""
export OPENAI_BASE_URL=""
fi
./evals/scripts/build-image.sh --binary-only
mkdir -p evals/jobs-out
export PYTHONPATH="$repo_root${PYTHONPATH:+:$PYTHONPATH}"
if [ "$compare" -eq 1 ] && [ "$mode" = "full" ]; then
job_config="evals/jobs/compare-terminal-bench.yaml"
elif [ "$compare" -eq 1 ]; then
job_config="evals/jobs/compare-terminal-bench-smoke.yaml"
elif [ "$mode" = "full" ]; then
job_config="evals/jobs/fx-terminal-bench.yaml"
else
job_config="evals/jobs/fx-terminal-bench-smoke.yaml"
fi
"${harbor_cmd[@]}" run -c "$job_config"
python3 evals/scripts/summarize-results.py evals/jobs-out
echo
print_viewer_command
fi
if [ "$view" -eq 1 ]; then
if [ "$run_job" -eq 0 ]; then
print_viewer_command
fi
"${harbor_cmd[@]}" view evals/jobs-out
fi
evals/scripts/summarize-results.py
-555
View File
@@ -1,555 +0,0 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import json
import re
from dataclasses import dataclass
from pathlib import Path
from typing import Any
AGENT_IMPORT_NAMES = {
"evals.agents.fx_agent:FxAgent": "fx",
"evals.agents.codex_gateway:CodexGatewayAgent": "codex",
"evals.agents.claude_gateway:ClaudeCodeGatewayAgent": "claude-code",
"evals.agents.opencode_gateway:OpenCodeGatewayAgent": "opencode",
}
@dataclass
class Trial:
task: str
agent: str
model: str | None
reward: float | None
source: str | None
exception: str | None
artifact_manifest_found: bool
artifact_failures: tuple[str, ...]
@property
def reward_value(self) -> float:
return self.reward if self.reward is not None else 0.0
@property
def passed(self) -> bool:
return self.exception is None and self.reward_value >= 1.0
def load_json(path: Path) -> Any:
try:
return json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
return None
def find_latest_job(path: Path) -> Path:
if (path / "result.json").exists() or (path / "config.json").exists():
return path
candidates = [
child
for child in path.iterdir()
if child.is_dir()
and ((child / "result.json").exists() or (child / "config.json").exists())
]
if not candidates:
raise SystemExit(f"no Harbor job found under {path}")
return max(candidates, key=lambda item: item.stat().st_mtime)
def find_key(value: Any, names: set[str]) -> Any:
if isinstance(value, dict):
for key, item in value.items():
if key in names:
return item
for item in value.values():
found = find_key(item, names)
if found is not None:
return found
elif isinstance(value, list):
for item in value:
found = find_key(item, names)
if found is not None:
return found
return None
def name_from_value(value: Any) -> str | None:
if isinstance(value, str):
return value.split("/")[-1]
if isinstance(value, dict):
for key in ("name", "task_name", "path"):
item = value.get(key)
if isinstance(item, str):
return item.split("/")[-1]
return None
def string_from_value(value: Any) -> str | None:
if isinstance(value, str):
return value
if isinstance(value, dict):
for key in ("source", "name", "path"):
item = value.get(key)
if isinstance(item, str):
return item
return None
def agent_info_name(result: Any) -> str | None:
if not isinstance(result, dict):
return None
agent_info = result.get("agent_info")
if not isinstance(agent_info, dict):
return None
name = agent_info.get("name")
return name if isinstance(name, str) and name else None
def agent_info_model(result: Any) -> str | None:
if not isinstance(result, dict):
return None
agent_info = result.get("agent_info")
if not isinstance(agent_info, dict):
return None
model_info = agent_info.get("model_info")
if not isinstance(model_info, dict):
return None
name = model_info.get("name")
provider = model_info.get("provider")
if not isinstance(name, str) or not name:
return None
if isinstance(provider, str) and provider:
return f"{provider}/{name}"
return name
def config_agent_name(config: Any) -> str | None:
if not isinstance(config, dict):
return None
agent = config.get("agent")
if not isinstance(agent, dict):
return None
import_path = agent.get("import_path")
if isinstance(import_path, str) and import_path in AGENT_IMPORT_NAMES:
return AGENT_IMPORT_NAMES[import_path]
name = agent.get("name")
return name if isinstance(name, str) and name else None
def read_reward(trial_dir: Path, result: Any) -> float | None:
reward_txt = trial_dir / "verifier" / "reward.txt"
if reward_txt.exists():
try:
return float(reward_txt.read_text(encoding="utf-8").strip())
except ValueError:
return 0.0
reward_json = trial_dir / "verifier" / "reward.json"
if reward_json.exists():
value = load_json(reward_json)
if isinstance(value, dict) and value:
try:
return max(float(item) for item in value.values())
except (TypeError, ValueError):
return 0.0
found = find_key(result, {"reward", "rewards"})
if isinstance(found, (int, float)):
return float(found)
if isinstance(found, dict) and found:
try:
return max(float(item) for item in found.values())
except (TypeError, ValueError):
return 0.0
return None
def read_exception(trial_dir: Path, result: Any) -> str | None:
exception_txt = trial_dir / "exception.txt"
if exception_txt.exists():
text = exception_txt.read_text(encoding="utf-8", errors="replace").strip()
if text:
lines = [line.strip() for line in text.splitlines() if line.strip()]
for line in reversed(lines):
if line.startswith(("Traceback ", "File ", "...<")):
continue
if re.match(
r"^([A-Za-z_][\w.]*\.)*[A-Za-z_]\w*(Error|Exception)(:|$)",
line,
):
return line[:240]
return lines[0][:240]
exception_info = find_key(result, {"exception_info"})
if isinstance(exception_info, dict):
message = exception_info.get("message") or exception_info.get("error")
exc_type = exception_info.get("type") or exception_info.get("exception_type")
if exc_type and message:
return f"{exc_type}: {message}"[:240]
if message:
return str(message)[:240]
if exc_type:
return str(exc_type)[:240]
if isinstance(exception_info, str) and exception_info.strip():
return exception_info.strip()[:240]
return None
def read_artifact_status(trial_dir: Path) -> tuple[bool, tuple[str, ...]]:
manifest_paths = [trial_dir / "artifacts" / "manifest.json"]
steps_dir = trial_dir / "steps"
if steps_dir.exists():
manifest_paths.extend(
sorted(steps_dir.glob("*/artifacts/manifest.json"))
)
failures: list[str] = []
found = False
for manifest_path in manifest_paths:
if not manifest_path.exists():
continue
found = True
label = manifest_path.relative_to(trial_dir).as_posix()
manifest = load_json(manifest_path)
if not isinstance(manifest, list):
failures.append(f"{label}: manifest is not a JSON array")
continue
for entry in manifest:
if not isinstance(entry, dict):
failures.append(f"{label}: manifest entry is not an object")
continue
status = str(entry.get("status") or "unknown")
if status.lower() == "ok":
continue
source = str(entry.get("source") or "unknown")
destination = str(entry.get("destination") or "unknown")
detail = f"{label}: {source} -> {destination}: {status}"
message = entry.get("error") or entry.get("message")
if message:
detail = f"{detail} ({message})"
failures.append(detail[:240])
return found, tuple(failures)
def read_trials(job_dir: Path) -> list[Trial]:
trials: list[Trial] = []
for child in sorted(job_dir.iterdir()):
if not child.is_dir():
continue
result = load_json(child / "result.json")
config = load_json(child / "config.json")
if result is None and config is None and not (child / "exception.txt").exists():
continue
reward = read_reward(child, result)
task = (
name_from_value(find_key(config, {"task_name", "task"}))
or name_from_value(find_key(result, {"task_name", "task"}))
or child.name
)
agent = (
agent_info_name(result)
or config_agent_name(config)
or name_from_value(find_key(config, {"agent_name", "agent"}))
or "unknown"
)
model = agent_info_model(result) or string_from_value(
find_key(config, {"model_name"})
)
source = (
string_from_value(find_key(config, {"source"}))
or string_from_value(find_key(result, {"source"}))
)
artifact_manifest_found, artifact_failures = read_artifact_status(child)
trials.append(
Trial(
task=task,
agent=agent,
model=model,
reward=reward,
source=source,
exception=read_exception(child, result),
artifact_manifest_found=artifact_manifest_found,
artifact_failures=artifact_failures,
)
)
return trials
def primary_agent(trials: list[Trial]) -> str:
agents = sorted({trial.agent for trial in trials})
if "fx" in agents:
return "fx"
return agents[0] if agents else "fx"
def dataset_label(value: Any) -> str | None:
if not isinstance(value, dict):
return None
name = value.get("name")
if isinstance(name, str) and name:
return name
path = value.get("path")
if isinstance(path, str) and path:
return Path(path).name
return None
def unique_labels(values: list[str]) -> list[str]:
seen: set[str] = set()
labels: list[str] = []
for value in values:
if value and value not in seen:
labels.append(value)
seen.add(value)
return labels
def infer_dataset(job_config: Any, trials: list[Trial]) -> str:
labels: list[str] = []
if isinstance(job_config, dict):
datasets = job_config.get("datasets")
if isinstance(datasets, list):
labels.extend(label for item in datasets if (label := dataset_label(item)))
labels.extend(trial.source for trial in trials if trial.source)
labels = unique_labels(labels)
if labels:
return ", ".join(labels)
return "unknown"
def group_stats(trials: list[Trial]) -> tuple[int, int, float, float, int, int]:
total = len(trials)
passed = sum(1 for trial in trials if trial.passed)
reward_sum = sum(trial.reward_value for trial in trials)
mean_reward = reward_sum / total if total else 0.0
pass_rate = passed / total if total else 0.0
exceptions = sum(1 for trial in trials if trial.exception is not None)
artifact_failures = sum(len(trial.artifact_failures) for trial in trials)
return total, passed, pass_rate, mean_reward, exceptions, artifact_failures
def job_error_count(job_result: Any) -> int | None:
if not isinstance(job_result, dict):
return None
stats = job_result.get("stats")
if not isinstance(stats, dict):
return None
value = stats.get("n_errored_trials")
return value if isinstance(value, int) else None
def job_total_count(job_result: Any) -> int | None:
if not isinstance(job_result, dict):
return None
value = job_result.get("n_total_trials")
return value if isinstance(value, int) else None
MetricMap = dict[str, Any]
METRIC_ORDER = ("mean_reward", "pass_rate", "n_trials", "n_passed", "mean")
def job_metrics_by_eval(job_result: Any) -> dict[str, MetricMap]:
if not isinstance(job_result, dict):
return {}
stats = job_result.get("stats")
if not isinstance(stats, dict):
return {}
evals = stats.get("evals")
if not isinstance(evals, dict):
return {}
metrics_by_eval: dict[str, MetricMap] = {}
for eval_name, eval_result in evals.items():
if not isinstance(eval_name, str) or not isinstance(eval_result, dict):
continue
metric_items = eval_result.get("metrics")
if not isinstance(metric_items, list):
continue
merged: MetricMap = {}
for item in metric_items:
if not isinstance(item, dict):
continue
for key, value in item.items():
if isinstance(key, str) and isinstance(value, (int, float)):
merged[key] = value
if merged:
metrics_by_eval[eval_name] = merged
return metrics_by_eval
def metrics_for_agent(metrics_by_eval: dict[str, MetricMap], agent: str) -> MetricMap | None:
matches = [
metrics
for eval_name, metrics in metrics_by_eval.items()
if eval_name.split("__", 1)[0] == agent
]
if len(matches) == 1:
return matches[0]
if not matches and len(metrics_by_eval) == 1:
return next(iter(metrics_by_eval.values()))
return None
def metric_sort_key(key: str) -> tuple[int, str]:
try:
return (METRIC_ORDER.index(key), key)
except ValueError:
return (len(METRIC_ORDER), key)
def format_metric_value(key: str, value: int | float) -> str:
if key.startswith("n_") and float(value).is_integer():
return str(int(value))
return f"{float(value):.2f}"
def format_metrics(metrics: MetricMap) -> str:
return " ".join(
f"{key}={format_metric_value(key, metrics[key])}"
for key in sorted(metrics, key=metric_sort_key)
)
def print_summary(job_dir: Path, trials: list[Trial]) -> None:
if not trials:
raise SystemExit(f"no Harbor trials found under {job_dir}")
job_config = load_json(job_dir / "config.json")
job_result = load_json(job_dir / "result.json")
dataset = infer_dataset(job_config, trials)
agent = primary_agent(trials)
primary = [trial for trial in trials if trial.agent == agent]
(
total,
passed,
pass_rate,
mean_reward,
exceptions_count,
artifact_failure_count,
) = group_stats(primary)
job_errors = job_error_count(job_result)
job_total = job_total_count(job_result)
error_count = job_errors if job_errors is not None else exceptions_count
metrics_by_eval = job_metrics_by_eval(job_result)
primary_metrics = metrics_for_agent(metrics_by_eval, agent)
title = "fx release evals" if dataset == "fx-release" else "Harbor evals"
print(title)
print()
print(f"dataset: {dataset}")
print(f"agent: {agent}")
if (
job_total is not None
and job_total != total
and len({trial.agent for trial in trials}) == 1
):
print(f"trials: {total}/{job_total}")
else:
print(f"trials: {total}")
print(f"n_trials: {total}")
print(f"n_passed: {passed}")
print(f"pass_rate: {pass_rate:.2f}")
print(f"mean_reward: {mean_reward:.2f}")
print(f"errors: {error_count}")
print(f"exceptions: {exceptions_count}")
print(f"artifact_failures: {artifact_failure_count}")
if primary_metrics:
print(f"harbor_metrics: {format_metrics(primary_metrics)}")
print()
failed = [trial for trial in primary if not trial.passed]
print("failed tasks:")
if failed:
for trial in failed:
reward = trial.reward_value
print(f"- {trial.task} ({reward:.2f})")
else:
print("- none")
exceptions = [trial for trial in primary if trial.exception]
print()
print("errors/exceptions:")
if exceptions:
for trial in exceptions:
print(f"- {trial.task}: {trial.exception}")
else:
print("- none")
artifact_failures = [
(trial, failure)
for trial in trials
for failure in trial.artifact_failures
]
artifact_manifest_count = sum(1 for trial in trials if trial.artifact_manifest_found)
print()
print("artifact collection:")
if artifact_failures:
for trial, failure in artifact_failures:
print(f"- {trial.agent}/{trial.task}: {failure}")
elif artifact_manifest_count:
print("- failures: 0")
else:
print("- no manifests found")
agents = sorted({trial.agent for trial in trials})
if len(agents) > 1:
print()
print("per-agent summary:")
print(
f"{'agent':<14} {'n_trials':>8} {'n_passed':>8} "
f"{'pass_rate':>9} {'mean_reward':>11} "
f"{'exceptions':>10} {'artifact_failures':>18}"
)
for name in agents:
group = [trial for trial in trials if trial.agent == name]
(
group_total,
passed,
group_pass_rate,
group_mean_reward,
group_exceptions,
group_artifact_failures,
) = group_stats(group)
print(
f"{name:<14} {group_total:>8} {passed:>8} "
f"{group_pass_rate:>9.2f} {group_mean_reward:>11.2f} "
f"{group_exceptions:>10} {group_artifact_failures:>18}"
)
if metrics_by_eval:
print()
print("harbor metrics by agent:")
for name in agents:
agent_metrics = metrics_for_agent(metrics_by_eval, name)
if agent_metrics:
print(f"- {name}: {format_metrics(agent_metrics)}")
print()
print("full results:")
print(job_dir)
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("path", nargs="?", default="evals/jobs-out")
args = parser.parse_args()
root = Path(args.path)
job_dir = find_latest_job(root)
print_summary(job_dir, read_trials(job_dir))
if __name__ == "__main__":
main()
evals/scripts/write-run-report.py
-387
View File
@@ -1,387 +0,0 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import importlib.util
import re
import shlex
import shutil
import sys
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from typing import Any
REPO_ROOT = Path(__file__).resolve().parents[2]
RUNS_DIR = REPO_ROOT / "evals" / "runs"
RUNS_INDEX = RUNS_DIR / "RUNS.md"
sys.dont_write_bytecode = True
def load_summary_module() -> Any:
path = Path(__file__).with_name("summarize-results.py")
spec = importlib.util.spec_from_file_location("fx_summarize_results", path)
if spec is None or spec.loader is None:
raise RuntimeError(f"could not load {path}")
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)
return module
SUMMARY = load_summary_module()
@dataclass
class AgentStats:
agent: str
model: str
n_trials: int
n_passed: int
pass_rate: float
mean_reward: float
exceptions: int
artifact_failures: int
@dataclass
class JobReport:
path: Path
dataset: str
agents: list[AgentStats]
exceptions: list[str]
artifact_failures: list[str]
started_at: str | None
finished_at: str | None
def rel(path: Path) -> str:
try:
return path.relative_to(REPO_ROOT).as_posix()
except ValueError:
return path.as_posix()
def slugify(value: str) -> str:
slug = re.sub(r"[^a-z0-9]+", "-", value.lower()).strip("-")
return slug or "harbor-run"
def model_for_agent(job_config: Any, agent: str) -> str:
if not isinstance(job_config, dict):
return "unknown"
agents = job_config.get("agents")
if not isinstance(agents, list):
return "unknown"
for item in agents:
if not isinstance(item, dict):
continue
if item.get("name") != agent:
continue
model = item.get("model_name")
return model if isinstance(model, str) and model else "unknown"
return "unknown"
def agent_stats(job_config: Any, trials: list[Any]) -> list[AgentStats]:
stats: list[AgentStats] = []
for agent in sorted({trial.agent for trial in trials}):
group = [trial for trial in trials if trial.agent == agent]
(
n_trials,
n_passed,
pass_rate,
mean_reward,
exceptions,
artifact_failures,
) = SUMMARY.group_stats(group)
stats.append(
AgentStats(
agent=agent,
model=next(
(trial.model for trial in group if getattr(trial, "model", None)),
None,
)
or model_for_agent(job_config, agent),
n_trials=n_trials,
n_passed=n_passed,
pass_rate=pass_rate,
mean_reward=mean_reward,
exceptions=exceptions,
artifact_failures=artifact_failures,
)
)
return stats
def read_job(path: Path) -> JobReport:
job_dir = SUMMARY.find_latest_job(path)
trials = SUMMARY.read_trials(job_dir)
if not trials:
raise SystemExit(f"no Harbor trials found under {job_dir}")
job_config = SUMMARY.load_json(job_dir / "config.json")
job_result = SUMMARY.load_json(job_dir / "result.json")
dataset = SUMMARY.infer_dataset(job_config, trials)
exceptions = [
f"{trial.agent}/{trial.task}: {trial.exception}"
for trial in trials
if trial.exception
]
artifact_failures = [
f"{trial.agent}/{trial.task}: {failure}"
for trial in trials
for failure in trial.artifact_failures
]
started_at = None
finished_at = None
if isinstance(job_result, dict):
started = job_result.get("started_at")
finished = job_result.get("finished_at")
if isinstance(started, str):
started_at = started
if isinstance(finished, str):
finished_at = finished
return JobReport(
path=job_dir,
dataset=dataset,
agents=agent_stats(job_config, trials),
exceptions=exceptions,
artifact_failures=artifact_failures,
started_at=started_at,
finished_at=finished_at,
)
def format_float(value: float) -> str:
return f"{value:.2f}"
def table_row(values: list[str]) -> str:
return "| " + " | ".join(values) + " |"
def harbor_view_command() -> str:
if shutil.which("harbor"):
return "harbor view evals/jobs-out"
if shutil.which("uvx"):
return "uvx harbor view evals/jobs-out"
return shlex.join(["uv", "tool", "run", "harbor", "view", "evals/jobs-out"])
def job_section(job: JobReport) -> list[str]:
lines: list[str] = []
lines.append(f"## {job.dataset}")
lines.append("")
lines.append(f"- Result path: `{rel(job.path)}`")
if job.started_at:
lines.append(f"- Started: `{job.started_at}`")
if job.finished_at:
lines.append(f"- Finished: `{job.finished_at}`")
lines.append("")
lines.append(
table_row(
[
"Agent",
"Model",
"n_trials",
"n_passed",
"pass_rate",
"mean_reward",
"exceptions",
"artifact failures",
]
)
)
lines.append(table_row(["---", "---", "---:", "---:", "---:", "---:", "---:", "---:"]))
for stats in job.agents:
lines.append(
table_row(
[
f"`{stats.agent}`",
f"`{stats.model}`",
str(stats.n_trials),
str(stats.n_passed),
format_float(stats.pass_rate),
format_float(stats.mean_reward),
str(stats.exceptions),
str(stats.artifact_failures),
]
)
)
lines.append("")
lines.append("Exceptions:")
if job.exceptions:
lines.extend(f"- {item}" for item in job.exceptions)
else:
lines.append("- none")
lines.append("")
lines.append("Artifact failures:")
if job.artifact_failures:
lines.extend(f"- {item}" for item in job.artifact_failures)
else:
lines.append("- none")
lines.append("")
return lines
def report_lines(title: str, jobs: list[JobReport]) -> list[str]:
today = datetime.now().strftime("%Y-%m-%d")
lines = [f"# {today} {title}", ""]
lines.append("Viewer command:")
lines.append("")
lines.append("```bash")
lines.append(harbor_view_command())
lines.append("```")
lines.append("")
lines.append("## Summary")
lines.append("")
lines.append(
table_row(
[
"Dataset",
"Agent",
"Model",
"n_trials",
"n_passed",
"pass_rate",
"mean_reward",
"exceptions",
"artifact failures",
"Result path",
]
)
)
lines.append(
table_row(["---", "---", "---", "---:", "---:", "---:", "---:", "---:", "---:", "---"])
)
for job in jobs:
for stats in job.agents:
lines.append(
table_row(
[
f"`{job.dataset}`",
f"`{stats.agent}`",
f"`{stats.model}`",
str(stats.n_trials),
str(stats.n_passed),
format_float(stats.pass_rate),
format_float(stats.mean_reward),
str(stats.exceptions),
str(stats.artifact_failures),
f"`{rel(job.path)}`",
]
)
)
lines.append("")
for job in jobs:
lines.extend(job_section(job))
return lines
def report_filename(title: str, slug: str | None) -> str:
date = datetime.now().strftime("%Y-%m-%d")
base = slugify(slug or title)
candidate = RUNS_DIR / f"{date}-{base}.md"
if not candidate.exists():
return candidate.name
for index in range(2, 100):
name = f"{date}-{base}-{index}.md"
if not (RUNS_DIR / name).exists():
return name
raise SystemExit("could not choose a report filename")
def index_row(report_name: str, title: str, jobs: list[JobReport]) -> str:
date = datetime.now().strftime("%Y-%m-%d")
if len(jobs) == 1 and len(jobs[0].agents) == 1:
job = jobs[0]
stats = job.agents[0]
notes = (
f"{stats.n_trials} trials, {stats.exceptions} exceptions, "
f"{stats.artifact_failures} artifact collection failures."
)
return table_row(
[
date,
f"[{title}]({report_name})",
f"`{stats.agent}`",
f"`{stats.model}`",
f"`{format_float(stats.pass_rate)}`",
f"`{format_float(stats.mean_reward)}`",
notes,
]
)
notes = ", ".join(
f"{job.dataset}: "
+ "/".join(f"{stats.agent} {format_float(stats.pass_rate)}" for stats in job.agents)
for job in jobs
)
return table_row(
[
date,
f"[{title}]({report_name})",
"`varies`",
"`varies`",
"`see report`",
"`see report`",
notes,
]
)
def update_index(report_name: str, title: str, jobs: list[JobReport]) -> None:
if not RUNS_INDEX.exists():
return
text = RUNS_INDEX.read_text(encoding="utf-8")
row = index_row(report_name, title, jobs)
if f"]({report_name})" in text:
return
marker = "| Date | Run | Agent | Model | Pass Rate | Mean Reward | Notes |\n"
index = text.find(marker)
if index == -1:
return
header_end = text.find("\n", index + len(marker))
if header_end == -1:
return
insert_at = header_end + 1
updated = text[:insert_at] + row + "\n" + text[insert_at:]
RUNS_INDEX.write_text(updated, encoding="utf-8")
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("paths", nargs="+", help="Harbor job output directories")
parser.add_argument("--title", default="Harbor eval run")
parser.add_argument("--slug")
parser.add_argument(
"--no-index",
action="store_true",
help="Write the report without updating evals/runs/RUNS.md.",
)
args = parser.parse_args()
RUNS_DIR.mkdir(parents=True, exist_ok=True)
jobs = [read_job(Path(path)) for path in args.paths]
report_name = report_filename(args.title, args.slug)
report_path = RUNS_DIR / report_name
report_path.write_text("\n".join(report_lines(args.title, jobs)) + "\n", encoding="utf-8")
if not args.no_index:
update_index(report_name, args.title, jobs)
print(rel(report_path))
if __name__ == "__main__":
main()