Files
zed/crates/eval_cli
Ben Brandt 39b5329322 acp_thread: Separate submission receipts from session activity (#64930)
Separate prompt submission from session activity without enabling v2
transport. A receipt acknowledges acceptance; it does not mean the agent
has finished working.

- Select an optional, session-bound receipt capability before sending.
Existing v1/native connections retain completion-based prompt behavior.
- Give submissions and foreground activity separate owners. Submission
records own response tasks, IDs, echo correlation, and recoverable
content; activity owns Running/RequiresAction/Idle state, timing,
generations, and cancellation waiters.
- Keep receipt-driven pending text outside the transcript. Only agent
echoes create canonical user messages, with receipt/echo correlation
working in either order. Pending and failed submissions have recovery
controls; restoring an editable copy keeps the original until explicit
Discard.
- Carry generation, duration, and stop reason in completion events.
Stale events cannot advance a newer activity's queue or display old
errors. Timers also reconcile correctly when Running and Idle arrive
before the UI observes either transition.
- Preserve legacy completion/cancellation semantics while running
completion directly in the record-owned task, removing a redundant
task/channel bridge. Native subagent and eval consumers explicitly
require completion rather than accepting a receipt as a finished run.

## Review guide

This is a cross-layer lifecycle change, not a wire adapter. Suggested
review order:

1. [Submission and activity
ownership](https://github.com/zed-industries/zed/blob/279f0074bae5ec7e85acac363fcadbefb04de6d5/crates/acp_thread/src/submission.rs#L98):
retention, echo correlation, task ownership, and idle waiters.
2. [Core
settlement](https://github.com/zed-industries/zed/blob/279f0074bae5ec7e85acac363fcadbefb04de6d5/crates/acp_thread/src/acp_thread.rs#L3910)
and [legacy turn
completion](https://github.com/zed-industries/zed/blob/279f0074bae5ec7e85acac363fcadbefb04de6d5/crates/acp_thread/src/acp_thread.rs#L5765):
receipt settlement must not finish activity; cancellation still waits on
the transport task.
3. [UI activity
reconciliation](https://github.com/zed-industries/zed/blob/279f0074bae5ec7e85acac363fcadbefb04de6d5/crates/agent_ui/src/conversation_view/thread_view.rs#L1446)
and
[recovery](https://github.com/zed-industries/zed/blob/279f0074bae5ec7e85acac363fcadbefb04de6d5/crates/agent_ui/src/conversation_view/thread_view.rs#L1533):
queue/error ownership, batched updates, and recoverable originals.

The receipt capability defaults to absent. Existing agents still use
optimistic v1 messages and completion-based send/command/retry behavior.
Legacy refusal, pending-action cleanup, and native subagent behavior
retain their compatibility paths.

Delta Thread:
https://delta.dev/trunk/join/thread_01m2swv74pzp4c6xk46sh19fmk#s=MYAgERzq

Release Notes:

- N/A
2026-09-30 08:42:44 +00:00
..
…
2026-09-11 09:13:44 +00:00
…

eval-cli

Headless Rust binary for running Zed's agent in evaluation and benchmark environments. It is designed for containerized harnesses such as Harbor and Pier, where the repository is already checked out and model API keys are provided via environment variables.

eval-cli uses the same NativeAgent + AcpThread pipeline as the production Zed editor: a full agentic loop with tool calls, subagents, and retries, without a GUI.

This directory also contains zed_eval/, the Python zed-eval package used to build this binary, launch remote benchmark runs on Modal/Harbor/Pier, and fetch results. For normal benchmark orchestration, start with zed_eval/README.md.

Building

Native, for local testing on the same OS

cargo build --release -p eval_cli

Linux x86_64, for Harbor/Pier sandboxes

Harbor and Pier containers run Linux x86_64. From the repository root, use the Docker-based build script:

crates/eval_cli/script/build-linux

This produces target/eval-cli, an x86_64 Linux ELF binary. You can also specify a custom output path:

crates/eval_cli/script/build-linux --output ~/bin/eval-cli-linux

Standalone usage

eval-cli \
  --workdir /testbed \
  --model anthropic/claude-sonnet-4-6 \
  --instruction "Fix the bug described in..." \
  --timeout 600 \
  --output-dir /logs/agent

eval-cli reads provider API keys from environment variables such as ANTHROPIC_API_KEY and OPENAI_API_KEY. It writes result.json, thread.md, and thread.json to the output directory.

Exit codes

Code Meaning
0 Agent finished
1 Error, such as model/auth/runtime failure
2 Timeout
3 Interrupted by SIGTERM or SIGINT

Running benchmarks

Most benchmark runs should use the Python zed-eval CLI instead of invoking eval-cli directly. From the repository root:

crates/eval_cli/script/install-zed-eval
zed-eval doctor --create-volume
zed-eval run rf --from local --n-tasks 2

For one-off source runs without installing the tool globally, use crates/eval_cli/script/zed-eval <args>.

See zed_eval/README.md for supported benchmarks, remote builds, Modal setup, reporting, rejudging, baselines, and Harbor/Pier installed agent usage.