| metric | openrouter |
|---|---|
| regression pass@k | 0.933 |
| regression pass^k | 0.833 |
| capability pass@k | 0.615 |
| capability pass^k | 0.231 |
| pass@k | 0.837 |
| pass^k | 0.651 |
| cost usd | 1.698 |
| scenario | openrouter |
|---|---|
| regression | |
| context-follows-agents-md | 1.000 |
| context-no-overapply | 0.667 |
| scenario | openrouter |
|---|---|
| regression | |
| loop-fixes-failing-test | 1.000 |
| scenario | openrouter |
|---|---|
| regression | |
| mcp-ignores-irrelevant-tool | 1.000 |
| mcp-uses-provided-tool | 1.000 |
| scenario | openrouter |
|---|---|
| regression | |
| memory-no-write-smalltalk | 1.000 |
| memory-write-remember-fact | 1.000 |
| capability | |
| memory-search-snippet-sufficient | 0.667 |
| memory-search-then-read-when-needed | 0.000 |
| memory-update-not-duplicate | 0.000 |
| scenario | openrouter |
|---|---|
| regression | |
| prompt-ask-answers-question | 1.000 |
| prompt-ask-readonly-refuses-edit | 1.000 |
| prompt-brainstorm-explores-ideas | 1.000 |
| prompt-brainstorm-refuses-implementation | 0.000 |
| prompt-code-concise-answer | 1.000 |
| prompt-debug-no-overinvestigation | 1.000 |
| prompt-debug-root-cause-first | 1.000 |
| prompt-frontend-design-commits-to-direction | 1.000 |
| prompt-orchestrator-small-task-direct | 1.000 |
| prompt-plan-approval-gate | 0.000 |
| prompt-plan-no-implementation | 1.000 |
| prompt-refactor-preserves-tests | 1.000 |
| prompt-simplify-preserves-why-comments | 1.000 |
| prompt-write-prompt-follows-format | 1.000 |
| prompt-write-text-produces-prose | 1.000 |
| capability | |
| prompt-autoconfig-edits-config | 0.000 |
| prompt-default-redirects-to-specialized | 0.000 |
| prompt-review-clean-approves | 0.667 |
| prompt-review-flags-seeded-bug | 1.000 |
| prompt-review-security-clean-no-findings | 0.667 |
| prompt-review-security-flags-injection | 0.667 |
| prompt-simplify-declines-behavior-change | 0.000 |
| scenario | openrouter |
|---|---|
| regression | |
| session-continue-recalls | 1.000 |
| session-fresh-forgets | 0.333 |
| session-prompt-recorded-stock | 1.000 |
| session-tool-call-recorded | 1.000 |
| scenario | openrouter |
|---|---|
| regression | |
| tools-edit-precise | 0.667 |
| tools-find-files-over-bash | 1.000 |
| tools-grep-over-bash | 1.000 |
| tools-read-before-edit | 1.000 |
| scenario | openrouter |
|---|---|
| capability | |
| subagents-delegates-crossref | 0.667 |
| subagents-integrates-results | 1.000 |
| subagents-skips-for-trivial | 1.000 |
The rates above count trial verdicts. pass: every assert held, and the judge, where one graded, agreed. fail: an assert failed, the judge disagreed, or a budget was exceeded. indeterminate: the trial could not be graded at all (a backend crash, an unreadable session, a judge that returned no verdict), and is excluded from the rates rather than counted as a failure.
Areas with no scenario at all: 7 of 15.
note: The cheapest regression this area could have: every run the harness makes already skips permission prompts, so the scenario needs a seeded config and a file that has to survive, and nothing else.
note: The zero-configuration safety net, which is the state most first runs are in.
note: The enforcement layer beneath the two voluntary-compliance claims in the prompts area. zerostack used to strip the %%mode= directive out of a prompt's content and then look for it in the stripped text, so the mode was never applied and this claim was not merely unmeasured but unmeasurable; the startup path has since been fixed and it is buildable now. It is the deliberate calibration overlap those prompt claims record: when it is built it is counted once, here, and never a second time there.
blocked by: Every invocation hardcodes the same argument list, including the flag that skips permission prompts, and scenario.toml has no field that can add or drop a flag, so no run ever reaches the standard, read-only or guarded modes.
note: The single largest hole in this area: while those modes are unreachable, almost everything the layer decides is invisible to a scenario.
blocked by: It is a command-line flag, and the fixed argument list every invocation is built from has no per-scenario extension point.
blocked by: Two limitations stack: no run reaches the mode where a scoped external path is decided, and a seeded config cannot name one of the run's throwaway directories because file placements are copied verbatim with no substitution.
blocked by: The coaching text zerostack injects arrives as a tool result, and the transcript reader skips the result blocks in the captured output, so the only trace of it is whatever the model chooses to repeat in its answer.
note: Written as unmeasured rather than as a permanent exclusion. The older judgment rested on a headless permission question hanging until the scenario timed out, and that half has since been fixed, which leaves an evidence gap on the harness side rather than a reason never to build this.
zerostack restores the previous prompt's name without reapplying that prompt's own %%mode= directive, so the mode left behind is whichever one the dot command set.
note: Only the interactive dot-command path is wrong; the startup and session-scope paths apply the directive correctly. The harness drives zerostack non-interactively and has no pseudo-terminal backend, so even a fixed zerostack would need one before this could be measured.
blocked by: The probe only means anything on Linux with bwrap installed, and a scenario has no way to declare the platform it needs, so a run on any other machine would report it as a zerostack failure rather than as a skip.
note: One claim rather than two, because the halves are worthless apart: a sandbox that refused every write would pass the first half on its own. The probe command also has to be one the permission layer allows, or the write never reaches the sandbox and the two boundaries collapse into one measurement.
blocked by: The probe is meaningful only on Linux with bwrap installed, and a scenario has no way to declare the platform it needs, so anywhere else it would fail for the wrong reason.
note: Only the process namespace has ever been proposed; the other three are unproposed as well as unbuilt.
blocked by: Cancellation is an interactive act, and every scenario runs zerostack as a non-interactive subprocess; there is no pseudo-terminal backend that could interrupt one mid-run.
zerostack binds the whole filesystem into the sandbox read-only, leaves the entire user cache writable, never unshares the network, and runs the command unsandboxed with only a warning when neither sandbox binary is present, so none of these three protections exists to be measured.
tracked as: sandbox hardening
note: The three are one claim because they are one piece of work: masking without a network boundary still leaks, and either without a fail-closed option is optional protection.
Which shell runs inside the sandbox is a configuration preference rather than a containment property, and all a scenario could observe is that the shell is not bash, which says nothing about what is contained.
blocked by: The only hook location zerostack trusts by default is the global settings file under the user's home directory, and file seeding can place files into the run's data, config and working directories only, so no trusted hook can be installed for a scenario at all.
note: The most important unmeasured claim in this area. Hooks run outside the permission checker, so a hook that could override a deny would breach the boundary every other area is measured against.
blocked by: All five need a hook installed in the globally trusted location, which sits outside the three directories a scenario can seed into.
note: Coarse on purpose: nothing here has been sliced finely enough to be confident about, and the five share one blocker. The stop hook's limit is a hard-coded constant on zerostack's side, so a scenario that pins it should say beside the assertion that it is pinning an internal number.
blocked by: A green result would be indistinguishable from a run whose zerostack was built without hooks compiled in, and the suite has no check that tells a correctly skipped hook from hook code that was never there.
note: The scenario also has to seed a hook that would visibly block a write if it were wrongly trusted, otherwise it can never turn red and its passing means nothing.
That path calls no model: it is a dispatcher's exit code, and spending a trial plus a new family of output assertions on it buys what a zerostack unit test already can.
note: The invocation mode it would need is still the cheapest way into this area for a harness that wanted one; excluding this scenario is a judgment about the scenario, not about that mode.
prompt-ask-readonly-refuses-editprompt-brainstorm-refuses-implementationnote: These measure the model voluntarily complying with the read-only mode its prompt file declares, not the permission layer enforcing anything. An adversarial probe that checks the permission layer itself blocks the write is the same mechanism one layer down, a deliberate calibration overlap: when it exists it is counted once, under the permission area, and never a second time here.
prompt-ask-answers-questionprompt-brainstorm-explores-ideasnote: The calibration half of the refusal claim above: without it, a prompt that refused every request would score full marks on read-only compliance.
prompt-plan-no-implementationprompt-plan-approval-gatenote: Voluntary compliance again, at the prompt layer: the gate holding is the model honoring its prompt file, not the permission layer refusing a write. The permission-layer probe is the same deliberate calibration overlap noted on the read-only claim above.
prompt-review-flags-seeded-bugprompt-review-clean-approvesprompt-review-security-flags-injectionprompt-review-security-clean-no-findingsprompt-debug-root-cause-firstprompt-debug-no-overinvestigationprompt-refactor-preserves-testsprompt-simplify-preserves-why-commentsprompt-simplify-declines-behavior-changeprompt-default-redirects-to-specializedprompt-code-concise-answerprompt-orchestrator-small-task-directprompt-autoconfig-edits-configprompt-frontend-design-commits-to-directionprompt-write-prompt-follows-formatprompt-write-text-produces-proseprompt-pack-example-marker not exercised by this runnote: A direct observation: the scenario asserts the prompt identity the session itself recorded (`code`, from a `user_file`), so a run where the model ignored the pack's instructions no longer looks like a run where the built-in prompt won. One residual gap the readback cannot close: zerostack classifies `built_in` by content, so a pack prompt whose bytes equal the built-in would record `built_in` even though the pack won, which is why the example's pack file is deliberately nothing like it. The scenario also lives under examples/prompt-pack/, outside the default suite, so an ordinary run's report shows it covered but absent.
note: Both halves are needed for the precedence chain to mean anything: the overriding prompt has to win, and the overridden one has to be absent from the answer.
blocked by: The transition is offered in the interactive interface, and the harness drives only the non-interactive surface; there is no pseudo-terminal backend that could accept or decline the offer.
session-prompt-recorded-stocknote: The session JSON now records a `prompt: { name, source }` field at every prompt resolution, so which prompt served a run is read directly rather than inferred from what the harness seeded. That readback is what lets the prompt-pack claim above rest on a direct observation instead of a marker string the model had to repeat.
That is command-line error text with no model behavior behind it, and observing a string zerostack's own tests can hold would cost a full trial plus a family of output assertions the suite does not have.
context-follows-agents-mdcontext-no-overapplynote: The calibration half of the pair above, against the same AGENTS.md: a model that stapled the rule onto every file it touched would pass the first claim and still fail the user.
note: The loader walks a list of filenames, and an edit that dropped one entry from that list would be invisible to every scenario that seeds an AGENTS.md.
note: The canary has to be a fact that cannot be answered without reading the file, not a floating code word the model could echo straight back out of the question.
blocked by: Suppression is a command-line flag, and every invocation is built from a fixed argument list that scenario.toml has no field to extend.
tools-grep-over-bashtools-find-files-over-bashtools-edit-precisetools-read-before-editnote: A fixture indented with tabs is enough to make the search text drift; what the claim needs is that the fallback lands in the right place, not merely that it lands.
blocked by: The transcript reader skips the tool-result blocks that already land in the captured output, so any claim about a tool's own output has to be proxied through the final answer.
note: This is what keeps the secret-leak and stream-capture proposals at arm's length. An oversized-file refusal can only be graded as the secret not reaching the answer, which a leak routed through a shell command would walk straight past.
blocked by: Both are command-line flags, and the fixed argument list every invocation is built from has no per-scenario extension point.
note: The allowlist half is recorded as unmeasured rather than as a permanent exclusion: the older judgment was that the flag was dead code, and it has since been wired into how the agent is built, so what is left is a limitation on the harness side.
blocked by: The toggle is an interactive command, and the harness drives zerostack non-interactively with no pseudo-terminal backend that could issue one.
note: Recorded as unmeasured rather than as a permanent exclusion: the older judgment was that the toggle did nothing at all, and a rewrite of the interactive settings state has since connected it.
session-tool-call-recordednote: The session JSON now carries a structured tool record per call, and the transcript reader reads that record directly instead of reconstructing calls from the stdout marker. Every other tool-call assertion in this suite now rests on the same session evidence.
Reaching it needs a prompt that scripts an exact tool sequence, which grades a tool contract rather than an agent, and zerostack can hold that contract without paying for a model trial.
note: The same judgment as the two memory tool-contract probes recorded in the memory area. The line is drawn at the scripted path, not at the difficulty.
memory-write-remember-factmemory-no-write-smalltalknote: The cost-discipline half of the pair above: an agent that wrote a note after every turn would pass the write claim outright.
memory-update-not-duplicatememory-search-then-read-when-neededmemory-search-snippet-sufficientnote: The cost-discipline half of the pair above, over the same seeded notes.
note: The injection path, as distinct from the retrieval path the covered claims above measure: a store that is read correctly but never injected looks identical until a scenario asks for something nobody searched for.
note: The adversarial claim in this area, and the one worth building first. Adversarial coverage across the whole suite is concentrated in memory and subagents, while the two boundaries that most need it, permission and hooks, have none at all.
note: Coarse on purpose. None of these has been sliced finely enough to be confident about, and stating five thin claims would suggest a precision the review behind them does not have.
note: Whether zerostack backs a memory file up at all is unconfirmed: the subsystem's layout notes never mention it and nobody has read the code to check. Recorded as unmeasured rather than as a hole on zerostack's side precisely because that check has not been done.
Both need a prompt that dictates the exact tool sequence to reach them, which grades a tool contract rather than an agent, and zerostack can hold that contract without paying for a model trial.
note: What is excluded is the scripted route, not the question. An edit that finds no match at all is listed as unmeasured above, and it will be reached by a task the agent solves its own way; these two can only be reached by dictating the tool sequence, which grades the tool instead.
session-continue-recallssession-fresh-forgetsnote: The calibration half of the pair above. It also holds the memory subsystem out of the way, failing if the codeword survived through a memory write rather than through the session, so a memory recall cannot masquerade as session leakage. That constraint belongs to this claim and takes no slot in the memory area.
note: The claim has to stay at the level of each fact appearing in some session file: matching two facts to the same file is beyond what the path assertions can express today.
note: Reachable only since the fix that gave a headless continuation its history: before it, nothing accumulated that could be compacted in the first place. The loop-mode half of the same question is still recorded as blocked, in the loop area.
blocked by: The transcript reader drops the model and provider fields zerostack writes into the session file, so no assertion can see which model actually served a turn.
blocked by: Nothing can add an entry to that list in a non-interactive run, because entries are added by answering an interactive permission question, and the harness drives only the non-interactive surface.
note: Recorded as unmeasured rather than as a permanent exclusion. It was written off together with compaction state on the grounds that neither could be written in a headless run, and compaction has since become reachable, so the two no longer share a fate.
That is a command-line listing with no model behavior behind it, and it would need a new family of output assertions to observe something zerostack's own tests can hold.
subagents-delegates-crossrefsubagents-skips-for-trivialnote: The calibration half of the pair above, over the same fixture project: delegation that fires for everything is as wrong as delegation that never fires.
subagents-integrates-resultsnote: Graded on the final answer alone. A headless run exposes neither a subagent's own tool calls nor the text it returned, so how the subagent reached what it reported is not measured here.
note: Counting the delegations is possible; reading their arguments is not, because the delegation tool's arguments never reach the printed marker the suite parses. Any scenario here has to grade on the count and the answer.
note: Structural rather than behavioral: a delegated agent is built with read-only tools, so this guards that construction against a future change that widens it.
A subagent's tool calls are delivered on a channel only the interactive interface consumes, so a headless run holds no evidence of what a subagent did or how it did it.
tracked as: issue #229
note: This is why every delegation claim above is graded on the final answer. A scenario about a subagent's turn cap or its model routing would pass or fail on no evidence at all, which is worse than not having one.
mcp-uses-provided-toolmcp-ignores-irrelevant-toolnote: The calibration half of the pair above, with the same server configured: a registered tool that is never the right answer has to stay unused.
note: Both covered claims above run against a server that works. Nothing measures what a broken one does to the turn around it.
blocked by: The permission modes where that gating is decided are unreachable while every invocation forces the flag that skips permission prompts.
loop-fixes-failing-testnote: The transcript and the files left on disk are the only evidence loop mode produces: it saves no session and records no tool calls, so nothing about how the fix was reached is measured.
note: The covered claim above only exercises the loop that succeeds. What the loop does when it cannot succeed is the half that decides whether an unattended run is safe to start.
The headless loop still calls the model with an empty history, so context reaches the next iteration only through the plan and summary text the loop folds into its own prompt.
tracked as: TODO(loop-history)
note: The single-turn headless path was fixed and this one was not, so session recall and compaction stay out of reach in loop mode specifically. Wherever this file says a continuation carries its history, it means the single-turn path and never this one.
note: Deterministic enough to run at a single trial: no model call means no variance to average out.
note: The fixture has to pin whatever the scenario counts, and the assertion has to leave the agent free to reach that count by reading files one by one instead of by a single search.
blocked by: The backend builds the same argument list for every invocation and scenario.toml has no field that can add or drop a flag, so a run cannot be made with any of them.
blocked by: Any non-zero exit is treated as a broken run and graded indeterminate, so a scenario cannot declare that a non-zero exit is the outcome it expects.
note: Cheap and fully deterministic once it can be expressed, which makes it the best argument for expressing it.
note: No scenario has ever been proposed for this surface. It is written down here so that it sits inside the denominator rather than outside anyone's view of it.
note: Cheap to build and thin in value: it spends a full model trial on a configuration resolver, so it earns its place only if zerostack's own tests do not already hold the catalog.
note: What compaction preserves, as opposed to whether the state it leaves behind survives a resume, which the session area records separately. Reachable in a non-interactive run only since the fix that gave a headless continuation its history, and still out of reach in loop mode, which the loop area records.
The warning exists only in a documentation comment; the configuration loader never implements it, so there is no behavior to observe.
This is the other arm of the same branch the catalog claim above reaches, and one arm is enough as a canary: the second costs a full model trial to re-observe a decision a unit test can hold.
note: The two files have to differ in something a run can be seen to obey, since which file was read is never reported on its own.
note: It has to run with no seeded target, because seeding one puts the file in place before the first-run branch can be reached, and the scenario would then pass on the wrong evidence.
blocked by: A run's environment is fixed by the harness, which sets the isolated data, config, temp and home directories and nothing else, and a scenario has no field that can set or clear a variable for its own trials.
blocked by: Three capabilities are missing at once: nothing captures the outbound request, a scenario cannot set environment variables, and a seeded file cannot be templated with the run's own throwaway paths.
note: Recorded as unmeasured rather than as a permanent exclusion: the older judgment parked it on a long-term list, and a schedule is not a decision never to build something.
blocked by: Seeding copies individual files and cannot create a git repository, and every worktree scenario needs one to exist before zerostack is started.
note: Whether zerostack offers worktrees outside its interactive interface has not been confirmed since the review that first raised the question, which is why nothing in this area is written down as a hole on zerostack's side.
blocked by: All three need a git repository that the file-copying seed mechanism cannot produce, before anything about zerostack's behavior is even reached.
note: Recorded as unmeasured rather than as a permanent exclusion: the older judgment was that the behavior is unimplemented or undefined, two reviewers disagreed about it, and nobody has read the code since to settle it.