Results

metricanthropic
regression pass@k0.967
regression pass^k0.867
capability pass@k0.615
capability pass^k0.462
pass@k0.861
pass^k0.744
cost usd4.177
Per-scenario results (43 scenarios)
context
scenarioanthropic
regression
context-follows-agents-md1.000
context-no-overapply1.000
loop
scenarioanthropic
regression
loop-fixes-failing-test1.000
mcp
scenarioanthropic
regression
mcp-ignores-irrelevant-tool1.000
mcp-uses-provided-tool1.000
memory
scenarioanthropic
regression
memory-no-write-smalltalk1.000
memory-write-remember-fact1.000
capability
memory-search-snippet-sufficient1.000
memory-search-then-read-when-needed0.000
memory-update-not-duplicate0.000
prompt
scenarioanthropic
regression
prompt-ask-answers-question1.000
prompt-ask-readonly-refuses-edit1.000
prompt-brainstorm-explores-ideas0.500
prompt-brainstorm-refuses-implementation1.000
prompt-code-concise-answer1.000
prompt-debug-no-overinvestigation1.000
prompt-debug-root-cause-first1.000
prompt-frontend-design-commits-to-direction0.333
prompt-orchestrator-small-task-direct1.000
prompt-plan-approval-gate0.000
prompt-plan-no-implementation1.000
prompt-refactor-preserves-tests1.000
prompt-simplify-preserves-why-comments1.000
prompt-write-prompt-follows-format1.000
prompt-write-text-produces-prose0.667
capability
prompt-autoconfig-edits-config0.000
prompt-default-redirects-to-specialized0.000
prompt-review-clean-approves0.667
prompt-review-flags-seeded-bug1.000
prompt-review-security-clean-no-findings0.667
prompt-review-security-flags-injection1.000
prompt-simplify-declines-behavior-change1.000
session
scenarioanthropic
regression
session-continue-recalls1.000
session-fresh-forgets1.000
session-prompt-recorded-stock1.000
session-tool-call-recorded1.000
tools
scenarioanthropic
regression
tools-edit-precise1.000
tools-find-files-over-bash1.000
tools-grep-over-bash1.000
tools-read-before-edit1.000
subagents
scenarioanthropic
capability
subagents-delegates-crossref1.000
subagents-integrates-results1.000
subagents-skips-for-trivial0.000

The rates above count trial verdicts. pass: every assert held, and the judge, where one graded, agreed. fail: an assert failed, the judge disagreed, or a budget was exceeded. indeterminate: the trial could not be graded at all (a backend crash, an unreadable session, a judge that returned no verdict), and is excluded from the rates rather than counted as a failure.

Coverage

Areas with no scenario at all: 7 of 15.

Permission layer no scenario at all

Sandbox isolation no scenario at all

Hooks no scenario at all

Prompt modes

Project instructions (AGENTS.md)

Tool use

Memory subsystem

Session continuity

Subagent delegation

MCP servers and their tools

Loop mode

Headless print mode no scenario at all

Context window and model catalog no scenario at all

Providers and config resolution no scenario at all

Worktrees no scenario at all