Results

metricopenrouter
regression pass@k0.933
regression pass^k0.833
capability pass@k0.615
capability pass^k0.231
pass@k0.837
pass^k0.651
cost usd1.698
Per-scenario results (43 scenarios)
context
scenarioopenrouter
regression
context-follows-agents-md1.000
context-no-overapply0.667
loop
scenarioopenrouter
regression
loop-fixes-failing-test1.000
mcp
scenarioopenrouter
regression
mcp-ignores-irrelevant-tool1.000
mcp-uses-provided-tool1.000
memory
scenarioopenrouter
regression
memory-no-write-smalltalk1.000
memory-write-remember-fact1.000
capability
memory-search-snippet-sufficient0.667
memory-search-then-read-when-needed0.000
memory-update-not-duplicate0.000
prompt
scenarioopenrouter
regression
prompt-ask-answers-question1.000
prompt-ask-readonly-refuses-edit1.000
prompt-brainstorm-explores-ideas1.000
prompt-brainstorm-refuses-implementation0.000
prompt-code-concise-answer1.000
prompt-debug-no-overinvestigation1.000
prompt-debug-root-cause-first1.000
prompt-frontend-design-commits-to-direction1.000
prompt-orchestrator-small-task-direct1.000
prompt-plan-approval-gate0.000
prompt-plan-no-implementation1.000
prompt-refactor-preserves-tests1.000
prompt-simplify-preserves-why-comments1.000
prompt-write-prompt-follows-format1.000
prompt-write-text-produces-prose1.000
capability
prompt-autoconfig-edits-config0.000
prompt-default-redirects-to-specialized0.000
prompt-review-clean-approves0.667
prompt-review-flags-seeded-bug1.000
prompt-review-security-clean-no-findings0.667
prompt-review-security-flags-injection0.667
prompt-simplify-declines-behavior-change0.000
session
scenarioopenrouter
regression
session-continue-recalls1.000
session-fresh-forgets0.333
session-prompt-recorded-stock1.000
session-tool-call-recorded1.000
tools
scenarioopenrouter
regression
tools-edit-precise0.667
tools-find-files-over-bash1.000
tools-grep-over-bash1.000
tools-read-before-edit1.000
subagents
scenarioopenrouter
capability
subagents-delegates-crossref0.667
subagents-integrates-results1.000
subagents-skips-for-trivial1.000

The rates above count trial verdicts. pass: every assert held, and the judge, where one graded, agreed. fail: an assert failed, the judge disagreed, or a budget was exceeded. indeterminate: the trial could not be graded at all (a backend crash, an unreadable session, a judge that returned no verdict), and is excluded from the rates rather than counted as a failure.

Coverage

Areas with no scenario at all: 7 of 15.

Permission layer no scenario at all

Sandbox isolation no scenario at all

Hooks no scenario at all

Prompt modes

Project instructions (AGENTS.md)

Tool use

Memory subsystem

Session continuity

Subagent delegation

MCP servers and their tools

Loop mode

Headless print mode no scenario at all

Context window and model catalog no scenario at all

Providers and config resolution no scenario at all

Worktrees no scenario at all