Evals
How much deft helps an agent. Each theme holds use cases, each use case holds scenarios with one prompt each, and each scenario has runs: one per agent, without deft and with it.
Claude Opus 5.5, every scenario
| Without deft | With deft, skill installed | |
|---|---|---|
| Solved | 139 of 143 | 143 of 143 |
| Time | 86 min | 40 min |
| New tokens | 1,788k | 1,598k |
| Failed commands | 119 | 16 |
| Wrote a script | 106 | 10 |
| Cost at API prices | $19.45 | $16.09 |
About these results
The complete run is Claude Opus 5.5 at medium effort, on 2026-11-07: every scenario, without deft and with it, on the current product. With deft means the skill is installed and nothing in the prompt mentions deft. Only the project's settings and skills load, so no personal skill reaches either side.
After that run the skill's description gained the words create and query, and the 7 scenarios that had skipped the skill ran again; 5 then read it. Those later runs are the ones shown.
Codex, with gpt-6-luna at medium effort, ran 42 scenarios on 2026-11-07 before its quota ran out. Its other runs are from before the audit of 2026-11-06, so they show older commands and an earlier skill description.
Evals are separate from the test cases. Tests prove a command is right. Evals are a smaller set that measures what an agent spends to finish a job. Every scenario is listed here, run or not.
So far only Codex has run, with gpt-6-luna at medium effort. The latest runs, on 2026-11-07, use the skill as a person would have it: installed, with nothing in the prompt about deft, and an empty home so that no other skills or instructions apply. Older runs handed the skill text over as AGENTS.md and show older commands. Claude Code has not run. New tokens are input tokens not served from the prompt cache, plus output tokens. Agents' files were kept only for some older runs; the reference output shows what a finished job looks like. How the benchmark works.