Evals / documents

Read a document

8 scenarios, 8 run by an agent. Each run shows its result, time and new tokens.

ScenarioCodexwithout deftCodexwith deft, skill installedClaude Opus 5.5without deftClaude Opus 5.5with deft, skill installedClaude Codewithout deftClaude Codewith deft
Read a Word documentExtract the text of letter.docx into letter.txt.pass16s · 6.4kpass13s · 7.4kpass13s · 9.2kpass8s · 9.7knot runnot run
Word to MarkdownConvert demo.docx to demo.md, keeping headings and tables.pass30s · 10.4kpass16s · 8.6kfail51s · 24.1kpass12s · 13.9knot runnot run
Read a deck with notesExtract all the text of show.pptx, including the speaker notes, into show.txt.pass19s · 5.8kpass15s · 15.0kpass16s · 10.9kpass10s · 10.0knot runnot run
Deck to an outlineMake a Markdown outline of show.pptx called show.md.pass21s · 6.1kpass14s · 5.9kpass26s · 11.2kpass20s · 10.0knot runnot run
Read an e-bookExtract the text of frankenstein.epub into book.txt.pass26s · 8.1kpass16s · 8.5kpass21s · 13.4kpass9s · 10.3knot runnot run
E-book to MarkdownConvert frankenstein.epub to book.md.pass38s · 11.8kpass13s · 4.2kpass58s · 24.3kpass13s · 9.9knot runnot run
Text of a web pageExtract the readable text of article.html into article.txt.pass15s · 6.5kpass14s · 6.8kpass17s · 9.4kpass19s · 10.1knot runnot run
Clean a web pageConvert the article in article.html to Markdown without the site navigation. Save it as article.md.pass13s · 3.8kpass14s · 9.2kpass11s · 9.7kpass13s · 10.0knot runnot run