Evals / pdf

Multi-step jobs

3 scenarios, 3 run by an agent. Each run shows its result, time and new tokens.

ScenarioCodexwithout deftCodexwith deft, skill installedClaude Opus 5.5without deftClaude Opus 5.5with deft, skill installedClaude Codewithout deftClaude Codewith deft
Assemble an application from mixed filesPut together one application.pdf for a portal: letter.docx first, then the photo DSCN0010.jpg, then fw9.pdf. Number the pages at the bottom. The portal rejects anything over 1MB.fail104s · 31.3kpass36s · 10.2kpass92s · 31.2kpass28s · 15.3knot runnot run
Send a scan to an accountantscan.pdf is going to our accountant. Make the text searchable, keep it under 1MB, and protect it with the password hunter2. Save it as for-accountant.pdf.pass64s · 8.7kpass41s · 9.6kfail600s · 21.7kpass50s · 18.7knot runnot run
Collect invoice line itemsThe invoices folder has three PDF invoices. Put every line item from all of them into items.csv, with a single header row.pass65s · 33.3kpass28s · 8.4kpass18s · 15.9kpass23s · 12.6knot runnot run