Evals / pdf

Security and redaction

3 scenarios, 3 run by an agent. Each run shows its result, time and new tokens.

ScenarioCodexwithout deftCodexwith deft, skill installedCodexwith deft, skill as agents.mdClaude Opus 5.5without deftClaude Opus 5.5with deft, skill installedClaude Codewithout deftClaude Codewith deft
Remove a password I knowlocked.pdf has the password hunter2. Save an unprotected copy as unlocked.pdf.pass13s · 6.5kpass15s · 5.4kpass15s · 10.5kpass113s · 26.2kpass12s · 9.9knot runnot run
Password-protect a PDFProtect paper.pdf with the password hunter2 and save it as paper-locked.pdf.pass17s · 4.1kpass28s · 17.4knot runpass34s · 11.7kpass12s · 9.8knot runnot run
Black out a regionOn page 1 of paper.pdf, black out the top 200 points of the page so the title and authors cannot be recovered. Save as safe.pdf.fail44s · 10.8kpass15s · 5.3knot runpass46s · 14.7kpass30s · 14.4knot runnot run