4 runs on real files, 6 tests on generated files, not shown. 0 failed.
Real-file tasks
Real files and a plain-language job, from the agent benchmark. These are the reference answers, not an agent run.
Put together one application.pdf for a portal: letter.docx first, then the photo DSCN0010.jpg, then fw9.pdf. Number the pages at the bottom. The portal rejects anything over 1MB.
deft doc to-pdf letter.docx -o letter.pdf
deft image to-pdf DSCN0010.jpg -o photo.pdf
deft pdf merge letter.pdf photo.pdf fw9.pdf -o joined.pdf
deft pdf number joined.pdf -o numbered.pdf
deft pdf compress numbered.pdf --target-size 1MB -o application.pdf
Input
Rendered by macOS Quick Look, first page
Text inside
I am a test document
This is page 1
I am Calibri (Body) in font size 11
This is page two
It’s Arial Black in 16 point
It’s also in blue
Automatic checks (2)
- max_bytes: file = "application.pdf", bytes = 1048576
- python: code = "import subprocess\ndef text(first, last):\n return subprocess.run(['gs', '-q', '-dNOPAUSE', '-dBATCH', '-sDEVICE=txtwrite', f'-dFirstPage={first}', f'-dLastPage={last}', '-sOutputFile=-', 'application.pdf'], capture_output=True, text=True, errors='replace').stdout\nn = int(subprocess.check_output(['gs', '-q', '-dNODISPLAY', '--permit-file-read=application.pdf', '-c', '(application.pdf) (r) file runpdfbegin pdfpagecount = quit'], text=True).split()[-1])\nw9 = subprocess.check_output(['gs', '-q', '-dNODISPLAY', '--permit-file-read=fw9.pdf', '-c', '(fw9.pdf) (r) file runpdfbegin pdfpagecount = quit'], text=True).split()[-1]\nassert n >= int(w9) + 2, n\nassert 'W-9' in text(n - int(w9) + 1, n - int(w9) + 1), 'the form is not last'\nassert 'W-9' not in text(1, 1), 'the letter is not first'\nassert str(n) in text(n, n).split(), 'last page has no number'\n"
Combine fw9.pdf and f1040.pdf, in that order, into tax-forms.pdf.
deft pdf merge fw9.pdf f1040.pdf -o tax-forms.pdf
Automatic checks (2)
- pdf_pages: file = "tax-forms.pdf", pages = 8
- pdf_text: file = "tax-forms.pdf", contains = ["W-9", "1040"]
Combine fw9.pdf and f1040.pdf, number the pages like 1/8, stamp every page with the word DRAFT, and save the result as packet.pdf.
deft pdf merge fw9.pdf f1040.pdf -o a.pdf
deft pdf number a.pdf --format {n}/{total} -o b.pdf
deft pdf watermark b.pdf --text DRAFT -o packet.pdf
Automatic checks (3)
- pdf_pages: file = "packet.pdf", pages = 8
- pdf_text: file = "packet.pdf", contains = ["1/8", "8/8"], visible = true
- pdf_changed: file = "packet.pdf", from = "fw9.pdf", page = 2, min_difference = 0.3