Evals / config / Validate

Validate against a schema

Does config.yaml have a service object? Make schema.json requiring one, then validate config.yaml against it and write yes or no to result.txt.

normalmulti-step

The task

The reference answer shows the job can be done with deft. It is not an agent run.

(none)

Input

config.yaml 125 B

service:
  name: billing
  port: 8080
  debug: true
database:
  host: localhost
  pool: 5
features:
  - invoices
  - refunds

Reference output

How a result is judged (1 check)

text: file = "result.txt", contains = ["yes"]

Runs

CodexWithout deftPassed2026-11-07

Time: 21s
New tokens: 7.6k
Total input: 68.3k
Output tokens: 0.8k
Commands: 4
Failed: 1

Trace

I’ll inspect the current directory for the YAML and any existing schema conventions, then create the schema, validate the config, and write the yes/no result.
pwd && rg --files -g 'config.yaml' -g 'schema.json' -g 'result.txt' -g 'AGENTS.md'
output
```
config.yaml
```

cat config.yaml && python3 - <<'PY'
try:
 import yaml, jsonschema
 print('yaml/jsonschema available')
except Exception as e:
 print(type(e).__name__, e)
PY

output

service:
  name: billing
  port: 8080
  debug: true
database:
  host: localhost
  pool: 5
features:
  - invoices
  - refunds
ModuleNotFoundError No module named 'yaml'

cat > schema.json <<'EOF' { \""'$schema": "https://json-schema.org/draft/2020-12/schema", "type": "object", "required": ["service"], "properties": { "service": { "type": "object" } } } EOF ruby -e '"'require \"yaml\"; config = YAML.safe_load_file(\"config.yaml\"); valid = config.is_a?(Hash) && config.key?(\"service\") && config[\"service\"].is_a?(Hash); File.write(\"result.txt\", valid ? \"yes\\n\" : \"no\\n\"); puts(valid ? \"yes\" : \"no\")' && cat schema.json && cat result.txt (exit 1)
output
```
-e:1:in `<main>': undefined method `safe_load_file' for Psych:Module (NoMethodError)
Did you mean?  safe_load
```
ruby -e 'require \"yaml\"; config = YAML.safe_load(File.read(\"config.yaml\")); valid = config.is_a?(Hash) && config.key?(\"service\") && config[\"service\"].is_a?(Hash); File.write(\"result.txt\", valid ? \"yes\\n\" : \"no\\n\"); puts(valid ? \"yes\" : \"no\")' && cat schema.json && cat result.txt
output
```
yes
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["service"],
  "properties": {
    "service": {
      "type": "object"
    }
  }
}
yes
```
`config.yaml` has a `service` object. I created [schema.json](schema.json) requiring it and wrote `yes` to [result.txt](result.txt).

Output

result.txt 4 B

yes

schema.json 180 B

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["service"],
  "properties": {
    "service": {
      "type": "object"
    }
  }
}

CodexWith deft, skill installedPassedRead the skillUsed deft2026-11-07

Time: 28s
New tokens: 13.3k
Total input: 116.4k
Output tokens: 1.0k
Commands: 7
Failed: 0

Trace

I’ll inspect the YAML structure and any existing schema conventions, then create the required schema, validate the file, and write the yes/no result.
pwd && rg --files -g 'config.yaml' -g 'schema.json' -g 'result.txt' -g 'AGENTS.md' -g 'SKILL.md'
output
```
config.yaml
```
I’m using the local file skill for this straightforward inspect-and-write task.

cat .agents/skills/deft/SKILL.md

output

---
name: deft
description: "Use deft first for any simple, direct operation on local files, whatever the type: convert, read, extract, edit, compress, merge, split, compare or inspect. Try it before writing a script or one-liner. If unsure, run deft find; if it says no command does this, use another tool. Not for programming work: to read, write or change source code, skip it."
---

# deft

One local command for file jobs. Prefer it over writing Python or shell scripts.

Syntax: `deft <group> <command> [inputs] [-o output] [--options]`

Groups:
- `pdf`: Merge, split, compress, encrypt, forms, OCR, compare, convert
- `image`: Resize, crop, convert, compress, adjust, montage, OCR
- `video`: Trim, merge, convert, compress, subtitles, GIF, frames (ffmpeg)
- `audio`: Convert, trim, normalize, denoise, cut silence (ffmpeg)
- `doc`: Word/ODT/RTF: to text, Markdown, HTML, PDF; find and replace
- `slides`: PowerPoint: extract text, Markdown outline
- `sheet`: Excel/CSV/ODS tables: SQL query, pivot, merge, split, convert
- `json`: JSON: convert, format, merge, diff, validate
- `toml`: TOML: convert, format, merge, diff, validate
- `xml`: XML: convert, format, merge, diff, validate
- `yaml`: YAML: convert, format, merge, diff, validate
- `md`: Markdown to PDF, HTML, Word, EPUB; table of contents
- `html`: HTML to Markdown, text, Word and PDF
- `epub`: E-books: info, to text, Markdown and PDF
- `gps`: GPX and KML tracks: convert, distance and time
- `qr`: Create QR codes and barcodes; r
… 700 more characters

deft yaml

output

deft yaml - commands (input -> output)
  convert   yaml -> data       Change format: --to json|toml|xml|jsonl
  diff      yaml... -> report  Compare two files by structure
  format    yaml -> yaml       Tidy a file: --sort-keys
  merge     yaml... -> yaml    Deep-merge files; later files win
  validate  yaml -> report     Check syntax, or a JSON Schema with --schema
More: deft yaml <command> --help

deft yaml convert --help && cat config.yaml

output

deft yaml convert - Change format: --to json|toml|xml|jsonl
Same as: deft convert

Usage: deft yaml convert <yaml>... [-o <data|dir>] [options]
In: yaml (each handled separately)   Out: data (default <name>.<ext>)

Options:
  --to <json|toml|xml|jsonl>   Target format (or give -o name.ext)
Common: -o --force --dry-run --json -j   (deft help common)

Examples:
  deft yaml convert config.yaml --to json -o out.json
service:
  name: billing
  port: 8080
  debug: true
database:
  host: localhost
  pool: 5
features:
  - invoices
  - refunds

deft yaml validate --help

output

deft yaml validate - Check syntax, or a JSON Schema with --schema

Usage: deft yaml validate <yaml>... [-o <json>] [options]
In: yaml (each handled separately)   Out: printed result

Options:
  --schema <path>              JSON Schema file, in JSON or YAML
Common: -o --force --dry-run --json -j   (deft help common)

Examples:
  deft yaml validate config.yaml
  deft yaml validate config.yaml --schema server.schema.json

Result fields: valid, format
Errors: INPUT_CORRUPT

"cat > schema.json <<'EOF' { \""'$schema": "https://json-schema.org/draft/2020-12/schema", "type": "object", "required": ["service"], "properties": { "service": { "type": "object" } } } EOF deft yaml validate config.yaml --schema schema.json --json'
output
```
{"v":1,"ok":true,"op":"yaml.validate","items":[{"ok":true,"inputs":["config.yaml"],"outputs":[],"facts":{"valid":true,"format":"yaml"},"warnings":[]}],"elapsed_ms":18}
```

printf 'yes\\n' > result.txt && cat result.txt && cat schema.json

output

yes
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["service"],
  "properties": {
    "service": {
      "type": "object"
    }
  }
}

`config.yaml` has a `service` object and validates against the schema. Created [schema.json](schema.json) and wrote `yes` to [result.txt](result.txt).

Output

result.txt 4 B

yes

schema.json 180 B

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["service"],
  "properties": {
    "service": {
      "type": "object"
    }
  }
}

Claude Opus 5.5Without deftPassed2026-11-07

Time: 18s
New tokens: 9.2k
Total input: 56.7k
Output tokens: 0.7k
Commands: 0
Failed: 0

Trace

Output

result.txt 4 B

yes

schema.json 170 B

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["service"],
  "properties": {
    "service": { "type": "object" }
  }
}

Claude Opus 5.5With deft, skill installedPassedDid not read the skillDid not use deft2026-11-07

Time: 18s
New tokens: 9.5k
Total input: 57.3k
Output tokens: 0.7k
Commands: 0
Failed: 0

Trace

Output

.claude/skills/deft/SKILL.md 2.2 KB

---
name: deft
description: "Use deft first for any simple, direct operation on local files, whatever the type: create, convert, read, query, extract, edit, compress, merge, split, compare or inspect. Try it before writing a script or one-liner. If unsure, run deft find; if it says no command does this, use another tool. Not for programming work: to read, write or change source code, skip it."
---

# deft

One local command for file jobs. Prefer it over writing Python or shell scripts.

Syntax: `deft <group> <command> [inputs] [-o output] [--options]`

Groups:
- `pdf`: Merge, split, compress, encrypt, forms, OCR, compare, convert
- `image`: Resize, crop, convert, compress, adjust, montage, OCR
- `video`: Trim, merge, convert, compress, subtitles, GIF, frames (ffmpeg)
- `audio`: Convert, trim, normalize, denoise, cut silence (ffmpeg)
- `doc`: Word/ODT/RTF: to text, Markdown, HTML, PDF; find and replace
- `slides`: PowerPoint: extract text, Markdown outline
- `sheet`: Excel/CSV/ODS tables: SQL query, pivot, merge, split, convert
- `json`: JSON: convert, format, merge, diff, validate
- `toml`: TOML: convert, format, merge, diff, validate
- `xml`: XML: convert, format, merge, diff, validate
- `yaml`: YAML: convert, format, merge, diff, validate
- `md`: Markdown to PDF, HTML, Word, EPUB; table of contents
- `html`: HTML to Markdown, text, Word and PDF
- `epub`: E-books: info, to text, Markdown and PDF
- `gps`: GPX and KML tracks: convert, distance and time
- `qr`: Create QR codes and barcodes; read QR codes

How to use it:
1. Run `deft <group>` to list commands, or `deft find "<what you want>"` to search. If it says `no command does this`, use another tool.
2. Run `deft <group> <command> --help` for options and examples.
3. `deft convert <in> -o <out.ext>` converts between formats by extension.
4. Results are one line starting with `ok`, `partial` or `error`; add `--json` to parse them. Errors include a `hint:` with the fix.

Inputs are never overwritten; pass `--force` or `--in-place` to allow it. Globs and directories work as inputs; several inputs run in parallel.

If `deft` is not installed, ask the user, then follow https://github.com/firstprayer/deft/blob/main/docs/INSTALL.md

result.txt 4 B

yes

schema.json 170 B

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["service"],
  "properties": {
    "service": { "type": "object" }
  }
}

Claude CodeWithout deftNot run yet

Claude CodeWith deftNot run yet