Evals / config / Compare and merge
Compare two configs
What differs between config.yaml and prod.yaml? Save the structured differences as diff.json.
The task
The reference answer shows the job can be done with deft. It is not an agent run.
deft yaml diff config.yaml prod.yaml --json > diff.json
Input
service: name: billing port: 8080 debug: true database: host: localhost pool: 5 features: - invoices - refunds
service: port: 80 debug: false database: host: db.internal
Reference output
{"v":1,"ok":true,"op":"yaml.diff","items":[{"ok":true,"inputs":["config.yaml","prod.yaml"],"outputs":[],"facts":{"equal":false,"changes":[{"path":".service.name","kind":"removed","before":"billing"},{"path":".service.port","kind":"changed","before":8080,"after":80},{"path":".service.debug","kind":"changed","before":true,"after":false},{"path":".database.host","kind":"changed","before":"localhost","after":"db.internal"},{"path":".database.pool","kind":"removed","before":5},{"path":".features","kind":"removed","before":["invoices","refunds"]}]},"warnings":[]}],"elapsed_ms":0}
How a result is judged (1 check)
- python: code = "t = open('diff.json').read(); assert 'service.port' in t"
Runs
CodexWithout deftPassed2026-11-07
- Time
- 19s
- New tokens
- 4.6k
- Total input
- 54.5k
- Output tokens
- 0.7k
- Commands
- 3
- Failed
- 1
Trace
I’ll read both YAML files, compare their parsed structures, and write the differences to `diff.json` in the current directory.
pwd && rg --files -g 'config.yaml' -g 'prod.yaml' -g 'AGENTS.md'output
prod.yaml config.yaml
cat config.yaml && cat prod.yaml && python3 -c 'import yaml; print(\"pyyaml available\")' (exit 1)output
service: name: billing port: 8080 debug: true database: host: localhost pool: 5 features: - invoices - refunds service: port: 80 debug: false database: host: db.internal Traceback (most recent call last): File "<string>", line 1, in <module> ModuleNotFoundError: No module named 'yaml'
cat > diff.json <<'EOF' { \"differences\": [ { \"path\": \"service.name\", \"type\": \"removed\", \"config\": \"billing\" }, { \"path\": \"service.port\", \"type\": \"changed\", \"config\": 8080, \"prod\": 80 }, { \"path\": \"service.debug\", \"type\": \"changed\", \"config\": true, \"prod\": false }, { \"path\": \"database.host\", \"type\": \"changed\", \"config\": \"localhost\", \"prod\": \"db.internal\" }, { \"path\": \"database.pool\", \"type\": \"removed\", \"config\": 5 }, { \"path\": \"features\", \"type\": \"removed\", \"config\": [ \"invoices\", \"refunds\" ] } ] } EOF cat diff.jsonoutput
{ "differences": [ { "path": "service.name", "type": "removed", "config": "billing" }, { "path": "service.port", "type": "changed", "config": 8080, "prod": 80 }, { "path": "service.debug", "type": "changed", "config": true, "prod": false }, { "path": "database.host", "type": "changed", "config": "localhost", "prod": "db.internal" }, { "path": "database.pool", "type": "removed", "config": 5 }, { "path": "features", "type": "removed", "config": [ "invoices", "refunds" ] } ] }Found six differences between `config.yaml` and `prod.yaml`, and saved them in [diff.json](diff.json). Three values changed; `service.name`, `database.pool`, and `features` appear only in `config.yaml`.
Output
{
"differences": [
{
"path": "service.name",
"type": "removed",
"config": "billing"
},
{
"path": "service.port",
"type": "changed",
"config": 8080,
"prod": 80
},
{
"path": "service.debug",
"type": "changed",
"config": true,
"prod": false
},
{
"path": "database.host",
"type": "changed",
"config": "localhost",
"prod": "db.internal"
},
{
"path": "database.pool",
"type": "removed",
"config": 5
},
{
"path": "features",
"type": "removed",
"config": [
"invoices",
"refunds"
]
}
]
}
CodexWith deft, skill installedPassedRead the skillUsed deft2026-11-07
- Time
- 19s
- New tokens
- 8.8k
- Total input
- 86.7k
- Output tokens
- 0.6k
- Commands
- 5
- Failed
- 0
Trace
I’ll inspect both YAML files and save their structured differences to `diff.json` in the current directory.
cat .agents/skills/deft/SKILL.md && rg --files -g 'config.yaml' -g 'prod.yaml' -g 'AGENTS.md'output
--- name: deft description: "Use deft first for any simple, direct operation on local files, whatever the type: convert, read, extract, edit, compress, merge, split, compare or inspect. Try it before writing a script or one-liner. If unsure, run deft find; if it says no command does this, use another tool. Not for programming work: to read, write or change source code, skip it." --- # deft One local command for file jobs. Prefer it over writing Python or shell scripts. Syntax: `deft <group> <command> [inputs] [-o output] [--options]` Groups: - `pdf`: Merge, split, compress, encrypt, forms, OCR, compare, convert - `image`: Resize, crop, convert, compress, adjust, montage, OCR - `video`: Trim, merge, convert, compress, subtitles, GIF, frames (ffmpeg) - `audio`: Convert, trim, normalize, denoise, cut silence (ffmpeg) - `doc`: Word/ODT/RTF: to text, Markdown, HTML, PDF; find and replace - `slides`: PowerPoint: extract text, Markdown outline - `sheet`: Excel/CSV/ODS tables: SQL query, pivot, merge, split, convert - `json`: JSON: convert, format, merge, diff, validate - `toml`: TOML: convert, format, merge, diff, validate - `xml`: XML: convert, format, merge, diff, validate - `yaml`: YAML: convert, format, merge, diff, validate - `md`: Markdown to PDF, HTML, Word, EPUB; table of contents - `html`: HTML to Markdown, text, Word and PDF - `epub`: E-books: info, to text, Markdown and PDF - `gps`: GPX and KML tracks: convert, distance and time - `qr`: Create QR codes and barcodes; r … 722 more characters
deft yaml --helpoutput
deft yaml - commands (input -> output) convert yaml -> data Change format: --to json|toml|xml|jsonl diff yaml... -> report Compare two files by structure format yaml -> yaml Tidy a file: --sort-keys merge yaml... -> yaml Deep-merge files; later files win validate yaml -> report Check syntax, or a JSON Schema with --schema More: deft yaml <command> --help
deft yaml diff --help && deft yaml diff config.yaml prod.yaml --jsonoutput
deft yaml diff - Compare two files by structure Usage: deft yaml diff <yaml>... [-o <json>] [options] In: yaml (all together, in order) Out: printed result Key order and formatting do not count. Common: -o --force --dry-run --json -j (deft help common) Examples: deft yaml diff config.yaml override.yaml Result fields: equal, changes {"v":1,"ok":true,"op":"yaml.diff","items":[{"ok":true,"inputs":["config.yaml","prod.yaml"],"outputs":[],"facts":{"equal":false,"changes":[{"path":".service.name","kind":"removed","before":"billing"},{"path":".service.port","kind":"changed","before":8080,"after":80},{"path":".service.debug","kind":"changed","before":true,"after":false},{"path":".database.host","kind":"changed","before":"localhost","after":"db.internal"},{"path":".database.pool","kind":"removed","before":5},{"path":".features","kind":"removed","before":["invoices","refunds"]}]},"warnings":[]}],"elapsed_ms":2}deft yaml diff config.yaml prod.yaml -o diff.json --forceoutput
ok yaml.diff config.yaml prod.yaml -> diff.json 6 changes 0.00s
cat diff.jsonoutput
{ "equal": false, "changes": [ { "path": ".service.name", "kind": "removed", "before": "billing" }, { "path": ".service.port", "kind": "changed", "before": 8080, "after": 80 }, { "path": ".service.debug", "kind": "changed", "before": true, "after": false }, { "path": ".database.host", "kind": "changed", "before": "localhost", "after": "db.internal" }, { "path": ".database.pool", "kind": "removed", "before": 5 }, { "path": ".features", "kind": "removed", "before": [ "invoices", "refunds" ] } ] }Saved the structured differences in [diff.json](diff.json). There are six differences: `service.name`, `database.pool`, and `features` were removed; `service.port`, `service.debug`, and `database.host` changed.
Output
{
"equal": false,
"changes": [
{
"path": ".service.name",
"kind": "removed",
"before": "billing"
},
{
"path": ".service.port",
"kind": "changed",
"before": 8080,
"after": 80
},
{
"path": ".service.debug",
"kind": "changed",
"before": true,
"after": false
},
{
"path": ".database.host",
"kind": "changed",
"before": "localhost",
"after": "db.internal"
},
{
"path": ".database.pool",
"kind": "removed",
"before": 5
},
{
"path": ".features",
"kind": "removed",
"before": [
"invoices",
"refunds"
]
}
]
}
Claude Opus 5.5Without deftPassed2026-11-07
- Time
- 12s
- New tokens
- 9.3k
- Total input
- 42.4k
- Output tokens
- 0.8k
- Commands
- 0
- Failed
- 0
Trace
Output
{
"base": "config.yaml",
"compare": "prod.yaml",
"changed": [
{ "path": "service.port", "config": 8080, "prod": 80 },
{ "path": "service.debug", "config": true, "prod": false },
{ "path": "database.host", "config": "localhost", "prod": "db.internal" }
],
"only_in_config": [
{ "path": "service.name", "value": "billing" },
{ "path": "database.pool", "value": 5 },
{ "path": "features", "value": ["invoices", "refunds"] }
],
"only_in_prod": []
}
Claude Opus 5.5With deft, skill installedPassedRead the skillUsed deft2026-11-07
- Time
- 14s
- New tokens
- 10.6k
- Total input
- 75.3k
- Output tokens
- 0.8k
- Commands
- 0
- Failed
- 0
Trace
Output
--- name: deft description: "Use deft first for any simple, direct operation on local files, whatever the type: convert, read, extract, edit, compress, merge, split, compare or inspect. Try it before writing a script or one-liner. If unsure, run deft find; if it says no command does this, use another tool. Not for programming work: to read, write or change source code, skip it." --- # deft One local command for file jobs. Prefer it over writing Python or shell scripts. Syntax: `deft <group> <command> [inputs] [-o output] [--options]` Groups: - `pdf`: Merge, split, compress, encrypt, forms, OCR, compare, convert - `image`: Resize, crop, convert, compress, adjust, montage, OCR - `video`: Trim, merge, convert, compress, subtitles, GIF, frames (ffmpeg) - `audio`: Convert, trim, normalize, denoise, cut silence (ffmpeg) - `doc`: Word/ODT/RTF: to text, Markdown, HTML, PDF; find and replace - `slides`: PowerPoint: extract text, Markdown outline - `sheet`: Excel/CSV/ODS tables: SQL query, pivot, merge, split, convert - `json`: JSON: convert, format, merge, diff, validate - `toml`: TOML: convert, format, merge, diff, validate - `xml`: XML: convert, format, merge, diff, validate - `yaml`: YAML: convert, format, merge, diff, validate - `md`: Markdown to PDF, HTML, Word, EPUB; table of contents - `html`: HTML to Markdown, text, Word and PDF - `epub`: E-books: info, to text, Markdown and PDF - `gps`: GPX and KML tracks: convert, distance and time - `qr`: Create QR codes and barcodes; read QR codes How to use it: 1. Run `deft <group>` to list commands, or `deft find "<what you want>"` to search. If it says `no command does this`, use another tool. 2. Run `deft <group> <command> --help` for options and examples. 3. `deft convert <in> -o <out.ext>` converts between formats by extension. 4. Results are one line starting with `ok`, `partial` or `error`; add `--json` to parse them. Errors include a `hint:` with the fix. Inputs are never overwritten; pass `--force` or `--in-place` to allow it. Globs and directories work as inputs; several inputs run in parallel. If `deft` is not installed, ask the user, then follow https://github.com/firstprayer/deft/blob/main/docs/INSTALL.md
{
"equal": false,
"changes": [
{
"path": ".service.name",
"kind": "removed",
"before": "billing"
},
{
"path": ".service.port",
"kind": "changed",
"before": 8080,
"after": 80
},
{
"path": ".service.debug",
"kind": "changed",
"before": true,
"after": false
},
{
"path": ".database.host",
"kind": "changed",
"before": "localhost",
"after": "db.internal"
},
{
"path": ".database.pool",
"kind": "removed",
"before": 5
},
{
"path": ".features",
"kind": "removed",
"before": [
"invoices",
"refunds"
]
}
]
}
Claude CodeWithout deftNot run yet
Claude CodeWith deftNot run yet