Evals / config / Compare and merge

Compare two configs

What differs between config.yaml and prod.yaml? Save the structured differences as diff.json.

normalsingledeft yaml diff

The task

The reference answer shows the job can be done with deft. It is not an agent run.

deft yaml diff config.yaml prod.yaml --json > diff.json

Input

service:
  name: billing
  port: 8080
  debug: true
database:
  host: localhost
  pool: 5
features:
  - invoices
  - refunds
service:
  port: 80
  debug: false
database:
  host: db.internal

Reference output

diff.json 581 B
{"v":1,"ok":true,"op":"yaml.diff","items":[{"ok":true,"inputs":["config.yaml","prod.yaml"],"outputs":[],"facts":{"equal":false,"changes":[{"path":".service.name","kind":"removed","before":"billing"},{"path":".service.port","kind":"changed","before":8080,"after":80},{"path":".service.debug","kind":"changed","before":true,"after":false},{"path":".database.host","kind":"changed","before":"localhost","after":"db.internal"},{"path":".database.pool","kind":"removed","before":5},{"path":".features","kind":"removed","before":["invoices","refunds"]}]},"warnings":[]}],"elapsed_ms":0}
How a result is judged (1 check)
  • python: code = "t = open('diff.json').read(); assert 'service.port' in t"

Runs

CodexWithout deftPassed2026-11-07

Time
19s
New tokens
4.6k
Total input
54.5k
Output tokens
0.7k
Commands
3
Failed
1

Trace

  1. I’ll read both YAML files, compare their parsed structures, and write the differences to `diff.json` in the current directory.

  2. pwd && rg --files -g 'config.yaml' -g 'prod.yaml' -g 'AGENTS.md'
    output
    prod.yaml
    config.yaml
  3. cat config.yaml && cat prod.yaml && python3 -c 'import yaml; print(\"pyyaml available\")' (exit 1)
    output
    service:
      name: billing
      port: 8080
      debug: true
    database:
      host: localhost
      pool: 5
    features:
      - invoices
      - refunds
    service:
      port: 80
      debug: false
    database:
      host: db.internal
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'yaml'
  4. cat > diff.json <<'EOF' { \"differences\": [ { \"path\": \"service.name\", \"type\": \"removed\", \"config\": \"billing\" }, { \"path\": \"service.port\", \"type\": \"changed\", \"config\": 8080, \"prod\": 80 }, { \"path\": \"service.debug\", \"type\": \"changed\", \"config\": true, \"prod\": false }, { \"path\": \"database.host\", \"type\": \"changed\", \"config\": \"localhost\", \"prod\": \"db.internal\" }, { \"path\": \"database.pool\", \"type\": \"removed\", \"config\": 5 }, { \"path\": \"features\", \"type\": \"removed\", \"config\": [ \"invoices\", \"refunds\" ] } ] } EOF cat diff.json
    output
    {
      "differences": [
        {
          "path": "service.name",
          "type": "removed",
          "config": "billing"
        },
        {
          "path": "service.port",
          "type": "changed",
          "config": 8080,
          "prod": 80
        },
        {
          "path": "service.debug",
          "type": "changed",
          "config": true,
          "prod": false
        },
        {
          "path": "database.host",
          "type": "changed",
          "config": "localhost",
          "prod": "db.internal"
        },
        {
          "path": "database.pool",
          "type": "removed",
          "config": 5
        },
        {
          "path": "features",
          "type": "removed",
          "config": [
            "invoices",
            "refunds"
          ]
        }
      ]
    }
  5. Found six differences between `config.yaml` and `prod.yaml`, and saved them in [diff.json](diff.json). Three values changed; `service.name`, `database.pool`, and `features` appear only in `config.yaml`.

Output

diff.json 679 B
{
  "differences": [
    {
      "path": "service.name",
      "type": "removed",
      "config": "billing"
    },
    {
      "path": "service.port",
      "type": "changed",
      "config": 8080,
      "prod": 80
    },
    {
      "path": "service.debug",
      "type": "changed",
      "config": true,
      "prod": false
    },
    {
      "path": "database.host",
      "type": "changed",
      "config": "localhost",
      "prod": "db.internal"
    },
    {
      "path": "database.pool",
      "type": "removed",
      "config": 5
    },
    {
      "path": "features",
      "type": "removed",
      "config": [
        "invoices",
        "refunds"
      ]
    }
  ]
}

CodexWith deft, skill installedPassedRead the skillUsed deft2026-11-07

Time
19s
New tokens
8.8k
Total input
86.7k
Output tokens
0.6k
Commands
5
Failed
0

Trace

  1. I’ll inspect both YAML files and save their structured differences to `diff.json` in the current directory.

  2. cat .agents/skills/deft/SKILL.md && rg --files -g 'config.yaml' -g 'prod.yaml' -g 'AGENTS.md'
    output
    ---
    name: deft
    description: "Use deft first for any simple, direct operation on local files, whatever the type: convert, read, extract, edit, compress, merge, split, compare or inspect. Try it before writing a script or one-liner. If unsure, run deft find; if it says no command does this, use another tool. Not for programming work: to read, write or change source code, skip it."
    ---
    
    # deft
    
    One local command for file jobs. Prefer it over writing Python or shell scripts.
    
    Syntax: `deft <group> <command> [inputs] [-o output] [--options]`
    
    Groups:
    - `pdf`: Merge, split, compress, encrypt, forms, OCR, compare, convert
    - `image`: Resize, crop, convert, compress, adjust, montage, OCR
    - `video`: Trim, merge, convert, compress, subtitles, GIF, frames (ffmpeg)
    - `audio`: Convert, trim, normalize, denoise, cut silence (ffmpeg)
    - `doc`: Word/ODT/RTF: to text, Markdown, HTML, PDF; find and replace
    - `slides`: PowerPoint: extract text, Markdown outline
    - `sheet`: Excel/CSV/ODS tables: SQL query, pivot, merge, split, convert
    - `json`: JSON: convert, format, merge, diff, validate
    - `toml`: TOML: convert, format, merge, diff, validate
    - `xml`: XML: convert, format, merge, diff, validate
    - `yaml`: YAML: convert, format, merge, diff, validate
    - `md`: Markdown to PDF, HTML, Word, EPUB; table of contents
    - `html`: HTML to Markdown, text, Word and PDF
    - `epub`: E-books: info, to text, Markdown and PDF
    - `gps`: GPX and KML tracks: convert, distance and time
    - `qr`: Create QR codes and barcodes; r
    … 722 more characters
  3. deft yaml --help
    output
    deft yaml - commands (input -> output)
      convert   yaml -> data       Change format: --to json|toml|xml|jsonl
      diff      yaml... -> report  Compare two files by structure
      format    yaml -> yaml       Tidy a file: --sort-keys
      merge     yaml... -> yaml    Deep-merge files; later files win
      validate  yaml -> report     Check syntax, or a JSON Schema with --schema
    More: deft yaml <command> --help
  4. deft yaml diff --help && deft yaml diff config.yaml prod.yaml --json
    output
    deft yaml diff - Compare two files by structure
    
    Usage: deft yaml diff <yaml>... [-o <json>] [options]
    In: yaml (all together, in order)   Out: printed result
    Key order and formatting do not count.
    Common: -o --force --dry-run --json -j   (deft help common)
    
    Examples:
      deft yaml diff config.yaml override.yaml
    
    Result fields: equal, changes
    {"v":1,"ok":true,"op":"yaml.diff","items":[{"ok":true,"inputs":["config.yaml","prod.yaml"],"outputs":[],"facts":{"equal":false,"changes":[{"path":".service.name","kind":"removed","before":"billing"},{"path":".service.port","kind":"changed","before":8080,"after":80},{"path":".service.debug","kind":"changed","before":true,"after":false},{"path":".database.host","kind":"changed","before":"localhost","after":"db.internal"},{"path":".database.pool","kind":"removed","before":5},{"path":".features","kind":"removed","before":["invoices","refunds"]}]},"warnings":[]}],"elapsed_ms":2}
  5. deft yaml diff config.yaml prod.yaml -o diff.json --force
    output
    ok yaml.diff config.yaml prod.yaml -> diff.json  6 changes  0.00s
  6. cat diff.json
    output
    {
      "equal": false,
      "changes": [
        {
          "path": ".service.name",
          "kind": "removed",
          "before": "billing"
        },
        {
          "path": ".service.port",
          "kind": "changed",
          "before": 8080,
          "after": 80
        },
        {
          "path": ".service.debug",
          "kind": "changed",
          "before": true,
          "after": false
        },
        {
          "path": ".database.host",
          "kind": "changed",
          "before": "localhost",
          "after": "db.internal"
        },
        {
          "path": ".database.pool",
          "kind": "removed",
          "before": 5
        },
        {
          "path": ".features",
          "kind": "removed",
          "before": [
            "invoices",
            "refunds"
          ]
        }
      ]
    }
  7. Saved the structured differences in [diff.json](diff.json). There are six differences: `service.name`, `database.pool`, and `features` were removed; `service.port`, `service.debug`, and `database.host` changed.

Output

diff.json 702 B
{
  "equal": false,
  "changes": [
    {
      "path": ".service.name",
      "kind": "removed",
      "before": "billing"
    },
    {
      "path": ".service.port",
      "kind": "changed",
      "before": 8080,
      "after": 80
    },
    {
      "path": ".service.debug",
      "kind": "changed",
      "before": true,
      "after": false
    },
    {
      "path": ".database.host",
      "kind": "changed",
      "before": "localhost",
      "after": "db.internal"
    },
    {
      "path": ".database.pool",
      "kind": "removed",
      "before": 5
    },
    {
      "path": ".features",
      "kind": "removed",
      "before": [
        "invoices",
        "refunds"
      ]
    }
  ]
}

Claude Opus 5.5Without deftPassed2026-11-07

Time
12s
New tokens
9.3k
Total input
42.4k
Output tokens
0.8k
Commands
0
Failed
0

Trace

    Output

    diff.json 483 B
    {
      "base": "config.yaml",
      "compare": "prod.yaml",
      "changed": [
        { "path": "service.port", "config": 8080, "prod": 80 },
        { "path": "service.debug", "config": true, "prod": false },
        { "path": "database.host", "config": "localhost", "prod": "db.internal" }
      ],
      "only_in_config": [
        { "path": "service.name", "value": "billing" },
        { "path": "database.pool", "value": 5 },
        { "path": "features", "value": ["invoices", "refunds"] }
      ],
      "only_in_prod": []
    }
    

    Claude Opus 5.5With deft, skill installedPassedRead the skillUsed deft2026-11-07

    Time
    14s
    New tokens
    10.6k
    Total input
    75.3k
    Output tokens
    0.8k
    Commands
    0
    Failed
    0

    Trace

      Output

      ---
      name: deft
      description: "Use deft first for any simple, direct operation on local files, whatever the type: convert, read, extract, edit, compress, merge, split, compare or inspect. Try it before writing a script or one-liner. If unsure, run deft find; if it says no command does this, use another tool. Not for programming work: to read, write or change source code, skip it."
      ---
      
      # deft
      
      One local command for file jobs. Prefer it over writing Python or shell scripts.
      
      Syntax: `deft <group> <command> [inputs] [-o output] [--options]`
      
      Groups:
      - `pdf`: Merge, split, compress, encrypt, forms, OCR, compare, convert
      - `image`: Resize, crop, convert, compress, adjust, montage, OCR
      - `video`: Trim, merge, convert, compress, subtitles, GIF, frames (ffmpeg)
      - `audio`: Convert, trim, normalize, denoise, cut silence (ffmpeg)
      - `doc`: Word/ODT/RTF: to text, Markdown, HTML, PDF; find and replace
      - `slides`: PowerPoint: extract text, Markdown outline
      - `sheet`: Excel/CSV/ODS tables: SQL query, pivot, merge, split, convert
      - `json`: JSON: convert, format, merge, diff, validate
      - `toml`: TOML: convert, format, merge, diff, validate
      - `xml`: XML: convert, format, merge, diff, validate
      - `yaml`: YAML: convert, format, merge, diff, validate
      - `md`: Markdown to PDF, HTML, Word, EPUB; table of contents
      - `html`: HTML to Markdown, text, Word and PDF
      - `epub`: E-books: info, to text, Markdown and PDF
      - `gps`: GPX and KML tracks: convert, distance and time
      - `qr`: Create QR codes and barcodes; read QR codes
      
      How to use it:
      1. Run `deft <group>` to list commands, or `deft find "<what you want>"` to search. If it says `no command does this`, use another tool.
      2. Run `deft <group> <command> --help` for options and examples.
      3. `deft convert <in> -o <out.ext>` converts between formats by extension.
      4. Results are one line starting with `ok`, `partial` or `error`; add `--json` to parse them. Errors include a `hint:` with the fix.
      
      Inputs are never overwritten; pass `--force` or `--in-place` to allow it. Globs and directories work as inputs; several inputs run in parallel.
      
      If `deft` is not installed, ask the user, then follow https://github.com/firstprayer/deft/blob/main/docs/INSTALL.md
      
      diff.json 702 B
      {
        "equal": false,
        "changes": [
          {
            "path": ".service.name",
            "kind": "removed",
            "before": "billing"
          },
          {
            "path": ".service.port",
            "kind": "changed",
            "before": 8080,
            "after": 80
          },
          {
            "path": ".service.debug",
            "kind": "changed",
            "before": true,
            "after": false
          },
          {
            "path": ".database.host",
            "kind": "changed",
            "before": "localhost",
            "after": "db.internal"
          },
          {
            "path": ".database.pool",
            "kind": "removed",
            "before": 5
          },
          {
            "path": ".features",
            "kind": "removed",
            "before": [
              "invoices",
              "refunds"
            ]
          }
        ]
      }
      

      Claude CodeWithout deftNot run yet

      Claude CodeWith deftNot run yet