Benchmarks and paper

Code navigation: 115 tasks, four models

In a benchmark with 115 tasks and four coding models from three providers (MiniMax-M3, GLM-5.2, Qwen3.6-27B, Kimi-K3), an agent with greppy answered 6 to 50 percentage points more questions correctly than the same agent with grep, at every tool-call budget. It reached the grep agent's best quality at 37 to 80% lower billed API cost; the other way round, the grep agent paid 1.6 to 5 times as much for its best quality.

The chart plots success rate against mean billed API cost per 1,000 tasks, with the agent with greppy in blue and the same agent with grep in green. Its dashed lines are the 2×2 ablation: an explicit method instruction changes nothing except the cost, so the effect comes from the tool surface.

Study design

The study is a pre-registered factorial design. For each of the four models it ran:

The comparison always holds the agent and the model fixed and changes the tool: grep or greppy. The 2×2 ablation also adds an explicit method instruction. The paper contains the frozen protocol and the panels of the four models.

The paper

The paper The Minimum Sufficient Code Context Problem — Complexity, Discovery Overhead, and Approximation in Coding Agents is by Michael Welsch, GPT-5.6 Sol and Fable 5.0 and appeared in July 2026.

Read the PDF · Cite it

The paper defines the minimum sufficient code context (MSCC) an agent needs for a task and proves that constructing it exactly is NP-complete. It gives a lower bound for what purely lexical navigation must pay for ambiguous entry points and unresolved relations, and it states the measurable conditions under which the combined strategy greppy implements is strictly cheaper without losing correctness. It also contains the full empirical evidence across four models.

The benchmark suites

Two pre-registered benchmark suites live in the repository:

bench/agent_efficiency/ contains 115 pinned navigation tasks in six real repositories plus deterministic controls. It measures answer correctness together with the cost of search, source reading and context.

bench/agent_coding/ contains 41 paired edit-and-test tasks derived from real commits of Flask, Hugo, Gson, Zod, Serde and Tokio; 18 of them are serious multi-file changes (80 to 800 lines) from real issues. Each task:

1. starts at the parent of the real commit;
2. takes the commit's test diff as the failing specification, machine-proven: red on the parent commit, green with the real change;
3. hides the code diff and gives the agent the intent from the real issue;
4. lets the test decide correctness after the agent run.

Setup does not count toward the measured agent time.

What is recorded

For the same agent and the same model, both suites record per-task correctness, tool calls, opened source files, input and output tokens, context or prompt volume and wall time. Task banks, prompts, binaries, runtime versions, setup commands and repository commits go into the manifests as hashes. The order of the arms is deterministically balanced per task, and the ordering scheme is versioned in the manifest.

Each diagnostic run publishes the exact repository commits, the task-bank hash, the prompt hash, the model ID, the hash of the greppy binary, the per-task rows, the grading, the aggregate and the forensics. Raw agent traces stay private and are not release artifacts.

Diagnostics, not a gate

Agent benchmarks run continuously on candidate commits as diagnostics. They never hold back the release under test. Their targets are:

• at least as many paired correctness wins as losses, with no paired regression alarm at p < 0.05;
• at least 20% fewer tool calls and opened source files on structural tasks;
• at least 20% fewer variable input tokens on structural tasks.

Before any product change, findings are classified as product, harness, prompt, usage or expected test behavior, and they flow into a later release as fixes. Building the index is one-time work per repository and is reported separately with its break-even.

The demo

In the demo, the same coding agent (MiniMax-M3, driven by Pi) answers a who-calls question on a real repository: on the left with grep, on the right with greppy.

Try it on your repository

greppy trial runs an isolated observation without and with greppy on your own repository, with Pi as the agent runner, and prints a greppy.project-trial.v1 JSON record:

greppy trial --help