Benchmarks

Measured, not claimed.

Headline numbers and reproduction commands. Full methodology — distinct-commit sampling, four baselines, bootstrap confidence intervals, budget sweep, hand-audited failure taxonomy — in benchmarks/EVAL_V2_REPORT.md. The 2026-07 hardening pass — leave-one-repo-out weight validation, a true dense baseline, calibration at n=1080, ground-truth validity, paired significance tests — is in benchmarks/RIGOR_REPORT_2026-07.md; where this page and that report disagree, the report is newer and wins.

Per-Signal Ablation

Each retrieval signal benchmarked alone against real commit history:

Hybrid numbers are at the shipped blend weights [0.3, 0.5, 0.2] — the leave-one-repo-out-validated choice from the 2026-07 rigor pass (the originally shipped [0.5, 0.35, 0.15] was mildly graph-overfit). Note that standalone, plain BM25 beats the call graph.

Against Real Developer Behavior, Across Repos

The core benchmark asks: a developer changed these functions together in one commit; shown one, can the tool find the others? Headline (per-commit hit / recall, hybrid):

Independent validation on repos never used for tuning or weight selection (frozen weights, results/loro/loro_3leg.json):

• black: hybrid hit 0.897 / recall 0.712
• requests: hit 0.953 / recall 0.762
• rich: hit 0.844 / recall 0.760
• starlette: hit 0.929 / recall 0.776

Head-to-Head vs grep, at Identical Token Budgets

The question that actually matters for an agent loop: given the same context window, does this beat what a developer does by hand?

30 real co-change queries from black’s history (n=30, no confidence interval — small sample; the 2.7× headline is the most-quoted number in this project and rests on the thinnest sampling). Recall of the true co-change partners inside the packed window:

Note the shape: grep plateaus — beyond ~4k tokens, more budget buys nothing, because name-matching cannot find co-change partners that don’t mention the name. Graph+BM25 retrieval keeps climbing.

Honesty audit at the tight 2k budget (128 ground-truth symbols): 34% made it into context, 66% were explicitly disclosed as dropped, 0% silently invisible.

I Audited My Own Benchmark, and Three of My Claims Lost

A 2026-07 pass attacked the evaluation instead of the tool. Three published numbers did not survive:

• Calibration — the only citable number (r=0.274, n≈25) was measured on a polluted index. Re-measured clean at n=1,080 the legacy score gets r=0.016 (p=0.60): no relationship at all. Components with zero evidence behind them had defaulted to a perfect 1.0. Fixed by shrinking toward “don’t know” → r=0.287 (p=0.0001) — a ranking signal, not a probability.
• Blend weights — the shipped [0.5, 0.35, 0.15] failed leave-one-repo-out; every fold picked a less graph-heavy blend. Standalone, plain BM25 beats the call graph (0.619 vs 0.558 recall). Now [0.3, 0.5, 0.2].
• Dense baseline — a TF-IDF stand-in had overstated dense retrieval (0.664, beating BM25 5/5). The real MiniLM encoder scores 0.597 and beats BM25 only 2/5. Two prior conclusions corrected on the record.

Plus one null reported as the null it is: adaptive per-query blending changes held-out metrics by nothing (p=1.000).

Raw pass: benchmarks/RIGOR_REPORT_2026-07.md.

Quality Can’t Silently Regress

benchmarks/check_regression.py enforces frozen hit/recall floors and runs in CI on every push. If a change to the heuristics drops retrieval quality below the floors, the build fails.

Reproduce It Yourself

pip install rank-bm25                          # benchmark-only dependency
python benchmarks/benchmark_runner.py --clone  # clone the five eval repos
python benchmarks/eval_v2_hardened.py          # full run (~10 min)
python benchmarks/budget_head2head.py benchmark_repos/black   # grep head-to-head
python benchmarks/check_regression.py          # the CI quality gate (~1 min)

Or skip our repos entirely and grade DiffContext against your repo’s real history:

diffcontext verify --from-history 20 --calibrate

See Verify for the full methodology.

Downstream: Does Better Context Improve LLM Task Outcomes?

Yes — context roughly quadruples pass@1. Measured on 128 ContextBench Python tasks (django, requests, flask), judged by each repo’s own test suite (Django runtests.py; pytest for requests/flask). No LLM-as-judge.

Two qualifiers (see RESULTS.md §6): (a) the seed functions given to every arm are oracle — extracted from the gold patch — so this measures “given correct localization, does context quality matter?”, not end-to-end issue solving; (b) 121 of the 128 effective tasks are django, so this is largely a django result.

Exact McNemar, context vs no-context (all arms p < 0.0001):

The honest companion: the three context variants are statistically indistinguishable from each other. Their Wilson CIs overlap almost completely, and exact McNemar gives p = 0.36 / 0.81 / 0.65 for diffcontext vs gap, diffcontext vs depboost, and gap vs depboost respectively. Only ~19 of 128 tasks are discordant on any context-variant pair — the benchmark is underpowered to resolve a 3pp difference, and more seeds will not help (the limit is discordant-pair count, not run-to-run noise).

Conditioning on successful patch application (the model produced a diff that git apply accepted), pass rate rises from 22% (no context) to 42% (with context): context halves the rate at which the model hallucinates code structure and produces a malformed patch.

Full methodology, provenance, and reproduction: benchmarks/contextbench/RESULTS.md. Every number is regenerated from JSONL by verify_results.py.

Known Limitations

From the failure taxonomy — 60 hand-audited Django co-change pairs with no call-graph connection:

• Thematic siblings (same feature, no call between them): the graph is blind; the BM25 leg recovers these partially. Adaptive per-query blending shipped for this, but the rigor pass measured it as a null (p=1.000) — no significant recall change on any benchmark repo.
• Dispatch/override pairs (same method name across a hierarchy): addressed by synthetic dispatch-sibling override edges, now shipped.
• Cross-subsystem conceptual links (e.g. a settings flag and the security check that reads it): graph, BM25, and hybrid all score 0/20. A structural ceiling for every static-analysis retriever — reachable only with signals like git co-change history, now shipped as an opt-in fourth signal (--with-history).
• Dynamic dispatch (getattr(obj, name)() with runtime name) and metaclass-generated code are statically unresolvable — this is why pydantic is the weakest benchmark repo for every method tested.
• Absolute recall at starvation budgets is low for everyone. At 1,000 tokens, grep manages 0.08 and DiffContext 0.12 — almost nothing fits in 1k tokens, and the meta header says so rather than pretending otherwise.

When in doubt: grep -rn "function_name(" --include="*.py" . before fully trusting “no callers found.”