diffcontext verify

compile answers “here is relevant context.”
verify answers the harder question: “is this context sufficient — and how would you know?”

It has three modes, each one step further up the evidence ladder:

The Honesty Contract

What can be measured statically are the known structural predictors of insufficiency:

1. Direct-neighbor closure — a caller/callee of a changed symbol that is not in context is the strongest predictor of a wrong or hallucinated patch.
2. High-score retention — symbols the ranker itself scored as relevant but the token budget cut. The ranker is telling you its own output is incomplete.
3. Local graph confidence — unresolved calls out of the changed symbols (externals, dynamic dispatch) mean the graph may be blind to real dependencies.
4. Parse health — files with SyntaxErrors are invisible to the graph.

The score becomes calibrated confidence only after --calibrate maps score buckets to empirically measured recall on your repo.

Mode 1: Sufficiency Report

# For the last commit's changes:
diffcontext verify --ref HEAD~1
 
# For a hypothetical change:
diffcontext verify --changed ./auth.py:validate_jwt --max-tokens 8000
 
# Machine-readable, for CI or a harness:
diffcontext verify --ref HEAD~1 --json

Example output:

=== DIFFCONTEXT SUFFICIENCY REPORT ===
Verdict : ⚠ DEGRADED  (structural score: 71/100)
  direct-neighbor closure : 83%  (2 missing)
  high-score retention    : 59%  (9 relevant symbols cut by budget)
  local graph confidence  : 100%
  parse health            : 100%

FINDINGS:
  ✗ [missing-direct-neighbor] 2 direct caller(s)/callee(s) of the changed
    symbols are NOT in context. ... Remediation: raise --max-tokens or --top-k.
      - ./api.py:get_user
      - ./middleware.py:check_auth

The exit code is 0 only for SUFFICIENT, so CI can gate on it:

# .github/workflows/context-gate.yml (sketch)
- run: pip install git+https://github.com/trakshan-mishra/Diffcontext.git
- run: diffcontext verify --ref origin/main --repo .

Verdicts:

• SUFFICIENT — score ≥ 80
• DEGRADED — score ≥ 55
• INSUFFICIENT — score < 55

Weights: 45% direct closure, 30% high-score retention, 15% local confidence, 10% parse health — direct closure dominates because a missing direct neighbor is the failure mode the eval_v2 benchmark observed most often.

Mode 2: Your Own Test Cases

A test case states something you know to be true about your repo: “when validate_jwt changes, a correct context must include get_user.” You know these because you wrote the code, fixed the incidents, reviewed the PRs. The tool is then graded against your knowledge, not its own.

Case File Format

JSON (always works) or YAML (if PyYAML is installed):

{
  "version": 1,
  "defaults": { "budget": 10000, "depth": 2, "top_k": 20, "min_recall": 1.0 },
  "cases": [
    {
      "name": "jwt-validation-change",
      "task": "tighten JWT expiry validation without breaking session refresh",
      "changed": ["./auth.py:validate_jwt"],
      "must_include": ["./api.py:get_user", "./middleware.py:check_auth"],
      "must_exclude": ["./billing.py:invoice_total"],
      "budget": 8000,
      "min_recall": 1.0
    },
    {
      "name": "order-total-refactor",
      "changed": ["./orders/pricing.py:compute_total"],
      "must_include": ["./orders/checkout.py:finalize", "./orders/tax.py:tax_for"]
    }
  ]
}

diffcontext verify --cases cases.json           # human-readable
diffcontext verify --cases cases.json --json    # for scripts

Exit code is 0 only if every case passes.

What Makes a Good Case

• Write cases from incidents. “The bug in PR #212 happened because the agent didn’t see refresh_session” → that’s a case, verbatim.
• One behavior per case. Three focused cases beat one case with nine must_include entries — failures stay diagnosable.
• Add must_exclude for your known false-positive magnets so precision regressions get caught too.

Mode 3: Calibration

# Mine up to 30 real cases from your git history and grade against them:
diffcontext verify --from-history 30 --calibrate
 
# Or generate them to a file first, prune the noise, then run:
diffcontext verify --from-history 50 --out cases.json
diffcontext verify --cases cases.json --calibrate

History cases come from co-change ground truth: if a past commit modified alpha() and beta() together, that’s external evidence they’re related.

--calibrate answers the meta-question: does the structural sufficiency score track measured recall on this repo?

=== CALIBRATION: structural score vs measured recall ===
Cases: 1,080
  score  40-60 : n=142  mean recall  34.1%  ######
  score  60-80 : n=503  mean recall  48.2%  #########
  score  80-100: n=435  mean recall  61.7%  ############
Pearson r (score vs recall): +0.287  (p=0.0001)
→ Moderate positive relationship. Treat the score as a coarse ranking signal...

That output is from a clean re-measurement (n=1,080 cases across all 9 repos). An earlier version of this page showed r=0.274 on n=25 — that number was measured on a polluted index (components with zero evidence defaulted to a perfect 1.0). It has been retracted. The tool measures itself against evidence it didn’t choose and reports the result even when unflattering. If calibration comes back flat or negative, verify says so in plain text (“NULL RESULT”) — a proxy that doesn’t track reality on your repo should not be trusted there.

How to Measure “Model Accuracy” Honestly

Q1 is a proxy for Q2 — a necessary-but-not-sufficient condition. The honest pipeline: maximize and measure Q1 → calibrate the structural score against Q1 → periodically spot-check Q2 on a small fixed task set with a real model.