Contents
Where the Errors Actually Live
A synthetic-first experiment program on SNAP payment accuracy, including the part where the headline numbers were wrong, and what correcting them produced. The views represented here are my own. Last updated: November 2026
Executive summary
SNAP’s payment error rate is the number state agencies are now financially exposed to: under H.R. 1, from FY2028 a state’s share of benefit costs scales with its error rate. I ran 122 experiments toward a small, domain-specific language model that could work as a QC copilot, trained on synthetic QC-style cases with deterministic ground truth and validated against the real QC public-use files. The program produced three things worth reporting, in ascending order of how much I trust them.
- The leak, and the correction. Comparing my triage rankers against Better Government Lab’s published results exposed a leak in the QC public-use file: the “reported” fact fields are the QC reviewer’s corrected values, so a model given both the facts and the issued benefit could partly see the error it was supposed to predict. On their test my rankers scored precision 1.00 with the leak and 0.40 without it. Every figure from the leaky experiments is labeled as such in the repo, the report carries a “Corrected October 26, 2026” banner, and one earlier finding is formally withdrawn.
- The rankers, on an honest basis. Rebuilt leak-free, with the official error definition and the label taken from the reviewer’s finding, per-jurisdiction triage models for all 53 SNAP jurisdictions rank cases with a median top-20% lift of 3.0× over random review. At official FY2025 error rates, 43 jurisdictions would owe roughly $10.6 billion a year under FY2028 cost-share; the central review scenario avoids about $4.47 billion of that. No state is running the rankers yet.
- Where the error lives. Treating the benefit formula as a black box and probing it, then reconstructing real cases: the statutory formula plus each case’s own deduction fields reproduces the QC minimodel on 99.87% of no-earnings cases. Issued benefits match it only 61.6% of the time, and with earnings in the case only 37.0%. The effective parameters fitted from issued benefits come back exactly $198 and 0.30, the statutory values. There is no parameter drift. The error is case-level: above all, a cluster of cases issued a full allotment where verified earnings imply the $23 minimum, earnings that never reached the issued benefit.
1. The question
A payment error has two possible homes. The formula can be wrong (the wrong standard deduction, the wrong percentage, a table a year out of date), or the facts can be wrong: the income, household size, or shelter cost fed into a correct formula was not the household’s real one. The remedies are opposites. Formula errors are fixed once, centrally, by correcting code. Fact errors are distributed across millions of caseworker–household interactions and are fixed, if at all, case by case.
Federal QC data can distinguish the two, if you ask it carefully. The QC public-use files hold roughly 45,000 reviewed cases a year (FY2020–2024: about 167,000), each with the agency’s issued benefit, the reviewer’s corrected facts, the reviewer’s finding, and a coded variance: element, nature, cause, and who was responsible. That file is the ground truth this whole program is validated against. It is also, it turned out, a trap.
2. The synthetic-first design, and its gates
The founding bet: real QC files are a validation resource, not a training one. They contain no reasoning traces, and a model that learns error patterns from real cases learns them with the review process baked in. So the language-model strand trains on synthetic QC-style cases, generated households and case records with planted variances, where a benefit calculator stands in for the QC minimodel and every label is deterministic ground truth.
A bet like that needs its failure conditions written down in advance. The repo’s north-star document sets three gates, and requires every experiment to move one or be recorded as not moving it:
- Calibration: the synthetic marginals (finding mix, variance mix, dollar-error distribution) stay near the real QC distributions. If the generator’s priors drift, anything trained on it inherits the drift.
- Transfer: models trained only on synthetic cases predict real QC labels above baseline. This is the core thesis test: signal must cross the synthetic-to-real gap.
- LM eval: the fine-tuned small LM beats its base model on a QC-variance eval and a policy benchmark.
Calibration held early and tightly (a decisive run: finding-mix distance 0.009; realized error rate 0.396 synthetic against 0.397 real). Transfer held from its first test (AUC 0.656 synthetic-to-real, against 0.614-level element accuracy for a real-to-real control on the companion metric). The LM gate, marked “future” when the gates were written, was met later: a 4-billion-parameter Gemma fine-tuned on synthetic cases only went from 0.5% to 33% overall accuracy predicting real reviewers’ findings on a pre-registered confirmation set of 1,200 fresh cases, clearing its 21% bar on the error-location subtask at 24%. That is a research result, not a product. 33% is well short of a human reviewer, and the half-size model missed its bar. It is reported here because the gate was pre-registered and the result, good and bad, is what the gate produced.
3. The leak
The program’s other strand ranks real cases for review: if an agency can only re-examine a fraction of its caseload, which cases most likely contain errors? I built per-jurisdiction rankers and, doing the obvious diligence, compared them against Better Government Lab’s snap_qc, which models the same files.
Their published results exposed the problem. In the public-use file, the fact fields prefixed RAW (RAWGROSS, RAWNET, RAWERND, and the rest) are not what the agency had. They are what the QC reviewer recorded, which for an error case means the corrected facts. RAWBEN, the issued benefit, was computed from the agency’s original facts. A model shown both can compare them and partially reconstruct the error it is meant to predict. It is the evaluation equivalent of grading with the answer key in the room.
On BGL’s test the effect was not subtle: precision 1.00 with the leak, 0.40 leak-free. Experiments 58 through 105, including the headline ranker figures, were leaky. The correction is part of the published record, and I want to be plain about its size: leak-free, the median top-20% lift fell from 3.97× to 2.70×, and the central cost-share estimate fell from $6.05B to $3.42B on the same any-error basis. A separate earlier result, a “feedback loop” finding, turned out to be the leak wearing a different hat and is withdrawn in the experiment index, by name.
I would rather have found it myself. The second-best outcome is what happened: a public comparison, a same-week correction, and a repo whose index tells you which rows not to believe.
4. The rebuild
The rebuild adopted BGL’s better practices wholesale, because they were better: their reconstruction of pre-review inputs (what the agency actually had before review) as the basis for like-for-like tests; the official over-tolerance error definition (the tolerance is $56 for FY2024, and it is the dollars over tolerance that the error rate counts); and a label taken from the reviewer’s finding rather than computed from the benefit gap. My gap-built label contradicted the reviewer in about 30% of cases, because the gap and the finding are genuinely different measurements.
On that basis (the current default configuration): median top-20% lift 3.0× across all 53 jurisdictions, every jurisdiction at 2.0× or better, Washington highest at 3.90×, Kansas lowest at 2.18×, and top-10% lift 4.1×. On BGL’s pre-review frame, like-for-like, the median lift is 2.61× on a 2024 test and 2.66× three years ahead, with dollar recall at a 10% review rate of 0.34 against 0.26 for BGL’s published rule lists. The comparison is predictions against lists, not model against model: BGL reports that its own boosted-tree models match numbers like ours, and the rule lists are what they publish.
Two honest residuals. First, even leak-free, an error case’s facts in the public file are the reviewer’s corrected values; a deployed tool sees uncorrected agency facts, and the one measurement of that gap (exp109) suggests the reviewer’s values flatter dollar recall by about 17%. The repo’s instruction stands: validate on a state’s own pre-review data before quoting deployment numbers. Second, the rebuild of the shipped rankers onto pre-review inputs (exp114) is still a design draft, not a run. It is the known unfinished step, and the four jurisdictions BGL’s frame omits (Alaska, Hawaii, Guam, the Virgin Islands) are an open decision inside it.
5. Recovering the formula
The last experiment series (116–122) asks the Section 1 question directly. First, on the synthetic side: treating the benefit calculator as a black box, adaptive probing recovered every parameter to the dollar (allotments, the standard deduction table, the $672 cap, the $35 medical threshold, the floored 20% earned-income staircase with zero sequence mismatches), and the recovered formula, compiled to network weights, reproduces the calculator exactly. Trained controls (an MLP, a gradient-boosted model) sit at about $6 mean error and break monotonicity in 7–9% of probes. A formula can be known in a way a fit cannot.
Then on real data. The statutory formula, plus each case’s own deduction fields from the QC file, reconstructs the QC minimodel’s benefit on 99.87% of 29,572 mainland no-earnings cases, and 99.85% of earnings cases. The formula layer is the statute, full stop. Later experiments recovered the territory tables from their own slices (Hawaii and Guam at 100.0%, the Virgin Islands at 97.1%) and identified Alaska’s three schedules from a benefit-max fingerprint at purity 1.000.
Issued benefits are another matter: they match the minimodel 61.6% of the time without earnings in the case and 37.0% with, the mismatch skewed to overissuance (40.0% over, 23.0% under). And the effective parameters fitted from issued benefits come back $198 and 0.30, identical to the minimodel control. No drift, no shadow formula. The residual is case-level variance, and its modal shape is specific: a cluster of 205 cases overissued by exactly $268, the full allotment where verified earnings (median $2,752) imply the $23 minimum benefit. Of those 205, 204 carry a coded variance; the first variance is earned income in 192; responsibility is the client’s in 170 (82.9%) and the agency’s in 34 (16.6%); the dominant natures are an unreported source of income (139) and employment status changed to employed (38).
So roughly five-sixths of the modal error cluster is earnings a household did not report, and one-sixth is an agency that did not act on what it had. That ratio, not a model score, is the finding a program designer can use, because the two sides have entirely different fixes: verification data matches and reporting UX on one side, workload and alerting on the other.
6. Where it stands
The deliverable that matters operationally is the ranker set: 53 jurisdiction models, an interactive map of lift and cost-share exposure, and a whitepaper, all public.12 At official FY2025 rates the FY2028 exposure is a national error rate of 10.62% and about $10.6 billion a year across 43 jurisdictions; a program reviewing 2% of cases monthly, ranked rather than random, avoids a central estimate of $4.47 billion of that (the scenario range is wide: $0.21B at 1% monthly, $8.24B at 3%, and dollar forecasts for the smallest jurisdictions run about 50% high). No state is running the rankers. The fairness floor built into them, which requires elderly households to get their share of reviews, lifts the elderly review ratio from 0.56 to 1.05 overall, and 8 of 52 jurisdictions still sit below 0.8; that is published too.
The LM strand should not be oversold in the other direction either. Its founding non-goal, that real QC microdata never enters its training, still holds, and its gate result is real but narrow: one task, one model family, a third of cases right. The rankers, separately, train on real QC records by design; conflating the two strands’ data rules would be exactly the kind of quiet slide this program exists to catch.
Footnotes
Working on an agentic or LLM-powered system?
I advise teams building these systems — architecture, tool and orchestration design, retrieval, and evaluation for high-stakes, compliance-sensitive domains.
Related Posts
- Synthetic Data Is Only as Good as Its ProvenanceI generate synthetic benefits cases from rules encoded out of primary sources and households sampled from real Census data, so every label is exact by construction. Verifying five years of those labels against the original FNS charts found two states that really did change policy in 2025, and a half-dozen parameters that had been quietly wrong. Synthetic data does not remove the hard problem. It moves it to source verification.11/17/2026
- Using Synthetic Government Benefits Data for Reasoning EvaluationWhy the question isn't 'can the model do arithmetic?' but 'does it know which rule to apply?'. Exploring synthetic data for evaluating reasoning in government benefits policy.3/27/2026
- Weight-Space Programming for the Social Safety NetAn independent test of the claim that AI models can write, read, and edit neural network weights directly, run against US benefit formulas where the correct answer is defined by law. Written weights were exact. Trained weights failed in ways average accuracy hides. And one step of the law cannot be represented at all.11/3/2026