Per-stage benchmark of the December migration for our highest-volume LLM pipeline, run by replaying real production generations.
Per-stage verdict: extraction and factuality move to gpt-5.6-luna; chain-of-key CCO generation moves to gpt-5.6-terra@medium; chain-of-key CCO reconciliation — the dominant stage — moves to gpt-5.5@low, because terra fails to produce cleanly-applicable reconcile operations on up to 58% of cases (an objective disqualification, not a judging call). Materiality scoring is deferred: it has no December deadline and every candidate flips 20–30% of material/not-material decisions. And 8 of the audit's 20 at-risk prompts turn out to be dead code — delete, don't migrate.
Case context is the pipeline that turns raw matter documents into the structured Case Context Object (CCO) behind Lawrence. Every stage is an LLM call whose model lives in a Langfuse prompt config — which is why the December 11 shutdown of gpt-5 / gpt-5-mini hits it for ~99.9% of our exposure, and why the migration itself is a set of prompt-version edits. The question this benchmark answers is which model goes into each config.
No synthetic corpus. Langfuse stores every production generation with its fully-compiled prompt and output, so we harvested recent real generations per prompt (85 per extraction/factuality prompt, 40 per CCO prompt, 60 per materiality prompt, stratified UK/US) and re-sent the exact same compiled messages to each candidate model. The production output, latency, and token usage are the baseline — no re-runs of the old models needed.
Objective metrics carry the load. Schema validity against the module's own pydantic contracts; statement-ID grounding (every supporting fact must reference a real input statement); whether chain-of-key reconcile operations actually apply using the module's own apply code; factuality agreement vs the baseline; materiality decision flip-rate. LLM judging (position-swapped pairwise, gpt-5.5 primary with a claude-opus-4-8 cross-provider pass) covers what objective checks can't.
Scope finding first: 8 of the audit's 20 at-risk prompts have zero 30-day volume — dead code to delete, not migrate:
case_context_document_context_extractioncase_context_document_reconciliationcase_context_summary_generationcase_context_summary_generation_USstatement_based_cco_generation_initialstatement_based_cco_reconciliationstatement_extraction_with_disambiguationstatement_factuality_classification
Exactly as in the ingestion benchmark, OpenAI's suggested drop-in gpt-5.4-mini is the weakest candidate: it loses pairwise against today's gpt-5-mini output under both judges, while gpt-5.6-luna wins under both.
Picks: gpt-5.6-luna (none or low) for extraction, gpt-5.6-luna@low for factuality. Both run 2–3× faster than today's prod calls.
Every candidate beats today's gpt-5 output here — the stage is a free upgrade. terra@medium takes it on the balance: near-zero losses, 100% schema validity, 97–100% statement grounding, half the latency of gpt-5.5 at 2.4× lower cost. One behavioural note: terra writes leaner CCOs (fewer entities/events than the old model's bulk) — the judges prefer them anyway.
This is 17.2k generations/month and the heart of case context: the model emits update/add/remove operations against the existing CCO. An operation set that doesn't apply cleanly is a failed generation, whatever a judge thinks of its prose. That metric is objective — we run the module's own operation-apply code — and it disqualifies terra outright:
Between the two safe options, gpt-5.5@low wins: judge-preferred or tied under both judges (gpt-5.5 judge: 24/49/6 uk; claude judge: 7/31/1 — mostly ties, no regression), 20% faster than today's gpt-5 (0.81× p50), while sol runs at 1.05–1.07× baseline latency with worse UK verdicts for the same money. Since gpt-5.5 is both a candidate and the primary judge here, the pick deliberately rests on the objective ops-apply gate plus the cross-provider judge — not on self-preference.
Materiality runs on gpt-4.1-mini — not in the December shutdown — and every candidate, including the same-family gpt-5.4-mini, flips 20–30% of material/not-material decisions with only 0.47–0.64 score correlation. Since this gate decides whether a CCO regenerates at all, that's a product-behaviour change. It needs a curated gold set and product signoff, on its own timeline.
| Stage | Recommended | $ / month (volume-weighted) |
|---|---|---|
| Extraction (65k gens/mo) | gpt-5.6-luna@low | ~2,500 |
| Factuality (62k gens/mo) | gpt-5.6-luna@low | ~340 |
| CCO initial CoK (519 gens/mo) | gpt-5.6-terra@medium | ~60 |
| CCO reconciliation CoK (17.2k gens/mo) | gpt-5.5@low | ~6,300 |
| Materiality (88k gens/mo) | deferred — stays gpt-4.1-mini | unchanged |
| Migrated total | ~9,200 |
Reconciliation dominates because its inputs (the existing CCO + new statements) routinely exceed 100k characters. Terra would have halved that line but is unsafe there. If that number matters, the lever is input size — the existing-CCO payload — not the model choice. And as with ingestion: today's prices retire with today's models, so "keep paying the current bill" is not on the menu.
gpt-5.6-luna; uk/us factuality → gpt-5.6-luna@lowgpt-5.6-terra@mediumgpt-5.5@lowfallback_models; re-run the harness on new prompt versions before promoting to productiongpt-5.4-mini anywhere — loses head-to-head under both judges, twice-confirmed pattern across two servicesterra on reconciliation — 32–58% of its operation sets fail to apply