Benchmarking the deprecated prod model against the GPT-5.6 family on 39 real documents, through the exact production code path.
Set summarisation to gpt-5.6-luna @ medium. It is a pure parameter change (no code, no deploy beyond a restart), roughly 3× faster than prod today, and nearly halves judge-flagged hallucinations. gpt-5.6-terra @ medium is the quality-first alternative at 2.4× the cost. The current model, gpt-5-mini, shuts down on December 11, 2026 and lost on every metric except raw price — and that price retires with it. OpenAI’s suggested drop-in, gpt-5.4-mini, was the weakest candidate head-to-head, so don’t take the default migration path.
Ingestion generates two summaries for every document it processes (a structured ≤500-word summary and a 1–3 sentence concise one), plus a vision description for images. All three run on OpenAI models. The summarisation model in production is gpt-5-mini at reasoning effort low, and its snapshot is shut down on December 11, 2026. We have to move; the question is where to.
Two findings from scoping worth keeping: ingestion’s models are not defined in Langfuse prompt configs (both prompts have empty config objects) — they come from WORKER__TEXT_SUMMARISATION_* env vars via Parameter Store, so a model switch is a parameter change plus restart. The image path is different: analyze_image.py hardcodes gpt-5.4-mini in code, and that model is not in the December shutdown, so vision has no deadline.
Corpus. 50 text documents: 31 real production PDFs (leases, affidavits, court filings, USPTO correspondence) plus S3 edge cases from the existing eval corpus — Urdu, Polish and Welsh forms, filled forms, DOCX and TXT. 39 had usable extracted text (the rest are corrupted or image-only PDFs, recorded as skipped). 4 scanned images (150dpi form scans, a screenshot) feed the vision task.
Execution. Every config ran through the real pipeline code: the two prompts fetched live from Langfuse at the production label, executed by the same split_and_aggregate library prod uses (litellm underneath), with per-call token usage and latency captured. Latency is total call time; ingestion is batch and non-streaming, so time-to-first-token is deliberately not measured.
Held constant to isolate the model variable: context window pinned to 250k for every config (prod’s value), identical text extraction, identical prompts, temperature 1.0.
Scoring. Deterministic checks (word-limit and sentence-count compliance, structure present) plus an LLM judge — gpt-5.5, outside the candidate family to reduce self-preference. Each output gets rubric scores 1–10 for coverage, faithfulness and structure with hallucination flags; each main summary is also judged head-to-head against the prod baseline, twice with positions swapped to cancel order bias.
Judge robustness. The three decision-relevant configs were re-judged with a cross-provider judge (Anthropic’s claude-opus-4-8, no stake in an all-OpenAI field). The ranking held and sharpened: hallucination-flagged docs 12/39 for the baseline vs 2/39 for luna@medium, faithfulness 8.92 vs 9.41. The only judge-sensitive axis was pairwise preference — the same verbosity effect already discounted below — so the recommendation does not rest on gpt-5.5’s opinion.
| Config | Role | $ in / out per 1M tokens |
|---|---|---|
gpt-5-mini@low | Prod baseline — deprecated 2026-12-11 | 0.25 / 2.00 |
gpt-5.4-mini@low | OpenAI’s recommended replacement | 0.75 / 4.50 |
gpt-5.6-luna @ none / low / medium | Candidate — fast tier | 1.00 / 6.00 |
gpt-5.6-terra @ none / low / medium | Candidate — balanced tier | 2.50 / 15.00 |
gpt-5.6-sol@low | Quality-ceiling reference | 5.00 / 30.00 |
| Config | Coverage | Faithfulness | Structure | Halluc. docs | ≤500w compliant | $ / 1k docs |
|---|---|---|---|---|---|---|
gpt-5-mini@low (prod) | 8.54 | 8.41 | 9.21 | 22 / 39 | 59% | 5.63 |
gpt-5.4-mini@low | 8.54 | 9.00 | 9.31 | 13 / 39 | 92% | 14.25 |
gpt-5.6-luna@none | 8.77 | 9.15 | 9.51 | 13 / 39 | 82% | 18.72 |
gpt-5.6-luna@low | 8.64 | 8.90 | 9.31 | 14 / 39 | 85% | 19.07 |
gpt-5.6-luna@medium ← recommended | 8.77 | 9.03 | 9.31 | 12 / 39 | 85% | 19.63 |
gpt-5.6-terra@none | 8.64 | 9.00 | 9.46 | 13 / 39 | 87% | 46.26 |
gpt-5.6-terra@low | 8.82 | 9.00 | 9.54 | 14 / 39 | 90% | 46.67 |
gpt-5.6-terra@medium | 8.85 | 9.00 | 9.62 | 14 / 39 | 87% | 46.80 |
gpt-5.6-sol@low | 8.90 | 9.10 | 9.46 | 10 / 39 | 97% | 94.94 |
Reading the head-to-head numbers honestly. In pairwise judging the old baseline “wins” against most candidates — but that is verbosity bias, not quality. It averages 484 words per summary against 379–409 for everything else, and only 59% of its summaries respect the 500-word limit the prompt itself declares (overflow is truncated in production). The judge rewards the extra detail in a side-by-side read; the rubric, which scores each summary against its source, tells the opposite story: worst faithfulness, most hallucination flags. We weight the rubric, compliance and latency — the metrics that reflect what production actually ships.
Same shape, compressed. The baseline again over-writes (72% sentence-count compliance); gpt-5.6-luna@medium has the fewest hallucination flags (5/39) at p50 1.2s vs the baseline’s 3.2s. gpt-5.4-mini posts the lowest coverage score anywhere in the run (6.67) — its concise summaries drop material facts.
| Config | Coverage | Faithfulness | Halluc. docs | 1–3 sent. compliant | p50 latency | $ / 1k docs |
|---|---|---|---|---|---|---|
gpt-5-mini@low (prod) | 7.59 | 9.00 | 7 / 39 | 72% | 3.22s | 3.95 |
gpt-5.4-mini@low | 6.67 | 9.13 | 7 / 39 | 97% | 1.36s | 11.21 |
gpt-5.6-luna@medium ← recommended | 7.21 | 9.21 | 5 / 39 | 92% | 1.17s | 15.03 |
gpt-5.6-terra@medium | 7.33 | 9.21 | 6 / 39 | 95% | 1.57s | 38.10 |
gpt-5.6-sol@low | 7.10 | 9.10 | 5 / 39 | 92% | 2.34s | 74.15 |
gpt-5.6-luna@low beats the hardcoded gpt-5.4-mini on speed (p50 4.3s vs 6.1s) at similar cost, but four images is a signal, not evidence. Since gpt-5.4-mini is not deprecated, there is no deadline here: either expand the image corpus and re-run before deciding, or switch to luna@low for family consistency whenever analyze_image.py is next touched (it needs a code change either way).
| Config | Mean words | p50 latency | p95 latency | $ / 1k images |
|---|---|---|---|---|
gpt-5.4-mini (current, hardcoded) | 567 | 6.05s | 7.48s | 5.34 |
gpt-5.6-luna@low | 474 | 4.32s | 5.78s | 6.90 |
gpt-5.6-luna@none | 510 | 4.69s | 11.01s | 6.75 |
gpt-5.6-terra@low | 524 | 7.07s | 8.43s | 17.48 |
Production runs roughly 190k documents a month through summarisation (Axiom, trailing 30 days: ~178k per-file runs plus ~15k batch runs), each generating both summaries. At the measured per-document costs:
| Option | $ / 1k docs (both prompts) | ~$ / month at 190k docs | vs today |
|---|---|---|---|
gpt-5-mini@low — today, retired Dec 11 | 9.6 | 1,850 | — |
gpt-5.4-mini@low — cheapest migration | 25.5 | 4,900 | +3.0k |
gpt-5.6-luna@medium ← recommended | 34.7 | 6,700 | +4.8k |
gpt-5.6-terra@medium | 84.9 | 16,400 | +14.5k |
Three things this table settles:
gpt-5-mini’s $0.25/$2). The choice is between migration options, not against today’s bill.gpt-5.4-mini, which measured weakest head-to-head and drops the most facts from concise summaries; it saves ~$1.8k/month against luna.none/low/medium on the same model, per-doc cost moves by about 5% (luna: $18.7 → $19.6 per 1k on the main summary). Pick effort for quality and latency, not cost.A model-independent lever if the delta matters: the pipeline currently sends the full document text twice, once per prompt. A single call producing both summaries would pay for input once — roughly 40% off any option above (luna lands near ~$4k/month). That is a small processing-task change, separate from the model decision.
gpt-5.6-luna@mediummedium fits the existing effort config type: parameter change only — WORKER__TEXT_SUMMARISATION_LLM_MODEL + _REASONING_EFFORT for the three worker services, context window stays 250kgpt-5.6-terra@mediumgpt-5-mini: hard deadline Dec 11, and it is dominated — worst hallucination rate, worst word-limit compliance (59%), slowest (p50 13.1s).gpt-5.4-mini (OpenAI’s suggested replacement): weakest candidate in head-to-head (7W/30L) and lowest concise-summary coverage in the whole run.gpt-5.6-sol: 17× prod’s cost for a marginal quality gain over terra.Follow-ups. If a future config wants effort none, xhigh or max, the Literal["low","medium","high"] type on the worker config needs widening first. Separately, the 5.6 family’s 1M-token window could reduce chunking for very large documents; that was deliberately pinned to 250k here and deserves its own experiment.