Case-context saturation on a normal Monday

lawrence-engine ran at 100% CPU for three hours under ordinary Monday traffic. Chat, case summaries, the admin, and a form upload all timed out against it. Nothing alerted; the incident was found because two people watched a form not appear.

Summary

For weeks, a record-keeping remediation (Project Hoover) has been filing old emails onto old cases. By 10 August, 444 cases carried 200 or more emails, and the statement lists on the biggest cases had outgrown every model's context window. Those cases failed case-context regeneration instantly and silently: an over-limit request is rejected before any work happens, so the failure cost nothing and produced nothing. Sixteen cases had been stuck this way since April.

PR #699 fixed that correctly: statements now fold into the case-context object (CCO) in token-bounded batches, so the stuck cases can finally complete. The cost is that a case which used to fail in zero seconds now runs a sequence of LLM passes inside one long HTTP request. The PR predicted 10 to 30 minutes for the worst case. It shipped on Friday 21 August at 15:05, in the first prod deploy since 31 July, into a quiet weekend that never exercised it.

Monday's ordinary traffic was the first real load. Every filing or update that touched a fat case started a multi-pass fold, and each fold occupied one of the service's two 1-vCPU asyncio event loops for tens of minutes. Health checks and interactive endpoints starve behind a busy loop, so the ALB returned 502s, Inngest retried, and the retries kept the service pinned from 07:30 until mitigation. The longest single request ran 95 minutes.

Mitigation was capacity and slot recovery: the service went from 2 tasks to 4 at 10:44, and the two CPU-pinned tasks were recycled at 11:39 and 12:04 so their stuck Inngest concurrency slots could retry onto fresh tasks. The backlog of roughly 1,700 queued runs drained to zero by 14:03. The service is holding at 4 tasks; the follow-ups below decide what happens after that.

The evidence: traffic was normal, the engine was not

Every input to the engine ran at its usual Monday level. Email filings through the Outlook add-in, statement extractions, and ingestion volume all sat inside their two-week weekday band. The two anomalies are on the output side: the number of case-context requests running over two minutes, and CPU. In the two weeks before this incident the busiest hour of any day never exceeded 56% CPU; on the 24th it hit 98%, with three hours above 80%. Monday's bars are partial, covering 00:00 to about 15:00.

The per-request cost tells the same story. The case-context endpoint averaged 3 to 4 seconds per request every day for a week; on the 24th the average was 14.7 seconds at a normal request count. Pre-saturation hours (00:00 to 07:00) averaged a normal 2.97 seconds, so the code did not get uniformly slower. What changed is the tail: 1,488 requests ran over two minutes (weekday range before: 270 to 1,054), and the longest ran 5,732 seconds against a prior two-week max of 1,805.

Before and after the fix, in Inngest's own numbers

Comparing every run of the two case-context functions across the deploy boundary (before: Mon 17 to Fri 21 15:05, on the old code; after: Fri 21 15:06 to Mon 24 15:30) confirms where #699's cost actually landed. Runs that complete take the same time as before. What changed is that failure collapsed into work.

Three things follow. First, the fix behaved as its PR claimed: typical folds are unchanged (p50 within 10%), and the context-window failures stopped (13 CCO failures before, 1 after). Second, the before world was quietly discarding roughly 350 asset extractions a day as failures; the after world does that work instead, which is the unfrozen backlog in one number. Third, the Monday-slice duration explosion is the incident measuring itself: an Inngest run's duration includes retry backoff and queuing against starved tasks, so runs alive during the saturation stretched by construction. The pre-saturation hours, at a normal 2.97s per request, are the cleaner evidence that the code's intrinsic cost did not jump.

One more number for the capacity decision: on the old code, the previous Monday's 09:00 to 11:00 slice already ran at a p50 of 6.7 minutes per asset run with a p90 of 21 minutes. The service was near its Monday-morning edge at 2 tasks before the deploy; #699 added the unfrozen failures on top rather than pushing a comfortable system off a cliff.

The remaining hump, measured in the case-context database at 16:30. Of 15,306 cases with 300 or more statements, 12,042 are fully folded and only 3 have no CCO at all. What is left folds lazily, one case at a time, whenever a case is next touched:

187,789 statements are pending in total across 6,385 stale cases. The tail is finite, lazy, and absorbable at 4 tasks; the consolidation follow-up is preventive, not firefighting.

What happened

Root cause

No single change was wrong. The incident is the composition of four things, each reasonable alone.

The amplifiers, in order of contribution:

1. One event loop serves everything. Each task is one Python asyncio process on 1 vCPU, and it serves health checks, chat retrieval, form endpoints, and Inngest step execution together. Any long CPU-holding step makes the whole task unresponsive, which is why a background fold took down a form upload.

2. Failure manufactures load. A starved loop fails its ALB health check and drops outbound LLM connections. The 502s make Inngest retry whole steps, and a failed asset is deliberately left unprocessed for the next event to retry. Saturation therefore feeds itself until capacity changes.

3. Concurrency slots pin to sick tasks. Inngest's 50-slot cap is the throttle protecting the service, but a slot is only freed when its step finishes. Slow steps on pinned tasks held the slots, so adding fresh tasks changed little until the pinned tasks were recycled. Capacity and slot recovery had to happen together.

4. The fat cases were grown upstream. The email back-fill is legitimate remediation work, but it concentrated hundreds of assets and thousands of statements onto cases whose CCOs nothing ever consolidates. The worst CCO's events array grew from 426k to 498k characters with no pruning, and CCO size directly sets fold cost.

Impact

From roughly 07:30 to 12:00, anything that called lawrence-engine synchronously degraded: admin case-context views timed out, lawrence-api logged fetch failures, chat and case summaries were slow or failed, and the forms admin upload was unusable. Background case-context processing was delayed by up to several hours. No data was lost: Inngest retries durably, the failed form upload left no partial state, and the same upload succeeds now. The clearest user-visible cost was silent, which is its own finding: the form upload appeared to do nothing, and only a human noticing that started the response.

What the investigation got wrong first

Three confident theories died on baselines, and the record should keep them because each sounded right at the time.

A weekend filing blitz. Mid-investigation the day was mistaken for Sunday, which made 481 filings look like ten times a normal weekend and implied a coordinated push by a dozen people. The 24th is a Monday. Its filing volume was ordinary, and the "dozen people on home IPs" were lawyers doing their jobs.

A bulk-import pipeline firing ingestion. The Hoover data pipeline writes records directly and deliberately skips AI ingestion, and it was not running on the 24th. Ingestion volume for the day was normal for a weekday.

A traffic surge at all. Every input series, compared against its own two-week history, was in range. The only anomalous series were request duration and CPU. The lesson is mechanical: pull the two-week baseline for a metric before treating its current value as a surge, and check what day it is.

Follow-ups