Document retrieval on large cases
Reading one document cost as much as reading the whole case it lived in. That hid documents quietly for eight months, then saturated the service for two hours once the hiding was fixed.
Summary
To return one document, retrieval listed every object stored under the case and filtered the results. The cost of a read scaled with the size of the case, never with the size of the document.
The listing was a single unpaginated call, capped at 1,000 objects. On a large case, a document sorting past that cutoff did not exist as far as retrieval was concerned. The endpoint answered HTTP 200 with an empty result, which looks exactly like a document that genuinely has no text, so nothing alerted for eight months. A solicitor eventually complained that Lawrence kept saying two witness statements had not been processed.
Pagination shipped on 5 August and removed the cap. Retrieval became correct and paid the full price of its own design for the first time: up to 35 sequential S3 LIST calls to find about 10 keys. Requests went from half a second to over a minute, a per-request S3 client pushed the tasks past Fargate's default file-descriptor limit, and the processes started aborting. Scaling from 3 to 6 and then 8 tasks carried the service through two waves of degradation between 15:40 and 17:35.
The fix at 19:09 made a read list only the folders a document can live in, with a fallback to the full listing if that misses. A read now scans 30 objects instead of 15,501, at a p50 of 0.16s, on the original three tasks.
What happened
Retrieval fetches one document by listing its whole case folder in S3 and stopping at the first 1,000 objects. The earliest proof still retained is case cas_l2h6e45x2zdmudxm, which already held 1,899 objects, with the 1,751 bytes of text it was asked for sitting past the cutoff.
Over the following eight months, 86 of the 8,209 cases created in the window were affected. That is 1.05%, about one case in 95. No alert ever fired, because the endpoint answered HTTP 200 with an empty result and that is indistinguishable from a document with no text.
A solicitor asks Lawrence to review two witness statements on case cas_1x9fga55f9ww9tx2 and is told repeatedly that they have not been processed. They had been. The case holds 5,213 objects across 274 files, mostly email traffic, so both statements sat past the 1,000-object cutoff.
Libaan Hassan publishes the root-cause analysis. Four fixes follow: #705 paginates the listing, #706 stops reporting failures as empty results, #707 returns one row per document instead of one per page batch, and platform #8197 covers root selection, tenancy scoping and honest failure messages.
Task definition :101. Pagination makes retrieval correct, and the endpoint pays the full cost of its own design for the first time. The biggest matters hold 34,896 objects, which is 35 sequential LIST calls, and 24,874 objects, which is 25 calls, to find roughly 10 keys.
The first Sentry alerts land on /api/v1/content. Wave one builds across the hour.
Across the 15:00 to 16:00 hour, 92 requests took more than 10 seconds, the slowest at 76.9s.
29 escalating 502 Bad Gateway events, plus BrokenPipeError, ConnectionDoesNotExistError and RetryError: too many 502 error responses from the indexing and preparation workers. The pipeline can no longer reach the management service.
File descriptor 56 is used by transport # uvloop ClientError: ListObjectsV2 # retrieval Failed to retrieve asset from S3 # /ingest
New uploads cannot be processed while this lasts.
Memory sat at 11% to 17% of 4 GB for the whole incident, so this was not an out-of-memory kill.
Work with nothing to do with S3 was also slow: kms-verification p95 11.8s, database SELECT 2.6s, connect 4.75s. Individual S3 calls ran at 3.7s against a healthy 50 to 200ms.
About 95% of the latency was contention rather than work, which is the part that scaling can fix. The database had room, with 25 of 400 connections in use.
get_s3_client is a per-request FastAPI dependency, so every request builds its own client with its own connection pool of 10 sockets, the botocore default. File descriptors therefore scale with how long requests run, not with how many arrive. When duration went from 0.5s to over 60s, concurrent sockets passed Fargate's default limit of roughly 1,024 (the task definition sets ulimits: null) and uvloop aborted the process on file-descriptor reuse.
Capacity had also been pinned at min = max = 3 in flightcontrol.prd.json, so there was no headroom to absorb any of this.
Raised the caller's upstream timeout from 10s to 30s and stopped retrying timeouts. The second part mattered more.
httpx_retries retries timeouts by default, so one slow inbound request became four upstream requests, each re-walking all 35 pages.
One upstream request per inbound request. Its test measures the count directly: four before, one after.
That cut the load fourfold without touching the cause.
No further crash loops. This is 1 hour 55 minutes after the first alert, and it is where user-visible degradation ends. Stable is not the same as fixed: the endpoint was still listing whole cases, on more hardware.
Task definition :103. Retrieval now lists only the folders a document can live in, and falls back to the full listing if that misses, so a wrong guess costs time and never a document. Flightcontrol reset capacity to the configured 3 tasks on this deploy, and 3 was enough.
15,501 objects scanned per read on average, up to 35 sequential LIST calls, on 8 tasks.
30 objects scanned per read, taken by 87.7% of requests (3,192 of 3,640), on 3 tasks.
That is better than the pre-incident baseline, on the original capacity.
What this tells us, and what happens next
The cost of reading one document was proportional to the size of the case it lived in, not the size of the document. That one flaw failed twice. It failed quietly for eight months, capped at 1,000 objects, so documents were invisible and no alert fired. Then it failed loudly for two hours, once pagination removed the cap and the endpoint paid its real price. The truncation was not protecting us from the scaling problem, it was hiding it, and the hiding was itself the customer-facing bug. Retrieval now costs what the document costs.
The change was reviewed four times, by humans and by bots, and every review checked correctness while none checked cost. Nothing measured how much work the endpoint did per request until the telemetry that made diagnosis possible shipped inside the offending change itself.
The cause of the file-descriptor exhaustion is still in the code, which is why it leads the list.
| Item | Why | Owner | Priority | Status |
|---|---|---|---|---|
| Share the S3 client across requests instead of building one per request | The cause of the file-descriptor exhaustion is still in the code. | AI platform | High | Open |
Set an explicit nofile ulimit and alarm on descriptor usage |
The failure currently surfaces as an opaque SIGABRT rather than naming itself. | AI platform | High | Open |
| Load-check the largest real matter before changes to hot read paths | Correctness was tested, cost was not. | AI platform | High | Open |
| Record derived-artefact keys at write time so reads stop listing S3 | Makes a read O(1) and removes this class of failure. | AI platform | Medium | Open |
| Alert on quiet wrong answers, not only on errors | A defect affecting one case in 95 ran for eight months because empty looks like success. | AI platform | Medium | Open |
| Clean up 54,908 duplicate page-batch rows and fix the retry path that creates them | 6,873 of 84,229 batched documents (8%) carry duplicate sets, inflating storage and listings. | AI platform | Medium | Open |
Extraction that yields no text should not report SUCCESS |
A scanned statement still reads as processed with almost no text, so the original complaint is not fully resolved. | AI platform | Medium | Open |