Production incident analysis of successfully processed documents being returned as missing or incomplete. The retained evidence now shows confirmed pagination-affected retrievals dating back to 5 December 2025.
ACTIVE. The other 56 affected cases pre-date the window but experienced a confirmed failure during it.
The retained Platform logs contain 951 times Lawrence asked for a file and received no usable text. Those requests cover 644 unique case-and-file combinations across 359 legal matters. I joined those IDs to PostgreSQL and reconstructed the S3 page that existed at each failure time using 1,096,488 retained object records.
Every matching text object for the requested file sorted after the first 1,000 S3 objects. In 271 combinations, a non-empty exact root text already existed; in six, only non-empty child-batch text survived under the root prefix.
These 277 combinations span 142 legal matters. The first retained example is from 5 December 2025.
The first surviving non-empty root text is timestamped after Lawrence asked: the median delay is 76 seconds, 90% appeared within 341 seconds, and the longest delay is 7,180 seconds.
This is the group where “it may still be processing” is supported by the retained timing evidence.
PostgreSQL had no V3 root with that file ID in the same legal matter. These may include legacy records, stale IDs or cases whose historical rows are no longer retained.
They cannot be classified as pagination from the evidence that remains.
These include roots deleted before the request, unsupported or zero-byte text, failed or incomplete processing, and one indexed root whose text was on page one but still returned empty.
The expandable breakdown below keeps these states separate rather than calling all of them “still processing”.
277 + 120 + 186 + 61 = 644 case-and-file combinations. They produced 447 + 142 + 253 + 109 = 951 failed retrievals because Lawrence asked for some files more than once.
A failed retrieval here means Platform received HTTP 200 but could not find non-empty plain_text for the selected file. It does not by itself prove that ingestion was still running.
cas_… ID.cas_… matter paired with one uploaded casfil_… record. Uploading the same document again creates another file ID, so this is not a count of unique real-world documents.86 of 8,209 Platform cases created from 5 December 2025 at 14:28 UTC through 4 August 2026 at 10:47 UTC were confirmed affected. That is 1.05%, or roughly one in every 95 cases created during the period.
All 86 were still in an open state when the query ran. Of those, 69 were ACTIVE; the other 17 were pending, opening or submitted for closing.
The historical investigation found 142 affected legal matters. 86 were created during the confirmed window; 56 had been created earlier but suffered a proven pagination failure during the window.
All 142 IDs resolved in the Platform Case table. Platform does not retain an openedAt field or case-status history, so this comparison uses createdAt and current status rather than inventing an opening date.
Platform: 86 of 8,209 cases created in the window were affected, or 1.05%. Ingestion: 142 of 20,807 matters with a V3 root present during the window were affected, or 0.68%; among matters first seen by V3 in the window, it was 95 of 11,214, or 0.85%.
The Platform figure is the useful case-opening comparison. The ingestion figures describe the document estate. None of these is the percentage of cases where Lawrence attempted document retrieval, which would require a separate Axiom denominator.
Case cas_l2h6e45x2zdmudxm
File casfil_9f64o3by956bg4xy
Root cfaa9382-3e43-4051-9e86-198e321d8657
Lawrence failed twice at 14:28 UTC. The matter already had 1,899 S3 objects and the non-empty 1,751-byte root text was beyond the reconstructed first-page boundary.
Case cas_1x9fga55f9ww9tx2
File casfil_bqvs1dzdzly23i14
Root a19e969a-65d7-4b0b-9141-1e406cd742ae
Lawrence failed at 10:47 UTC. The matter had 5,025 objects and the complete 29,230-byte root had existed for 29 minutes, but its a19e… prefix sorted beyond page one.
Case cas_lq2ronf5cn7rs3tf
File casfil_oo8zckee3brvfgak
Root baf8e1cd-cbde-4fdf-b751-61ece0156a45
This file had no child rows. Its non-empty 5,901-byte root existed before the 09:35 UTC failure, but the 1,207-object matter was truncated at page one. This independently rules out child selection as the explanation.
Case cas_1x9fga55f9ww9tx2
File casfil_w558a2omemh6dfcm
Selected child f9ecbfb1-aad4-4f8e-b0a5-d2c8e72010b0
Platform returned a 711-byte child batch instead of the 29,230-byte root. This was logged as success because some text was present, so it is not counted in the 951 empty retrievals.
The S3 bucket does not have versioning enabled. Deleted objects cannot be reconstructed, and overwriting a key replaces its old LastModified timestamp. That can hide old pagination evidence or make an overwrite look like delayed processing. I have therefore counted only cases the retained database and object history can prove.
If Platform returns a small child batch, the request is logged as a success because some text exists. The 711-character example therefore sits outside the totals above.
This historical analysis measures empty retrievals. It does not provide a complete count of documents that returned incomplete text.
Lawrence never reads the uploaded PDF directly. It asks Platform for a caseFileId; Platform calls ingestion-search; ingestion-search calls ingestion-management; management finds asset rows in PostgreSQL and loads derived plain-text objects from the V3 ingestion S3 bucket.
caseFileId and asks for document content.matter_ids and external_asset_ids./assets/content.matters/<case>/ once. Maximum: 1,000 keys. No continuation..find() can select a child, or return null.In plain English: ask for one folder, open the whole filing cabinet, read only page one, then claim later pages may still be processing.
ingestion-management content router
where(
or_(
Asset.matter_id.in_(matter_ids),
Asset.external_asset_id.in_(file_ids),
)
)
The matter clause is true for every asset in the case, so the file clause no longer narrows anything.
list_response = await s3_client.list_objects_v2(
Bucket=bucket,
Prefix=f"matters/{matter_id}/",
)
all_keys = list_response.get("Contents", [])
No paginator and no continuation token means every matter is capped at the first 1,000 derived objects.
cas_1x9fga55f9ww9tx2This is the production timeline for the matter that triggered the investigation. The files were uploaded twice because Lawrence kept reporting they were unavailable. The evidence shows two separate browser submissions five minutes apart, not an automatic ingestion retry.
Axiom + PostgreSQL + S3 Both witness statements received V3 roots and derived plain text. The full root for statement B existed by 10:12:55.
Axiom + PostgreSQL + S3 Platform has no filename/hash idempotency guard. The duplicate of statement B had complete root text by 10:17:58.
Langfuse + PostgreSQL + S3 The model chose unrelated historical IDs retained in conversation context. Both roots sorted beyond S3 page one, so retrieval returned no content and Lawrence diagnosed a processing delay.
Langfuse + Platform + S3 Statement A returned 7,941 wrapped characters. Statement B returned 771 wrapped characters containing only exhibit labels and an image marker.
Langfuse Lawrence correctly observed that its extracted payload lacked narrative, but that did not mean the PDF lacked narrative.
Langfuse + Axiom + S3 Its complete 29,230-byte root existed, but root a19e… sorted after the first-page boundary 36e….
PostgreSQL + S3 It reused the root, appended another six child rows and emitted success. The same unpaginated retrieval path still could not reach the root.
This second timeline records how I moved from the customer report to a systemic classification. Times are BST.
A repeated document-review failure, multiple uploads and a formal complaint triggered on-call investigation.
I confirmed four successful ingestion events, mapped the upload IDs and separated wrong-ID selection from the later partial and empty retrievals.
The original prefix contained a 29,230-byte root and a 711-byte child. This ruled out image-only PDF, OCR and aggregation failure.
Its first 29,230-byte root existed at 10:17:58, before the failed 10:47 request. Reingestion later wrote the same root output again.
A single-page S3 request returned KeyCount=1000, IsTruncated=true and a continuation token. The target root sorted after that page.
Root IDs, child counts and matter sizes were collected for every recent no-content file, followed by read-only root-prefix and page-boundary checks.
Root-only files confirmed pagination independently of the child-selection defect. All 36 file records in the first sample were classified: 28 pagination, four processing races, three missing artefacts and one legacy file.
This example is from matter cas_1x9fga55f9ww9tx2, statement B. The original upload was casfil_w558a2omemh6dfcm; the duplicate upload was casfil_bqvs1dzdzly23i14.
Bytes are the stored size of each extracted .txt object in S3. Characters are the length of the text string Platform handed to Lawrence.
The complete root was 29,230 bytes. The selected child batch was only 711 bytes, about 2.4% of the root and roughly 41 times smaller. Platform logged 711 characters because the fragment was predominantly ordinary text; Langfuse recorded 771 characters after Lawrence’s document wrapper was added.
Case file casfil_w558a2omemh6dfcm
Root asset 21850e8d-e341-441b-9e07-bd6fac527f3c
Selected child f9ecbfb1-aad4-4f8e-b0a5-d2c8e72010b0
S3 contained both the full 29,230-byte extracted statement and smaller page-batch files. Platform selected the 711-byte batch containing only exhibits.
Root and children share the same external file ID. Platform chooses the first matching row without checking that it is the root.
Case file casfil_bqvs1dzdzly23i14
Root asset a19e969a-65d7-4b0b-9141-1e406cd742ae
First-page boundary 36e59f79-0d60-4852-bba1-46eacb8d96b0
The duplicate also had a complete extracted root before Lawrence requested it. Its root UUID sorted after the first 1,000 S3 keys, so the unpaginated listing never reached it.
Axiom showed no GET for that root. The endpoint still returned HTTP 200, and Platform converted the missing text in that response to null.
Platform sends both identifiers, then selects the first matching response row.
url.searchParams.append("matter_ids", caseId)
url.searchParams.append("external_asset_ids", caseFileId)
const asset = data.assets?.find(
(a) => a.external_asset_id === caseFileId
)
if (!asset?.content?.plain_text) return null
No root marker is checked, ordering is not guaranteed, and HTTP 200 says nothing about completeness.
The first trace chose two unrelated June files instead of the new August PDFs. Every turn in the long thread exceeded the context limit and was pruned, but generation inputs still remained roughly 140k–176k tokens.
Even a perfect retrieval service would have returned the wrong documents for that attempt.
Later responses attributed substantive positions to statement B despite never receiving its narrative. The claims appear to have been synthesised from statement A or conversation history.
Incomplete retrieval did not trigger a grounding stop.
retrieve_specific_document turns every empty result into “It may not have been processed yet.” In this sample that meant: pagination, genuine processing, missing artefacts, or legacy fallback. The copy happened to be correct for four files and was misleading for the majority.
The links below open the model traces or the relevant source. Axiom does not expose a stable query deeplink through the connected read-only API, so the exact APL is included for copy/paste rather than inventing a URL.
['vercel']
| where ['message'] has_cs "No plain text content found for document"
| extend body=parse_json(['message'])
| extend case_id=tostring(body.caseId),
case_file_id=tostring(body.caseFileId),
case_file_pair=strcat(case_id, "|", case_file_id)
| summarize failed_retrievals=count(),
case_file_combinations=dcount(case_file_pair),
legal_matters=dcount(case_id),
first_failure=min(['_time']),
last_failure=max(['_time'])
Retained window used: 2025-12-02 16:03 UTC to 2026-08-04 10:47 UTC. Result: 951 failed retrievals, 644 case-and-file combinations and 359 legal matters.
['ingestion-prd']
| where ['trace_id'] == "197c0040968344ce35d08974254a070c"
| extend url=tostring(['attributes.custom']['http.url'])
| project ['_time'], ['trace_id'], ['service.name'],
['name'], url, ['status.code']
| order by ['_time'] asc
The trace contains one ?list-type=2&prefix=matters/… URL, no continuation token, many unrelated first-page GETs and no GET under requested root f8eaed29….
I ran this read-only query against the V3 ingestion PostgreSQL database. It links each external case-file ID to its internal root, children, status, timestamps, source key and hash.
SELECT a.asset_id, a.external_asset_id, a.workflow_id, a.matter_id, a.parent_asset_id, a.s3_bucket, a.s3_key, a.asset_name, a.asset_size, a.asset_hash, a.status, a.created_at, a.last_updated_at, a.batch_index, a.total_batches, d.deleted_at FROM asset AS a LEFT JOIN asset_deletion AS d ON d.asset_id = a.asset_id WHERE a.external_asset_id IN ( 'casfil_emz5y0hy0rbrp4qi', 'casfil_f5q1s5s8cy91vjnu', 'casfil_w558a2omemh6dfcm', 'casfil_bqvs1dzdzly23i14' ) ORDER BY a.created_at;
Result used in the finding: duplicate statement B had root a19e969a-65d7-4b0b-9141-1e406cd742ae, six initial children, SUCCESS status and the same source hash as the first upload. Reingestion reused that root and appended six more children.
This query separated root-only controls from batched files and measured each matter’s asset graph. Root-only failures are important because they cannot be explained by Platform selecting a child.
WITH candidates(case_id, case_file_id) AS (
VALUES
('cas_gpmyhu6p44yhe4bm', 'casfil_w1ez3w35n0zielv9'),
('cas_x8csc4yun26y8sfe', 'casfil_cpckkrvacbwwtaz2'),
('cas_x8csc4yun26y8sfe', 'casfil_sjiygypct5e086v1'),
('cas_lq2ronf5cn7rs3tf', 'casfil_oo8zckee3brvfgak'),
('cas_lq2ronf5cn7rs3tf', 'casfil_y2c6svxsu9yf2599'),
('cas_lq2ronf5cn7rs3tf', 'casfil_ofi5j1tt81wh4vot'),
('cas_lq2ronf5cn7rs3tf', 'casfil_1irfzrg8b43edvjp')
)
SELECT
c.case_id,
c.case_file_id,
COUNT(a.asset_id) AS matter_asset_rows,
COUNT(a.asset_id) FILTER (
WHERE a.external_asset_id = c.case_file_id
) AS requested_asset_rows,
COUNT(a.asset_id) FILTER (
WHERE a.external_asset_id = c.case_file_id
AND a.parent_asset_id IS NULL
) AS requested_roots,
COUNT(a.asset_id) FILTER (
WHERE a.external_asset_id = c.case_file_id
AND a.parent_asset_id IS NOT NULL
) AS requested_children,
STRING_AGG(a.asset_id::text, ', ') FILTER (
WHERE a.external_asset_id = c.case_file_id
AND a.parent_asset_id IS NULL
) AS root_asset_ids
FROM candidates AS c
LEFT JOIN asset AS a
ON a.matter_id = c.case_id
LEFT JOIN asset_deletion AS d
ON d.asset_id = a.asset_id
WHERE d.deleted_at IS NULL
GROUP BY c.case_id, c.case_file_id
ORDER BY matter_asset_rows DESC, c.case_id, c.case_file_id;
Result excerpt: cas_gpmy… had 647 matter rows and four children; each cas_x8… target had 330 matter rows and eight children; all four cas_lq2… targets were root-only. Every listed target still failed when its root sorted after S3 page one.
I ran this read-only command in AWS CloudShell against the V3 ingestion data bucket.
aws s3api list-objects-v2 \
--bucket lawhive-ingestion-prd \
--prefix 'matters/cas_1x9fga55f9ww9tx2/21850e8d-e341-441b-9e07-bd6fac527f3c/' \
--region eu-west-2 \
--query 'Contents[?contains(Key, `.plain_text.`)].{Key:Key,Size:Size,LastModified:LastModified}' \
--output table
# Output excerpt gathered in CloudShell 21850e8d…/21850e8d….plain_text…txt 2026-08-04T10:12:55Z 29230 21850e8d…/f9ecbfb1….plain_text…txt 2026-08-04T10:12:43Z 711
The 711-byte object is the exact child batch Lawrence received. The 29,230-byte root proves the complete extracted statement was already present.
I forced a single S3 API request to reproduce the production implementation exactly.
aws s3api list-objects-v2 \
--bucket lawhive-ingestion-prd \
--prefix 'matters/cas_1x9fga55f9ww9tx2/' \
--region eu-west-2 \
--no-paginate \
--query '{KeyCount:KeyCount,IsTruncated:IsTruncated,FirstKey:Contents[0].Key,LastKey:Contents[-1].Key,NextToken:NextContinuationToken}' \
--output json \
--no-cli-pager
{
"KeyCount": 1000,
"IsTruncated": true,
"LastKey": "matters/cas_1x9…/36e59f79…/345ae597….plain_text….txt",
"NextToken": "present"
}
The missing duplicate root was a19e969a…, which sorts after the last returned prefix 36e59f79…. The application did not use the continuation token.
# Output excerpt from the duplicate root prefix a19e969a…/a19e969a….plain_text.20260804T101757Z.txt 2026-08-04T10:17:58Z 29230 a19e969a…/a19e969a….plain_text.20260804T135820Z.txt 2026-08-04T13:58:21Z 29230
The first complete root pre-dated Lawrence’s 10:47 request by 29 minutes. The second was produced by forced reingestion, confirming reingestion duplicated derived objects without fixing retrieval.
| Layer | Confirmed fault | Customer consequence |
|---|---|---|
| ingestion-management | Whole-matter OR query and unpaginated S3 listing. | Processed documents beyond key 1,000 appear missing. |
| Platform | First matching external ID can be a child asset. | Lawrence receives a fragment rather than the complete document. |
| Lawrence | Opaque historical ID selection, generic diagnosis and synthesis beyond retrieved evidence. | Wrong documents, repeated retries and unsupported legal analysis. |
| Upload / reingestion | No duplicate-upload idempotency; reingestion appended children. | More ambiguous IDs and a larger asset graph without repair. |
Sources: Platform document retrieval · ingestion-management content router · Axiom · Langfuse