previews · RCAs · Lawrence US family-law test session

Lawrence on a US family-law matter — trace triage of the 2026-08-31 test session

James Peters ran a Texas divorce matter end-to-end through Lawrence and reported four complaints in Slack: rules of civil procedure missing from the first draft, an alarmist tone, documents reported as gone while still visible in his Documents tab, and no access to his own email address. We pulled all 42 traces from the session and traced each complaint to a root cause in platform-v3, the agents repo, or a legal-research provider.

Date · 2026-09-01 Window · 2026-08-31 19:07–22:21 UTC Env · prd Prompt · agent/lawrence-2 v90 · Sonnet 5 User · James Peters (test matter, fictional parties) Traces reviewed · 42 · $13.64 Author · Adolfo Final

One-liner

The four complaints have mechanical causes. The "vanished" documents were drafts James had finalised: finalisation removes a document from the agent's view and the error says only "not found", so Lawrence recreated them and produced duplicates. Two turns returned zero output after ~5-minute hangs at 188k and 207k input tokens; the compaction guard resolves to a 737k threshold and never fires. CourtListener timed out for eight minutes and the agent saw Tool execution failed: with an empty reason, twelve times. OpenLaws keyword search returns nothing for insupportability, the title of the Texas divorce ground. The first-draft petition skipped procedural rules because neither the prompt nor the drafting skill asks for them. Each finding below has trace evidence and a file-level root cause. The OpenLaws results were reproduced live on 2026-09-01; the CourtListener timeouts are confirmed in Axiom logs.

Turns

one user, one matter, 3.2 h

Dead turns

0 output after ~300 s, shown as completed

Consecutive tool timeouts

one turn, empty error text

20 : 2

Creates vs edits

documents were recreated, not edited

1. The session at a glance

All times UTC. The matter is a fictional uncontested Texas divorce (Ed and Karen Baldwin, respondent in Arizona), created that afternoon with one intake note and one contact.

Time	What happened	Outcome
19:07–19:15	Platform questions: "add a team to this matter", "where is Support?". The prompt's static UI reference (§ 11) covers layout and tabs but not admin workflows; a web search returned help articles for LawVu and Filevine. James pushed back until it answered plainly.	Admin how-tos missing
19:15–19:33	Arizona spousal-maintenance research. Loaded `legal-research`, read statutes and four opinions, ran a citator check, flagged the TX/AZ jurisdiction split before starting.	Strong
19:35–19:53	Explained the uncontested-divorce process, created 9 tasks, then drafted the petition, waiver of service, agreement incident to divorce, and decree. Statutes were researched; the TRCP, standing order, and mandatory-content checks were not.	Procedure skipped
19:57 + 20:05	James pasted a detailed external review of the petition. Two verification turns hung for ~5 minutes each at 188k and 207k input tokens and ended with no output. The second streamed the "case will unravel" preamble first, then nothing.	2 dead turns
20:09–20:11	Third attempt verified every point in the review against the statutes and confirmed the jurisdiction defect. Asked "did you not reference the TRCP info?", Lawrence answered that it drafted from general knowledge without researching procedure.	Recovered
20:13–20:21	Consent-to-jurisdiction research. CourtListener timed out 12 times in a row; the agent rewrote the query each time, then answered from statutes and TRCP text with the gap flagged.	Provider outage
~21:00	James finalised the four drafts in the UI (each becomes a docx + pdf pair in the Documents tab).	Off-trace
22:10–22:21	Asked for revisions. Reads and stats on all draft IDs returned "not found"; a documents search returned nothing. Lawrence recreated five documents from scratch, James saw duplicates, and the two could not reconcile which documents existed.	Docs mismatch

2. Finalised documents are invisible to the agent

This caused the duplicate documents and the disagreement at the end of the session. At 22:10 Lawrence tried to read the three drafts it had created at 19:41–19:53:

GET /vfs/read?path=documents://mat_lw121wcqixxf4pel/matdoc_lpp3ckwlapcl87sf → 500 {"error":"Failed to read content","details":"Matter document matdoc_lpp3ckwlapcl87sf not found in matter mat_lw121wcqixxf4pel"}

A fresh search over the documents namespace returned []. Meanwhile James's Documents tab showed ten files, five documents as docx + pdf pairs, which is what finalisation produces. Finalising a document snapshots a version, renders docx and pdf, and sets currentFinalisationId on the row (matter-service/src/matterDocuments/services/finalizeMatterDocument.ts). Nothing is deleted, but the list every VFS read, stat, and search goes through excludes finalised rows:

// services/matter-service/src/matterDocuments/services/listMatterDocuments.ts:24-36
where: {
  matterId,
  ...(includeFinalised ? {} : { currentFinalisationId: null }),
  ...
}

The lawrence-api adapter never passes includeFinalised, so once a lawyer finalises a draft it disappears from the agent's view, and the error says "not found" rather than "finalised". Lawrence reported the documents as gone and rebuilt all of them. The rebuilt petition then disappeared the same way six minutes after creation, because James finalised that one too.

Three adjacent defects were found while verifying this path:

Search is silently capped at the 50 newest documents. searchEntries.ts discards the adapter's nextCursor (apis/lawrence-api/src/agent-retrieval/services/searchEntries.ts:378-387), and the adapter's default page is 50, ordered newest first. Any namespace error is also swallowed into [], so partial results are indistinguishable from complete ones.
The agent's glob pattern could never match. It searched **Original Petition**; minimatch only treats ** as a globstar when it is a whole path segment, so this pattern matches nothing on any input. **/*Original Petition* works. Nothing in the tool description warns about this.
Finalised output is also invisible to content search. The finalisation event ships the file to ingestion with asset_type: "matter_document", which ingestion's enum rejects with a silently-swallowed 422 (found 2026-08-19, still open), so the finalised docx never gets indexed either.

Fixes, in order of value

Pass includeFinalised: true in the adapter's read/stat lookup and return finalised documents read-only, with metadata saying so. The content fetch itself has no finalisation filter, so this is a lookup change, not a pipeline change.
Make the not-found error state the reason: "this document was finalised on <date>; the final version lives in the matter files as <name>.docx".
Follow nextCursor in searchEntries.ts (or raise the page limit and surface truncation), and stop mapping namespace failures to empty results.
Fix the ingestion asset-type enum gap so finalised documents are searchable by content.

3. Two turns returned no output and no error

When James pasted the external review, the verification turn ran six minutes, cost $0.71, and produced nothing. The retry streamed one preamble sentence (the "unravel" line James screenshotted) and then also produced nothing, at $1.25. The traces show the same signature on both:

Trace	Input tokens	Output tokens	Duration	completionStartTime	Recorded status
4a77f50a	188,188	0	300.0 s	never	completed (!)
2d609089	206,556	0	311 s	never	completed (!)

Three layers line up to produce this:

Compaction exists but is calibrated to a threshold that can never fire. maybe_compact_stream runs every iteration, but the max-input lookup asks LiteLLM for anthropic/claude-sonnet-5, which is not in the registry, so it falls back to 1,000,000 tokens (packages/compaction/src/compaction/compactor.py:19). The compaction threshold works out to ≈737k, so a turn that grows to 200k from statute reads and web results never triggers compaction. (The registry entry that does exist, claude-sonnet-5, also says 1M, so fixing the name alone does not fix the calibration.)
The model call has no timeout of its own, and the deaths are completely unlogged. The call goes direct to Anthropic through the LiteLLM SDK (prod logs: LiteLLM completion() model= claude-sonnet-5; provider = anthropic); acompletion passes no timeout, and the SDK default is 6,000 s, so the ~300 s cut comes from the connection path, most plausibly an idle timeout on the internal SSE hop while no tokens are flowing. Both failures left zero log lines in agents-chat application logs, uvicorn access logs, and v3-prd error spans, which is consistent with a silent request cancellation. The absence of logging is itself a defect, because we currently cannot tell a provider problem from a network cut from a code bug.
An empty stream is recorded as success. When the stream ends with no content, the loop yields an empty assistant message, which passes the falsy-check and ends the turn as completed (agents/chat/src/chat/agent/agent.py:1513-1518). The user sees nothing; the dashboards see a healthy turn.

Fixes

Set explicit timeouts on the model call: a first-token deadline (say 120 s) and an inter-chunk inactivity deadline (say 60 s), one retry with jitter on timeout or overload, then a surfaced failure.
Treat a zero-length completion or a cancelled stream as a failure: log it with the trace id, retry once compacted, and if it still fails, tell the user the turn failed. "Completed, empty" must not be a terminal state, and every stream termination must leave a log line.
Send SSE heartbeats while waiting on the model so no intermediate hop sees an idle connection during long prompt-processing waits.
Pin real context-window numbers per model in compaction config instead of trusting the LiteLLM registry, and compact well below what the provider sustains comfortably (these 190–207k requests are the reason the waits got long in the first place).

4. CourtListener timed out for eight minutes; the error text was empty

In the consent-to-jurisdiction turn, search_case_law failed twelve times in a row over eight minutes. Every wall-clock duration was 30.0 s, the client's read timeout, and Axiom logs show the exception:

exc_type: "ReadTimeout", exc_value: "" ← str(exc) is empty frames: … case_law_tool/tool.py:145 → legal_sources/adapters/courtlistener.py:126

The agent loop catches every exception as f"Tool execution failed: {str(e)}" (packages/agent-definition/src/agent_definition/agent.py:965-984), and httpx timeout exceptions stringify to nothing, so the model saw twelve identical blank failures. It had no way to distinguish "the provider is down, stop" from "your query was malformed, rephrase", so it rephrased twelve times, at $1.18 and eight minutes. It then answered from statutes and TRCP text and flagged the missing case law explicitly.

The HTTP retry layer only retries status 429 (packages/legal-sources/src/legal_sources/clients/retry.py:14); timeouts are not retried, and nothing caps how often the model can retry the same failing tool. The same empty-error signature appeared twice earlier in the session at 19:16, so CourtListener was degraded for hours and nothing alerted. CourtListener publishes no status page (status.courtlistener.com does not exist; their docs mention only a Thursday-night maintenance window, and this was a Sunday), so detection has to come from our own alerting.

Fixes

Include the exception class in the tool error: Tool execution failed: ReadTimeout (provider did not respond in 30 s). One line in the generic handler.
After 2–3 consecutive failures of the same tool in a turn, replace the error with an instruction: "this provider is unavailable right now, do not retry it this turn, work from other sources and say so".
Retry timeouts once at the HTTP layer, and alert on provider timeout rate so a degraded CourtListener shows up in our monitoring rather than in a lawyer session.

5. OpenLaws keyword search misses Family Code sections

James asked whether Lawrence has gotten worse since the CourtListener/OpenLaws move. For legislation search, the tool returns useful results only when the model already knows the citation. We re-ran the session's failing queries live against OpenLaws (TX corpus, 2026-09-01):

Query	Mode	Result
residency requirement suit for dissolution of marriage	and (prod default)	404 / zero hits
residency requirement suit for dissolution of marriage	or	Family Code §§ 85.061–.063 (protective orders, not § 6.301)
protective order statement required in petition	and	Radiation-control regs, utility conduct codes, capacity auctions
eligibility for maintenance duration of marriage	and	Insurance termination standards
insupportability	and	0 results — this is the literal title of Tex. Fam. Code § 6.001

The adapter sends every query as boolean AND over the whole jurisdiction corpus, administrative code included; there is no corpus filter, no OR/phrase fallback (the client supports both but the adapter never passes them), and a 404 is returned as a silent empty list (packages/legal-sources/src/legal_sources/adapters/openlaws.py:48, clients/openlaws.py:55-77). In the session the model coped by guessing exact section numbers and calling read_legislation with a citation, which works well and is unmetered. Reading a known citation is reliable; finding the citation by keyword is not. Court rules (TRCP) are not in the corpus at all, which is why the model fell back to web_search for rules 57, 108a and 190.2.

Fixes

Fallback ladder in the adapter: and → or → drop the least-specific terms, before returning empty. Cheap, no prompt change. (Verified against the live API on 2026-09-01: an invalid type returns "Valid query types are: 'and', 'or', and 'phrase'", and the or run of the protective-order query returns Family Code sections.)
Post-filter administrative code in the adapter: each result carries a law_key (TX-STAT vs TX-RR), so dropping or down-ranking non-statute divisions is a client-side change. There is no server-side corpus filter today — a law_key query parameter is silently ignored (verified live) — so also ask OpenLaws for one. Their own docs recommend citation-string lookup over keyword search, and their status page (status.openlaws.us) monitors a "Laws Search" check that showed intermittent degraded marks on Aug 31.
Teach the secondary-source path to drive discovery: vector_search over the practice-manual corpus surfaces the right section numbers, then read_legislation grounds them. The session's best research turns already worked this way.
For court rules, either add a rules corpus or make the skill explicit that procedure rules come from web_search against official sources.

6. The first draft skipped procedural research; no skill or prompt requires it

The petition was drafted at 19:41 with statutes researched (§ 6.301 residency, § 6.001 grounds, § 6.4035 waiver, § 7.006 agreement) but no TRCP, no Travis County Standing Order, no § 6.405 protective-order statement, no discovery-level pleading, no TRCP 57 signature block. Every one of those was caught by the external review, and every one was verifiable with the tools Lawrence already has, because it verified all of them 20 minutes later when challenged. Its own account of the gap was accurate:

"No, I didn't. I drafted that petition directly from a standard pleading structure without first researching Texas Family Code jurisdictional requirements, TRCP 190.2, TRCP 57, TRCP 108, or checking the Travis County Standing Order. […] It should have happened before the first draft went out, not after."

To be precise about what is and is not required: prompt § 6.2 does mandate loading the legal-research skill for legal drafting, and substantive statute research did happen before the draft (§ 6.301, § 6.001, § 6.4035, § 7.006 were all read). What nothing requires is procedural research — the legal-research skill scopes procedure out (its US variant hands procedure to web_search as an aside), and the drafting skill's from-scratch path has one research line: "Research the matter and the relevant law (load legal-research if jurisdiction-specific case law or legislation is needed)". The system prompt's drafting workflow (§ 7.2) mandates a precedent check and a forms check, both of which happened and both of which were empty, and then lets the model free-draft.

Skill tuning (concrete)

Add a court-filing pre-flight to drafting Path B, before "Build the ProseMirror document":

If the document will be filed with a court or tribunal: identify the forum, then research (a) the mandatory content the governing statute requires in this pleading type, (b) the applicable rules of procedure for formalities (signature blocks, discovery-plan statements, service), and (c) local and standing orders for the specific county or court, via web_search against official sources.
Write the checklist into the turn before composing, and carry each item into the draft or flag it as an open question in the hand-back.
Facts that touch jurisdiction (a nonresident party, an out-of-state marriage, service across state lines) trigger a jurisdiction-basis check before drafting, not after.

7. Smaller defects

The acting lawyer's email is not in the prompt

Asked to email the drafts, Lawrence said it has no access to James's email address, which is accurate: the prompt's team context carries name and role only (packages/agent-schemas/src/agent_schemas/chat.py:27-32); client participants carry email and phone, the acting lawyer does not. Add the current user's email (and the firm's domain conventions) to the identity block.

Tasks are write-once

Lawrence created nine well-formed tasks, then could not mark one done or add a due date; tasks have no edit grant and the model omits optional dueAt on create. James had to tick his own agent's tasks manually. Grant task edit, and have the skill set due dates where the statute implies one (the 60-day waiting period task is a natural key date).

forms:// dumps a ~20KB schema on every list read

The search tool refuses the forms namespace and tells the agent to read forms://<matter>; that read returns forms: [] plus the full case_data_schema, every time (vfs-forms-adapter.ts:106). It happened three times this session; thousands of junk tokens per occurrence. The fix is small and safe: drop the schema from the list response entirely. The form-filling flow reads it from the single-form response (forms://<matter>/<slug>, which also carries fields_outline), and FormsListContent has no consumer other than the adapter itself.

Every turn logs 12 bogus validation errors

On each request, message validation flags every historical tool part with "Found providerExecuted field (this field doesn't exist in actual UIMessages)", 12 error-level log lines per turn in this session, identical each time. Meanwhile the two failed turns above logged nothing at all, so healthy turns produce error logs and failed turns produce none. Fix the serialisation (or the validator) and downgrade the noise.

First note edit 500s with a raw Zod dump

The first edit sent block_id: null and got a 500 with an internal Zod validation dump (known class, LEX-831). The model recovered by re-reading with format: "map", but the recovery cost a full iteration. Return a 422 with a plain instruction instead.

Platform knowledge is one hand-maintained prompt section

Fifteen minutes of the session were platform questions ("add a team", "where is Support?"). The prompt's § 11 is a static UI reference (layout, sidebar icons, matter tabs, the Documents tab's two sections) — it is how Lawrence knew where Support should sit — but it covers no admin workflows (team management, permissions), says nothing about finalisation, and nothing checks it against the shipped product, so it drifts silently. Web search surfaced LawVu and Filevine help pages instead. Fix: a product-help corpus for the how-tos, and a periodic check of § 11 against the product (or generating it from one source of truth).

Late-session turns cost ~$1 each

The full history is resent every turn, so by 22:10 a ten-second "I don't have your email" answer cost $0.97. The compaction fix in finding 3 is also the cost fix; the forms-schema and statute-dump trimming compound it.

8. What worked

Research quality when the tools worked. The Arizona maintenance answer framed the annulment route correctly, quoted § 25-319(C) verbatim, read four opinions in full, and ran a citator check on the case the argument depends on (Foughty). The post-review verification confirmed every point in the external review with primary sources.
Accurate self-reporting. It described the drafting gap correctly when asked, named CourtListener and OpenLaws when pressed about sources, refused to invent case law during the outage, and refused to draft the settlement agreement while the asset inventory was unknown.
Jurisdiction awareness. It stopped to flag the Texas/Arizona split before researching, and asked which state's law James actually wanted.
Clarifying before drafting. Each document was preceded by targeted factual questions rather than invented placeholders.

James's tone complaint ("proclaiming the case will unravel") attaches to the opening sentence of a turn that then returned nothing (finding 3), so the dramatic claim was never followed by the analysis that would have justified it.

9. Ranked recommendations

	Fix	Where	Size
P0	Finalised documents visible to the agent read-only, and not-found errors that say "finalised, lives at <file>"	`platform-v3` lawrence-api documents adapter + matter-service list	S
P0	Zero-output completions and cancelled streams are failures: log them, retry compacted, then surface the failure to the user	`agents` chat loop	S
P0	Explicit first-token and inter-chunk timeouts on the Anthropic call (SDK default is 6,000 s), one retry with jitter, SSE heartbeats while waiting	`agents` agent-definition + chat API	M
P0	Compaction threshold pinned per model instead of trusting the LiteLLM registry (currently resolves to ≈737k and never fires)	`agents` compaction	S
P1	Tool errors carry the exception class; consecutive-failure circuit breaker per tool per turn	`agents` agent-definition	S
P1	OpenLaws fallback ladder (and → or → trimmed) and a statutes-first corpus filter	`agents` legal-sources	S
P1	Court-filing pre-flight in the `drafting` skill (procedure rules, standing orders, mandatory content)	skills	S
P2	Search follows `nextCursor`; namespace errors are not empty results; glob guidance in the tool description	`platform-v3` searchEntries + prompt	M
P2	Acting lawyer's email in the identity block; task edit grant + due dates; forms schema stripped from list reads	`agents` + `platform-v3`	M
P2	Ingestion accepts `matter_document` asset type so finalised docs are content-searchable	`ingestion` + lawrence-api	S (known issue)
P3	Product-help corpus for platform questions; provider-latency alerting for CourtListener/OpenLaws	new	M

10. Prompt and skill tunings

From reading agent/lawrence-2 v90 (79k chars) and the shipped skills against this session:

Nothing in the prompt or skills mentions finalisation or document versions. Until the platform fix lands, add one paragraph to § 5.3 (read/search tools): "A document a lawyer has finalised no longer appears in documents searches or reads. If a document you created is reported as not found, say it may have been finalised and ask the lawyer to check the Documents tab, and do not recreate it without their say-so." This covers the failure in finding 2 until the platform fix lands.
Guard against recreating documents. § 7 lets the model create freely; add: "Never recreate a document that previously existed on this matter without the lawyer confirming the original is gone." The 20:2 create-to-edit ratio is the measurable symptom.
The tool-failure guidance in § 5.4 assumes failures are query problems. "Do not conclude that no information exists after a single failed search" is correct for empty results but wrong for repeated infrastructure timeouts; the model followed it into twelve retries. Split the guidance: reformulate on empty results, stop and disclose on repeated identical failures.
search_legislation's description oversells it. It already says "try 2–3 reformulations"; add "results can include administrative code unrelated to your question — prefer confirming section numbers via secondary sources (vector_search), then read_legislation by citation, which is authoritative". That matches what the successful turns actually did.
drafting skill gets the court-filing pre-flight (finding 6), and its Path C should say what to do when the target document is not readable (the finalised case) rather than leaving recreation as the default.
§ 9's citation paragraph is still garbled mid-sentence (carried over from the Aug-13 audit, unchanged in v90) — worth fixing while touching the prompt.

11. Methodology and data

How this triage was produced, so the findings can be checked or extended:

Complaint intake. Read James's full group-DM history and thread replies (2026-08-31 20:00–22:37 UTC), including all five screenshots, and turned them into four testable complaints: procedure missing from the first draft, alarmist tone, documents reported gone while visible in the UI, and no access to his own email.
Trace pull. Listed prd chat-response traces since Aug 20 from the Langfuse REST API (chat project) and matched the session by user and matter: 42 traces for pers_ulau97w4xlnytxkc on mat_lw121wcqixxf4pel, all on 2026-08-31. Pulled each trace with its full observation tree (plain httpx with Basic auth; the SDK's trace list endpoint 400s through our proxy). Every claim about what the agent saw, called, or answered comes from these observations, not from screenshots or memory.
Digest, then drill. Compressed each trace to user turn → tool calls with arguments and result snippets → final answer, read all 42, and drilled into the anomalies: the two zero-output generations, the twelve identical tool failures, and every document read, create, and search in the 22:10–22:21 window.
Code verification. Traced each suspected mechanism to source before asserting it: platform-v3 at trunk for the documents adapter, finalisation flow, search pagination, and glob matching; agents at origin/develop for the tool error handling, retry policy, compaction thresholds, model-call parameters, and skill contents; prompt agent/lawrence-2 v90 fetched from Langfuse. File-and-line references in each finding are from those revisions.
Live reproduction. Re-ran the failing OpenLaws queries against their API with our prod key on 2026-09-01 (the table in finding 5 is that run, not the trace replay), and probed for a CourtListener status page. Confirmed the CourtListener timeouts and the per-turn validation noise in Axiom's prd dataset, and confirmed the dead turns left no log lines in agents-chat, uvicorn, or v3-prd error spans.
Known limits. The exact hop that cuts the connection at ~300 s is not yet attributed (nothing logs it, which is finding 3's point). The finalisation timeline (~21:00 UTC) is inferred from the docx+pdf pairs and the disappearance pattern, not from an audit log. Costs are Langfuse-computed.

The pulled data, kept alongside this page: all 42 traces with per-turn summaries, tool calls, and Langfuse deep links, and the condensed per-turn digest (tool arguments and result snippets, 176 KB). The raw trace JSON (70 MB with full observation payloads) is not committed; each row of the trace index links to its live Langfuse record, which remains the source of truth.