Day-one of Lawrence-2 in the US — what we learned.

Adoption was clean for two of three users, output quality was strong, one Anthropic outage took the agent off the air mid-stream, and the read tool silently returned an empty PDF on the showcase memo. Cost numbers are inflated by known Langfuse accounting bugs.

1. Adoption picture

Per-user Lawrence-2 turns since the flag landed on 2026-05-12:

Sabrina and Georgia engaged substantively on day one. Scott made one turn (the showcase memo, below) and hasn't returned to the sidebar in the window — worth a check-in to understand why.

2. Per-user behaviour

Sabrina · 3 sessions, exploratory then production

Georgia · 1 session, IP-classification advice

Scott · 1 Lawrence-2 turn, the showcase memo

3. Failures & quality issues

Hard failure MidStream Anthropic overload (Sabrina, trace ac66f580…)

• Error originated on orchestrator-call (GENERATION) and propagated up iteration-1 → chat-response.
• Trace output stored is {"error": "...", "status": "failed"} — i.e. the raw litellm error string lands in Langfuse. The user-facing emit in packages/agent-definition is the generic "Error generating response" string, so the user did not see the raw payload, but they also got no actionable cue or retry control.
• The error name MidStreamFallbackError suggests a fallback chain tried and failed — likely same-provider (Sonnet → Opus or Opus → Sonnet) and Anthropic was overloaded across the board.
• This was the trigger for the evening's Sonnet → Opus swap; the Langfuse prompt has since been reverted to Sonnet.

Improvements

• Catch MidStreamFallbackError / ServiceUnavailableError at the orchestrator boundary; emit a typed provider_overloaded stream event with a "retry" affordance.
• Confirm the litellm fallback chain reaches a non-Anthropic provider (Anthropic API → Anthropic Bedrock) so a regional Anthropic outage is recoverable in-place.
• For mid-stream failures, let the user "retry this turn" as a first-class FE control — the user's question is unchanged.

Silent failure Empty PDF read (Scott, read on matfil_1c79x1rn3lacrwd6)

The read tool returned:

{"file": {"file_data": "data:application/pdf;base64,"},
 "name": "2026-05-01 Notice of Appeal",
 "extension": "pdf", ...}

A well-formed envelope with an empty base64 payload. The agent worked around it via vector search and surfaced "the notice of appeal PDF didn't return readable content" to Scott. Right user-facing behaviour from the model, but the underlying failure should not happen.

Investigate

• Is the file actually present in S3 / blob storage with content, or did the upload-side complete with zero bytes?
• If the source has bytes, the read tool's PDF extraction is the suspect — content-type or size bug producing empty file_data.
• Either way, the tool should return {"status": "failed", "error": "..."} instead of an envelope with an empty payload — the current shape forces the model to infer failure.

Gap Citations not structured (all 14 traces, reference_count = 0)

The model output is rich with citations (statutes, case law, regulations, USPTO Manual sections) but the structured references array is always []. This matches the LEX-264 / granular-citations workstream. Concrete impact:

• Citations in the UI are not clickable — users can't verify or click through to Justia / TMEP / etc.
• We cannot programmatically audit hallucination rate per citation.
• We cannot easily generate citation indices for export or for client-facing versions of a memo.

The parser pipeline exists in agent-definition and is wired through to trace metadata. reference_count = 0 means the model isn't emitting the marker syntax yet. Either the Langfuse agent/lawrence-2 prompt isn't instructing it to, or the chat-agent-embed-citations flag isn't on. Confirm both.

Polish Streaming-text artefact in the saved final response

Multiple traces show final_response_content as a wall of concatenated text where inter-tool narration ("Let me check the relevant Florida statutory framework…", "I have enough research…") is inlined directly before the next narrative paragraph, with no separator. Live this probably renders cleanly because the FE shows tool-running indicators between segments. But the saved/transcripted form is harder to read — which matters for Langfuse review, customer audit trails, and matter-note exports. Consider persisting as discrete typed events instead of one flat string.

UX nits

• Georgia's "yes, please draft" produced an A/B clarifying question, but the original turn had already proposed Class 9 primary with secondary 42/45. A confirmation should authorise the proposed plan, not re-elicit scope.
• Sabrina's drafting flow had the agent flag scope concerns three times before drafting. Good lawyering, but verbose — once the user has acknowledged a concern in the thread, the next acknowledgement should be a single sentence at the top.

4. Cost / latency

Even with the over-count caveat, the shape is informative: multi-iteration drafting turns are the dominant relative cost driver (Sabrina's lease, Scott's memo). What we don't yet know — and should check once PR #564 is in — is what fraction of the per-iteration input on Scott's 7-iteration trace was cache_read vs fresh. If caching was already working server-side but uncounted by Langfuse, his real spend is likely well under $5.

5. Top recommendations

Prioritised by impact on adoption / quality / cost:

1. Check in with Scott. One Lawrence-2 turn then silence in the window. Worth understanding whether that's a workload thing or whether something put him off after the showcase memo. The Notice-of-Appeal PDF failure (#2) is a candidate contributor.
2. Fix the empty-PDF read failure. Either the storage path for matfil_1c79x1rn3lacrwd6 or the read tool's PDF extraction. The user-visible workaround was OK but the cost of working around it was several iterations.
3. Handle Anthropic overload gracefully. Wrap MidStreamFallbackError at the orchestrator boundary; verify the cross-provider fallback chain works; expose a "retry" control in the sidebar for mid-stream failures. This is the failure mode that forced last night's outage-driven Sonnet→Opus swap — surviving Anthropic overloads in-place removes the need for emergency prompt-version flips.
4. Ship structured citations. All 14 traces emit textually-cited output but reference_count = 0. The pipeline exists; the model just isn't emitting the marker syntax yet. Largest remaining production-quality gap.
5. Land agents PR #564 and bump Langfuse for #13367. Today's displayed costs are inflated by the cache_read drop + the stale Sonnet 4.6 large-context tier. Cost decisions (model routing, budgets) shouldn't be made off Langfuse numbers until both fixes are in. Once they are, revisit per-turn cost budgets.
6. Persist orchestrator events as structured items instead of one concatenated final_response_content. Helps Langfuse review and matter-side transcripts.
7. Suppress redundant scope re-flags. Once a user has acknowledged a scope concern in a thread, downgrade subsequent flags to a one-liner.

Was: "confirm Sonnet→Opus was intentional" — confirmed. Outage-driven, since reverted on the Langfuse prompt. No action.

6. Where to look

High-level orientation — see the full Notion write-up for file/line pointers and the long-form analysis.

Request flow

Which area owns each finding

When a Lawrence-2 trace surfaces trouble — quick triage

• Unexpected tool used / write in a read-only context → check surface + use_vfs_tools resolution and the prompt name that actually got picked.
• Missing or duplicated citations → check whether the model emitted the citation marker syntax in its output. If yes, parser side; if no, the prompt.
• Laggy stream / wrong batching feel → frontend smoothing hook, backend fine-grained-streaming flag, gateway buffering headers.
• Hallucinated content / missing matter context → CCO retrieval failure path silently falls back to empty context; check the fetch-case-context observation.
• Generic "Error generating response" → root cause is upstream (LLM provider exception or context-window retry); the route-level OTel span carries the real error.message.