What lawyers actually do with Lawrence 2, trace by trace.

A living review of production chat-response traces since the Lawrence 2 sidebar went to 100% on 12 August. Each observation window gets a dated section below; the issue register at the top carries every finding until it is fixed. Client, lawyer, and counterparty names are deliberately absent; traces are referenced by ID so anyone with Langfuse access can verify.

Traces: Langfuse chat project, name=chat-response, environment=prd, userId != eval-user, from 2026-08-12. Costs on this page are Langfuse-displayed and over-count real spend.

Issue register

Every finding lives here until it is fixed, whichever window found it. New windows append rows; fixes flip the status and link the PR. Severity is about user harm, not code size.

Evidence column shows Langfuse trace IDs (first 8 chars). R4 carries over from pre-release traffic on the same code path; details in the pre-release section.

Window 1 · 12–13 August · first two days at 100%

Fourteen traces from two users. One user toured a demo matter on 12 August (the files include memes and an unrelated ad, and the agent itself flagged them as likely test data): capability questions, task creation, folder organisation, five traces, all completed. The signal is the other user.

One lawyer, two hours, one live matter

On 13 August a US attorney (pers_6dea18ewlaqk87h0) worked a live quiet-title and lien-removal matter for two hours straight, nine traces from 17:32 to 19:23 UTC, all on claude-sonnet-5 via the agent/lawrence-2 prompt. The arc: draft an attorney letter, check the exact language of a contract paragraph, hand the agent a four-point critique of its own drafts, ask for corrected drafts, then work out what to request from the client and draft that message too.

Bar length is latency, scaled to the 444.6s worst case. Green ran clean, amber recovered from tool errors, red failed or returned nothing.

The lawyer also confirmed three of his four critique points were real defects in the agent's earlier drafts, which is worth sitting with: the agent's own quality control is currently the user. He stayed engaged for two hours anyway, which reads as a user who finds the tool valuable and is compensating for it.

Measurement gap

Nothing on this page came from a metric. Every trace in the window reports completed except one; latency and cost fall when the agent stops retrieving; and no trace carries a score of any kind. The two worst turns of the window (the hard failure and the empty response) are invisible to any dashboard we currently have. Until product feedback lands in Langfuse scores (R9), this review only exists because someone read the transcripts.

Pre-release signal on the same code paths · 3–11 August

Before the 12 August release, 42 more prd traces ran: six users on the legacy gpt-4.1 path (out of scope here) and early sonnet traffic on the same VFS stack Lawrence 2 now uses. Two things from that traffic stay relevant because the code paths are live:

• R4, the /vfs/edit silent no-op. A form-filling run got a 200 with zero edits applied, guessed a workaround, and spent 17 iterations and $2.79 on one form (trace 3f74a05e, 5 August). Same endpoint Lawrence 2 uses for document and form edits.
• A behaviour worth protecting with an eval. Asked to fill a Section 8 possession notice on a matter where the client is the tenant, the agent refused and explained why filling it would harm the client's position (trace 5caf4ceb, 5 August). Exactly the judgment we want; exactly the kind that gets tuned away if we optimise completion rate.

Appendix · every trace analysed

Each ID opens the trace in Langfuse (requires Langfuse access). Window 1 is the report's scope; the pre-release traces fed the same analysis and are kept here for provenance.

Window 1 · 12–13 August · 14 traces

Updating this report

This page is designed to be appended to. When new traces land, add a window section and update the register; don't rewrite history in old windows.

• Pull. The Langfuse Python SDK's trace.list gets a proxy 400 on this host; plain HTTP works. From ~/Projects/agents/agents/chat, run under doppler run (the Doppler config carries the chat project's Langfuse keys) and page through GET /api/public/traces?name=chat-response&environment=prd&fromTimestamp=… with Basic auth public:secret. Always pass fromTimestamp; unbounded queries time out. Oversized traces 422 on the detail endpoint; stitch them from GET /api/public/observations?traceId=… instead. Filter userId != eval-user client-side.
• Analyse. Condense each trace to: user message, ordered tool calls with statuses, non-default-level observations, final response, latency, models. Read for four things: what was asked, what the agent did, how the user reacted in follow-ups (rephrasing and corrections are the satisfaction signal, since there are no scores), and anything that failed.
• Append. New dated <h2> window section above the pre-release one, newest first. New findings become register rows with fresh IDs; recurring findings get their evidence column extended. Fixed items flip to fixed with a PR link. Update the TL;DR, the stat tiles, the "Last updated" date, and give any new blocks fresh data-hpc-ids so the review watermark can diff the page for returning readers. Link every trace ID to its Langfuse trace (traces?peek=<full-id>) and extend the appendix.