A real Lawhive Legal matter travelled all four stages into dev: dumped from v2, shaped and exported by a one-click Dagster job, imported into v3 with its files and the firm's people, and the first activation machinery landed behind it. The architecture grew a deliberate fourth stage this sprint: activation, which turns imported rows into things lawyers can log into, search, and ask about.
Sprint 1 built the engine's parts; Sprint 2 connected them. A real v2 matter, with its 36 files and the firm's 92 staff records, was dumped from the v2 dev database, shaped into boundary packets by the new one-click firm_import_shape_and_export Dagster job, and imported into v3 on dev: database rows, member records, and every file byte copied into place. 64 PRs landed across 5 repos, up from 27 last sprint, and two milestones closed to 100%.
The fourth stage, activation, went from idea to machinery: five tech plans agreed, member-login verbs on trunk, a dedicated ingestion backfill lane proven end to end in dev, and the file-activation chain staged as approved draft PRs. Two things are deliberately not claimed: file virus-scan verdicts in dev are blocked on a bucket-policy decision (fix raised, decision Sep 2), and nothing has touched production yet, by design.
Why no ticket counts: this sprint's Linear board does not reflect the work. The import-service lane in platform-v3 (22 of its 34 landed PRs) ran with no tickets behind it, and the merge queue never moves tickets on its own, so several finished tickets still read In Review. Rather than quote numbers we know are wrong, this report counts landed PRs, git-verified on each trunk; getting the board back in step is a named item in Lessons learnt.
Sprint 1's picture ended when rows landed in v3. This sprint made explicit what has to happen after that: an imported row is inert until someone can log in to see it, until its files have previews and are searchable, and until the AI has case context to answer questions about the matter. That work is now a named stage with its own tech plans, its own infrastructure, and its own controls, and it reuses the product's normal machinery (Clerk, Knock, previews, ingestion, case context) rather than reimplementing any of it.
Stages 1 and 2 live in the data platform (BigQuery, dbt, Dagster), stages 3 and 4 in platform-v3. The halves still only meet at the packet: this sprint's end-to-end run is the first time all four stages ran against the same real matter on a deployed environment.
Sprint 1's report ended with four fronts for this sprint. Here is how each landed, validated against trunk.
The pipeline and the import service met on a real environment for the first time. A real matter from the v2 dev database was dumped raw (DM-72), shaped and exported (DM-68, closed Aug 24), and imported into v3 on dev on Sep 1: database rows, 92 staff records, and all 36 files physically copied into the matter store.
On Aug 19 the warehouse side was a deploy fix away from nothing on trunk. By Sep 1 it can shape a firm's entire caseload and export it unattended: 19 PRs landed in data-dbt building the canonical package, the exporter, and the Dagster job around them.
The service Lukas handed over went from "on trunk" to "running in dev" in the sprint's first two days: CI registration, ECS deploy, configurable dump bucket, Axiom telemetry. Then the two missing halves landed as an 11-PR stack on Sep 1: file placement and member activation groundwork.
The verification net is being built alongside the files lane rather than as a separate front. Reconciliation models (DM-54) and runtime gates (DM-57) are in progress; the blob catalogue, its matter_file join, and the unclaimed-binaries test (DM-56/82/83) are open PRs; business-readable run reports (DM-80) are next in the queue. The coverage loop over the full book has not started.
The squad's definition of "proven" is a real run on a real environment, not a passing unit test. Nothing has touched production yet, by design: the project's gate order is local, then dev, then prod behind sign-offs.
| Capability | Local | Dev | Prod |
|---|---|---|---|
| Stage 1 · Sourcing | |||
| v2 database dump into raw/the real v2 dev database, dumped by Daniele and Cesar | ✓ proven | ✓ proven | not yet |
| File binaries via S3 Batch Copy + SHA-256Dagster one-off job, resumable, Launchpad pre-filled; first full-book copy is next | n/a | ◐ landed, first run pending | not yet |
| Stage 2 · Transformation | |||
| Shape + export: one-click Dagster jobwhole-caseload dbt build → JSONL packets in target/ | ✓ proven | ✓ proven | not yet |
| Contract sync: Zod → dbt daily crondrift opens a review PR automatically | ✓ live in CI, daily | n/a | |
| Stage 3 · Importing | |||
| Rows + members into v3714-matter rehearsal (Sprint 1) and this sprint's real-matter run | ✓ proven | ✓ proven | not yet |
| File placement: copy, scan, quarantine, report36/36 files placed in dev; scan verdicts blocked on the bucket-policy fix (#573) | ✓ proven | ◐ placed, scans blocked | not yet |
| Stage 4 · Activation | |||
| Member logins (Clerk) + invites (Knock)lifecycle verbs, ledger, worker, admin panel on trunk and deployed; first pilot activation is next | ✓ proven | ◐ deployed, pilot next | not yet |
| Ingestion backfill lanededicated queues + workers so migration volume never delays live traffic | ✓ proven | ✓ proven | not yet |
| File activation: previews + AI searchthe chain that wakes a run's files, newest matter first | ✓ proven | ○ drafts in review | not yet |
| Case context per matterbuilt once a run's files settle | ○ draft in review | not yet | not yet |
The sprint's biggest decisions were made in writing before the code was cut, in firm-migrations. Read in order, they are the story of how files and activation went from open question to build contract:
| Doc | What it settles |
|---|---|
| Dev dry-run lessons · Aug 20 | The import service's first deployed runs, measured: real work rate ~160 rows/s, 97% of the wall clock lost to dispatch idle, and a process-affinity bug behind 302 phantom failures. Set the instrumentation agenda. |
| Import instrumentation plan · Aug 21 | How run time per firm becomes predictable: three small PRs, four kinds of measurement run, one calibration formula. |
| Files from v2 to v3 · Aug 20 | Daniele's investigation of the byte hop: stage, scan, copy, hydrate, with the approach proven against the real v2 estate (3 TB, 4M objects). |
| File migration framework · Aug 24 | The source-blind design for how bytes behind migrated matters move into v3, with the decisions, alternatives, and the local tracer that proved it end to end. |
| Files migration tech plan · Aug 24 | The locked build contract after team alignment: destination-only scanning, import-bucket staging, data classes with firm-configurable thresholds, and the build sequence. |
| Firm provisioning · Aug 25 | A migration run never creates its firm: provision in admin first, point the run at it, delete the seed script. |
| Imported staff activation · Aug 26 | How a migrated firm's people get a login: one credential per identity, minted from the imports admin, invited through Knock, unwound by teardown. |
| Imported file activation · Aug 30 | The interfaces that wake a completed run's files: one operator action fans out to previews and ingestion, a second builds case context, all on the backfill lane. |
| Activation at migration scale · Aug 30 | The numbers companion: measured book sizes, per-document and per-matter economics of previews, AI search and case context, and which shared dependencies give way first. |
Milestone progress from the Linear project as of Sep 1, with Sprint 1's close as the left number. Two milestones closed to 100% this sprint. Six late-stage milestones (verification, live cutover, login migration, sign-offs, post-migration) remain at 0%; the project targets 20 October for the full Lawhive Legal load.
| Work | What | State |
|---|---|---|
| #14400–05, #14410 | The Stage-1 file-activation chain: wake a run's files newest matter first, fan out to previews and ingestion on the backfill lane, report progress, build case context once the run settles, drive it all from the run page | Drafts, approved |
| ingestion #745 | One Langfuse trace per file across the whole ingestion pipeline (nested per-stage spans, deterministic trace ids, flag-gated) | In review |
| infra-v3 #573 | Matters-bucket scan-policy fix; the fail-open versus fail-closed decision is scheduled for Sep 2 | Open · decision Sep 2 |
| DM-56 / 82 / 83 / 80 | The files verification wave in data-dbt: catalogue staged binaries into BigQuery, join the catalogue into matter_file so packets only reference genuinely staged files, test for unclaimed binaries, and ship a business-readable run report per packet | Open PRs |
| DM-54 / 57 / 81 / 89 | Reconciliation models, runtime gates, the full-book binary copy run, and the operator runbook for it | In progress |
| DM-66 / 79 / 16 | Built and merged (contract sync cron live; firm-provisioning write-up published); awaiting the ticket sync and the sync smoke test | Awaiting sync |
Pending sprint planning; these are the fronts the open work already points at.
| Front | Work |
|---|---|
| Close the file loop | Settle the matters-bucket scan posture (Sep 2), requeue the demo's 36 files to real verdicts, run the full-book binary copy (DM-81/89), and land the blob catalogue joins so packets only reference genuinely staged bytes (DM-56/82/83) |
| First activation in dev | Land the Stage-1 drafts, wake the demo run's files into previews and search on the backfill lane, build its case context, and run the one-member login pilot |
| Scale toward the full load | Full Lawhive Legal load on dev (the milestone sits at 13%), business-readable run reports (DM-80), and the instrumentation that predicts run time per firm |
| Observability & prod prep | Land per-file pipeline tracing (ingestion #745) and start the prod backfill-lane checklist: workers, config syncs, and the KMS prod arm |