The database migration moves rows through four stages and ends with a matter_file row in matter-service. That row points at bytes the pipeline never touches. For v2 we own the bucket the bytes sit in; for LEAP and Woodstock we do not know how we will get access, and a physical drive is one live possibility. This page is the byte half of the migration: the goal, the architecture, the decisions with their alternatives, and the tracer bullet that ran the whole design locally. Daniele is working the same problem independently on purpose; the final shape is whatever survives both sets of notes on Monday. Team context in the Slack thread.
TL;DR. Copy what made the row migration generic. Rows go through a neutral dump; every source-specific quirk dies at the dump boundary. Bytes go through a staging bucket we own: each source delivers files into it however it can (S3 copy, export, a shipped drive), and one shared pipeline gates each file on a malware verdict, verifies its digest against the dump, copies it into the destination bucket, and accounts for it in a ledger. The import run itself never touches bytes — it promises them and finishes, so rows and bytes can arrive in any order. The design is built and proven locally: a real import run, real S3 copies, a quarantined infected file, and late-arriving bytes settling on their own, 75 placed / 1 quarantined / nothing waiting.
The goal
Move every file a migrated firm's matters reference, from any source system, with integrity proof per file, malware scanning before anything reaches a product bucket, and an account of where every byte got to. Do it without over-indexing on v2 or LEAP: v2 is just the first source adapter.
The scale, from v2 production (excluding files already marked deleted):
| What | Count | Bytes | Notes |
|---|---|---|---|
| Case files | 1,331,623 | 1.54 TB | referenced by matter_file rows |
| Email attachments | 301,428 | 233 GB | ride the comms entities, not matter_file |
| Median file | 177 kB | small files, many of them |
Where the numbers come from, since they read smaller than Daniele's 3–4 TB: the table is measured from the v2 production File table (2026-08-21), and the bucket disagrees with it for a reason. The whole cases bucket measures 3,018,032 objects / 2.33 TB (full listing, 2026-08-24). The ~1.35M objects and ~0.5 TB between the two are per-case filed-emails/, call-recordings/, agreements/ and preview/ objects — classes that migrate through other pillars (filed emails as communications, recordings with calls), get regenerated (previews), or are separate products. The acquirer copies by manifest reference, so the files migration pays for the table's numbers; only a whole-bucket sync would pay for the bucket's. Both figures are right; they answer different questions, and cost planning should say which one it is using.
The design pressure comes from the count. The byte total is hours of copying — by arithmetic rather than benchmark: the copies are server-side (no bytes pass through our compute), S3 sustains thousands of copies per second per key prefix, and even a single worker at a conservative 100 copies per second clears 1.3M objects in under four hours. The dev acquirer run turns that arithmetic into a measured rate. The per-object costs are where this goes wrong if we let it: 1.6 million objects is 1.6 million scan verdicts, ledger rows, and pipeline triggers. LEAP and Woodstock volumes are unknown.
How an upload works today
The context everything below leans on: this is what happens when a lawyer drops one file onto a matter in the app.
sequenceDiagram
participant App as app (browser)
participant MS as matter-service
participant S3 as matters bucket
participant GD as GuardDuty
participant WH as Hookdeck → webhook-api
participant IN as Inngest consumers
App->>MS: ask to upload (matter, filename)
MS-->>App: presigned POST, 15 min
App->>S3: PUT the bytes, matters/{matterId}/matterFiles/{filename}
App->>MS: confirm
MS->>MS: File + MatterFile + FilePreview rows<br/>preview status decided from content type
MS->>IN: matter-service/file.uploaded (domain event)
Note over IN: lawrence-api registers the asset<br/>for ingestion and case context
S3-->>GD: new object — bucket-level, automatic
GD->>S3: scan, write GuardDutyMalwareScanStatus tag
GD-->>WH: verdict via EventBridge → SNS
WH->>IN: webhook/s3/malwareScan.completed
IN->>MS: fileUploaded runs on the verdict
MS->>MS: clean + preview PENDING? invoke pdf-converter Lambda
MS->>MS: FilePreview gets s3Key + SUCCESS
Four facts from this flow shape the migration design. The malware scan is bucket-level and automatic — anything written to the matters bucket gets scanned, whoever wrote it, which is what makes bulk placement a cost decision (the triangle below). Preview generation runs downstream of the verdict, not the upload, and short-circuits on a terminal FilePreview row — PDFs and images are marked previewable at row creation without any Lambda. Ingestion and case-context ride the domain event only this path sends, so a bulk copy cannot trigger them by accident; that stays with the activation layer by construction. And the read path has no scan gate in this repo's code or Terraform: downloads are signed against matter participation and visibility only, and the File row has no field to record a verdict. Whether the bucket itself enforces one — a tag-conditioned deny like the content bucket's, defined in the infra repo — could not be verified from here and is open question 5. Either way the design holds: the placement copy carries the staging scan's clean tag onto the placed object, so imported files pass a tag-conditioned gate if one exists.
The architecture
The row pipeline is generic because of the dump: extractors are source-specific, and everything downstream of the dump is one shared pipeline. Bytes get the same waist — a staging bucket we own, with the manifest riding in the dump itself.
An acquirer's contract is one sentence: deliver the bytes for a dump into the staging bucket, keyed by the same externalRef the dump uses, with the digest computed on the way in. For v2 that is a bucket-to-bucket copy we could start today. For LEAP it is whatever access materialises, and a shipped drive satisfies the contract exactly as well as an API. The shared side never learns which one happened. Email attachments fit without a second design: more entries in the manifest, more objects in staging, consumed by the comms importer's rows.
The dump contract already carries the manifest: each matter_file entry requires a payloadUri and a payloadSha256, and the catalogue's docstring says the URI points into a staged store the run owns. The v2 extractor pointing at v2's live bucket was the shortcut; this framework makes the existing contract true.
How a file moves
The import run itself never touches bytes. It writes one ledger row per promised file (the staged object's URI, its declared digest, the destination key) and creates the matter-service File row with its final key already in place — then finishes. A separate pipeline, driven by Inngest the way the run itself is, makes the promises true:
sequenceDiagram
participant Run as import run
participant Ledger as ledger (import_svc)
participant Inngest as Inngest
participant Sweep as placement sweep
participant Staging as staging bucket
participant Dest as matters bucket
Run->>Ledger: promise 76 files (EXPECTED)
Run->>Inngest: finished, request a sweep
Inngest->>Sweep: sweep run 12
Sweep->>Ledger: read EXPECTED rows
Note over Sweep: gate on, no verdicts yet: every row waits
Note over Staging: scanner tags objects,<br/>posts verdict events (GuardDuty in prod)
Staging-->>Inngest: webhook/s3/malwareScan.completed × 38
Inngest->>Ledger: stamp verdicts (first one wins)
Inngest->>Sweep: rows moved, resweep
Sweep->>Staging: HeadObject + read the GuardDuty tag
Sweep->>Sweep: sha256 vs the dump's manifest
Sweep->>Dest: CopyObject (clean files only)
Sweep->>Ledger: settle PLACED / QUARANTINED, compare-and-set
Per file, the sweep's decision is one pure function, and every branch below is a pinned test:
flowchart TD
A[EXPECTED row] --> B{bytes in staging?}
B -- "bad URI / wrong bucket" --> F1[FAILED, reason names both buckets]
B -- "not yet" --> W1[stays EXPECTED, next sweep retries]
B -- yes --> C{threat, from verdict or tag?}
C -- yes --> Q[QUARANTINED, never copied]
C -- no --> D{"gate on: verdict CLEAN and tag clean?"}
D -- "not yet" --> W2[stays EXPECTED, verdict event will resweep]
D -- "yes, or gate off" --> E{sha256 matches the dump?}
E -- no --> F2[FAILED, digest mismatch in failureReason]
E -- yes --> P[CopyObject, settle PLACED]
Because the run only promises, rows and bytes run on their own clocks — which is the whole point for a source whose bytes arrive on a drive weeks after the dump:
Decisions madesoft — argue with the specific step
Each of these is a lean with the alternative named, so a reviewer can attack the step rather than the whole shape.
1. Bytes move beside the import run, not inside it
The run promises and finishes; the pipeline delivers. The alternative — keep placement inline in the run's batches, as the code originally sketched — buys one property (a completed run means a complete firm, bytes included) at three structural costs: the DB import becomes hostage to byte logistics (a LEAP drive arriving weeks late blocks cutover for rows that are ready today); the run has nowhere to wait for asynchronous scan verdicts without holding its single concurrency lane open; and at 1.3M files the run's wall clock becomes byte throughput. The trailing window where a listed file has no bytes yet is the deliberate trade, and how long it may be at cutover is a product call (open question 2).
2. Scan in staging, before placement, on two signals
The gate demands the malware verdict event and the object's own GuardDutyMalwareScanStatus tag before a copy; either lagging means wait, a threat from either is terminal even when the gate is off. Alternative A, scan on placement via the destination bucket's existing trigger, delivers the verdict after the bytes are already in a product bucket and detonates the per-object cost (see the destination triangle below). Alternative B, skip scanning for migrated files, fails production's require gate as configured and leaves never-scanned bytes downloadable; anyone arguing for it should argue in review.
3. Activation is a separate layer, not part of migration
Migration ends at a per-case signal: rows landed, bytes landed, digests verified. Previews, case-context generation, and ingestion consume that signal with their own controls — eagerly for active cases at cutover, lazily on first open, or by operator list. Carrying v2's existing CCOs over is an optional v2-only cost saver inside that layer; the designed-for path is the CCO backfill pipeline per activated case.
4. The acquirer is the digest authority
Verification compares the staged object against the dump's payloadSha256 — so those digests must be computed by whoever fills staging, during the copy, not trusted from the source system (v2's own values are placeholders). Production hardening is S3-native ChecksumSHA256 on upload with ChecksumMode verification at placement. How strict the requirement is for a weak source (a drive with no checksums) is held open until we know what LEAP can produce.
Open decisionssettled at the 2026-08-24 alignment
Settled — see the tech plan
The alignment picked the options; the buildable contract is the files migration tech plan. In short: staging is the import bucket's per-firm raw/ area with no scanning there and no scan gate at placement. The destination bucket's scan-on-write is the only malware scan, its verdict settles the ledger after landing, a threat verdict disables downloads via a tag-conditioned IAM deny, and the import marker keeps the upload machinery inert. Matters carry a data class (live / recent / archive, firm-configurable cutoff dates, not-migrated never leaves the import bucket), and the activation layer wakes them by class: live gets previews + ingestion + CCO, recent gets previews, archive nothing. Free-tier previews are written at placement; only the pdf-converter tier waits for activation. The sections below are kept for the reasoning that led there.
Where the bytes land, and who scans them there
The rows always land in matter-service; the bytes have two candidate buckets, and the scanning model differs between them. GuardDuty has no per-object opt-out (it writes the scan tag, never reads one), so "mark it pre-scanned" is not on the table. The naive version of option 1 — copy into the matters bucket and let its bucket-level scan fire — costs this, per object:
| 1 · One layout, pay the re-scan | 2 · Import prefix excluded from the scan | 3 · Content bucket | |
|---|---|---|---|
| Bytes land | matters/{id}/matterFiles/…, same as uploads | firms/{firmId}/matters/{id}/{fileId} | content-service's files/… layout |
| Scan on placement | yes, again (already scanned in staging) | no, plan scoped to the app's prefix | no, scanning there is app-managed |
| One-time cost at v2 scale | ~US$1k of GuardDuty plus 1.6M webhook + Inngest events, the metered part to check | none | a matter-service read-path change |
| Permanent cost | none (preview rows + an import marker make placed bytes inert, below) | provenance baked into keys forever | couples to the artifact-storage direction |
| In its favour | uniform bucket; destination bytes independently re-verified | cheapest by far | the storm is impossible by construction |
Facts that soften the edges: CopyObject carries the staging scan's clean-verdict tag onto the placed object either way, so the receipt travels; the matters bucket is already not uniform (previews under their own prefix; uploads keyed by sanitized filename, imports by file id — and id-keying is arguably the better canon if a single layout is ever the goal); and the dev proof measures option 1's event volume at 76-file scale before anyone commits at 1.3M.
A refinement that strengthens option 1: an import marker on the placed object, for free. Staged objects already carry user metadata (the sha256), CopyObject propagates metadata to the destination by default, and the verdict consumer already does a HeadObject per file for the content type — so a lawhive-source=import metadata entry plus a three-line early-return in fileUploaded makes imported bytes inert to the upload machinery even where a preview row is missing or still pending, and turns eager-versus-deferred previews into an explicit choice instead of whatever the preview-row short-circuit implies. Metadata beats an object tag here: it rides the copy at no cost and does not share the object's tag budget with GuardDuty. What the marker does not fix: the re-scan itself still fires (GuardDuty consults no tags before scanning) and the verdict events still arrive, each now ending in a cheap early-return — option 1's per-object cost shrinks to a scan plus a short run, with the Lambda gone. Ingestion never needed a marker at all: it rides the domain event only the app's upload path sends (the upload flow above).
Previews, sized
For an app upload, the preview decision is one pure function of content type, made at row-creation time; generation only ever runs downstream of a malware verdict. That mechanics plus the v2 extension mix splits migrated files into tiers:
| Tier | What | v2 volume | Cost |
|---|---|---|---|
| Previewable as-is | PDF, images, video, audio: FilePreview row with SUCCESS, the original object is the preview | ~731k files, ~45% | one DB row each, no bytes touched |
| Needs the pdf-converter Lambda | Office docs, HEIC — and 752k .eml/.msg filed emails, if they stay matter files | ~136k (~8%) if filed emails become communications; ~888k (~54%) if not | one Lambda call each, at a rate we choose |
| Never previewable | archives and similar | ~4k | a NOT_SUPPORTED row |
Three consequences: tier-1 rows should probably be created at placement time (one extra row per file, closes the gap that makes imported files download-only today, and stops the Lambda fallthrough under destination option 1); tier 2 has to be eager-per-activated-case unless someone builds on-demand machinery, and the placement ledger is already the worklist a backfill worker needs; and the .eml mapping question moves tier 2 by a factor of six, so it goes to the transform side first.
Provisioning and usersflagged from dev
Trying to add himself to a dev test-import firm, Jaime hit the two workstreams that sit after files (thread); a code scout turned the symptoms into gap lists.
Provisioning: the seed script writes one database, a real firm spans four. Normal firm creation is a platform-api orchestration creating a dozen entities across identity, assignment, and transaction services; the migration seed creates four, all in identity. The blockers: no firm bank accounts, and no INVOICE counterparty — which blocks adding members and blocks the "Edit firm to continue" remediation, because the edit path runs the same uncaught query. The proposed split, per entity: relax what's dying, provision what's real. The counterparty gates cannot be honestly provisioned past (the member gate wants a real account number and sort code, so seeding rows means inventing financial details) and counterparties are legacy-ledger machinery the new cashiering path no longer reads — so the fix is two small gate changes the ledger retirement needs anyway. Firm bank accounts, EIN, address, and logins are real facts about a real firm: they belong in provisioning, fed by migration or onboarding data. The precedent exists — fee plans and product routing are already conditional on enrolledToMarketplace; the counterparty gates just predate that discipline. Longer term, "provision a firm for migration" should be a real procedure reusing the admin createFirm fan-out, replacing today's path of a laptop with SSM-fetched database credentials.
Users: without a linking operation, every path mints a duplicate person. The Clerk↔identity link is a Credential row, and imported identities have none; identity lookup is by Clerk id only, never email. So adding a migrated lawyer as a member creates a second identity and a second firm seat for the same human — and if that lawyer simply logs in, the login path creates a fresh empty identity while their imported one holds all the matters. No claim or merge operation exists, and v2 and v3 run separate Clerk instances (dev-verified), so migrated staff must get v3 Clerk users created, not discovered. The missing capability has a clean shape: given an imported identity that already holds a firm seat, create its Clerk user and attach the credential to that identity — every building block (an unused createMembership seam, the credential write pattern, admin email search) already exists.
The rest of the open list
| # | Question | Blocked on |
|---|---|---|
| 1 | LEAP / Woodstock access: export, API, or drive? Can they produce checksums? | George / commercials |
| 2 | Cutover posture: may bytes trail the rows by hours or days, or must a visible case be byte-complete? | product call |
| 3 | Destination bucket and scanning (the triangle above) | Monday + artifact-storage direction |
| 4 | Byte teardown: does tearing down a run delete its placed objects? (Today they orphan; the ledger knows their keys.) | design call |
| 5 | Scope of the GuardDuty plan on v3-matters-* (defined in the infra repo; unverified) | infra repo read |
| 6 | Where filed .eml/.msg emails map: communications or matter files? | transform side |
| 7 | Do the 36k already-deleted v2 files migrate at all? | extractor scope call |
The tracer: the design run as a test
The whole shared side of the architecture is built and staged for review on feature/import-20-file-placement-tracer (62 files, +4,658/−454, on top of the dry-run-registry branch). What changed in the import service, in one picture:
The live proof ran against real S3 semantics (LocalStack), a real local Inngest dev server, and a firm imported from scratch during the proof — so the 76 ledger rows were minted by the importer itself, not seeded. The fixture's files were deliberately staged in two waves, because "bytes trail the rows" is the scenario the design exists for:
The reconciliation surface closed the loop — this table is the byte-complete signal the activation layer will consume:
| matter | placed | quarantined | failed | cancelled | verdictPending | awaitingTagOrBytes | awaitingSweep |
|---|---|---|---|---|---|---|---|
| cas_zlkoq0q3ih8lb4o4 | 75 | 1 | 0 | 0 | 0 | 0 | 0 |
Beyond the happy path, the proof also pinned: a deliberately tampered staged object failing with the digest mismatch in its failureReason; a wrong-bucket URI failing loudly with both bucket names instead of waiting forever; a torn-down run's sweep placing nothing; the infected file's later clean re-tag changing nothing (the first verdict wins); and real filenames with spaces and parentheses surviving S3's notification encoding — a class of silent stall the fixture's own v2 data exposed.
The code went through three independent adversarial review rounds (412 tests at the end, both typechecks and lint clean, the migration replayed on a scratch database). The catches were all one family — states that wait forever instead of failing loudly — which is exactly the failure a reconciliation ledger exists to prevent, and each became a pinned test. Everything is staged, uncommitted, with the commit message drafted; nothing is pushed.
Reproducing the demo
With the local stack up (LocalStack on 4566, the Inngest dev server, import-service with the three bucket variables — the service README's "Local file placement" section has the exact env block):
1 · Stage half the bytes
import:stage-files --half synthesizes the fixture's 76 files, computes real digests, uploads 38 to staging, and writes the rewritten dump packet.
→
2 · Stage the dump
import:stage-s3 puts the rows and manifest into the dump bucket under a fresh prefix.
→
3 · Seed the firm
import:seed-firm creates the v3 firm and the migration actor identity.
→
4 · Run the import
import:run --apply — rows land, 76 promises are minted, and the finish requests the first sweep, which correctly places nothing yet.
→
5 · Scan
import:scan-sim --infect <ref> tags the staged objects and posts the verdict events; the pipeline stamps, resweeps, places 37 and quarantines 1 on its own.
→
6 · Watch it close
import:file-status --run <id> shows the table. Re-run stage-files without --half, then scan-sim again: the late 38 settle with no manual trigger.
What dev proves next
The dev version swaps the three simulated pieces for real ones, in two stages. First, gate off: a small infra change (staging bucket, env, IAM), the v2 acquirer copying the fixture matter's real files, a run driven from the admin UI, and the moment that matters — opening the migrated matter in the app and downloading a real v2 document. Then, gate on: a GuardDuty plan on the staging bucket delivering verdicts through the existing Hookdeck → webhook-api chain (the one hop no local test can reach), with an EICAR test file proving quarantine with nothing simulated. Along the way, the matters bucket's per-object cost gets measured at 76-file scale — the number the destination triangle needs.
Sources
Team thread · provisioning/users thread · dev dry-run report · instrumentation plan · branch feature/import-20-file-placement-tracer · pipeline: services/import-service/src/placement/ · dump file contract: services/import-service/src/catalogue/matterContent.ts · storm mechanics: services/matter-service/src/integrations/inngest/functions/fileUploaded.ts · v2 numbers: prod File table, 2026-08-21 · run of record: imprun_platform-v2_firm_szntlk8kzj2bsbml_12_20260821T213325Z.