Files migration: tech plan

The framework page carries the architecture, the tracer proof, and the option analysis; the team alignment (whiteboard, 2026-08-24) picked the options. This page locks the interfaces those decisions imply. The placement pipeline is built, reworked to this plan, and proven live (PRs #13982 → #14052 → #14058 — the stack as pushed is the final shape described in contract sections 1–3); what follows locks those interfaces and the layers still to build around them: the data classes, activation, ingestion and CCO wiring, and the teardown byte-deletion option.

The pre-stage, before anything above runs. Getting a firm's data into the import bucket is its own leg, and the one a firm's CIO will ask about: production-grade channels end to end — a system we operate, with audited access, never a developer's laptop. For v2 it is a bucket-to-bucket copy inside our own AWS estate, reconcilable object-for-object with S3 Inventory (Daniele is researching exactly this). For LEAP and other sources the channel is agreed per source when the engagement is real; what is fixed now is the destination and the bar — everything lands in the per-firm raw/ area through a channel we can name to the firm's security reviewer.

Problem, goals, non-goals

Problem. The placement pipeline is built against decisions that were provisional. The alignment changed one of them (where scanning happens) and settled the rest (staging location, data classes, activation policy). Nothing downstream of placement (activation, ingestion routing, CCO backfill) has an interface yet.

Goals. Once shipped: a firm's files migrate from any source through the import bucket into v3-matters-{env} with per-file integrity proof and a reconciliation ledger; a threat verdict makes the object undownloadable; imported bytes trigger no accidental previews or ingestion; matters wake up by data class, on demand, without touching live-firm ingestion capacity; not-migrated matters never leave the import bucket and are retained there for the legal period.

Non-goals. Escape-hatch file producers and exotic types (call recordings, transcripts): they plug into the same jsonl + raw/ contract later and need no new machinery here, so they are explicitly deferred. Byte deletion on teardown is designed but deferred: an operator option on the teardown (ledger-driven, never a default), and the option matters because ids — and so target keys — are derived per run, meaning a re-import never overwrites a torn-down run's bytes; tearing down to re-import without deleting leaves orphans. The class thresholds themselves: the system takes dates as configuration, and the business picks the values.

Domain

Two new concepts, one changed one:

DataClass: how much of the platform wakes up for a migrated matter. LIVE (work in progress, any age), RECENT (closed on or after a firm-configured cutoff date), ARCHIVE (closed before it). A fourth state, not-migrated, deliberately has no enum value in v3: those matters are filtered out by Dagster and exist only in the import bucket's raw/ area (retained for the 7-year legal period). Thresholds are dates rather than rolling windows, computed once at transform time, so a matter's class cannot drift between the dump being cut and activation running.
The classes are an educated guess. The business may rename them, add one, or change what earns one (practice area, say). A class means something in exactly two places — Dagster's classifier and the activation policy map — so any of those changes touches two configuration points and no schema.
MatterActivation: the activation layer's ledger, one row per imported matter recording its class and what has been requested for it. It mirrors FilePlacement's pattern (create-only rows, compare-and-set settling, an Inngest sweep) because it faces the same replay problem, applied to matters instead of files.
FilePlacement (exists): the scan verdict's meaning changes. It is now stamped after placement, from the destination bucket's verdicts, so PLACED → QUARANTINED becomes a legal transition.

The seams between owners: Dagster owns classification and filtering (the class arrives on the wire; v3 never computes it); matter-service owns previews and the upload machinery the marker must not trip; the ingestion repo owns the backfill queues; the CCO backfill coordinator (agents side) owns whole-case context generation. Activation only requests work; each owner executes its own.

Contract

1. The boundary (Dagster → import service)

matterImport gains one required field; not-migrated matters are absent from every jsonl, whole matter and children, because Dagster filters before emission:

// catalogue/matter.ts (boundary schema)
dataClass: z.enum(["live", "recent", "archive"])

matterFileImport is unchanged. Its payloadUri (decoded-key contract, enforced by the existing refine) now points at s3://{IMPORT_DUMP_BUCKET}/firms/{firm}/raw/{snapshotId}/files/{externalRef}, payloadSha256 computed by whatever wrote the bytes into raw/. Threshold configuration (the cutoff dates per firm) lives with Dagster; v3 never sees it, only the resulting class.

2. Placement (import service: the delta from the built tracer)

The gate simplifies: with no pre-placement scan, assessPlacement's decision table loses its verdict/tag arms. Copy when the bytes are staged and the digest matches; the verdict arrives later.

// per EXPECTED row, verdict checked first — the whole table:
verdict THREATS_FOUND   -> QUARANTINED (no head, no copy)
bytes absent            -> stays EXPECTED (next sweep retries)
bad URI / wrong bucket  -> FAILED (reason names both buckets)
sha256 mismatch         -> FAILED (digest mismatch in failureReason)
otherwise               -> CopyObject, settle PLACED

The copy carries Metadata: { sha256, "lawhive-source": "import" } (the marker; propagated free by CopyObject). In the same files-pillar transaction that writes File + MatterFile, the pillar now also writes the free-tier FilePreview row: SUCCESS pointing at the original object for image/video/audio/PDF content types, NOT_SUPPORTED for archives, using the same pure decision function as the app upload. The pdf-converter tier stays PENDING and waits for activation. IMPORT_SCAN_GATE and the staging-side tag read are deleted; IMPORT_STAGING_S3_BUCKET collapses into the dump bucket: one env var fewer, and the raw/ prefix is a convention rather than configuration.

Built, with three refinements the code settled. The copy also carries the file's ContentType (the ledger records it from the filename at run time; CopyObject with metadata replacement would otherwise drop it and every file would download as a generic binary). The target key is matters/<matterId>/matterFiles/<fileId> — the same prefix family a live upload gets, so an imported file is indistinguishable from an uploaded one by its path; it is built only from minted ids, so a replayed hop re-mints the identical key. And the operator surface gained a second verb: requeueFailedPlacements puts FAILED rows back to EXPECTED for the next sweep, with one invariant guarding both verbs — a row that has ever carried a THREATS_FOUND verdict is never put back in service.

Reruns and deltas cost no copies. Ledger rows are create-only on (run, externalRef) and a sweep reads only EXPECTED rows, so asking for placement again — a replayed hop, an operator's sweep, a second look after late bytes — re-copies nothing that already landed. A delta emission from Dagster adds only the new rows, and only those get S3 calls. The one rerun that does pay full price is teardown-then-reimport, because a new run mints new keys; that is the case the teardown byte-deletion option exists for. And sweeps are operator-driven by design: nothing watches raw/ for arriving bytes, because an import is something a person runs and watches from the admin screen, not something that starts because bytes landed.

3. Verdict settlement (destination scan → ledger)

The existing recordFileScanVerdict consumer re-targets: it now matches verdicts whose s3Bucket is the target bucket and whose objectKey equals a FilePlacement.targetKey. The match is exact: the import service mints these keys itself, and they contain nothing that needs URL encoding. What this costs, per million files: GuardDuty scans each object once as it lands (its own service, no calls from us), each scan emits one event, and the consumer settles it with a single indexed database update — FilePlacement carries an index on targetKey for exactly this. Nothing probes S3 at verdict time; the only metadata probe placement ever makes is the one HeadObject per file at copy time. Stamping:

NO_THREATS_FOUND -> scanVerdict CLEAN     (status stays PLACED)
THREATS_FOUND    -> scanVerdict THREATS_FOUND, status PLACED -> QUARANTINED
FAILED/SKIPPED   -> scanVerdict FAILED    (status stays PLACED; report shows it)

No resweep is requested (nothing downstream waits on a verdict any more); byteComplete is unchanged (a quarantined row is settled). The run report grows a scanned / clean / threats line so an operator sees verdict coverage converge after placement.

4. Downloads disabled on a threat (Terraform, platform-v3)

A deny on matter-service's task role, the content module's pattern inverted from allowlist to blacklist so legacy untagged objects are unaffected:

# tf/.../matter/malware-deny.tf — deny GetObject only on confirmed threats
statement {
  effect  = "Deny"
  actions = ["s3:GetObject", "s3:GetObjectVersion"]
  resources = ["${matters_bucket_arn}/*"]
  condition {
    test     = "StringEquals"
    variable = "s3:ExistingObjectTag/GuardDutyMalwareScanStatus"
    values   = ["THREATS_FOUND"]
  }
}

This statement does not exist yet — the content module carries the same pattern for its own bucket, but the matters bucket has no tag-conditioned deny today, and adding it (with its own security review) is what build-sequence step 2 is. Presigned URLs inherit the signer's role, so every URL for a tagged object dies the moment GuardDuty tags it. It needs no schema field and no read-path code change, and it covers app uploads too — closing a real gap that predates the migration: today nothing on the read path checks the scan tag, so a matter file GuardDuty has flagged can still be downloaded until someone acts.

5. The upload machinery stays inert (matter-service)

fileUploaded (the verdict-triggered preview generator) gains one early return: HeadObject already happens; if metadata["lawhive-source"] === "import", log and stop. Ingestion and case-context were never at risk (they ride matter-service/file.uploaded, which only the app path sends).

6. Activation (import service orchestrates, owners execute)

Events (work-request style, matching file-placement.batch.requested):

// import-service internal, batch-chained sweep over MatterActivation rows
"import-service/activation.batch.requested"  { importRunId: string; afterMatterRef?: string }

The class policy is data, with the agreed defaults, overridable per invocation:

const DEFAULT_ACTIVATION_POLICY: Record<DataClass, ActivationAction[]> = {
  LIVE:    ["previews", "ingestion", "caseContext"],
  RECENT:  ["previews"],
  ARCHIVE: [],
}

Control surface (tRPC, admin-gated like the placement procedures):

imports.admin.requestActivation({
  importRunId: ImportRunId,
  dataClasses?: DataClass[],          // default: all
  actions?: ActivationAction[],       // default: the policy per class
}) => Promise<{ requested: number }>  // matters whose rows moved to REQUESTED

imports.admin.getActivationReport({ importRunId: ImportRunId })
  => Promise<ActivationReport>        // per-class counts of each action state

Errors: ImportRunNotFound; ActivationRefused for dry runs, torn-down runs, and runs that are not byte-complete for the requested matters (activating a matter whose files are still landing would ingest half a case; the refusal carries the per-matter counts in its message).

What each action does when the sweep picks up a REQUESTED row:

previews: walk the matter's PLACED ledger rows whose FilePreview is PENDING, invoke the pdf-converter Lambda at a concurrency we set (the ledger is the worklist; nothing goes through fileUploaded). Settle DONE when no PENDING previews remain.
ingestion: enqueue one asset-registration message per placed file onto the ingestion repo's backfill queue (plain SQS sends, no per-file Inngest runs, no CCO side effects). Message schema is the cross-repo contract: { matterId, fileId, s3Key, contentType, firmId, source: "migration" }. Settle DONE on enqueue (delivery is the queue's job).
caseContext: one request per matter to the existing CCO backfill coordinator (its current input contract; this plan adds no shape to it). Settle DONE on acceptance.

7. Backfill queue wiring (ingestion repo)

The backfill queues exist but no worker consumes them. The change: workers poll the live queue first and take backfill work only when the live queue is empty (two-queue priority receive, no new infrastructure). The cross-repo contract is the message schema above plus the invariant that backfill consumption must never delay a live-queue message by more than one in-flight batch.

Risks

A threat object exists in the product bucket between landing and verdict (minutes). Accepted at the alignment: the deny kills downloads at verdict time, and matter files are only ever served when a person opens them. The quarantined ledger row is the operator's signal; deleting the object stays a manual step for now.
The deny is new IAM on the product's read path. A mistake blocks real downloads. Mitigation: the blacklist form means only THREATS_FOUND objects can match, it ships to dev first, and it is one statement, so reverting it is trivial. It changes a trust boundary, so it gets an independent security review before merge.
Verdict-consumer retarget must not double-match a staging URI and a target key. The consumer matches exactly on bucket plus full key, and it is the only matcher: nothing else in the service interprets a scan event.
FilePreview writes grow the files pillar's transaction. One extra row per previewable file, in the same database and transaction; the existing per-hop timings measure it before anyone worries.
Class misconfiguration migrates the wrong book. Thresholds are Dagster config reviewed with the firm; the not-migrated filter is the only one whose mistake loses data by leaving it out, and raw/ retention means a wrongly-skipped matter can be re-emitted without going back to the source.

Alternative recorded: pre-placement scanning (the framework page's original decision 2) lost at the alignment because destination scanning already exists, costs one scan instead of two, and needs no GuardDuty infrastructure on the import bucket. Its price is the landing-to-verdict window, judged acceptable given the deny.

Build sequence

Land the current stack: #13982, #14052, #14058. The placement rework this plan describes is already folded into it — the stack as pushed reads as the final shape (gate simplification, marker + content-type metadata, free-tier FilePreview in the pillar transaction, staging env collapse, verdict-consumer retarget with PLACED → QUARANTINED, upload-aligned target keys, the requeue verb). Proven live locally end to end.
Inert upload machinery + the deny (platform-v3): fileUploaded early-return; the tf deny statement. Small PR + independent security review for the IAM change.
Boundary dataClass (with Daniele): schema field, Dagster classification + not-migrated filtering, MatterActivation rows written by the data pipeline. Blocked on nothing; can run parallel to 2–3.
Activation v1 (import-service + admin): the sweep, the policy, requestActivation/getActivationReport, previews action only. Admin panel row on RunDetail beside the placement panel.
Ingestion wiring (ingestion repo + activation's ingestion action): priority receive on the workers, the message contract, enqueue from the sweep.
CCO action: the coordinator request per live matter.
Teardown byte deletion (import-service + admin): the opt-in deleteBytes on the teardown run, ledger-driven, with the admin checkbox showing what it would remove. Needed before any teardown-then-reimport on a real firm, because re-imports never overwrite a torn-down run's keys.
Dev proof, end to end: real GuardDuty verdicts on the destination (including an EICAR quarantine and a dead download URL), a classed firm activated live-only, ingestion visibly draining from the backfill queue only while live traffic is idle.

Each step ships alone and the system is coherent after every one: after 1 the bytes flow under the new scan model; after 2 a threat verdict locks downloads; after 4 an operator can wake matters up; the rest add the remaining actions.

Success

The dev proof in step 8 is the acceptance test. Monitoring: the run report's verdict-coverage line converging to 100% scanned; the activation report reaching DONE for every requested action; ingestion's live-queue age staying flat while the backfill queue drains; zero fileUploaded Lambda invocations attributable to import-marked objects.

Scaling placement

A question worth answering on the page: does the ledger need row locks, so two workers cannot move the same file twice?

No — and the reason shapes how placement scales. Correctness never depended on exclusivity. Every state change is a conditional update — WHERE status = 'EXPECTED' AND scanVerdict <> 'THREATS_FOUND' — so of two racing workers, one settles the row and the other matches nothing. The copy itself targets a key derived only from ids, so a duplicate copy writes the same bytes to the same place. A race can waste an S3 call; it cannot corrupt the ledger. On top of that, Inngest runs one sweep chain per run (concurrency: { key: importRunId, limit: 1 }), so in practice the race never happens — that limit protects cost, not correctness.

The sequential chain is sized for one firm at a time, and per-firm that holds: a batch of files per hop, tens of thousands of files in hours. The whole v2 book is a different order — the bucket holds around three million objects, about 1.6 million of them matter files a manifest will reference — and there the single chain per run is the bottleneck.

When that day comes, the scale-out is assignment, not locks: keep the single-flight dispatcher, but have it deal disjoint batches instead of processing them — one event per cursor range, N workers each consuming their own. Exclusivity comes from no two events naming the same rows, so nothing new is locked and everything built (the cursor, the per-run key, the event chain) is reused. The alternative — workers claiming batches with FOR UPDATE SKIP LOCKED plus a lease reaper for dead claimants — buys the same guarantee at the cost of new columns and new machinery, and earns its keep only if something outside Inngest ever pulls from the ledger directly.

One honest limit either way: "copy exactly once" is not a promise two systems can make. A worker can die between the S3 copy and the ledger settle, and the retry re-copies that batch. Every design here is at-least-once with idempotent copies — which is why the conditional updates and derived keys stay load-bearing no matter what orchestration sits above them.

Sources

Framework page · Daniele's investigation · alignment whiteboard 2026-08-24 · PRs #13982 / #14052 / #14058 · placement pipeline: services/import-service/src/placement/ · content-module deny precedent: tf/…/content/malware-scan.tf.