Files from v2 to v3: stage, scan, copy, hydrate

Goal

We are designing the files hop of the firm import engine, not a v2-only copy script. v2 is the first source and the cost proving ground. The same contract must later carry files from other firms and systems (LEAP, dumps, SFTP, another PMS) without a new blob store or a new GuardDuty story.

That split already exists on the import boundary: source transform is per system; the packet and the import-service are generic.

Success looks like:

• v2 files usable on v3 on day one (list and download), at 3 TB / 4 million object scale
• files processed so Lawrence can see them: per-file ingest, then CCO backfill, not left as opaque blobs
• a later firm or system reuses the hop

A few terms used below. The dump is the regulated S3 packet for one import (firms/{firm}/…). A files cell is the files pillar of one matter in the unit model grid: that is the failure unit. CCO is the Case Context Object Lawrence builds for a matter. ICI is Initial Case Information, Lawrence’s initialCaseInformation asset (the matter’s opening facts, not a document).

Recommendation

Stage, then server-side copy into matter-service. Do not re-enter the organic upload lifecycle.

What generalises versus what stays per source:

Efficiency, versus re-uploading every file through the product: one manual CopyObject v2 → dump raw/, scan on dump then dest auto-scan on CopyObject (cost both), no organic upload lifecycle, no live CCO dump, no 4 million Inngest events 1:1.

Bytes leave the source before the load run, into firms/{firm}/raw/{snapshotId}/files/. GuardDuty scans that landing. JSONL under target/…/data/ has payload_uri pointing at the raw object. There is no second blob tree under target/…/files/. Serving keys are minted before place. Display name, visibility, and folder live on the JSONL row.

Load-time order: matter core is already committed (walker: core before the files pillar). Then commit the files unit (MatterFileFolder, File, MatterFile, FilePreview). Then BlobStore.place CopyObject onto File.s3Key. Today the writer still places before writeUnit; invert that.

A bad, missing, or infected blob fails the files cell, not the matter core. Scan at raw landing: unscanned objects are listable but not readable. Import copies from raw only after NO_THREATS_FOUND. IMPORT_SCAN_GATE=require is the load-time backstop.

Hydrate after the row exists: list and download on day one. Files unit writes FilePreview; dest GuardDuty drives preview. Per-file Lawrence ingest via a post-commit file.imported event, not organic file.uploaded (live CCO sits on that event). CCO uses the existing case-context backfill coordinator. Who fires that coordinator is an open question (lean: explicit and manual). No participant “new file” notifications.

Not content-service File (PENDING to READY, _external keys). That model is templates and content uploads. Matter documents stay on matter-service.

Buckets

The transform default s3://lawhive-v2-case-files/… is a label, not the live bucket. Confirm v2 names in Doppler. There is a newer Terraform bucket {env}-v3-matter-{suffix}; that is not what the env var points at yet, so import CopyObject should target v3-matters-* until matter-service stops using the shim.

Copy path for v2: case-files bucket → dump raw/…/files/ → v3-matters-{dev|prd}. Same-region CopyObject on both hops (cost in Costs, volume, and S3 limits).

S3 key conventions

Product folders are not S3 directories on either platform. Keys are case- or matter-scoped prefixes; folder membership is a database join.

v2 (CASE_FILES_S3_BUCKET_NAME, everything under cases/{caseId}/):

• Organic upload: cases/{caseId}/{uuid}-{url-encoded filename}
• Preview PDF: cases/{caseId}/preview/{same-leaf}.pdf
• Other writers in the same bucket: filed-emails/, emails/{token}/, bundles/, agreements/, call-recordings/{id}-{sid} plus sibling -transcription.json / .txt
• Object metadata: uploader, case-id, original-filename. Cap 200 MB.
• Copy File.s3Key. Do not rebuild keys from display names.

v3 matter documents:

• Organic: matters/{matterId}/matterFiles/{timestamp}-{sanitized filename}
• Import prototype today: firms/{firmId}/matters/{matterId}/{fileId}
• Recommendation: mint the serving key before place in organic shape, with fileId (or timestamp) so two firms cannot collide. Do not pre-bake v3 keys in dbt. Do not rewrite JSONL payload_uri.

Dump:

• Binaries: firms/{firm}/raw/{snapshotId}/files/… (shard this prefix; 4 million objects in one prefix will hotspot)
• JSONL: firms/{firm}/target/{exportId}/data/matter_file.jsonl

How v2 manages files

From platform (packages/api/src/modules/cases/files/) and caseFiles.tf. Story first, then the records.

• Upload: the API mints a 15-minute presigned POST into the case-files bucket, key cases/{caseId}/{uuid}-{filename}, cap 200 MB. The browser posts the bytes. No malware scan.
• Register persist: HeadObject, then two database rows. File is the blob. CaseFile is the place on the case (who uploaded, who can see it).
• Kick-off, after the rows exist, before the mutation returns: request preview (SNS to the LibreOffice lambda) and queue AI ingestion. The API waits for those requests to be accepted, not for preview or ingest to finish.
• Download: a 10-minute presigned GET, gated by case party and visibility.

Also true, and not on the happy-path diagram:

• Product folders are not S3 directories. A lawyer or firm employee with writeOwnCases names the folder in the Documents UI (title plus colour; colour is usually hashed from the name). Order is assigned automatically. There is no default folder on case create. Clients cannot create folders.
• Putting a file in a folder is a later database join, not part of the S3 key. Organic upload does not take a folder: the file lands unfiled. The same lawyer then setFolders / bulkSetFolders, or creates a folder and assigns the file in one step. Outlook import can pass caseFolderId at register. A file can sit in several folders, or none.
• Other writers use the same bucket with different key prefixes, not product folder names. Same File plus CaseFile pair, source stamped on CaseFile.
• Soft delete hides the row; the S3 object stays (production object lock).

What that means for migration:

• Copy live documents only (File.isDeleted is false). Soft-deleted files stay in v2 S3. Whether they should come across for retention is an open question.
• Folder many-to-many collapses to one folder on the boundary (lowest order), already in the v2 mapping.
• v2 ingestion success is dropped; v3 processes again so Lawrence can see the files.
• Call recordings and transcripts sit in that same bucket as fields on Call, not CaseFile. They are not in the documents list. The 2.5 TB / 3.9 million object inventory is the whole bucket, so it includes this call data. What we do with those bytes is an open question.

Schema cheat-sheet: File (s3Key, names, fileSize, isDeleted, preview and ingestion statuses); CaseFile (caseId, fileId, visibility, uploader, source); CaseFileFolder (flat, many-to-many). The current transform still uses placeholder payload_sha256 and payload_uri pointing at the v2 key.

How v3 manages files

Same shape as v2: presigned upload, register the rows, then download with a presigned GET. The important difference is when preview runs: v2 kicks it in the register mutation; v3 waits for a GuardDuty webhook on the matter bucket. Lawrence ingest and live CCO both listen to matter-service/file.uploaded, not to that webhook.

Two independent fans after the bytes exist:

GuardDuty starts on the S3 PUT, so it can finish before register. Preview persist then misses File / FilePreview until a later retry. That is the same race import CopyObject has if you copy before the files unit commits.

• Upload: 15-minute presigned POST into MATTER_DOCUMENTS_S3_BUCKET, key matters/{matterId}/matterFiles/{timestamp}-{filename}, cap 200 MB. ObjectCreated starts dest GuardDuty immediately, not after register.
• Register persist: HeadObject, then File, MatterFile, FilePreview. Then matter-service/file.uploaded (fileId, matterFileId, matterId, s3Key, bucket, filePreviewId). Listeners: lawrence-api ingest and live CCO. Optional Knock notify after this, still before the HTTP response.
• Dest preview chain (not on file.uploaded): GuardDuty → webhook-api → webhook/s3/malwareScan.completed|failed|skipped → matter-service fn id malwareScan.completed → preview lambda only if NO_THREATS_FOUND.
• Download: 10-minute presigned GET, gated by matter participant and ALL vs FIRM_SIDE_ONLY.

Folders are a flat list per matter, one folder per file (not v2 many-to-many). Lawyers name them; matter templates can seed titles; LEGL can auto-create a Legl folder.

Import does not walk the presigned-POST upload. After matter core is committed, it commits the files unit (File, MatterFile, FilePreview; import-service creates FilePreview, not dbt), then CopyObjects onto File.s3Key. Dest GuardDuty drives preview. Then the ingest event. Do not call insertMatterFile. Do not run live CCO off that event.

Malware scanning

v2 case files and the import dump are not scanned today.

Recommended: scan when the raw packet lands.

Dump GuardDuty (steps 1–4) is the import scan control plane. Dest GuardDuty (step 7) is the organic preview path from the v3 section, not a second copy of dump policy. Preview is not on file.uploaded in either path.

What import CopyObject into v3-matters-* actually fans out: dest GuardDuty auto-scan, Slack if infected, malwareScan.completed then PDF lambda if File plus FilePreview already exist. Not by itself: Lawrence ingest, live CCO, Knock. Those need an event after commit, and live CCO must not run for import.

Organic is the opposite order (PUT then register) because the browser owns the presigned POST and insertMatterFile HeadObjects first. Do not change that path.

A later source that must transform bytes (a format v3 cannot store) can still write a derived object. That is per-source, not the v2 path.

Inventory, then dbt

Inventory is not dbt. Copy objects from v2 to dump raw/{snapshotId}/files/ one time. Do this by hand. Do not put this copy in Dagster. Do not put this copy in the import worker. After that copy, the inventory job writes SHA-256 of the object bytes into BigQuery blob_inventory (source key, dump URI, sha256, size). Hash from the v2 source (or a copy-job sidecar), not from unread dump raw/. Later add scan status. Do not put that table under raw/…/files/. GetObject is denied there until scan. Put the live v2 bucket name on this copy job.

Wait for GuardDuty tags in Malware scanning. Then run dbt.

The v2 mapping already has matter_file and matter_file_folder. Keep that tabular mapping. One folder (lowest order). Visibility all becomes ALL. Live files only (not isDeleted and fileSize > 0).

Today's matter_file JSONL still points at the v2 bucket. The hash is a placeholder from the key, not the object bytes. Import-service cannot CopyObject that packet. It reads payload_uri and payload_sha256 from the JSONL.

dbt joins blob_inventory in an earlier model. That join fills payload_uri, payload_sha256, and file_size. The dump export SQL that writes import JSONL runs later. Inventory does not write filename, folder, or visibility. Those stay on the v2 mapping.

dbt does not emit FilePreview. dbt does not emit v3 serving keys. dbt does not emit ingest status. JSONL is the source-shaped packet: payload_uri, hash, size, filename, folder, visibility.

1. Join File.s3Key to blob_inventory. Set payload_uri to the dump raw object. Do not mint a v3 serving key. Do not write binaries under target/…/files/. Do not add a file_preview JSONL.
2. Set payload_sha256 to the inventory hash. Replace to_hex(sha256(concat('v2:s3Key:', …))). Placeholder hashes are not loadable.
3. If an object is missing, exclude the row with a reason, or fail the model. If size or hash does not match, do the same.
4. Add not_null tests on payload_uri, payload_sha256, file_size, and filename.
5. Point payload_bucket at the dump. Do not put the live v2 bucket name in the exported JSONL.

The dump export SQL still sorts parent, then pk. That is the existing dump contract. It is not part of the inventory join.

Run these models after inventory exists and after the malware wait. dbt writes dump raw/ payload_uri from inventory. Import-service must not rewrite JSONL.

Import-service changes

The files pillar writer already exists (importers/files.ts): validate JSONL, fail the files cell on a bad blob. Today place runs before writeUnit (dest-preview race). Dump Terraform is its own section below.

Locked order (units before blob copy):

1. Matter core unit is already committed. The walker does not run the files pillar until core succeeded. matterId exists. Do not CopyObject with no matter row.
2. Scan gate plus mint dest key on each new file (IMPORT_SCAN_GATE, fileId, platformFileKey). No dest write yet.
3. Commit the files unit: MatterFileFolder, File (s3Key = minted key, displayName from JSONL filename), MatterFile (source: IMPORT), FilePreview PENDING. Import-service creates this row; it is not in the dump. This is writeUnit, not insertMatterFile.
4. Then BlobStore.place: CopyObject from dump raw onto that key. Dest GuardDuty sees FilePreview. If place fails, fail the files cell and keep the committed rows. Heal retries place on the same key.
5. Then ingest event (file.imported). Not organic file.uploaded unless live CCO is skipped for IMPORT.

Do not invert organic (client PUT still before register).

Already in place: IMPORT_SCAN_GATE, assertBlobIsImportable, createLocalBlobStore (placesBytes: false), createRun refuses a real prod run (BLOB_STORE_CANNOT_PLACE), dump reader for target/…/data JSONL.

Build in import-service:

1. createS3BlobStore. scanState via GetObjectTagging on payload_uri (raw). place: CopyObject from raw into MATTER_DOCUMENTS_S3_BUCKET, verify size / sha256, return dest key. Set placesBytes: true. Wire it in processNextBatch for non-dry runs; keep local no-op for dry run and laptop.
2. Mint the serving key in platformFileKey. Dump keys stay dump keys. Dest: matters/{matterId}/matterFiles/… with fileId (or timestamp) so firms cannot collide.
3. IAM and env. Task role: dump raw GetObject plus GetObjectTagging (only after NO_THREATS_FOUND), matter bucket CopyObject / PutObject / GetObject for verify. MATTER_DOCUMENTS_S3_BUCKET on the worker. Change the README: load copies raw, not target/files.
4. Load from raw is required. GetObject / CopyObject only after NO_THREATS_FOUND. Do not stage a second blob tree under target/…/files/.
5. Collision. Fail the files cell if the dest key already exists (HeadObject / If-None-Match). Do not overwrite another firm's object.
6. Teardown. ImportedRecord lists the File rows this run wrote. Teardown today only deleteMany those Postgres rows. Before dropping a File, DeleteObject on File.s3Key (and any preview object under matters/…/preview/ if we wrote one). Dest GuardDuty and Inngest are not cancelled. If rows and dest object are already gone, the preview fn has retries: 0; it fails or no-ops. Harmless noise, not a resurrect.
7. Hydrate is not the writer. After successful place, emit ingest (file.imported). CCO backfill is a separate coordinator; whether anyone fires it automatically is an open question. Do not call insertMatterFile.
8. Throughput follows the unit walk. Load-time CopyObject is after each files unit commits. Do not Batch Copy into the matter bucket ahead of the walker. The v2 → dump raw/ copy is one manual job, not a step of each load. Hash verify must not download 3 TB onto the task. A short window exists where File.s3Key has no object yet; download 404s until place succeeds.

Infra changes

Lives next to the dump bucket in platform-v3 import Terraform, not in infrastructure-v3 and not in the shared infra S3 module.

1. New platform-v3/tf/modules/service/services/services/import/malware-scan.tf, based from content's GuardDuty IAM (assume role for malware-protection-plan.guardduty.amazonaws.com, GetObject, PutObjectTagging, EventBridge managed-rule, validation object) but: protection plan on module.dumps.name; scan the whole dump bucket in Terraform (object_prefixes omitted, because keys are firms/{firm}/raw/… so a prefix of raw/ would not match); tagging ENABLED. GetObject deny stays on …/raw/…/files/* only.
2. Required bucket policy, based from content DenyExternalFileReadsUntilGuardDutyMarksClean, scoped to …/raw/…/files/* (not target/.../data/ JSONL). Deny GetObject / GetObjectVersion unless tag NO_THREATS_FOUND. Do not deny ListBucket. Exempt the GuardDuty malware-protection role. Deny PutObjectTagging of GuardDutyMalwareScanStatus except GuardDuty. Import-service needs GetObjectTagging. CopyObject from raw only after clean; S3 403 otherwise.
3. EventBridge THREATS_FOUND to the existing org Slack SNS. Skip Hookdeck / webhook-api on the dump for v1.
4. Import-service task role (the worker that runs place). Give it only what scanState and CopyObject need; do not give it GuardDuty's “scan this object” API.
   • On dump …/raw/…/files/*: GetObjectTagging and GetObject. The bucket policy still 403s those gets until the tag is NO_THREATS_FOUND. JSONL under target/ is a different prefix.
   • On MATTER_DOCUMENTS_S3_BUCKET: PutObject (CopyObject dest), GetObject to verify, DeleteObject for teardown.
   • Do not attach guardduty:SendObjectMalwareScan. Auto-scan on ObjectCreated already tags the dump object; the worker only reads that tag.
5. Apply order: dump bucket already exists; protection plan is additive. Document in the import Terraform README.
6. Open: keep raw and target on this one bucket, or split into two (open questions). Current Terraform assumes one module.dumps.

The wait for scan tags before writing the import packet lives in Dagster, in Malware scanning. The loader's IMPORT_SCAN_GATE is only a backstop.

Follow-up processing after imported files land

Organic v3 does more than write File plus MatterFile. insertMatterFile also creates a FilePreview row (PENDING), sends matter-service/file.uploaded, and optionally notifies other matter participants.

That event kicks Lawrence per-file AI ingestion (lawrence-api fileUploaded.ts: safeIngestAsset, flips aiIngestionV3Status to PENDING). It also currently fans into caseContextUpdateRequired, which is the live incremental CCO pipeline: one new asset on an existing case. That path is the wrong one for a migrated matter.

CCO: the live pipeline is one new asset on an existing case. Dumping a full migrated matter onto that path is expensive and lower quality. Use the incremental backfill in lawrence-engine (incremental_backfill/main.py) instead. Same statement extraction and CCO generation as organic, but the case is walked in chronological rounds so each CCO sees prior context. Local seed docs warn not to run it casually just to populate fixtures.

ICI is required for the first CCO. Backfill round 0 is ICI alone, then later rounds add files, notes, and comms on top of that first CCO.

How the backfill pipeline runs

Who fires the coordinator is an open question. Do not send process-case from import-service; the coordinator does that.

1. Wait until the matter’s assets exist for Lawrence. Files cell succeeded, file.imported ingest has run (aiIngestionV3Status), and any notes or comms the CCO reads are on the matter. Backfill fetches asset IDs from platform; empty or not-yet-ingested assets produce a thin or empty CCO.
2. Fire one coordinator event, not one event per file: case-context-service/backfill/coordinator.requested. Payload: case_ids (v3 matter ids, min 1) and max_buckets (default 5: hard ceiling on CCO rounds after ICI). Batch many imported matters on one event. Coordinator concurrency is 1; it fans every case at once and Inngest queues them.
3. Coordinator invokes the per-case worker (incremental-backfill-case, also triggerable as …/backfill/process-case.requested for a single matter). Worker limits: 10 cases in flight, 1 run per case_id. Extra worker fields (from_scratch, bucket size) are for manual resets, not what import should set.
4. Per case, sequential rounds (idempotent: already-processed assets are skipped):
   • Optional wipe only if from_scratch.
   • Round 0: extract the ICI asset, generate the first CCO. Skip if ICI already processed; warn and skip if the matter has no ICI.
   • Rounds 1…N: remaining assets split into chronological buckets. Each round extracts that bucket with the current CCO as context, then regenerates the CCO. Abort remaining rounds if a CCO generate fails.
5. When it is done: statementbasedcco plus summary exist. Status already_complete if every asset was already processed. Failed ingest or delayed CCO is not a failed files cell.
6. Never attach caseContextUpdateRequired to file.imported. Stampede: 4 million files versus one backfill per matter.

PDF preview on import

Preview is dest GuardDuty, not file.uploaded. After matter core is committed:

1. Import-service writes File plus MatterFile plus FilePreview PENDING and commits that files unit. After a succeeded files cell, File is already there. Hydrate does not create File.
2. Then CopyObject from dump raw onto File.s3Key. Dest GuardDuty auto-scans. malwareScan.completed with NO_THREATS_FOUND runs the preview lambda against the existing FilePreview row. Preview persist needs that row (fileId unique FK on File). Without it the lambda can still run and then cannot save.
3. Dest GuardDuty can still beat the insert if you copy before commit. That is why import inverts organic: rows first, then CopyObject. After the cell succeeds, File is present; the lasting writer gap without this invert is FilePreview.
4. Dump GuardDuty on raw is the malware gate. Dest scan is the preview trigger. Files cell success must not wait on dest tags. Cost both hops.
5. If CopyObject fails after commit, keep the rows. Fail the files cell. Heal retries place onto the same File.s3Key. Until place succeeds, download 404s.
6. After a successful place, emit file.imported for per-file Lawrence ingest only. That is not CCO backfill. Do not send file.uploaded (live CCO and Knock sit on that event). No participant notify. Skip if the ImportedRecord already existed.

Cutover bar: files are usable when list and download work. Preview and AI are progressive: expected after a succeeded files pillar, not in the same database transaction. Preview on hour zero is a capacity question (lambda plus Inngest), not a reason to block the load.

Costs, volume, and S3 limits

This section gives the cost of the files hop. It does not give the cost of Lawrence, CCO, or preview.

Cost the production dump, not the dev dump. Production is env prd-us (US market). The dump bucket uses var.ctx.aws_region. Terraform for prd-us sets that region to eu-west-2. Use the S3 Standard list price for eu-west-2. The dump is regulated (is_regulated = true).

Plan the work for 3 TB and 4 million objects. The measured values are 2.5 TB and 3.9 million objects. The mean object size is about 0.6 MB. The bucket count can contain recordings, previews, and objects with no CaseFile row.

The two copy steps are in Recommendation and Import-service changes. This section only gives their cost.

S3 CopyObject in eu-west-2 has no data-transfer charge. This is true when the AWS accounts are different. Each copy has a COPY/PUT request charge. The import worker does not pull the file through the task. Different accounts need IAM. Different accounts do not add a GB charge.

These values are list prices. They are not a quote.

• S3 Standard: about USD 0.024 per GB per month (about USD 74 for 3 TB).
• COPY: about USD 0.005 per 1,000 requests.
• GuardDuty: about USD 0.09 per GB plus USD 0.215 per 1,000 objects. The object fee is larger than the GB fee.

From the table: about USD 220 per month during the move (dump raw/ plus dump's backup plus v3). One-off about USD 2,320 (USD 40 COPY plus two scans). Skipping dest prefixes would cut GuardDuty in half; that is a saving, not the baseline.

During the move the data is in v2, dump raw/, the dump's backup, and v3. JSONL under target/…/data/ is small. There is no extra blob copy under target/.../files/ for v2.

Limits:

• A single CopyObject can copy 5 GB.
• GuardDuty does not scan an object that is larger than 100 GB.
• The one-time v2 → raw/ copy can write more than 3,500 objects per second on one prefix. Split the raw/ keys if the job hits that limit.
• Do not copy from v2 to raw/ again. Do not use S3 Batch to copy into the matter bucket.
• After the one-time v2 → dump raw/ copy, the inventory job writes SHA-256 of the object bytes into BigQuery blob_inventory (source key, dump URI, sha256, size). Hash from the v2 source (or copy-job sidecar), not from unread dump raw/. dbt copies that onto canonical matter_file.payload_sha256. Do not put the inventory under raw/…/files/. Do not hash the key string. Do not GET 3 TB through the import worker.
• Do not copy or scan again when you heal an object that is already in place.

How to decrease cost:

• The object count sets the GuardDuty cost, per hop. Cost the page at two hops.
• Do not GET 3 TB through the import worker.
• If you keep the dump, the dump and the dump's backup cost about USD 150 per month.
• Do not put the v3 serving copy in Glacier. Lawyers download and preview from it.
• After the one-time copy, and after v2 is no longer the live documents store for that firm, you can lifecycle the v2 source bucket to Glacier (or Deep Archive) to drop that ~USD 74 per month. Object lock still allows a storage-class change; it blocks delete. Restore first if you ever need to CopyObject from v2 again. Do not Glacier v2 while the copy job still needs Standard reads.

Risks and blockers

Blockers for a real files load:

1. Production BlobStore (CopyObject plus SHA verify plus scan tag). Without it createRun refuses non-dry prod runs.
2. Blob staging: copy v2 objects into dump raw/{snapshotId}/files/ one time, by hand, scan there, set JSONL payload_uri at raw, replace placeholder hashes.
3. Inventory join (missing / hash mismatch / orphans fail verify).
4. GuardDuty protection plan plus tagging on module.dumps. Without tags the prod gate can never see clean.
5. Import worker IAM: dump read plus GetObjectTagging plus matter-bucket write. Terraform today provisions dump bucket plus database, not a full copy role.
6. Invert files.ts: commit the files unit (including FilePreview) before place.

Risks (not necessarily blockers):

• Production v2 object lock / versioning: copy still works; deletes in dump are denied on the regulated bucket.
• Orphan S3 objects vs missing objects: fail the file, record uri; do not invent a dangling File row.
• Key-layout drift if we keep firms/.../{fileId}.
• Stampede if we emit one organic file.uploaded per file (notifications plus live CCO plus Inngest flood).
• Hydrate: do not dump a full imported matter onto caseContextUpdateRequired. Use the existing backfill coordinator. Per-file ingest should still run first so the backfill can retrieve assets.
• Content-service confusion: writing imported matter documents there would hit the wrong lifecycle and ACL.
• Recordings and transcripts: same v2 bucket, Call rows not CaseFile. Do not silently drop them from the 2.5 TB story.
• Cost: regulated dump that cannot expire; GuardDuty twice (dump raw plus v3 matter bucket). Production dump AWS region is eu-west-2 (env name prd-us).

Follow-up work to name

• Invert files.ts: commit files unit including FilePreview, then place
• dbt matter_file after inventory and scan (URI and real hash, not v2 placeholders)
• S3 BlobStore plus IAM
• One manual copy v2 → dump raw/ plus hash rewrite
• Inventory join
• Dump GuardDuty plus GetObject deny
• Dump retention / lifecycle exception
• file.imported for per-file ingest (not live CCO)
• CCO backfill coordinator as an explicit later step
• Confirm destination bucket (v3-matters-* vs new matter module)
• CaseFile-only vs bucket total

Open questions

These are still forks. Locked path is not repeated here.

Collisions when firm B imports after firm A. Shared prod bucket. CopyObject overwrites silently. Locked: fail the files cell if the dest key already exists (If-None-Match / HeadObject). Still open: dest key shape. Lean: mint from ids we own (matterId plus fileId). Other forks: put firm or run id in the key, or rely on nanoid matterId uniqueness with organic {timestamp}-{filename}.

Infected files, and wait timeout. Locked: Dagster waits for a tag on every inventoried raw/ key before target/ JSONL. Loader gate is a backstop. Still open: how long to wait, and what to do with THREATS_FOUND / UNSUPPORTED / FAILED (drop the row, fail the run, keep unread JSONL, quarantine). Failed tags are not clean. Do not ship unscanned rows.

Is there a GuardDuty cap that stops a 2.5 TB / 4 million dump? Locked: land everything, wait for the AWS queue, then export. Per-object skip is 100 GB. Still open: account quotas, backlog, protected-bucket count. Confirm max(fileSize) on the whole bucket, not only CaseFiles.

Do we run GuardDuty on the v3 matter object if dump raw/ is already clean? Locked: dump scan is the malware gate. Files cell does not wait on dest tags. Costs assume dest scan stays on (second ~USD 1,140). Still open: skip imported prefixes. If dest scan is off, preview cannot ride malwareScan.completed. Then: wire preview from file.imported, or generate previews later. Organic uploads on the same bucket must keep a scan.

Source files bigger than 200 MB. Locked: organic cap is 200 MB. Still open: max(fileSize) and counts over 200 MB / 5 GB / 100 GB. Lean: dbt fails the join if inventory file_size is over 200 MB. Other forks: drop the row, fail at load, copy anyway, raise the v3 serving cap.

Call data. Recordings and transcripts are in the same v2 bucket on Call, not CaseFile. They are in the 2.5 TB count. This hop does not copy them. Still open: later comms pass, or reuse this dump hop onto Call rows. Do not surface transcripts as matter files.

Soft-deleted documents. Locked for this hop: live files only. Still open: copy hidden objects for retention (off the documents list). That needs a deleted flag on the boundary.

Who fires CCO backfill? Locked: never live incremental CCO for the initial dump. Never one fire per file. Lean: explicit / manual after ingest. Other fork: import-service fires coordinator.requested automatically.

One dump bucket, or split raw from target? Locked for now: one module.dumps bucket, prefix deny, JSONL readable. Still open: a later split (raw malware bucket vs JSONL-only target).

Dump retention, and CaseFile-only vs whole-bucket bytes. Still open. Dest serving bucket for this hop is v3-matters-* until matter-service stops using that env var.