Imported file activation: tech plan

The files tech plan gets a firm's matters and file bytes into v3, and the imported firm-member activation plan gives its people a login. This page covers the files after they land, because today they land inert: no preview, never ingested, no case context. It locks the interfaces that wake them up and the order the work ships in. No new preview or ingestion engines are built here: activation triggers the upload-path machinery, and the only changes to the upload path are two bug fixes it needs anyway.

Decisions

Two riders on those decisions. The extractor default changes everyday ingestion behaviour too, not just the migration, so it lands in the ingestion repo behind the dev experiment's quality gate and needs the ingestion and search owners' sign-off. And the cheapest mechanical home for the live-only scope is run scope: import and place the live book, and if the closed book is imported later it is its own run that an operator simply does not activate. If one run ends up carrying both, the activation chain needs a matter-status filter on its work list. That is the one open detail in the contract below.

Architecture and domain

No new tables. The receiving services already keep the per-file activation state: FilePreview.status (plus canPreview, now maintained) for previews, and File.aiIngestionV3Status for ingestion. FilePlacement rows at PLACED with verdict CLEAN are the work list, and the run report reads those receiving columns back instead of duplicating them.

A FileActivation ledger table was the obvious alternative and it is rejected: it duplicates state FilePreview.status and aiIngestionV3Status already maintain, a fourth ledger drifts, and the receiving columns are what the UI trusts anyway. Re-asks are idempotent end to end instead: an ingest 409 means “already there”, and the preview handler's final-status gate makes it one-shot.

Ordering: most-recent matters first. However long the drain takes, the matters a firm is actively working land first, so the newest cases are usable within the first minutes or hours while the tail fills in.

The mechanism: the run stamps a matterRecency onto FilePlacement rows at promise time, taken from the source system's last-activity or opened date, which is already in the staged matter payload the run walks (catalogue/matter.ts:20 carries lastActivityAt as a required field). The chain's cursor then walks that order instead of bare externalRef. Null recency sorts last, so already-promised dev runs degrade to ref order.

That ordering also sets up a later refinement. Since activation completes matter by matter, “Build case context” could eventually go per-matter as each one settles, rather than waiting for the whole run. Run-level for v1.

Contract

Events

file.activated is a past-tense fact: the import has activated this file, and what each consumer does about it is its own business. Only files at PLACED with scan verdict CLEAN are ever named. Batch, cursor and single-flight semantics copy the placement chain exactly: DONE only on an empty page, and a teardown guard refuses while a teardown is in progress.

Verbs

import-service, gateway passthroughs behind importOperatorProcedure as usual:

imports.admin.requestFileActivation({ importRunId })
  => { expected: number }            // files at PLACED+CLEAN not yet asked; 0 sends no chain

imports.admin.getFileActivationReport({ importRunId })
  => {
    importRunId,
    previews:  { ready, converting, failed, notSupported },              // FilePreview.status of the run's files
    ingestion: { notRequested, pending, success, failed, unsupported },  // aiIngestionV3Status
    settled: boolean,                // nothing converting and nothing pending
  }

imports.admin.requestCaseContextBuild({ importRunId })
  => { matters: number }             // fires the backfill coordinator once per run matter

matter-service, internal:

matterFile.internal.getImportedFileEnrichmentCounts({ fileIds: FileId[] /* chunked ≤1000 */ })
  => { previews: Record<FilePreviewStatus, number>, ingestion: Record<FileAIIngestionStatus, number> }

Consumers

matter-service gets a new Inngest function on import-service/file.activated that reuses the existing converter step, the Lambda call and the FilePreview write, with retries: 3 where the upload path has 0. The final-status gate is preserved, and it writes canPreview alongside every transition.

lawrence-api gets a new Inngest function on the same event, calling the same ingest-request service as the upload path's file.uploaded function (a shared function, not a copy), with asset_type: "matter_file", reingest: false, and the backfill lane. It writes PENDING only when the request succeeded, and logs plus reports to Sentry otherwise; a 409 leaves the status untouched, because the asset is already known. The upload path's function moves onto the same shared service, picking up the same fix. Give this consumer explicit Inngest concurrency, so it does not become the enqueue bottleneck.

The one data change

FilePlacement.matterRecency, one nullable column, plus an index on (importRunId, matterRecency, externalRef) for newest-first ordering, stamped at promise time from the staged matter payload. Nothing else. No new tables, and NOT_REQUESTED already exists as the imported files' aiIngestionV3Status value.

Surface

Run page (admin-app). A files-activation panel next to the placement panel. An “Activate files” button, enabled when placement is byte-complete and scans have settled (the same hasLiveImportedRows gating), preview and ingestion tiles from the report, and polling while unsettled. Then “Build case context”, enabled when settled. Confirm dialogs follow the members-panel copy pattern.

legal-os. Files with aiIngestionV3Status: NOT_REQUESTED stop rendering as “Analysing” and get their own copy, “Not analysed”. IngestionStatusCell.tsx:23 currently renders NOT_REQUESTED, PENDING and null identically, which is why an imported file reads as permanently in progress. An independent small slice.

Teardown and failure semantics

Refusals use the same shape as the placement verbs. An unknown run, a rehearsal, or a torn-down run returns NOT_FOUND or PRECONDITION_FAILED. requestCaseContextBuild additionally refuses while getFileActivationReport().settled === false, because building context from half-ingested documents is the exact hole this exists to close. The activation chain's teardown guard refuses to advance while a teardown is in progress, and re-running activation after a partial run is safe by construction: the ingest 409 and the preview final-status gate make every re-ask a no-op.

What can still go wrong, and what we do about it:

• The cross-repo contract is untyped. The backfill coordinator event lives in ai-platform. One thin sender function, with the payload shape pinned by a test and a cross-reference comment on both sides. The same treatment the app/case/batch-* hop already gets.
• Ingestion load. A Woodstock-size run is tens of thousands of /ingest calls. The chain paces exactly like placement, one page per hop, and ingestion's own SQS pipeline absorbs the rest. Timing instrumentation on the chain feeds the import-run scalability workstream.
• Interim context pollution if anything else fires the incremental pipeline mid-migration, a user touching the matter for instance. Accepted: the throttle and singleton make it cheap, and the backfill replay reconciles. Not worth a global mute switch.
• Retries can burn themselves. Every prd queue allows three receives before dead-lettering, but a worker that retries a 429 immediately spends all three in seconds. Stage 0 adds worker backoff on 429 so each receive is a real attempt.

Rollout

Stage 0: platform readiness

Ingestion and case-context work we own as ai-platform. All of it is independent of the import stack, most of it is mergeable now, and each item executes one of the constraints the scale numbers impose.

1. lawrence-api truth fix. The shared ingest-request service; PENDING only on success; Sentry on failure. Closes one of the two upload-path bugs activation would inherit.

2. legal-os NOT_REQUESTED copy. Imported files stop reading “Analysing”.

3. Ingestion Langfuse tracing. The clean seam is the worker's context-injection block (worker.py:484-537), where every external call is bound as a partial, so wrapping there traces everything without touching task code; trace context already propagates across the three queue hops via traceparent, so per-document cost aggregates across stages. Generations carry model, token usage and cost, tagged with the lane and external_asset_id. Without this there is no per-document ingestion cost to read at all.

4. Case-context locale fix, incremental and backfill. Retire infer_jurisdiction_from_case_id. Jurisdiction becomes an explicit field on both the incremental batch event and the backfill coordinator event, threaded from the platform, since lawrence-api knows the firm's market. Prefix inference stays only as a fallback for legacy cas_ traffic. This corrects every v3 matter's prompts today, and it must land before any migrated matter generates context.

5. Retry resilience, both lanes. Add worker backoff on 429 so the three receives every prd queue already allows are three real attempts. Raising maxReceiveCount to ≥ 5 on top is a one-line change in the same terraform file, worth taking but not a prerequisite.

6. The backfill lane. Capacity isolation, not priority: a parallel set of workers on their own queues, scaled independently, so live-traffic latency never sees the migration. The six queues already exist in dev and prd, a default and a backfill per stage, each with its DLQ, at a 1000s visibility timeout suited to long batch tasks. So there is no queue to create and no IAM to widen: the sqs module grants across the whole queue list already. What is left:

   • Infra repo, one file (stacks/ingestion/main.tf): three new rows in input_output_queue_mapping pointing each backfill worker at its input and output queue, three new Doppler worker projects for the secret module that mapping drives, and three new entries in queue_autoscaling_config keyed to the backfill queues.
   • Ingestion repo, schemas: register the backfill queue names in QUEUE_SCHEMAS (they resolve to the same message schemas; today an unknown name fails worker startup), or set WORKER__*_QUEUE_SCHEMA_NAME per deployment. Registering is the less trappy option.
   • Management service: add lane, an enum of live (the default) and backfill, to IngestAssetRequest, plus a second cached queue dependency, and route by lane. lawrence-api's import consumer sends the backfill lane; today's upload-path caller does not change.
   • Flightcontrol: three new worker services (ingestion-{preparation,processing,indexing}-backfill-worker), same images, env pointing input and output at the backfill queues.
   • Scaling: the backfill autoscaling entries reuse the same queue-depth module the default-lane workers use, with min_capacity 0, so the lane sits at zero between migrations and bursts on queue depth during one. Organic stays untouched by construction.
   The indexing self-loop

   The indexing worker's output queue must be its own backfill input queue, because of the summary-generation self-loop. Pointing it at the default queue leaks backfill summaries into the live lane.

7. Embedding batching, both lanes. embed_chunks sends arrays of ~16 chunks per call. It is shared worker code, so live ingestion gets the same 16× headroom.

8. Infra resizes, before any burst. Sizing evidence for all four.

   The first two have a known home: stacks/ingestion/main.tf declares both, module "opensearch" at r7g.large.search with 500 GB and a public endpoint, and module "rds" at db.t4g.medium with 20 GB allocated and storage autoscaling to 80 GB. Each resize is a one-file terraform change.

   • The prd ingestion OpenSearch domain → 3 × r7g.xlarge.search data nodes (4 vCPU / 32 GB), 1 TB gp3 each, 1 replica, zone-aware, plus 3 dedicated masters. That is ~$1.7k/mo against today's ~$220 (eu-west-2 on-demand rates, masters included), and it also retires the zero-replica availability risk AI search runs today. A scale-in candidate once the migration settles, and the storage spike may shrink this target before we buy it. Bulk-load knobs for the burst: refresh_interval 30–60s, and optionally replicas at 0 during the initial load (safe, because the backfill is re-runnable) then enabled.
   • The prd ingestion database, db.t4g.medium → db.m7g.large + 100 GB gp3, for the migration window (~+$90/mo). Revert after.
   • The shared platform database, prd-db, → at least m-class with a ≥ 500 GB ceiling, or split lawrence_case_context onto its own instance. ~+$150–250/mo, largely permanent, because the rows stay. The migration consumes this ceiling twice, so this resize gates the import, not just activation. Also env-ify the backfill concurrency constants (15 cases / 5 per round) so the burst can be dialled without a deploy.
   • Management service, 3 → 6 instances during enqueue and burst, because it stream-hashes every object on /ingest and takes every stage's status callbacks. Back to 3 after.

9. Efficiency fixes E1 to E3. The summary-call merge, the context-echo cap, and the backfill round idempotency fix, all three described here, landed before the Stage 2 experiment measures the baseline.

Stage 1: the activation feature

This stacks on the import stack.

10. matter-service preview ask. The file.activated consumer, converter reuse, and canPreview as a fact, with insert semantics aligned and the handler writing it.
11. lawrence-api ingestion ask. The file.activated consumer on the shared service, sending the backfill lane.
12. import-service activation chain, report and verbs, including the matterRecency column and its index.
13. Admin panel: activate button, tiles, case-context button, plus platform-api passthroughs.
14. Case-context build verb. The backfill coordinator sender, behind the settled gate.

Stage 2: experiments and spikes

Before or alongside scale-up. Each one is scoped, with what it settles, in what is still unmeasured.

15. Dev cost, extractor and capacity experiment. One experiment, four answers: the cost model, the extractor verdict, the capacity number, and the lane sizing.
16. Extractor unification, after that gate, in the ingestion repo. All PDFs to Unstructured, Textract as the failure fallback, high-value classification as a later opt-in.
17. v2-reuse spike, covering ingestion and case context — conditional, not committed. It belongs to the stage that migrates the full v2 book, and that stage is itself an if: Woodstock is the committed migration, and v2 may run only as a proof-of-concept slice (a burst of ~1,000 files that proves cost, scale and time ahead of Woodstock). If the full v2 book is committed, the spike runs first and ends in a decision doc with measured cost and time against regenerate. Until then this plan prices regenerate, with ~$20–40k on the table if reuse wins.
18. OpenSearch storage spike. Quantization plus a mapping review; it feeds the resize target in Stage 0.
19. Woodstock document census. Required before quoting Woodstock cost or timeline.

Rollback and success

Every slice is additive, the chain is operator-triggered so “don't press the button” is the kill switch, and the only schema change is one nullable column with an index.

Done means all six of these hold:

1. An operator activates a completed run's files from the run page and watches progress there, using the same chain, report and panel shape as placement and member activation.
2. Every imported file an everyday upload would get a preview for, gets one.
3. Every imported file an everyday upload would get ingested, gets ingested, and is visible in AI search and to Lawrence.
4. Each migrated matter gets its case context built once its documents are readable: one controlled chronological pass, not an accidental trickle.
5. The UI stops lying. A file nobody has asked ingestion about reads as “not requested”, not “analysing”.
6. The whole thing is idempotent, teardown-aware and re-runnable, like every other import stage.

The gate on all six is a local end-to-end demo, import, place, activate, previews render, files searchable, case context present, and then the dev-firm migration doing the same on dev with cost numbers read from Langfuse.

The import-run scalability workstream is deliberately out of scope here. It follows, informed by the timing instrumentation this chain adds.

Follow-up tickets

Real work, tracked separately from this plan.

• Ingestion reassign gap, a live bug. A v3 file moved between matters never updates ingestion-management or the OpenSearch routing, because reassignDocument only calls the legacy service. Deletes were dual-written; reassign was not.
• Case-context backfill round retry re-extracts whole buckets and likely duplicates statement rows, because extraction mints fresh uuid4 ids. A correctness risk for ai-platform, and E3 above is the fix.
• Artefact metadata hardcodes "model": "gpt-5.4-mini" regardless of the configured model, so stored provenance lies if the SSM value differs.
• The case-context asset inventory caps messages at 50, leaving long migrated histories partially invisible to case context.
• .eml / .msg as communications or as files. A 6× swing on the preview-converter tier; the decision is owned by the files migration plan.
• The matter_document / matter_artifact_precedent ingestion 422, where finalised artifact documents are silently never indexed. Pre-existing, and it does not affect imported files, which resolve to matter_file.
• Reconcile the file-migration design with reuse. It was settled before reuse was on the table and says imported files get a deliberate full re-ingest via the backfill queues.

Sources

Every measurement, and the dates it was taken, is in Activation at migration scale. Contract details on this page were read directly from catalogue/matter.ts, IngestionStatusCell.tsx, packages/id-schemas/src/matter-service/matter-id.ts and ingestion worker.py; the queue, autoscaling and resize wiring from stacks/ingestion/main.tf in the infrastructure repo.

Companion pages: files migration tech plan · imported firm-member activation plan · activation at migration scale.