Companion docs: data provenance (the per-service ImportedRecord registry that references runs) and human-readable ids.
Ask: approve or veto one decision
The ImportRun record lives in the migration engine's own database, not in any platform service. A run always targets exactly one firm that already exists, and the importRunId every ImportedRecord row carries is a self-describing string, so the basics (source system, firm, run number, start time) survive in the service databases even without the engine. Fast path: read the TL;DR and the options table.
TL;DR. The registry rows in every service database say which run wrote a row, but nothing yet says what a run is: when it happened, from which source snapshot, dry or real, finished or torn down, what its report said. That record belongs to the engine, in one place, because a run writes into many service databases and none of them owns it. The run id embedded everywhere is self-describing (
imprun_<sourceSystem>_<firmId>_<seq>_<startedAt>), so even if the engine's database is gone, every imported row still names its source system, firm, run and start time on its own. <startedAt> is the run's start instant in compact UTC ISO 8601 (20260806T141230Z): sortable, unambiguous, and free of characters that ids cannot carry.A run targets one firm, and the firm exists first
A run is bounded to exactly one firm. Its data cannot span firms, and one migration cannot import several firms; a firm is the unit of migration. That is already implicit in the run id format, which has a firmId in it. This section makes it explicit: creating the empty firm is a prerequisite step, done before the first run, and the engine never creates a firm as part of a run.
Why the firm is not auto-created by the run:
- The run id needs the firm id before the run starts.
imprun_<sourceSystem>_<firmId>_<seq>_<startedAt>cannot be minted for a firm that does not exist yet. - Re-runs need a stable anchor. The whole point of teardown-and-rebuild is running the import against the same firm more than once: tear down what previous runs wrote, rebuild, or later apply an incremental update. If run 1 had created the firm, tearing run 1 down would delete the anchor every later run hangs off, along with anything else attached to it in the meantime (users who logged in, configuration someone set).
- Teardown stays well-defined. "Delete what runs wrote inside this firm" is a clean boundary only if the firm container itself sits outside the run's write set.
- Firm creation is provisioning, not import. Setting up a firm touches things a data import should not: the auth organisation, memberships, third-party systems (Clerk, Knock), and a commercial agreement about who this firm is. That happens once, through the normal provisioning path. Imports happen many times against the result.
What a run record has to hold
Enough to answer the questions operations and support actually ask:
- Which runs have touched firm X, and what state is each in (running, completed, failed, torn down)?
- Was run Y a dry run or a real one?
- What did the run read (which staging snapshot) and what did it write (per-service counts, errors)?
- Who started it, when did it start, when did it finish or get torn down?
A sketch, in whatever store the engine ends up with:
model ImportRun {
id String @id // "imprun_<sourceSystem>_<firmId>_<seq>_<startedAt>", the value in every ImportedRecord
sourceSystem String // "platform-v2", "leap", ...
firmId String // the target v3 firm; must exist before the run starts
label String? // human-friendly name for the run
snapshotRef String? // the staging snapshot the run read from
dryRun Boolean @default(false)
status ImportRunStatus // RUNNING | COMPLETED | FAILED | TORN_DOWN
startedBy String? // operator
report Json? // per-service counts and errors
startedAt DateTime @default(now())
completedAt DateTime?
tornDownAt DateTime?
@@index([firmId])
}ImportedRecord.importRunId stays a plain string with no foreign key: the run table is in a different database, and the registry already treats the id as opaque. What is new here is pinning down the id's format so it carries the basics by itself.
Where the table lives
The engine writes into many service databases in one run, and there is no shared platform database to fall back on (platform-api and webhook-api have no database at all). The two closest precedents in the codebase split exactly along this line:
MessageBackfillin email-service is a real job-state table (status, progress cursor, timestamps). It works because that job writes to exactly one service, so its home is obvious.backfillMatterTeamAccesshas arunLabel, a dry-run flag and a per-firm report, and keeps all of it in memory and logs. Once the Inngest run scrolls out of retention, nothing durable remains. That gap is what this table closes.
| Option | Why it was attractive | Why it lost |
|---|---|---|
| The engine's own database (chosen) | The run is engine lifecycle state: it starts, progresses, fails, gets torn down under the engine's control. The engine already needs its own storage for staging snapshots and transform state, so the table has a natural home with no new infrastructure | Platform services cannot join to it. Mitigated by the self-describing id: the question v3 can answer alone ("which system, firm and run did this row come from") never needs the run table |
| A designated platform service (identity-service was the candidate, since firms live there) | Queryable from inside the platform; survives the engine | Puts engine lifecycle state in a domain database that does not own it, and couples engine operation to that service's deploys and migrations. No platform feature needs to read run records; support tooling can ask the engine |
A copy of the run row in every service database, next to ImportedRecord | Every database fully self-contained | N copies of mutable state (status, report) that must be kept in sync across databases, for a record that is read rarely. Registry rows are append-once; the run row is not |
| No table: self-describing id plus logs | Zero schema anywhere | Loses exactly what operations needs: durable status, the dry-run flag, the teardown bookkeeping, the report. Logs expire; "was this firm's run torn down?" must not |
Where the engine itself runs, and what its database technology is, are engine-design questions this doc does not decide. The decision here is only that the run record is engine-owned, that a run is bound to one pre-existing firm, and that the id format embeds source system, firm, sequence and start time so the service databases stay self-describing without the engine.