Companion docs: the unit model (the grid's cells, and dangling-row reporting), data provenance (the registry, and teardown), the v2 boundary mapping (the fixture round zero runs on, and the upstream coverage numbers), and migration readiness (the measured volumes).
Ask: approve one decision, and answer the two questions it forces
The decision: import progress is measured as source-row coverage, developed inside the loop below. The forced questions: cross-service registry resolution, and the five fields the run report must carry. Fast path: the TL;DR, the metric section, and the two warning boxes.
TL;DR. The contracts define what a correct import writes, and the loader walk defines how rows stream in. What nothing defines is how we develop the importer: how we know each week that we are closer to a full migration, and by how much. This doc claims that slot. Every round imports the firm's whole book, reads one coverage number anchored in source rows, tears the run down, fixes the biggest gap, and runs again. Teardown, the run report grid, and dangling-row reconciliation are already specified in the other docs; the coverage loop is those three composed, with a number on it. Round zero runs on the fixture the v2 mapping already ships, so the loop is alive weeks before the S3-scale plumbing exists.
The slot this claims
Nine docs now cover the migration engine's contracts and mechanics. Between them, "did every staged record land" is designed: the registry makes writes idempotent, the unit model contains failure to one matter-and-pillar cell, and the run finishes regardless and emits a grid.
None of them says how the importer gets built. Build everything and run it once, and every mapping gap, enum mismatch and unresolvable reference in a 48,000-row book surfaces together, at cutover, under a deadline. The realistic path is dozens of runs against real data, meeting those problems one class at a time, months earlier and under no pressure. That path needs a loop that is cheap to go around, and a measure that makes each round comparable to the last.
The loop
flowchart LR
S["Stage the book<br/>dbt → sorted JSONL on S3"] --> I["Import<br/>the whole book"]
I --> R["Coverage report<br/>N% of source rows landed<br/>matters × pillars grid<br/>dangling · unmapped · failed"]
R --> P["Pick the<br/>biggest gap"]
P --> F["Fix one thing<br/>a schema, a mapping,<br/>an insert routine"]
F --> T["Tear down the run<br/>registry entries,<br/>reverse seq"]
T --> I
classDef hot fill:#fbdcb2,stroke:#c08552,stroke-width:2px,color:#5c3a1f
class R hot
Every round imports the whole book, not a sample. The point is that the report is always about the real firm, so "45% coverage" is a statement about Lawhive Legal's actual data rather than a fixture. Rounds are cheap because teardown is cheap: delete the run's registry entries in reverse sequence, exactly as data provenance specifies.
This is what one round of the loop does to the report (six matters shown; the real grid has thousands of rows):
| core | files | notes | tasks | comms | |
|---|---|---|---|---|---|
| #8241 | ✓ | ✗ | ✓ | – | ✓ |
| #8244 | ✓ | ✗ | ✓ | ✓ | ✓ |
| #8250 | ✗ | · | · | · | · |
| #8263 | ✓ | ✓ | ✓ | ✓ | ✗ |
| #8271 | ✓ | ✗ | ✓ | ✓ | ✓ |
| #8302 | ✓ | ✓ | – | ✓ | ✓ |
fix file checksums, resolve one person reference, tear down, re-run→
| core | files | notes | tasks | comms | |
|---|---|---|---|---|---|
| #8241 | ✓ | ✓ | ✓ | – | ✓ |
| #8244 | ✓ | ✓ | ✓ | ✓ | ✓ |
| #8250 | ✓ | ✓ | ✓ | ✓ | ✓ |
| #8263 | ✓ | ✓ | ✓ | ✓ | ✗ |
| #8271 | ✓ | ✓ | ✓ | ✗ | ✓ |
| #8302 | ✓ | ✓ | – | ✓ | ✓ |
✓ landed ✗ failed, visible in the report · not run, its core failed – nothing to import
The checksum fix lands three files cells, and resolving one person reference unblocks the whole of #8250's row, core and pillars together. One comms failure persists into run 7, and a new tasks failure appears on #8271. New failures appearing each round is expected: fixing one class of problem uncovers the next. The coverage number is this grid summarised.
Each catalogue gap the v2 mapping names (signed agreements, case addresses, tags) becomes one such round.
We have run this loop before
This is the working pattern from the ingestion DLQ work, applied to the import. There, a full dead-letter queue was the starting point, not a crisis: classify the messages by failure class, fix the biggest class, deploy, redrive, watch the queue shrink, repeat as new document shapes arrive. The loop worked, and the team knows how it feels to operate.
| Ingestion DLQ loop | Import coverage loop |
|---|---|
| A message in the DLQ | A failed cell in the matters × pillars grid |
| Message payload plus its error | Error samples with example ids in the run report |
| A failure class (oversized request, 400-class LLM error) | A failure class (enum mismatch, unresolvable person reference) |
| Fix and deploy | Fix a schema, a mapping, or an insert routine |
| Redrive the queue | Re-run the book: landed units no-op on the registry key, failed cells run for real |
The reingest flag, for "processed but wrong" | Teardown and rebuild, for when a fix changes rows that already landed |
| New traffic brings new failure shapes | Each fresh snapshot brings the rows the firm created since the last one |
Two rules fall out of the mapping:
Redrive when the fix fills gaps; tear down when the fix changes landed data. A failed cell wrote nothing, so re-running it after a fix is cheap and safe, and the registry gives it for free. But a wrong row that landed cannot be healed by a re-run that no-ops on it. That is ingestion's "processed but wrong" case, and its answer here is teardown. Most rounds only fill gaps, so most rounds skip the teardown. Failure classes are also independent of each other, so they parallelise between rounds: one person takes checksums, another takes enum mismatches, and both fixes land in the same next run.
flowchart LR
Q{"Does the fix change rows<br/>that already landed?"}
Q -- "no, it fills gaps" --> RD["Redrive<br/>re-run the book: landed units no-op,<br/>failed cells run for real"]
Q -- "yes, landed rows are wrong" --> TD["Tear down the run,<br/>then re-run"]
classDef go fill:#d8ead2,stroke:#5a8a64,stroke-width:2px,color:#24401f
classDef stop fill:#fbdcb2,stroke:#c08552,stroke-width:2px,color:#5c3a1f
class RD go
class TD stop
The loop is only as good as its refusal to fail silently. Ingestion's chunking-timeout path deleted messages without dead-lettering them, and those documents were simply gone: nothing in the DLQ, nothing to redrive. The loader's "no implicit skip" rule and the dangling-row report are that lesson applied: every staged row is either landed, failed visibly, or reported as unconsumed.
The metric: two numbers, one owner each
The trap in "coverage" is a moving denominator. If coverage is canonical rows imported over canonical rows staged, then improving the dbt mapping lowers the number: staging more rows makes the importer look worse. The fix is to anchor the denominator upstream, in the source system, and split the measure into the two halves the pipeline already has.
Upstream: source → staged (the mapping owns it)
Source rows accounted for, over source rows in the firm's book. The v2 mapping already computed exactly this shape: 28% of the ~48,000-row book maps today, 17% could map with catalogue additions, 52% is money.
"Accounted for" includes deliberate exclusions. A row excluded with a reason (soft-deleted, credentials) counts, because the measure counts decisions made, not just rows moved.
Downstream: staged → landed (the importer owns it)
Staged rows consumed by a unit that committed, over staged rows shipped. The design already contains the complement: dangling-row reconciliation reports staged rows consumed by no unit, and the grid marks failed cells.
So downstream coverage is 1 − dangling − failed, arithmetic over a report the engine already has to produce. No new bookkeeping.
The headline number is source rows resolved, over the book. A row is resolved when it landed in v3, or was excluded with a written reason. So 100% means every row in the firm's book is accounted for, not merely that most rows moved. Exclusions count once, upstream, and never enter the downstream denominator; when nothing is excluded, the headline is exactly the two numbers multiplied. One honest number per run: "run 7: 41% of the firm's book is resolved; here is the grid; round 8 adds custom fields."
The split is what makes the number arguable and the work parallel. Each number has one owner, the Data-team mapping upstream and the import service downstream, so a drop always names whose regression it is. And because neither owner waits on the other, the two move at the same time: the mapping raises the ceiling while the importer closes the gap to it.
Drawn over a series of rounds, the two numbers are bars chasing a ceiling:
What the number deliberately excludes: blob bytes. File rows count; whether 1.8 TB of content moved is a separate pipeline with a separate measure (checksums verified over files referenced). Mixing them would let a metadata regression hide behind a byte-copy success.
Rounds start before the plumbing exists
The v2 boundary mapping ships a fixture: one referentially closed matter across all 17 canonical files, described as exactly what the import service would receive for a one-matter run. The harness starts there, and the triple below runs in CI forever as the harness's own correctness test.
1 · Import the fixture matter
Assert the expected v3 rows and registry entries exist, service by service.
→
2 · Tear it down
Assert zero residue in every touched database. This is the assertion that flushes out the two-database teardown gap below.
→
3 · Import it again
Assert the second run's outcome is identical to the first. Idempotency proven by test, not by argument.
Round zero needs no S3 streaming, no cursor resume, no run cache: just the insert routines and teardown. The 714-case export the prototype already produced is round one. The full book is round two. Every loader feature then lands inside an already-working loop instead of before it.
What this resolves
The teardown-versus-healing tension between two sibling docs. Data provenance says re-runs are teardown-and-rebuild; the unit model describes additive healing of failed cells. The redrive-versus-teardown rule above is the resolution: both are right, for different fixes. Both stay simple before cutover, when no organic writes exist inside imported matters. True additive healing on live data earns its complexity only at production cutover, when torn-down matters would take organic data with them; that is cutover machinery, designed later.
The first half of the run report's design pass. The matter boundary schema parked the report format for its own pass. The coverage metric is a report format decision: per run, the report must carry the two coverage numbers, the matters-by-pillars grid, dangling rows by model, unmapped columns by count with example ids, and failures by cell. Whatever else the report becomes, the loop needs those five fields machine-readable and diffable between runs.
What this surfaces
Cross-service registry resolution blocks round zero, not eventually
The unit model already says owning-service lookups versus a central read copy "must be settled before the run loop is built". The loop makes that concrete and urgent: even the first increment, firm plus matter cores, spans identity-service and matter-service, so the question bites in round zero. This is the decision to force while the infra and scaffolding work lands.
Teardown across a two-database unit is undesigned
The notes pillar commits Content to content-db and MatterNote to matter-db as two transactions. A crash between them leaves a half-pair; teardown must delete both halves and detect the orphan when the registry entry for one side never got written. Round zero's zero-residue assertion is exactly the test that finds this. Simplest candidate rule: the registry entry for the second write is the commit marker for the pair, and an orphaned first write with no marker is torn down as residue. Unlike the registry question, this needs no meeting: the round-zero change can propose the rule, and its zero-residue test proves it.
What this is not
- Not a change to the loader, the contracts, or the unit model. The five runtime amendments stand as-is.
- Not the production cutover process. This is the development loop; cutover adds freeze, reconciliation sign-off, and, later, additive healing.
- Not a blob-migration measure. Bytes get their own measure when the staging step exists.
Asks
- Approve the coverage definition: resolved rows over the book, split into the two owned numbers, exclusions counting once.
- Settle cross-service registry resolution now; it blocks round zero.
- Agree the report carries the five coverage fields, machine-readable, as the first slice of its design pass.