Companions: hierarchical unit load from sorted S3 dumps (Daniele's cursor mechanics, DM-14), units, buckets, and pillars (what a unit is and why), import boundary schemas (the canonical tables this run reads).
The setup
One firm, firm_9y2, created by hand before the run. The dump on S3 holds one sorted JSONL file per canonical schema from the boundary catalogue, singular names and all: matter.jsonl, matter_participant.jsonl, identity.jsonl, and so on. Roots sort by pk, children by parentFk, pk.
- Three matters are staged; the run is scoped to
case_71 and case_72, because "migrate these N matters" is the real operational shape. case_73 sits in the dump but not in this run.
- One failure is planted:
doc_991 on case_72 fails its blob checksum mid-copy. Watch what it costs.
- The middle column is the part to watch: what the import service holds at each step, which registry entries each transaction reads, and what the run cache keeps versus evicts.
What the loader holds, and why it stays flat
At any instant the import service holds exactly three things:
- One buffered row per open cursor (13 files in this demo, roughly 30 in a real forest). A cursor sits on its current row until the row is consumed; a non-matching row waits in place for the next parent.
- The one unit being assembled. The peak in this run is three rows: the matter core with its two participants. The moment the transaction commits, the unit is forgotten.
- The run cache, covered below: the externalRef → v3 id pairs the run itself has written, kept only for the record types that later rows reference.
Nothing else. The dump can hold twenty thousand matters and the footprint does not move, because everything already inserted lives in the ImportedRecord registry inside each receiving service, written in the same transaction as the rows it describes.
The loader reads too: registry lookups and the run cache
Writes are only half the traffic. Almost every transaction also reads the registry, because every cross-unit reference resolves through it, and the simulation shows each lookup as it happens. The pattern:
- A person unit reads its team refs (
dept_prop) to fill TeamMember.teamId.
- A matter core reads every participant's person ref against identity-db's registry.
- Every pillar of that matter then re-reads the same hot refs: the matter's own id, plus the uploader, author, assignee, and message authors, which are usually the same two or three people.
Left naive, that is real load on the service databases: a 1,000-matter firm with five pillars each is on the order of ten thousand point reads, most of them fetching the same few hundred people over and over. The reads are indexed single-row lookups, so even the naive version survives, but there is no reason to pay for it. The run wrote every one of those registry entries itself, so it can simply remember them:
- Write-through: when a unit commits, keep the externalRef → v3 id pairs it just wrote, but only for types that downstream schemas reference: identities, teams, contacts, and the current matter. Pillar rows (files, notes, tasks, messages) are never referenced by anything else, so they are never cached.
- Bucket eviction: a matter's id is only referenced by its own pillars, so it leaves the cache the moment the bucket closes. Firm-level entries stay for the whole run.
- Lazy fallback for incremental runs: a comms-only run months later starts cold. A cache miss falls back to one registry read, then caches the result, so the worst case is one db read per referenced record per run, not per cell.
The bound is easy to state: cache size is the firm's referenced people, teams, and contacts, plus one matter. Even a large firm's address book is tens of thousands of entries at a few dozen bytes each, single-digit megabytes. In this simulated run the counter ends at 17 lookups, all 17 served by the cache, zero db reads.
The five amendments, read against DM-14
Daniele's walk is the right mechanism: sorted dumps, one forward cursor per file, drain on equality, resume by byte offset. The amendments are the places where the write-up and the unit model disagree, and each one is visible in the simulation.
- 1 · Commit at the core and at each pillar, not at the root. DM-14's
scanAndBuildUnit appends the root and all of its descendants into one unit, then commits it whole. For a matter that is the mega-unit the unit model rejected as option C: one bad blob rolls back the entire case, and blob copies stretch the transaction to minutes. The drain machinery stays exactly as designed; only the commit points move. In the simulation, case_71 lands as five transactions, not one. This is the load-bearing amendment.
- 2 · A failure marks a cell and the run continues. DM-14 raises
MissingChildError and stops. The unit model's contract is that the run always finishes and always produces the matters × pillars grid. When doc_991 fails its checksum, the files cell goes red, the notes pillar of the same matter still lands, and the cursors advance past the failed unit's rows because those rows were consumed into the failed attempt. Healing is a later run; the registry makes it idempotent.
- 3 · "No forward-skip" becomes "no implicit skip". DM-14's invariant forbids the cursor from moving past unconsumed rows, which makes scoped runs ("migrate these 50 matters") impossible. The run must move cursors past out-of-scope data somehow. The simulation drains and discards
case_73 and its children deliberately: five rows parsed, counted, logged, none inserted. Planned batches can scope the dump itself instead; drain-and-discard is what healing and partial runs use.
- 4 · Assert EOF on every cursor at end of pass. The walk only ever tests equality, never order, so a mis-sorted dump strands rows on a cursor silently: parents insert childless and the run finishes clean. DM-14's dbt
not_null tests catch null keys but not ordering drift between files. The last step of the pass checks that every cursor is exhausted; leftover rows fail the pass loudly.
- 5 · Nullable parents split at dump time. The boundary catalogue makes
task.matter_external_ref nullable because firm-level work exists; DM-14 requires child files to carry no null parent keys. Both are right. Matter-scoped tasks dump as a child file sorted by matter_ref, pk; firm-level tasks dump as their own small root pass. The simulation carries only the matter-scoped file.
Verdict
The mechanism holds. Cursor mechanics, the Zod annotation codegen, and byte-offset resume from DM-14 are keepable as-is. The five amendments turn the walk into the unit model's runtime rather than a table streamer, and the run cache removes the one cost the walk never priced: reading the registry it writes. Discussion on this page.