Instrumenting the import: plan

Follows from the dev dry-run report: a 19-minute run held 38 seconds of work, and the rest was Inngest dispatch gaps between batches. This plan adds the instrumentation to measure that split on every run, and defines the runs that turn the measurements into a per-firm recommendation.

What the scout found

Two facts change how much we have to build.

A batch ("hop") is three Inngest executions, not one: the step.run that does the work, the step.sendEvent that triggers the next batch, and the final return. Each execution re-enters the queue and contends for the same concurrency: 1 slot, so one batch pays up to four queue waits. The existing OTel middleware already emits a span per execution with inngest.run.id attached, so yesterday's gaps are queryable in Axiom today, with no code change. That query is the first task on this plan and needs nothing deployed.

The heartbeat column already exists. ImportRun.lastProgressAt is in the schema with a comment saying the run module is expected to stamp it per batch. Nothing writes it.

Constraints the scout pinned down: OTel metrics are unusable in this stack (the exporter sends an encoding Axiom's metrics endpoint rejects), so tuning data lives in span attributes and the report, not a metrics pipeline. Span attributes are not PII-scrubbed, so spans carry ids and refs only. Prisma auto-instrumentation already emits thousands of spans per hop, so our own spans stop at unit granularity, with stage timings as attributes rather than child spans.

The model

wall(B) ≈ (units ÷ B) × gap  +  Σ_entity rows_e × rate_e × k_e

• B — batch size in units. A matter is up to 6 units (core + 5 pillars) and a matter bucket is atomic, so B cannot split a matter. Today's default B=100 processes ~17 matters per hop.
• gap — idle time per hop. A property of the Inngest configuration, not of the firm's data. Measured 37s average in dev at concurrency: 1.
• rows_e, rate_e — the firm's row count and the measured dry-run work rate for entity e. Both come out of one dry run of the firm's dump.
• k_e — dry-to-real multiplier for entity e. A dry run skips the receiving-database transactions and the note-content conversion, so its work rate understates a real run's. k is a property of the pipeline, calibrated once and reused across firms until the write path changes. Start with one global k; split out k_notes if a note-heavy firm diverges.

Batch size enters only through the hop count, so once the constants are measured the curve over B is arithmetic. The final report can print it: "this firm at batch 100 ≈ 19 min, at 500 ≈ 5 min, at 1000 ≈ 3 min".

The same shape says where each lever stops paying: raising B divides the gap term but never touches the floor, and past the point where a hop holds a whole matter bucket the curve flattens early. Shrinking the gap itself (higher concurrency, or one execution per hop) lowers the whole curve. Density decides where a given firm sits on it.

The instrumentation

PR 1 — durable timings on the run row

• Stamp lastProgressAt in the per-hop run-row update (processNextBatch.ts:387). The admin screen gets a "last progress" row and a stalled indicator; the next hop gets a measured dispatch gap for free (now − lastProgressAt at entry).
• Record per-hop timings in the in-progress report JSON, which accepts new fields without a migration: {hop, queueLagMs, coldRebuild, rebuildMs, workMs, unitsProcessed, unitsDiscarded, cursorPass, rowsByEntity delta}. queueLagMs is Date.now() − event.ts at handler entry; Inngest stamps ts on every event and the handler currently ignores it.
• Summarise in the final report: work vs idle, gap distribution, rows/s by entity, matters per hop. batchSize is already stored per run, so every run is a self-describing experiment.
• Two log lines: one per hop with {runId, hop, workMs, queueLagMs, unitsProcessed}, and one on the currently-silent S3 body restart in streamLines (a mid-run ranged re-GET is a latency spike with no fingerprint).

PR 2 — spans

Via the tracer the Inngest middleware already injects into the function context (typed, no new dependency):

• import.batch around processNextBatch — attrs: runId, firmId, hop, cursorPass, batchSize, unitsProcessed, queueLagMs, coldRebuild, dryRun.
• import.rebuild inside buildRunState — the cold-rebuild cost, dominated by the walker's fast-forward, which re-reads the active pass up to the cursor and is invisible today.
• import.unit per unit — attrs: bucketRef, pillar, importer, outcome, rows, and assemble/validate/write durations as attributes.
• Per-service write duration on the unit record, so the receiving-database pressure side of the trade-off gets numbers (needed when the DBs are resized before the full v2/LEAP load — one re-run re-prices the write rates).
• appVersion on the Inngest client, so gaps can be attributed to deploys.

Every span carries runId: hops are separate traces and nothing links them, so the run id attribute is the only way to assemble a run-level view in Axiom.

PR 3 — surfaces

• Admin UI: "last progress" and rows/s tiles on the run detail, a stalled badge on the list.
• A saved Axiom query for the per-run gap breakdown.

The measurement runs

1. Gap query, now. APL over the existing execution spans from the 714-matter run of 2026-08-20. Tells us which of the four queue waits dominates, which decides whether the first experiment is raising concurrency or collapsing the hop to one execution.
2. Gap table. Fixture-dump runs, one per configuration under consideration (current; raised IMPORT_GLOBAL_CONCURRENCY; collapsed hop if we build it). The gap is size-independent, so the fixture's 107 rows suffice.
3. Density profile. One dry run per firm at default settings. Produces units, rows_e, rate_e, matters per hop.
4. Calibration. Dry run, real run, teardown of the same 714-matter dump against the dev test firm. Same dump and same instrumentation on both sides; dividing the per-entity work rates gives the k table. Teardown exists so the real run leaves nothing behind.
5. Confirmation. One run at the recommended B per firm before its real import. Prediction printed next to actual.

Order of work

1. The gap query (nothing to build).
2. PR 1. From here every run collects tuning data.
3. Calibration runs in dev (needs PR 1 only).
4. PR 2, PR 3.
5. Gap-table runs for any configuration change we decide to test.

Not in scope: fixing the OTel metrics exporter (spans and the report cover this need), per-query span sampling policy (a cross-service decision), and the run-creation checksum stream (tracked as its own item on the dry-run report's next-work list; the scout added that it downloads the entire dump inside the operator's tRPC call).