Forms as a content format

Meeting outcomes · 9 Jul

Discussed with Lukas, Adolfo and Petros. Decided now:

• FirmTemplate is the per-firm mapping (renamed from FirmForm so any template type can be firm-associated later). Slug-keyed until the migration, then templateId; the slug column is temporary and dropped after. Home: content-service, revisit if the firm domain's architecture settles elsewhere.
• Firm tags stay on the association table, joined onto the global definition. Petros flagged that categories and jurisdictions may really be global, Lawhive-finalised attributes, which contradicts the earlier PM steer; parked as a product question rather than a schema one.
• Provisional lean: one canonical typed Template rather than a separate form-template entity, "unless we figure out a way why it's really not good". Lukas's counterpoint stands: form-only columns sit oddly on a shared table, so this stays provisional.
• Deferred: the wider migration itself (more thought first), scoped templates and scope groups (one template shared with N scopes; a neat future shape, kept out of scope), identity-service ownership of firm associations, and custom firm-uploaded forms.

Questions 1 to 3 below therefore remain open in substance; question 3's surface shrank because firm tags stay on the association table, but the definition still needs a globally-readable scope when the migration runs.

Post-meeting refinement: scoped child Templates

Sketched by Lukas on the Lawyer Efficiency FigJam after the meeting. It replaces the FirmTemplate mapping table entirely: the firm's library entry is itself a Template row.

Template  + parentTemplateId String?   // self-reference
          contentId becomes optional
// Invariant (Lukas): a Template WITHOUT contentId must have parentTemplateId.
// With parentTemplateId set, the parent is the content source and this row is
// all about assignment to a scope, and tracking who added it.

// the example on the board
Template(scope: global)              // the global form definition
   ▲ parentTemplateId
Template(scope: firm/fir_1234)       // the form in one firm's library

// before the migration: Template carries a temporary slug column
// referring to the Lawrence Engine entities (dropped after)

What this buys: one uniform query. "Show my firm's library" becomes getTemplates(scope: firm/<id>) and returns documents and forms alike; no join against a separate assignment table, and the firm's categories reuse TemplateCategoryAssignment on the child row directly. Lukas's stickies add two properties: the child is always up to date with the global parent (a pointer, not a copy: library association only; matter instances still copy at fill time), and the shape generalises to any template marketplace.

What it costs: Template's invariants become conditional (contentId nullable, required-unless-parented), which is the same class of concern that argued against merging forms into Template in the first place. And the global-scope question (question 3 below) remains: the parent row still needs a scope every firm can read.

Relative to the meeting decision: this is a refinement of, not a contradiction to, the FirmTemplate table. FirmTemplate is the two-table spelling of the same idea; child Templates are the one-table spelling. The slug-first, re-point-later mechanics work identically in both.

That makes three spellings of the same association, differing in what the per-firm row carries:

• Child Templates (the board): the firm row is a full Template. Tags get a natural home and the library query is uniform, but name, code, issuer and status exist on every child unless children deliberately null them and defer to the parent, which is the duplication caveat above.
• TemplateScope(template_id, scope) (the comment): the leanest spelling, a pure grant, and the closest to SkillAccess. But it has no home for the firm-editable jurisdictions and categories; the moment those exist the join row grows columns and becomes…
• FirmTemplate (the meeting decision): exactly that, TemplateScope plus the firm's tags. Which suggests the decision already sits at the right point on this spectrum: leaner than child Templates, one column set richer than a pure grant.

The deciding question between them is whether firm tags stay firm-editable. If Petros's point lands and tags become global Lawhive-finalised attributes, TemplateScope is the honest shape and FirmTemplate shrinks into it. If Sedona's steer holds, FirmTemplate is already right.

The suggestion

From Lukas, on the migration thread: Content was always meant to back any content type, and Template was always meant to be typed. Both enums have one value today (PROSEMIRROR, DOCUMENT) because documents shipped first, not because the model is prose-only. So before building Form/FormVersion/MatterFilledForm beside his tables, check whether extending them does the same job. His open questions: partial updates and Yjs, live agent edits, versioning.

This is not the "Template row with an empty Content" idea the standalone plan already rejected. That draft faked the content. Here a form has real Content of a new format.

What the model already does

Verified against the schema, the storage layer is format-agnostic. Only the service layer (markdown, OOXML, the editor bindings) is ProseMirror-specific.

Content         scope · format (enum, today: PROSEMIRROR) · latestSnapshotId
ContentUpdate   append-only binary Yjs updates, batch-idempotent
ContentSnapshot full Y.Doc state, epoch-ed
ContentVersion  named pointer to a snapshot · sourceFileId → File
Template        type (enum, today: DOCUMENT) · status · currentVersionId ·
                publishedVersionId · editors · categories · scope
MatterDocument  matterId · contentId · finalisation // instantiation copies template content

Agent edits to documents already run through server-side applyContentYjsUpdate. Nothing in the update/snapshot path parses ProseMirror.

Yjs is not ProseMirror

The reuse claim rests on this distinction. A Yjs document is a container for CRDT shared types: prose lives in it as a Y.XmlFragment (the y-prosemirror binding), a form lives in it as a Y.Map. Updates and snapshots encode operations and state, not prose: the bytes in ContentUpdate and ContentSnapshot are the same encoding whatever shared types are inside. The format column tells the service layer which root type to read: getXmlFragment() for PROSEMIRROR, getMap("fields") for FORM_JSON.

A form edit therefore reuses the pipeline whole: load snapshot + newer updates → materialise the doc → validate against the map (the guardrail gate) → map.set("21", …) in a transaction → append the emitted update as a ContentUpdate row. Reads are getMap("fields").toJSON(). Compaction, epochs and version pinning operate on opaque bytes and never change. What is genuinely ProseMirror-specific, the editor binding and the markdown/OOXML converters, is replaced by a thin encode/decode layer per format.

The honest flip side: reusing these tables commits forms to Yjs as their storage encoding even if live collaboration never ships. That buys the update-log audit trail and per-field merge; it costs warehouse opacity and a yjs dependency in every reader. That trade IS this proposal.

What the standalone plan would rebuild

Where everything lives

Same convention as the other plans: amber chips are shared precedents, solid accent chips are new, dashed chips retire when this lands.

The schema change

The complete table set. Everything marked shipped exists today from LEX-677; the rest is this proposal:

// content-service · new enum values
enum ContentFormat  { PROSEMIRROR, FORM_JSON }
enum TemplateType   { DOCUMENT, FORM }

// content-service · Template hosts the global definition
// (new columns are FORM-only, null for DOCUMENT rows)
Template   + slug String?            // the stable public id FirmTemplate + filledforms:// use
           + code String?            // "N5"
           + issuer String?          // "HMCTS"
           + defaultJurisdictions String[]  // copied onto FirmTemplate at assign
           + guidanceFileId String?  // → File · the how-to-fill document
           scope gains the value "global" (open question 2)
           // name, status (PROCESSING is already the default!), publishedVersionId,
           // categories: already there

// content-service · shipped in LEX-677, survives unchanged
FirmTemplate   formSlug → templateId at Migration A (one-column re-point)
           scope "firm/<id>" · jurisdictions[] · @@unique([templateId, scope])
FormCategoryAssignment  firmTemplateId → FirmTemplate · categoryId → TemplateCategory

// matter-service · the instance row, mirroring MatterDocument's anatomy
MatterFilledForm  id · matterId · templateId · contentId → content-service
                  Content(FORM_JSON) · currentFinalisationId · metadata
// or: reuse MatterDocument itself with a discriminator; the thin row keeps
// matter lists and finalisation form-aware without a type sweep

// retired when the migrations complete
Lawrence Engine  form table + blank-PDF bucket (Migration A)
matter-service   MatterArtifact FILLED_FORM + version blobs (Migration B)

The Y.Doc inside a FORM_JSON content is one top-level Y.Map "fields", one entry per field id. The definition and the instance use the same keys:

// definition content · frozen structure, shared by every matter
"21" → { field_name: "Petitioner", field_annotation: "Full legal name of the
        person filing", field_type: "Case Information", page: 1,
        coordinates: { x1: 72.0, y1: 214.5, x2: 396.0, y2: 232.0 },
        max_chars: 60, pdf_widget_type: "text", pdf_on_state: null }

// instance content · the same field on one matter: the structure carried
// over by instantiation, plus the mutable answer state
"21" → { field_name: "Petitioner", page: 1, coordinates: { … },   // frozen copy
        max_chars: 60, pdf_widget_type: "text", pdf_on_state: null,
        value: "Jane A. Mitchell", status: "Complete",              // answer state
        changedBy: "Lawrence", changedAt: "2026-06-12T14:03:22Z",
        explanation: "Named as petitioner in the intake note",
        citations: [{ ref: 1, source: { type: "note", id: "matnt_4f" } }],
        confidence: { score: 0.98 }, previousValue: null, reviewedBy: null }

Each map entry is a plain JSON value replaced whole on edit. Two concurrent edits to different fields merge cleanly; two edits to the same field resolve last-writer-wins, which is today's behaviour too. The review metadata (previousValue, changedBy, citations) is the same shape the blob carries now.

"Copy" means instantiate, not byte-copy

The definition and instance value shapes differ on purpose, so starting a fill is a transform, not a Yjs clone: for each definition entry, carry over the structure the renderer and validator need (name, page, coordinates, widget type, on-state, max_chars) and add the empty answer slots (value: "", status: "Incomplete"). This is exactly the skeleton build the filledforms:// create runs today; only its output target changes.

The instance is deliberately self-contained (a "fat" instance). The FormEditor renders and validates from it alone, never re-reading the definition, which is what makes pinning by value real: a matter owns the structure it was filled against. The thin alternative (answers only, joined to a pinned definition version at render) was considered and loses on every axis: two reads to render, pinning degrades to a pointer, and Migration B stops being trivial. Fat is also today's exact blob shape, so the 476 existing filled forms seed straight across.

Groups and collections are kept

A definition is not just a flat field list. Lawrence Engine stores it twice today: the nested annotations (collections → groups → fields), which the agent's fields outline and the correction UI read, and flat_fields_annotations, which the skeleton build reads. FORM_JSON mirrors that split inside one Y.Doc:

Y.Map "fields"     flat, one entry per field id     // the mergeable half
"structure"        collections → groups → ordered field ids
                   // one JSON value; read by the outline and the correction UI,
                   // edited only by ops, so whole-value LWW is fine

Instances stay flat, exactly as today's filled_fields already are; grouping is a definition-side concern the instance never needed.

A microworld: the machine, running

The same machinery, two formats. Step through the lifecycle; watch what each action creates. DOCUMENT is how templates work today. FORM is this proposal: identical entities, different content inside them.

How each flow works

The agent's wire contract (value/checked/skip, citations, confidence) does not change. Only what lawrence-api writes underneath changes: Yjs updates instead of a JSON blob version.

The two questions that decide it

1 · Guardrails under CRDT

Today's edit resolver rejects invalid edits before persisting: max_chars overflow, checkbox/text widget mismatch, never overriding a reviewed field. A CRDT apply path cannot reject a merged update. Three candidate designs, none free:

• Gate at lawrence-api (validate, then generate the update). Covers the agent. A direct Yjs client bypasses it, so the FormEditor must stay on the server path or duplicate the rules.
• Validate-on-apply in applyContentYjsUpdate for FORM_JSON: decode, check the resulting doc, reject the batch. Works for tRPC-submitted updates; breaks if a live websocket provider is ever attached.
• Validate-on-read + repair: accept everything, flag violations in the review UI. Weakest; overflow would reach the PDF render.

Recommendation if this option proceeds: gate at lawrence-api AND keep the FormEditor on server mutations. Revisit only if live collaborative filling becomes a product goal.

2 · Scope for a global catalogue

Template already has a scope column, and it is a string, so writing global into it is trivial. The gap is that the access model has no global semantics: assertScopeAccess answers "does this identity belong to this firm or identity scope?" It has no concept of "readable by every firm, writable by Lawhive ops only". Adding that entry to the scope grammar is the real change, and it touches middleware every content surface shares.

How the two tables then divide the work. Template answers what the form is; FirmTemplate answers which firm has it and how they organise it:

Template (type: FORM, scope: global)        // ONE row for the whole platform
  name · code · issuer · content (annotations) · publishedVersionId
        ▲                          ▲
        │ referenced by (formSlug → templateId at Migration A)
        │
FirmTemplate (scope: firm/abc)    FirmTemplate (scope: firm/xyz)   // one row per firm
  jurisdictions: ["England & Wales", "Fulton County"] · categories: [L&T]

Why not use Template.scope for the assignment itself? Scope puts a template in one owner's library. For forms that would mean one FORM template per firm, duplicating the definition and its Content. Assignment is many firms to one definition, which needs a join row. This is exactly the Skill/SkillAccess shape in lawrence-api: global Skill rows, per-firm access grants. FirmTemplate is SkillAccess plus the firm's editable tags, and it is shipped and unaffected.

The hybrid option exists largely because of this question: a thin Form table owning the definition's identity means no global Template row, so the scope grammar never changes.

Smaller, still real

• Warehouse visibility. Filled fields are queryable JSON in the CDC mirror today. Yjs binary is opaque to BigQuery. Field-level analytics would need a materialised JSON projection, e.g. written on snapshot.
• Shared-machinery risk. Extending Content touches code every prose surface relies on (snapshot compaction, epochs, shells). Someone owns proving FORM_JSON does not destabilise it.
• Ordering and size. A definition has hundreds of fields; the Y.Doc for FL-100 is ~100KB of structure. Fine for Yjs, but snapshots store it whole per epoch.

Getting there: the same two migrations, different target

The shape of the standalone plan's rollout carries over whole: additive, born hidden, validated, reversible, using the proven artifacts → documents backfill machinery. Only the write target changes.

The one honest loss in Migration B: historical per-save versions become Yjs history going forward, but the 2,398 existing version rows don't convert to meaningful update logs. They stay queryable on the source until retirement.

Every AGS column has a home

PR slicing

The rework surface: FormEditor and friends

How much code moves, and for whom. The honest cut is against Migration B, because most of this rework is owed under either plan the moment instances leave the artifact blob. The last column is what FORM_JSON adds on top of the standalone tables.

Reading the last column: the FORM_JSON-specific rework concentrates in exactly two places, the FormEditor's save path and the adapter's write path. Everything else is either owed to Migration B regardless or untouched. And one row is negative: the agent-edit merge machinery (the clobber guard, the re-baseline logic) exists because whole-blob saves race; it gets deleted, not ported.

All sizes assume the editor stays on server mutations (question 1's recommendation). Making the FormEditor a live Yjs client is the one genuinely large piece (provider wiring, offline/undo semantics, presence) and is not required for any of the above; it is the optional upgrade this architecture leaves open.

Four options

The hybrid deserves attention: the thin rows answer question 2 without a global Template scope (the definition row owns identity; its content is just storage), and the guardrail gate stays where the standalone plan puts it. Its cost is that Template's publish flow is not reused, only Content's version/File machinery.

What is unaffected either way

This decision only changes the Migration A/B end-state, which is still a proposal. The shipped PRs need no redo under any option.

Why FirmTemplate survives every option

FirmTemplate is the one layer neither end-state can absorb. The catalogue is one global definition seen by many firms, but categories and jurisdictions are per-firm editable (PM requirement). A single global row, whether a Form or a Template(FORM), can only carry global tags: TemplateCategoryAssignment attaches categories to the template, not to a (template, firm) pair. The only Template-native alternative is one FORM template per firm, which duplicates definitions per firm and both plans reject. So one row per (form, firm) holding the firm's tags and standing as the assignment is required everywhere. That row is FirmTemplate.

• FirmTemplate.formSlug re-points at Migration A time: to Form.id in the standalone plan, to the FORM template here. The same one-column migration either way, which is why the shipped slice is keyed by slug rather than any store's id.
• The interim engine catalogue client retires at Migration A in both plans; the list join flips local.
• If this option wins, a later follow-up could generalise FirmTemplate into a per-firm view of any global template (assignment + firm tags) for future curated catalogues. A rename, not a redo.

Questions to resolve in the meeting

• 1 · Which option are we building towards? Standalone, full extension, or hybrid. If we can't decide outright, agree the smallest spike that would decide it (a FORM_JSON walking skeleton behind a flag is PR 1 of this plan anyway).
• 2 · Guardrails under CRDT. Accept the recommendation (gate at lawrence-api, FormEditor stays on server mutations) or push for validate-on-apply? Whoever wants a live Yjs editor later inherits this constraint.
• 3 · Global scope. Teach assertScopeAccess a global value (touches middleware every content surface shares, so Lukas's call), or pick the hybrid precisely to avoid it?
• 4 · Instance row: thin MatterFilledForm or MatterDocument with a discriminator? Thin row keeps matter lists and finalisation form-aware without a type sweep; a discriminator means one model but touches every MatterDocument consumer.
• 5 · Field-level analytics. Does anyone query filled values in the warehouse today? If yes, the JSON projection moves from "later" into Migration B's scope.
• 6 · Ownership of the shared machinery. Who validates that FORM_JSON contents don't destabilise snapshot compaction, epochs and shells? This is the one part LEX can't safely own alone.
• 7 · Alignment with the flat-form ingestion workstream. Its correction UI writes to the engine's annotations_raw; under any migration option that write target flips. Agree when, before it puts down deeper roots.
• 8 · Sequencing. Does choosing this option change the Migration A/B order or the LEX roadmap? (It shouldn't: A and B stay independent, and the shipped LEX-677 slice is end-state-agnostic.)

Questions 1 to 3 decide the architecture; 4 to 8 are scoped once 1 lands. Everything else raised during review (instantiation, groups, Yjs vs ProseMirror, column disposition, FirmTemplate's survival) is answered inline above.