NetMesh/docs/designs/convergent-sync-crdt.md

Convergent Sync CRDT Core

Status: experimental core; not connected to persistence or cloud providers yet.

Issue: #2245

Goal

The existing sync engine compares local and remote snapshots against a stored base. That is useful for two replicas, but folding more replicas or providers in different orders is not algebraically safe. The v2 core defines a state-based join so every replica reaches the same state regardless of message order, duplication, or grouping.

This first change intentionally contains only pure domain logic. Encryption, legacy migration, provider verification, persistence, and UI are separate follow-up changes after this core is reviewed.

Mutation callers must provide the wall-clock sample used to advance the HLC. The domain layer never reads Date.now(), so replaying the same state, device, mutation batch, and timestamp produces identical serialized state.

State model

Each device owns a monotonically increasing counter. A write allocates a unique dot (deviceId, counter). The global version vector allocates collision-free dots, while each candidate records the exact prior dots observed in its own register. The register context is an exact dot set rather than another compact version vector because one device's global counters can interleave writes to different registers. A Hybrid Logical Clock (HLC) supplies a user-facing ordering hint without defining causality.

The replica contains:

• one global dotted version vector and HLC;
• a compact dot-origin index mapping every device counter to its register;
• an MV-register for entity presence;
• an MV-register for collection position;
• an MV-register for every top-level entity field;
• an MV-register for every settings leaf path (arrays are atomic leaves);
• observed-remove string entries with their own presence and position registers.

A string-entry remove is emitted only after the replica has observed a currently visible add. Deleting a locally absent entry is a no-op, so a concurrent add on another replica survives without creating an artificial value/tombstone conflict.

Mutation application is idempotent for already-selected entity fields, string entries, and settings. A same-value settings write still tombstones active ancestor or descendant paths, and a same-value write against an MV-register conflict still emits a causally dominating resolution.

A full entity upsert follows the same rule for presence, fields, and collection position: unchanged conflict-free registers are preserved, while accepting a currently selected conflicted value emits a new candidate that dominates every retained alternative.

String-entry add mutations likewise resolve visible presence and position conflicts even when the selected values are unchanged; conflict-free repeated adds remain no-ops.

Deletion is a register candidate, not absence from the serialized structure. Tombstones are retained indefinitely in v2. A later recreation replaces a tombstone only when its new dot causally observes the deletion.

Settings writes keep the active leaf set prefix-free. Replacing an object leaf with an atomic parent (or the reverse) causally tombstones the overlapping paths. Deleting a settings path tombstones that path and every causally observed descendant, so deleting a subtree cannot leave stale leaf registers visible; deleting a nested path does not implicitly remove an atomic ancestor. Independent replicas can still create a parent/descendant shape conflict; materialization then selects a deterministic maximal prefix-free set, keeps non-overlapping siblings, and reports the competing paths and candidates for explicit resolution.

Entity field updates also write a fresh present candidate. Consequently, an offline deletion racing an offline edit becomes a presence conflict; it cannot silently hide the edit. No-op field writes allocate no dot and do not refresh presence. Deleting a field from a non-present entity may tombstone stale field data but never recreates the entity.

Join

For each register, the join keeps:

1. candidates present on both sides;
2. left-only candidates not covered by the right register's causal context;
3. right-only candidates not covered by the left register's causal context.

Candidates causally dominated by another surviving candidate are removed. The replica vector is the pointwise maximum. A global vector is never used as proof that a candidate from an absent register was superseded; only a candidate in that same register can carry such proof. This makes partial provider states fail validation instead of silently deleting unrelated local data. Join remains commutative, associative, and idempotent. Property tests exercise those laws directly and also reduce 2-20 randomly generated offline replicas using reordered, partitioned, and duplicated joins.

Reusing a dot for different data or different register addresses is an invariant violation and fails closed. Every global vector counter must also be witnessed by a retained candidate dot or same-register candidate context. Hydration rejects dangling observations so a malformed or partial state cannot use an unsubstantiated vector to discard local candidates during join. It also rejects a context that references any currently retained candidate dot: exact contexts contain dominated history only, so retained references would represent invalid causal dominance or a cycle. The permanent dot-origin index proves that each candidate and context dot belongs to the register claiming it, so copying an omitted register's dot into unrelated context cannot satisfy validation. Origin indexes join by dot and reject conflicting register identities.

Materialization and conflicts

Concurrent candidates remain in the CRDT state. A deterministic materialized snapshot is selected for legacy readers and immediate application:

1. a value sorts after a tombstone;
2. then HLC wall time and logical counter;
3. then device ID using locale-independent UTF-16 code-unit order;
4. then device counter.

Candidate ordering in canonical serialization uses the dot, not the selected winner order. Dot and HLC objects are rebuilt with fixed property order so provider JSON key ordering cannot change identity or serialized bytes. Conflicts are emitted in collection, entity, field-path, and dot order. Resolving a conflict creates a new write whose causal context covers all observed candidates, so the resolution remains stable when stale replicas return.

Internal field and position conflicts are emitted only while their parent entity or string entry is materialized. An accepted parent deletion retains the underlying causal metadata for future explicit recreation but suppresses non-actionable child conflicts from the current conflict list. A concurrent delete/update whose selected presence remains visible still exposes both the presence conflict and all active internal conflicts.

Complexity

State validation and canonical serialization are linear in registers, candidates, and retained context dots. Join additionally compares the bounded set of concurrent candidates within each register. Sorting is bounded by keys within each map. Batch mutation clones the replica once, avoiding a full-state copy per imported entity. npm run bench:sync-crdt reports non-gating measurements for 1,000, 5,000, and 10,000 entities so accidental quadratic behavior is visible during review.

Follow-up boundaries

The encrypted v2 envelope, legacy baselines, migration preview, protection snapshots, key rotation, and fail-closed protocol rules are described in convergent-sync-protocol-v2.md. The final change will integrate provider read-merge-write-verify loops, multi-window locking, conflict resolution state, and localized settings UI.