NetMesh/docs/designs/convergent-sync-protocol-v2.md

# Convergent Sync Protocol and Migration

Status: experimental end-to-end implementation.

Issue: [#2245](https://github.com/binaricat/Netcatty/issues/2245)

## Compatibility contract

A v2 cloud file remains a normal `SyncedFile`. Its plaintext metadata adds only
`syncSchemaVersion: 2`; the complete materialized v1 `SyncPayload`, CRDT
metadata, conflicts, and candidate values remain inside the existing
AES-256-GCM ciphertext.

The decrypted payload keeps all v1 fields so an older Netcatty client can read
hosts, keys, snippets, settings, and other synchronized collections without a
new parser. The adjacent `convergentSync` envelope records causal metadata and
concurrent alternatives. For a visible entity field or settings leaf, the
selected winner is normally omitted from the envelope and reconstructed from
the materialized v1 field. Structural presence and position values stay inline;
they are CRDT metadata rather than duplicated user records.

Hydration fails closed when:

- plaintext metadata advertises an unknown or malformed schema;
- metadata and the encrypted envelope disagree;
- a materialized winner cannot be reconstructed;
- the envelope violates any core dot, context, origin, vector, or HLC
  invariant;
- an unknown collection would be discarded by the current materializer.

## Legacy import

Initial v1-only migration downloads every connected provider and runs the
existing smart merge against each provider-specific trusted base. A provider
without a trustworthy base may be adopted as a fresh-device seed, or accepted
when it exactly matches the current merge; divergent data without a base blocks
instead of guessing whether an absent entity was deleted. Any unresolved
entity/settings conflict, unavailable provider, or shrink guard blocks
initialization. A successful result becomes one v2 CRDT lineage and is written
back as a full v1 materialized snapshot plus the compact envelope.

If one or more providers already contain v2, their states are joined with the
CRDT join. A legacy local or provider snapshot may then contribute writes only
when a trusted materialized baseline is available. The baseline-to-snapshot
field diff is applied on an independent branch with a stable synthetic device
ID and then joined, so multiple legacy writers remain concurrent instead of
being ordered by provider iteration. Without a trustworthy baseline, upload is
blocked rather than guessing whether an absent field means deletion.

Optional top-level collections omitted by an old client, or present only as an
in-memory `undefined` property, are treated as unsupported and left unchanged.
Explicitly present empty collections are real deletions. Arrays inside settings
remain atomic, matching the CRDT core. Every legacy branch also passes the
suspicious-shrink guard before its writes can join an existing v2 state; fields
omitted by that client inherit the trusted baseline for this safety check.
A device with no cloud entities and no trusted local baseline is treated as a
fresh install: v1 and v2 migrations seed from cloud instead of turning local
first-launch settings into edits. With a trusted baseline, an empty local
snapshot remains a causal deletion. A v1 provider selected as the fresh-device
seed still passes the same suspicious-shrink guard before it can initialize v2.

## Local persistence and key rotation

The canonical replica and provider-specific v2 baselines are encrypted with the
same master-derived AES-GCM key used for existing sync bases and sync snapshots.
Loading an existing v2 record is strict: corruption and unsupported schemas are
errors, never `null` fallbacks.
Provider-specific v2 baselines are invalidated together with the existing merge
base and remote anchor whenever an account, endpoint, bucket, or connection is
replaced, so a new remote identity can never inherit trust from the old one.

Master-key rotation prepares replacement ciphertext for all derived-key sync
records before writing anything. It snapshots both existing records and absent
keys, then verifies neither changed during preparation. Only then does it commit
the new ciphertext and publish the new master configuration. A write failure
restores the exact prior ciphertext and configuration.

The experimental enabled/paused flag is device-local and is intentionally not
part of `SyncPayload.settings`. Disabling an initialized replica pauses it; it
does not delete local or cloud metadata. Clearing v2 storage requires explicit
downgrade confirmation.

## Backup and restore

Local vault backups remain materialized snapshots and never carry the active
replica. Migration initialization uses the existing protected-apply transaction:
under the convergent Web Lock it rebuilds the current cloud-sync payload and
compares it with the snapshot used for the preview. Any intervening local edit
aborts initialization and requires a new preview. An unchanged vault gets a
required encrypted safety backup, holds the cross-window restore barrier,
applies the previewed materialized payload, persists the canonical replica, and
only then marks v2 initialized. Before releasing the same Web Lock, migration
forces a convergent read/merge/write/verify cycle so unchanged v1 materialized
data still receives a v2 envelope on every connected provider. Concurrent
provider edits discovered by that cycle are protected and applied locally;
partial publication remains visible as a provider error and pending sync. The
sync manager must still be unlocked immediately before this transaction;
otherwise initialization fails before any backup, sentinel, or local mutation.
A crash or failure after mutation starts leaves the existing apply sentinel set
so auto-sync cannot publish a partial migration.

Before a local backup restore mutates local data, the restored snapshot is
diffed against the current materialized replica and prepared as normal device
writes without persisting them. The replica load is therefore validated before
the protective backup and partial-apply sentinel, while the prepared writes are
committed only after every local import step succeeds. A preparation failure
leaves no sentinel because the vault is still untouched. An import failure
leaves the replica unchanged; a later replica commit failure leaves the
protected-apply sentinel set so the partial restore cannot be published. Causal
history and tombstones survive restore instead of being replaced by an unrelated
replica copied from the backup.

Trusted legacy diffs also compare collection positions. A reorder-only edit is
converted into position-register writes for entity and string collections,
rather than disappearing because the values themselves are unchanged.
In-memory entities are normalized with the same JSON serialization semantics
as encrypted sync payloads before validation, so optional `undefined` model
fields are omitted instead of preventing migration.

## Provider convergence state machine

An initialized device holds one canonical replica shared by every provider.
Each sync acquires an exclusive Web Lock; environments without Web Locks fail
closed so two renderer windows cannot allocate and upload competing local
states. Disabling the experimental switch pauses the v2 path and never falls
through to the legacy writer.

The runtime downloads every connected provider before choosing an outgoing
state. `smartMerge` joins local writes and all remote branches, `preferLocal`
joins first and then creates causal local writes that dominate the joined
registers, and `preferCloud` adopts the unordered remote join. The canonical
state containing locally generated dots is encrypted and persisted before any
provider upload. Downloaded remote-only dots are committed to the local replica
only after at least one provider verifies the joined state; a total network
failure therefore leaves the durable replica aligned with the unchanged local
vault and safely retries the remote branch later.
Before `smartMerge` or `preferLocal` turns a local snapshot into writes, the
existing suspicious-shrink guard compares it with the materialized replica.
Mass deletion is blocked before dots are allocated or persisted unless the
user performs the existing one-shot force operation.

Providers then run at most three read-merge-write-verify rounds. Every round:

1. downloads and joins in memory any state that appeared since the initial read;
2. uploads the same expected vector to available providers;
3. reads each provider back and accepts the write only when the returned vector
   dominates the expected vector;
4. joins verified remote supersets and repeats when they contain new concurrent
   state.

The retry delay uses short full jitter. One unavailable provider remains an
error and leaves local sync pending, but it does not roll back providers that
verified successfully. Because locally generated causal writes are durable
before network I/O, application restart retries the same dots instead of
regenerating them. Provider baselines and the joined canonical replica advance
only after read-back verification.

## Conflict resolution and downgrade

Materialization exposes retained conflicts by register address. Choosing a
candidate writes a new device value whose context observes every candidate;
the resolution therefore dominates stale replicas and propagates through the
normal provider state machine. If propagation discovers additional concurrent
provider writes, Netcatty applies the final canonical materialization locally
before releasing the same Web Lock. Secret-bearing fields are detected from
their address and nested field names, including objects nested inside atomic
arrays. Their UI renders only “set” or “empty”; values are never formatted,
logged, or inserted into DOM text.

Explicit downgrade holds the same Web Lock and downloads every connected
provider before writing anything. Netcatty first converts edits made while v2
was paused into causal writes over the local replica, then joins those writes
with every remote state. It applies the joined payload behind a protective
backup and blocks downgrade until any newly discovered field conflicts are
resolved. It then writes the joined materialized v1 snapshot to every provider,
downloads it again, and verifies both the absence of v2 metadata and equality
of cloud data.
Only after every provider verifies does Netcatty clear the local replica,
provider baselines, and experimental configuration, still inside the same Web
Lock. A partial downgrade keeps the joined local v2 state and refreshed
provider baselines so the user can retry safely without losing remote-only
dots.