[Init] Initial commit - NetMesh terminal manager

docs/designs/convergent-sync-protocol-v2.md

@@ -0,0 +1,182 @@
+# Convergent Sync Protocol and Migration
+
+Status: experimental end-to-end implementation.
+
+Issue: [#2245](https://github.com/binaricat/Netcatty/issues/2245)
+
+## Compatibility contract
+
+A v2 cloud file remains a normal `SyncedFile`. Its plaintext metadata adds only
+`syncSchemaVersion: 2`; the complete materialized v1 `SyncPayload`, CRDT
+metadata, conflicts, and candidate values remain inside the existing
+AES-256-GCM ciphertext.
+
+The decrypted payload keeps all v1 fields so an older Netcatty client can read
+hosts, keys, snippets, settings, and other synchronized collections without a
+new parser. The adjacent `convergentSync` envelope records causal metadata and
+concurrent alternatives. For a visible entity field or settings leaf, the
+selected winner is normally omitted from the envelope and reconstructed from
+the materialized v1 field. Structural presence and position values stay inline;
+they are CRDT metadata rather than duplicated user records.
+
+Hydration fails closed when:
+
+- plaintext metadata advertises an unknown or malformed schema;
+- metadata and the encrypted envelope disagree;
+- a materialized winner cannot be reconstructed;
+- the envelope violates any core dot, context, origin, vector, or HLC
+  invariant;
+- an unknown collection would be discarded by the current materializer.
+
+## Legacy import
+
+Initial v1-only migration downloads every connected provider and runs the
+existing smart merge against each provider-specific trusted base. A provider
+without a trustworthy base may be adopted as a fresh-device seed, or accepted
+when it exactly matches the current merge; divergent data without a base blocks
+instead of guessing whether an absent entity was deleted. Any unresolved
+entity/settings conflict, unavailable provider, or shrink guard blocks
+initialization. A successful result becomes one v2 CRDT lineage and is written
+back as a full v1 materialized snapshot plus the compact envelope.
+
+If one or more providers already contain v2, their states are joined with the
+CRDT join. A legacy local or provider snapshot may then contribute writes only
+when a trusted materialized baseline is available. The baseline-to-snapshot
+field diff is applied on an independent branch with a stable synthetic device
+ID and then joined, so multiple legacy writers remain concurrent instead of
+being ordered by provider iteration. Without a trustworthy baseline, upload is
+blocked rather than guessing whether an absent field means deletion.
+
+Optional top-level collections omitted by an old client, or present only as an
+in-memory `undefined` property, are treated as unsupported and left unchanged.
+Explicitly present empty collections are real deletions. Arrays inside settings
+remain atomic, matching the CRDT core. Every legacy branch also passes the
+suspicious-shrink guard before its writes can join an existing v2 state; fields
+omitted by that client inherit the trusted baseline for this safety check.
+A device with no cloud entities and no trusted local baseline is treated as a
+fresh install: v1 and v2 migrations seed from cloud instead of turning local
+first-launch settings into edits. With a trusted baseline, an empty local
+snapshot remains a causal deletion. A v1 provider selected as the fresh-device
+seed still passes the same suspicious-shrink guard before it can initialize v2.
+
+## Local persistence and key rotation
+
+The canonical replica and provider-specific v2 baselines are encrypted with the
+same master-derived AES-GCM key used for existing sync bases and sync snapshots.
+Loading an existing v2 record is strict: corruption and unsupported schemas are
+errors, never `null` fallbacks.
+Provider-specific v2 baselines are invalidated together with the existing merge
+base and remote anchor whenever an account, endpoint, bucket, or connection is
+replaced, so a new remote identity can never inherit trust from the old one.
+
+Master-key rotation prepares replacement ciphertext for all derived-key sync
+records before writing anything. It snapshots both existing records and absent
+keys, then verifies neither changed during preparation. Only then does it commit
+the new ciphertext and publish the new master configuration. A write failure
+restores the exact prior ciphertext and configuration.
+
+The experimental enabled/paused flag is device-local and is intentionally not
+part of `SyncPayload.settings`. Disabling an initialized replica pauses it; it
+does not delete local or cloud metadata. Clearing v2 storage requires explicit
+downgrade confirmation.
+
+## Backup and restore
+
+Local vault backups remain materialized snapshots and never carry the active
+replica. Migration initialization uses the existing protected-apply transaction:
+under the convergent Web Lock it rebuilds the current cloud-sync payload and
+compares it with the snapshot used for the preview. Any intervening local edit
+aborts initialization and requires a new preview. An unchanged vault gets a
+required encrypted safety backup, holds the cross-window restore barrier,
+applies the previewed materialized payload, persists the canonical replica, and
+only then marks v2 initialized. Before releasing the same Web Lock, migration
+forces a convergent read/merge/write/verify cycle so unchanged v1 materialized
+data still receives a v2 envelope on every connected provider. Concurrent
+provider edits discovered by that cycle are protected and applied locally;
+partial publication remains visible as a provider error and pending sync. The
+sync manager must still be unlocked immediately before this transaction;
+otherwise initialization fails before any backup, sentinel, or local mutation.
+A crash or failure after mutation starts leaves the existing apply sentinel set
+so auto-sync cannot publish a partial migration.
+
+Before a local backup restore mutates local data, the restored snapshot is
+diffed against the current materialized replica and prepared as normal device
+writes without persisting them. The replica load is therefore validated before
+the protective backup and partial-apply sentinel, while the prepared writes are
+committed only after every local import step succeeds. A preparation failure
+leaves no sentinel because the vault is still untouched. An import failure
+leaves the replica unchanged; a later replica commit failure leaves the
+protected-apply sentinel set so the partial restore cannot be published. Causal
+history and tombstones survive restore instead of being replaced by an unrelated
+replica copied from the backup.
+
+Trusted legacy diffs also compare collection positions. A reorder-only edit is
+converted into position-register writes for entity and string collections,
+rather than disappearing because the values themselves are unchanged.
+In-memory entities are normalized with the same JSON serialization semantics
+as encrypted sync payloads before validation, so optional `undefined` model
+fields are omitted instead of preventing migration.
+
+## Provider convergence state machine
+
+An initialized device holds one canonical replica shared by every provider.
+Each sync acquires an exclusive Web Lock; environments without Web Locks fail
+closed so two renderer windows cannot allocate and upload competing local
+states. Disabling the experimental switch pauses the v2 path and never falls
+through to the legacy writer.
+
+The runtime downloads every connected provider before choosing an outgoing
+state. `smartMerge` joins local writes and all remote branches, `preferLocal`
+joins first and then creates causal local writes that dominate the joined
+registers, and `preferCloud` adopts the unordered remote join. The canonical
+state containing locally generated dots is encrypted and persisted before any
+provider upload. Downloaded remote-only dots are committed to the local replica
+only after at least one provider verifies the joined state; a total network
+failure therefore leaves the durable replica aligned with the unchanged local
+vault and safely retries the remote branch later.
+Before `smartMerge` or `preferLocal` turns a local snapshot into writes, the
+existing suspicious-shrink guard compares it with the materialized replica.
+Mass deletion is blocked before dots are allocated or persisted unless the
+user performs the existing one-shot force operation.
+
+Providers then run at most three read-merge-write-verify rounds. Every round:
+
+1. downloads and joins in memory any state that appeared since the initial read;
+2. uploads the same expected vector to available providers;
+3. reads each provider back and accepts the write only when the returned vector
+   dominates the expected vector;
+4. joins verified remote supersets and repeats when they contain new concurrent
+   state.
+
+The retry delay uses short full jitter. One unavailable provider remains an
+error and leaves local sync pending, but it does not roll back providers that
+verified successfully. Because locally generated causal writes are durable
+before network I/O, application restart retries the same dots instead of
+regenerating them. Provider baselines and the joined canonical replica advance
+only after read-back verification.
+
+## Conflict resolution and downgrade
+
+Materialization exposes retained conflicts by register address. Choosing a
+candidate writes a new device value whose context observes every candidate;
+the resolution therefore dominates stale replicas and propagates through the
+normal provider state machine. If propagation discovers additional concurrent
+provider writes, Netcatty applies the final canonical materialization locally
+before releasing the same Web Lock. Secret-bearing fields are detected from
+their address and nested field names, including objects nested inside atomic
+arrays. Their UI renders only “set” or “empty”; values are never formatted,
+logged, or inserted into DOM text.
+
+Explicit downgrade holds the same Web Lock and downloads every connected
+provider before writing anything. Netcatty first converts edits made while v2
+was paused into causal writes over the local replica, then joins those writes
+with every remote state. It applies the joined payload behind a protective
+backup and blocks downgrade until any newly discovered field conflicts are
+resolved. It then writes the joined materialized v1 snapshot to every provider,
+downloads it again, and verifies both the absence of v2 metadata and equality
+of cloud data.
+Only after every provider verifies does Netcatty clear the local replica,
+provider baselines, and experimental configuration, still inside the same Web
+Lock. A partial downgrade keeps the joined local v2 state and refreshed
+provider baselines so the user can retry safely without losing remote-only
+dots.