NetMesh/docs/research/issues-3213-3155-sftp-responsiveness.md

SFTP responsiveness: issues #3213 and #3155

Investigated on 2026-08-31 against 39d7c38a6acea2f59524566d117345e1ced21fbd.

Evidence and scope

#3213 reports slow/unstable large-file transfers, ineffective pause/resume and failed recovery after killing the app. #3155 reports a freeze with many files. Neither report supplies a direction, server configuration or transfer log. The following are reproduced code defects, not proof that every reported symptom has one cause. Keep both issues open for reporter confirmation.

1. globalTransferScheduler.run scanned the entire waiting queue on every insertion. Enqueuing 10,000 jobs behind two active jobs took 1,237 ms and 49,985,005 limit checks locally. Coalescing queue pumps reduces the same case to 10,001 checks (3-6 ms in local runs). Priority, owner fairness and per-host limits remain unchanged; immediately completed batches also yield to input.
2. The transfer-center popover mounted every top-level row, even far below the viewport. The actual Electron component took about 1,885 ms to display 1,000 rows. A measured, bounded viewport mounts around nine rows instead. All jobs remain in the store; scrolling and bucket selection still expose them. Folder expansion is held outside the recycled row.
3. Remote source/prefix verification used ssh2's serial createReadStream, even though body downloads already used pipelined reads. Pause captures a complete identity; resume verifies it before continuing. A 128 MiB loopback SFTP test with delayed READ replies spent 41,493 ms in resume on the old code. Reuse the existing 64-request, 32 KiB, ordered SHA-256 verification helper. This still checks every required byte with bounded memory, cancellation and inactivity deadlines. It does not substitute metadata, sampling or file size for content verification.

Correctness boundaries retained

• Only contiguous acknowledged ranges are checkpoints; aggregate displayed progress and sparse file length are not safe offsets.
• First-run force-kill recovery still restarts at zero when no complete source identity was captured. Recovery persistence is unchanged: progress since the last lifecycle save can still be lost. This PR improves the verified resume path's latency, not every whole-app force-kill recovery scenario.
• SCP and legacy fastPut paths do not gain unsupported pause/resume.
• Source changes, staged-prefix mismatch, missing staging, cancellation, replacement, permissions and conflict handling retain their existing checks.
• No claim is made about arbitrary server power-loss durability, every Windows server, or a universal throughput multiplier.

Mature-client comparison

• FileZilla: official SVN revision 11556 consumes directory results in batches and posts a continuation to the UI loop; its queue fills available transfer slots under total/directional/site limits. Adopt cooperative work and bounded admission, not an unrestricted Promise.all or a larger packet. Sources: directory consumption, lines 69-129, queue admission, lines 544-632 and 2452-2493.
• OpenSSH: 32 KiB requests and a 64-request default window; interruption stops new requests, drains replies and tracks the contiguous acknowledged prefix separately from the highest acknowledged position. Its manual warns that mismatched partial content can corrupt a resumed file. Netcatty keeps its stronger content checks. Sources: defaults and transfer loops, resume warning.
• WinSCP: two simultaneous background operations is also its default; eligible transfers use temporary files before publication. These choices do not establish that two files is Netcatty's bottleneck, or that every killed process can safely resume. Sources: background queue, resume requirements and temporary-file tradeoffs.
• rclone: documents the same 32 KiB/64-request defaults, server compatibility concerns and possible deadlock when checks/transfers compete for a capped connection pool. Future adaptive windows or a server compatibility matrix should be separate measured work. Source: SFTP documentation.

Reproduction and verification

The regression tests exercise scheduler admission/order/yielding, complete remote prefix verification, and actual transfer-center DOM bounds/scrolling. Each new defect was observed failing before its fix.

The opt-in real SSH/SFTP fixture listens only on loopback, uses generated test credentials and isolates all temp/home state. It creates a saved prefix, checks resume and (for sufficiently long transfers) live pause/resume, then checks the complete output SHA-256. It is not a simulation of a whole-app force-kill.

NETCATTY_SFTP_LIVE=1 SFTP_LIVE_MIB=128 SFTP_LIVE_FILES=12 \
  node scripts/sftp-transfer-resume.live.test.cjs
NETCATTY_SFTP_LIVE=1 SFTP_LIVE_MIB=128 \
  SFTP_LIVE_BASELINE_REF=39d7c38a6 \
  node scripts/sftp-transfer-resume.live.test.cjs

Twelve 128 MiB files completed with matching hashes, two submitted at a time. Each exercised pause/resume: pause acknowledgements took 6-15 ms; resume took about 1.45-3.18 seconds in this run. These are fixture-specific observations, not performance promises. Electron verification also exercised the production popover, pause-all/resume-all, scrolling to the last file, bucket switching and 20,000-job admission while the popover was open.

Review follow-up

• Preserve serial prefix verification when a server rejects range OPEN/READ with an ordinary protocol error. Cancellation and request timeouts do not fall back; timed-out channels are abandoned before another attempt.
• Count every positive short READ as inactivity-watchdog activity, without changing ordered full-range hashing. Finished windows cannot publish late progress, rearm their watchdog or issue another partial READ.
• Allocate the live fixture inside the managed temp directory, then isolate its own staging/home state. Cleanup removes only that unique fixture child, including when fixture setup fails.
• Keep transfer-row separators based on the actual list position, not the temporary viewport wrappers. The final list row alone omits its separator.
• Remove the proposed periodic full-history save: with 20,000 queued files, serialization alone occupied about 80 ms every five seconds. A compact journal would additionally need cross-window ownership and retry-attempt coordination. Those recovery semantics are not changed in this focused responsiveness PR. There is no new persistence timer, storage key or schema migration.