Wire Protocol & Remote Transfers

For remote transfers, bcmr implements a binary frame protocol (bcmr serve) that replaces per-file SSH process spawning with a persistent connection over stdin/stdout. The protocol uses length-prefixed frames ([4B length][1B type][payload]) and supports: Stat, List, Hash, Get, Put, Mkdir, Resume, plus the extensions covered on this page.

Key properties of the base design:

• Persistent sessions: operations share a long-lived SSH session instead of spawning one process per file. Multi-file batches may opt into several independent sessions with --parallel.
• Server-side hashing: the remote bcmr computes BLAKE3 hashes locally, avoiding data round-trips for verification.
• Visible fallback: if the remote lacks a matching bcmr serve, the client warns and falls back to legacy SCP before transfer processing starts.
• Frame size limit: read_message rejects frames $> 16$ MiB to prevent memory exhaustion from malicious peers.

The current wire version is 2. Hello / Welcome carry a capabilities byte (LZ4 = 0x01, Zstd = 0x02, Dedup = 0x04, Fast = 0x08, Sync = 0x10, Direct TCP = 0x20, AEAD = 0x40, PUT offset = 0x80). Capability intersection still disables optional features safely, but it is not a substitute for versioning message semantics.

Version 2 deliberately breaks wire compatibility with version 1: Put now carries an explicit overwrite policy. A mismatched client/server pair rejects the serve handshake rather than guessing, then the high-level copy command may fall back to legacy SCP. This keeps old devices usable without pretending that old protocol semantics are safe.

sequenceDiagram
    participant C as Client
    participant S as Server
    Note over C,S: Both hold their own caps mask.<br/>The intersection drives behaviour.
    C->>S: Hello { version: 2, caps: client_caps }
    S->>S: effective = server_caps AND client_caps
    S->>C: Welcome { version: 2, caps: effective }
    C->>C: algo = negotiate(client_caps, effective)
    Note over C,S: For each Data frame:<br/>encode_block(algo, ...) emits<br/>Data (raw) or DataCompressed.

When both peers advertise CAP_ZSTD it wins; if only one does, LZ4 (if both have it) is the fallback; otherwise raw Data. The encoder also runs a per-block auto-skip: if compressed.len() exceeds 95 % of the original (random / already-compressed bytes) the block goes raw to save the receiver's decompress pass.

Current v2 publication and liveness contract

• PUT defaults to no-clobber. Only --force --yes sets overwrite=true.
• The server writes through a handle-bound AtomicFile transaction, validates declared length and hash, applies security metadata, and only then publishes.
• Multi-file pipelined PUT remains parallel and atomic per file. The older single-file striped PUT is disabled in production because its Truncate + PutChunked shape exposed the final path before all workers completed. Re-enabling it requires a server-owned transaction token shared across connections.
• SSH sessions use ServerAliveInterval=15 and ServerAliveCountMax=20: a black-holed connection becomes a bounded failure after roughly five minutes, while a very slow connection that still exchanges data is not killed by a total wall-clock deadline.
• Release Linux binaries use musl targets. A protocol-v2 static x86-64 artifact was executed successfully on both a current Linux host and an older-glibc Linux host; a dynamically linked build failed on the latter before this control.

Parallel SSH with Independent Connections

SSH's ControlMaster multiplexing serializes all channels through one TCP connection and one encryption context. For $P$ parallel workers, throughput is bounded by a single core's encryption speed regardless of $P$.

The serve pool forces ControlMaster=no and ControlPath=none for each member, so $P$ workers really create $P$ independent TCP and cipher streams even when the user's SSH config enables multiplexing:

$\text{throughput} \approx \min(P \cdot T_{\text{single}},\; T_{\text{link}})$

The mscp project measured 5.98x speedup with 8 independent connections on a 100 Gbps link.

See the Remote Copy guide for end-user configuration.

Experiment 9: Wire Compression for Data Frames

Hypothesis: Per-block LZ4/Zstd encoding pays for itself whenever the network is slower than the codec. On modern CPUs LZ4 decodes at multiple GB/s, so the receiver is never compute-bound; the only question is ratio.

Method (Part A --- codec probe): encode then decode a single 4 MiB block three times (random, text-like, mixed) for each algorithm. Ratios and throughputs measured on Apple Silicon:

Interpretation:

1. Random data. All three codecs return ratios indistinguishable from 1.0. Sending compressed is pure CPU waste, so the wire path must auto-skip when the encode output is within 5 % of the input.
2. Text-like data. Zstd-3 reaches 5x reduction at 320 MB/s encode. For anything under ~2.5 Gbps of effective network throughput, compression is the bandwidth bottleneck, not the CPU.
3. Zstd-9 is consistently worse than -3 for file content: encode drops by 7x (to 47 MB/s) for only a ~2 % ratio gain. Skip it.

Decision: Default to auto-negotiation advertising both LZ4 and Zstd. The handshake picks Zstd when both sides speak it (better ratio at acceptable encode cost), falls back to LZ4 when only one does, and to raw Data frames otherwise. Zstd level fixed at 3 --- the library's own default, and our measurement agrees.

Method (Part B --- auto-skip in vivo): a unit test encodes a 4 MiB pseudo-random block through encode_block(Lz4, ...) and asserts the emitted message type is Data (raw), not DataCompressed. Covers the happy path where the codec's frame header + payload overshoots the 0.95 × original threshold and the encoder falls back.

Experiment 11: Content-Addressed Dedup for Repeat PUT

Hypothesis: For dev workflows where the same artifact is uploaded to a remote host repeatedly, the second-and-onward upload can avoid the wire entirely if the receiver remembers what it has seen. BLAKE3 is already computed per 4 MiB block, so a tiny pre-flight that exchanges hashes lets the server short-circuit to a local CAS read.

Design: Negotiate CAP_DEDUP in the Hello/Welcome caps byte. When active and the file is at least 16 MiB:

sequenceDiagram
    participant C as Client
    participant S as Server
    participant CAS as Server CAS<br/>~/.local/share/bcmr/cas
    C->>C: hash file in 4 MiB blocks
    C->>S: Put { path, size }
    C->>S: HaveBlocks { block_size,<br/>hashes: [h0, h1, h2, ...] }
    loop for each hash
        S->>CAS: exists?
    end
    S->>C: MissingBlocks { bits: 0b...10110 }
    Note over C,S: 1 = block needed on wire,<br/>0 = server already has it
    loop for each block i
        alt bit i is 1
            C->>S: Data / DataCompressed (block i)
            S->>CAS: write block i (mtime = now)
            S->>S: write block i to dst
        else bit i is 0
            S->>CAS: read block i (mtime → now)
            S->>S: write block i to dst
        end
    end
    C->>S: Done
    S->>C: Ok { hash: <BLAKE3 of dst> }

The composite hash returned in Ok covers the full file regardless of which blocks took which path. The 16 MiB threshold protects small uploads from the round-trip cost of HaveBlocks/MissingBlocks itself.

Method: 64 MiB pseudo-random file uploaded twice from a macOS laptop to host-L (Linux NVMe over the public internet, ~30 ms RTT, ~10 MB/s effective WAN bandwidth). Cold cache via rm -rf ~/.local/share/bcmr/cas between runs.

The savings track the eliminated wire bytes: 64 MiB at ~10 MB/s ≈ 6 s, which matches the observed delta. The remaining 13 s is local hash + CAS read + dst write + protocol round trips, all on either side of the network. For higher-bandwidth links the relative win shrinks; for slower / metered ones (cellular tethering, transoceanic SSH) it grows.

Correctness check: SHA-256 of source matches both destinations across the two runs.

CAP_FAST: Skip Server Hash + Linux Splice

--fast advertises CAP_FAST in the client's caps byte. When the server also has it (it always does on supported platforms), GET responses skip the inline BLAKE3 entirely and on Linux the file → stdout payload moves through splice(2) with no userspace buffer. The server's Ok carries hash: None; clients that need end-to-end integrity get it via -V (re-hash dst client-side).

The 4 MiB pipe buffer (set via fcntl(F_SETPIPE_SZ)) means each chunk needs one splice call from the file and one from the pipe to stdout, with no copy ever touching userspace. Compression is mutually exclusive with this path --- the encoder needs userspace bytes --- so CAP_FAST only activates the splice variant when --compress=none. CAP_FAST without splice (compression on, or non-Linux) still wins from skipping the server's BLAKE3.

Experiment 14: CAP_FAST Real Numbers

Hypothesis: Skipping the server-side BLAKE3 should always be a small win (server CPU saved) and on Linux the additional splice(2) zero-copy on the file → stdout payload should be a larger win when the network isn't the bottleneck.

Method (WAN): 1 GiB random file pulled from host-L over a ~10 MB/s residential link. Default vs --fast, both with --compress=none so the codec doesn't dominate.

Result: 1.07x. The wire is the bottleneck; saving ~700 ms of server-side hash on top of ~150 s of network is noise.

Method (loopback): same file, but bcmr copy localhost:src dst running on host-L itself. SSH-encrypted localhost peaks around ~500 MB/s on AES-NI hardware so this isolates server behaviour from real-network jitter.

Result: --fast is slower. Two compounding causes:

1. Pipe sizing falls back silently. fcntl(F_SETPIPE_SZ, 4 MiB) needs root or a raised /proc/sys/fs/pipe-max-size; the default cap on Ubuntu is 1 MiB. The call silently caps at the max and the actual buffer stays small (default 64 KiB), so each 4 MiB chunk needs ~64 paired splice rounds.
2. spawn_blocking per chunk. The splice loop currently lives in its own tokio::task::spawn_blocking per chunk --- the same anti-pattern that Experiment 13 identified for the local copy path. With 256 chunks per 1 GiB that's 256 thread bounces, more than enough to wipe out the savings from skipping hash and memcpy.

Decision: Keep --fast for the hash-skip benefit. The splice path stays in; the per-chunk spawn_blocking + pipe-sizing fixes are tracked on the Open Questions page and land in later releases.

Experiment 15: CAS LRU Eviction Under Load

Hypothesis: A monotonically-growing CAS makes dedup unusable on disk-constrained boxes. A simple LRU-by-mtime scheme should keep the store at the configured cap while preserving the most recently hit blocks (which are also the most likely to recur).

Method: end-to-end integration test with three distinct 24 MiB files (each = 6 blocks) PUT in sequence to a local serve process, with BCMR_CAS_CAP_MB=32 (=8 blocks). Cap is enforced on the server side at the start of each PUT. After all three uploads the CAS is walked and totals are checked.

The mtime touch on cas::write and cas::read means a block hit during PUT N+1 stays warmer than untouched blocks from PUT N, matching the dev-loop pattern (re-upload the same artifact).

Decision: ship the LRU as default. The hit rate degrades gracefully as cap shrinks, and the BCMR_CAS_CAP_MB=0 escape hatch restores the v0.5.8 unbounded behaviour for users who explicitly want it.

Experiment 12: Wire Compression Across Real Hosts

The earlier Experiment 9 measured codec ratios in isolation; this one re-runs the protocol over real SSH connections to confirm the prediction. 64 MiB of source-text-like content from host-N to three peers, three runs each.

Zstd-3 wins on every link. LZ4 wins over None but loses to Zstd because the bandwidth saving from Zstd's extra ratio more than pays for the lower encode throughput. host-M's smaller relative win comes from the path's variance dominating the small absolute duration --- the absolute saving is similar to the others.

Experiment 17: Per-File fsync as the Many-Files Tax

Hypothesis: bcmr serve was 7.86× slower than scp -r on a 10000 × 64 KiB benchmark on host-L loopback. --fast and default were both at ~24 s while scp finished in 3 s. The CPU breakdown (~3 s total CPU on a 24 s wall) ruled out compute. Likely culprit: per-file fdatasync in both directions --- server handle_put calls file.sync_all() per file, client GET callback calls dst_file.sync_all() per file. At ~1-2 ms per fdatasync × 10000 files, that's 10-20 s of pure barrier time — matching the gap.

cp/rsync/scp don't fsync per file by default. The local copy path in bcmr already gates fsync on --sync (see Experiment 7). The serve path was over-promising durability silently.

Method: 10000 × 64 KiB random files, host-L loopback ssh (Ubuntu 22.04, NVMe ext4, kernel 6.8). Warm cache, hyperfine, 2 runs each.

Implementation: New CAP_SYNC = 0x10 advertised by server; client OR's it into Hello caps when --sync is set. Both sides gate their per-file fsync on the negotiated bit. Default is off, matching cp/scp/local-copy default behavior.

Decision: Ship. Closes the many-files gap from 7.96x → 2.04x with no functionality loss — --sync users still get exactly the durability they asked for (and pay the 5x cost they implicitly opted into).

The single-file path also benefits — the GET fsync at end-of-file was ~0.5 s on a 1 GiB stream:

Experiment 18: Client-Side Request Pipelining

Hypothesis: After Experiment 17 (CAP_SYNC) closed the per-file fsync overhead, host-L still showed bcmr serve at 2.04× scp on the 10000 × 64 KiB many-files bench. CPU breakdown ruled out hash/encode work — the wall time was dominated by RTT-style serialization. The single-file put() does send Put → send Data* → send Done → await Ok, then the next file. Server's dispatch loop is FIFO and would happily process queued requests, but the client never queues more than one. SFTP (what scp -r uses under the hood) keeps a window of ~64 in-flight requests; that's the gap.

Method: Same 10000 × 64 KiB host-L loopback bench, plus the mirror upload direction and a single-1-GiB regression check. Three runs each, hyperfine, warm cache.

Implementation: Two new methods on ServeClient:

• pipelined_put_files(files, on_complete): takes ownership of stdin, spawns a writer task that emits Put / Data* / Done for every file in order. The reader task (the caller's task) collects FIFO Ok hashes from stdout.
• pipelined_get_files(files, sync, on_complete): mirror — the writer task sends every Get request up-front, the reader demuxes the Data* / Ok stream into per-file dst handles.

The OS pipe between the client process and the SSH child plus SSH's own send window provide natural backpressure when the server hasn't drained yet — no explicit channel needed.

Server unchanged: the dispatch loop in serve.rs:159-198 already processes requests in FIFO order. Pipelining is purely a client-side win.

Failure path: if the reader bails mid-batch (server emits Error for file K), both pipelined methods call writer.abort() before writer.await to avoid a deadlock where the writer keeps pushing into a stdin buffer nobody drains. The connection is left indeterminate after a pipelined failure — caller drops, not retries, on the same client.

Decision: Ship. Many-files closed from 7.86× scp to 1.18× in three steps (CAP_SYNC, GET pipelining, PUT pipelining). bcmr serve is now within 18 % of scp on multi-small-files while still offering resume, content-addressed dedup, inline integrity, and a progress UI scp doesn't have.

Experiment 19: Parallel SSH Connections Break the Single-Stream Ceiling

Hypothesis: After Experiment 18 closed the per-file RTT gap through client-side pipelining, bcmr serve ran at ~1.18× scp -r on the happy path (light load) and, we discovered, fell back to ~3-5× slower under heavy contention. The remaining gap has a structural cause: SSH gives you one cipher stream per TCP connection. AES-NI tops out at ~500 MB/s/core and ChaCha20 at ~200-500; a single SSH session is walled in by exactly one crypto thread on each side. Opening N independent SSH connections in parallel — what mscp does — multiplies the crypto ceiling by N until NIC or disk takes over. No protocol change.

Method: 10000 × 64 KiB files, host-L loopback ssh, same dataset as Experiment 17. Run under genuine production contention (load average ~93 from unrelated minimap2 -t 30 jobs on the box) so the crypto bottleneck actually materialises — Experiment 17's "load ~1" numbers hid the single-stream ceiling behind abundant headroom. 2 runs per N, /usr/bin/time for wall + CPU breakdown, scp -r as baseline.

Near-linear through N=4, diminishing past that (box already running 90+ threads of other users' CPU work — we're taking what we can). At N=8 bcmr beats scp by 24% on the same loaded box.

The scaling curve:

---
config:
  xyChart:
    width: 700
    height: 340
  themeVariables:
    xyChart:
      titleColor: "#222"
      plotColorPalette: "#7E6EAC, #CABBE9"
---
xychart-beta
    title "Exp 19: 10000 × 64 KiB on host-L under load ~93 (lower = faster)"
    x-axis "Pool size N" [1, 2, 4, 8]
    y-axis "Wall time (s)" 0 --> 80
    line [67.3, 40.8, 23.7, 14.7]
    line [19.4, 19.4, 19.4, 19.4]

The upper line is bcmr copy -r --parallel N, the flat line at 19.4 s is the scp -r baseline — bcmr crosses below it between N=4 and N=8. The diminishing return past N=4 is the box telling us the NIC and disk queue have started bounding us; the crypto ceiling that limited N=1 is already gone by then.

Implementation: New ServeClientPool { clients: Vec<ServeClient> } in src/core/serve_client.rs:

• connect_with_caps(target, caps, n) opens N connections concurrently via futures::try_join_all — total handshake latency ≈ one connection's, not N×, because they auth in parallel.
• pipelined_put_files_striped / pipelined_get_files_striped partition input files round-robin across the N clients and drive all buckets concurrently via another try_join_all. Each bucket runs the existing single-client pipelined method unchanged; the pool is purely a dispatch-and-scatter layer.
• PUT hashes come back in input-index order: each bucket saves its original indices, results are re-scattered into a size-N output vec at the end.
• One-shot protocol ops (stat, list, mkdir) stay on pool.first_mut() — no parallelism win for a single round trip, so don't waste the other N-1 connections.

Server unchanged: every connection talks the same protocol it always did. The pool is a client-side construct; bcmr serve on the other end sees N independent sessions exactly as if N separate users had opted in concurrently.

UX: --parallel N on the CLI now applies to both transports. For the serve fast path, default stays N=1 (no surprise behavior change for small batches — opening 8 SSH connections for a 3-file copy is pure handshake tax with no payoff). Users who know they're moving many small files or need to saturate a fat pipe opt in explicitly.

The part that's not free:

• N× SSH handshakes on connect (done concurrently so wall latency is ~1 handshake, but CPU + auth cost is N×).
• N× memory for per-client buffers, stdin/stdout pipes, tokio tasks. For N=8 that's ~16 MiB of pipe buffers on the box — trivial. Each bcmr serve subprocess adds a few MB of its own.
• Progress callbacks fire from multiple tasks simultaneously, so they need Fn + Send + Sync + Clone + 'static (our runner callbacks already are). on_complete no longer fires in input order — documented.

What doesn't help: OpenSSH ControlMaster=auto multiplexes channels over one TCP connection, which means they share a single cipher stream. Multi-channel ≠ multi-crypto. To lift the crypto ceiling you genuinely need N TCP connections = N SSH sessions.

Decision: Ship. First release where bcmr serve is faster than scp -r on both many-small-files and a realistically contended system. Single-connection baseline (--parallel 1) unchanged; users who want the win opt in. For single large files, --direct=direct lifts the stream ceiling instead (Exp 20).

Experiment 20: Direct-TCP Data Plane (Path B) Beats scp 2.84× on a Single Large File

Hypothesis: OpenSSH serialises an entire session through one cipher stream, so single-file copies over SSH are walled in at a fraction of the link capacity even when the CPU has plenty of AES-GCM headroom. Carving the data plane onto a dedicated AEAD-framed TCP socket (SSH kept only for auth + rendezvous) should lift that ceiling to within noise of the raw TCP rate.

Method: 1 GiB random file, host-N → host-L GET, LAN link (single-stream iperf3 -t 5 reports 41 MiB/s as the physical ceiling). --compress=none --fast to keep the number about transport cost, not Zstd or BLAKE3. 3 iterations per mode (best-of-3 reported — WiFi variance is enough that a median blurs the peak-achievable rate¹), fresh dst file each run.

Reading: SSH mode tracks scp — same OpenSSH stream, same ceiling — and lands within noise of it. Direct-TCP is 2.84× scp and reaches 65 % of the raw TCP rate; SSH mode stays at 23 %. --parallel=4 doesn't help a single 1 GiB file: no per-file chunk routing exists on this branch, so the extra sessions only add handshake cost. --parallel earns its keep on multi-file batches (Exp 19), not on one big file.

Integrity: BLAKE3 of the received bytes matches the source on every run (cross-checked with md5sum). AEAD framing is active on every direct-TCP run — is_aead_negotiated() returns true, enforced by the serve_direct_tcp_put_get_roundtrip assertion.

Decision: Ship. --direct=ssh stays the default (backwards compat, no new sshd config on the server). Users on LAN-class links flip --direct=direct and take the ~3× win on single large files. AEAD is mandatory on direct-TCP so the CLI can't silently drop into plaintext.

Experiment 21: Protocol v2 Atomic PUT and Failure Semantics

Problem found on a real peer: protocol v1 did not carry the caller's overwrite policy. Uploading different content to an existing isolated destination without --force returned exit 0 and replaced the file. This violated the CLI contract and made a retry capable of destroying the last good remote copy.

Change:

1. bump the protocol from 1 to 2;
2. encode overwrite: bool in every Put, including pipelined batches;
3. default to AtomicFile::new_no_replace, with AtomicFile::new reachable only through explicit force;
4. receive into a private handle-bound transaction, validate length/hash, apply inherited or preserved security metadata, then publish;
5. disable production single-file striped PUT until independent connections can join one server-owned transaction.

Method: current v2 client and a static-musl v2 server on an older-glibc Linux peer. All paths were isolated under a per-revision test cache. SHA-256 was checked independently with the host's sha256sum; downloads were also checked with cmp.

Successful and known-rejection cases left no .bcmr.receive.* entry. The killed transfer under a group-writable (0775) parent retained an empty private transaction directory. That is deliberate fail-closed behavior: after unlinking the exact payload through its retained directory handle, the implementation will not remove a directory name from a shared namespace where another principal could race a substitution. The test verified the directory was empty and removed only that exact path manually.

Security metadata controls:

• macOS: an initially missing destination receives exactly the serialized inheritable ACEs of a directly-created reference file; the in-flight transaction itself has no effective inherited ACL.
• Linux ext4: a parent default ACL is retained only long enough to create the empty payload; the resulting access ACL matches a direct child byte-for-byte, then the transaction's default ACL is removed before payload bytes are written.
• Existing destinations retain non-privileged ACL/xattr metadata; setuid/setgid and privileged content-bound xattrs are stripped.

Reliability control: SSH now probes a silent connection every 15 seconds and gives up after 20 unanswered probes. A third test peer transferred only 4.22 MiB in roughly two minutes but continued making progress; it was not misclassified as dead. This is why the policy is an idle/liveness bound rather than a total transfer deadline.

Performance boundary: this experiment establishes correctness, not a post-change throughput win. Earlier 20 MiB smoke runs on the v1 transport reached 20.25 MiB/s for compressible Zstd content and 13.48 MiB/s for random content on two real SSH paths, but those single runs are not used as a v2 before/after claim. Full adaptive compression, congestion response, and transactional single-file striping require their own controlled ablations.

Decision: ship v2 and reject v1 at the serve handshake. Preserve device reach through static musl releases and visible legacy-SCP fallback, not by accepting ambiguous PUT semantics.

Experiment 22: Size-Aware Multi-File Scheduling

Problem: ServeClientPool assigned files to connections by input index (i % N). For a traversal ordered 15, 1, 14, 1, 13, 1, 12, 1 MiB, two workers therefore received 54 MiB and 4 MiB. The faster worker cannot take work from the slow bucket, so the last oversized bucket determines completion.

Change: schedule Longest Processing Time first (LPT). Sort job indices by descending declared size and repeatedly assign the next job to the currently least-loaded worker. Buckets retain original input order internally, while callbacks and PUT hashes are remapped to their original indices. Every file counts for at least one scheduling unit so a batch of empty files still uses the whole available pool.

This changes no wire messages and adds only $O(n \log n + nP)$ client-side scheduling work for $n$ files and $P$ connections. It applies equally to pipelined PUT and GET.

Deterministic model control:

Unit controls also cover deterministic ties, u64 saturation, empty input and distributing zero-length files across all workers. The existing four serve-pool integration tests then verify PUT, GET, original-index hash ordering, the one-client case and sibling cancellation.

Real SSH method: compare release binaries from 9b87ce6 (round-robin) and d94c5e4 (LPT) on the same peer. Use -P 2, --direct ssh, and --compress none; all eight random-content files remain below the 16 MiB CAS threshold. Exclude one warm-up per binary, then run eight trials per scheduler in an interleaved order, always into a fresh directory. The received 60,817,408 bytes were checked file-by-file with independent SHA-256.

The raw mean is not the primary result because one round-robin run took 4.03 s. The modest median gain, despite the 46.3% modeled bucket reduction, shows that this peer was dominated mostly by shared link throughput rather than a per-connection ceiling.

Decision: keep LPT as a protocol-free tail-risk improvement, not as a claim of universal large speedup. A future adaptive work-stealing scheduler would handle heterogeneous connection rates better, but issuing files one at a time would discard the small-file pipelining benefit; it needs a batched-window ablation before replacing LPT.

Experiment 23: High-Bandwidth Compression Crossovers

Problem: --compress auto advertises LZ4 and Zstd, after which the current negotiation always selects Zstd-3 when both peers support it. That is a strong WAN default, but the send loop currently performs encode + write sequentially. On a modern fast link, saved bytes can cost more CPU time than sending raw data.

For a codec with encode rate $E$, size ratio $r$, and effective link rate $B$, the present sender cost per raw byte is:

$T = 1/E + r/B$

The codec/raw break-even is therefore $B = E(1-r)$. Two codecs cross where their respective $T$ values are equal. The updated compress_probe prints both quantities and unit-tests the equations instead of leaving the interpretation to hand calculation.

Method: release-mode 4 MiB codec probe on Apple Silicon, ten paired repetitions. Each repetition processes 256 MiB per algorithm and workload. Ratios were stable, while encode rates varied materially with host load; report the median derived crossovers rather than selecting a fast run.

The ten derived Zstd-3 -> LZ4 / LZ4 -> raw samples (MB/s), in run order, were:

• text-like: 315.0/426.7, 232.0/398.5, 590.2/249.6, 247.7/441.7, 298.2/391.2, 352.1/391.5, 270.2/444.6, 254.5/436.9, 146.6/209.2, 87.8/376.0;
• mixed: 73.2/289.4, 89.5/377.3, 150.1/378.4, 141.3/354.8, 143.5/381.5, 135.9/398.2, 120.1/375.3, 112.5/336.0, 100.8/323.7, 115.9/338.9.

These are sender cost-model crossovers, not end-to-end link benchmarks. SSH encryption, disk speed, receiver CPU and future pipeline overlap can move them. The earlier real-WAN workload in Experiment 12 was around 10 MB/s, far below every crossover, and correctly favored Zstd.

Negative result: changing auto globally from Zstd to LZ4 or raw would improve some 2.5/5/10 GbE paths but regress the slow, lossy and relayed environments that need compression most. A single compile-time bandwidth threshold is also invalid: encode rates moved enough across the ten runs to shift individual crossovers substantially.

Adaptive promotion gate:

1. sample 64--256 KiB from several file positions, not just an extension or the first block;
2. calibrate sender encode and receiver decode rates, then update actual link goodput with EWMA transfer windows;
3. compare raw, LZ4, Zstd-fast, Zstd-1 and Zstd-3 using a bounded memory/CPU pool;
4. require a 10--15% predicted margin plus hysteresis before switching, and re-sample only on a distribution change;
5. ablate 5/20/100 Mbps and 1/2.5/10 Gbps paths, old single-core servers, Cloudflare/Tailscale relays, and incompressible data.

Until that controller and the encode/network overlap pipeline are measured together, explicit --compress lz4 or --compress none remains the honest fast-LAN control.

Summary

Each row below states the specific workload behind the number. Don't lift this table out of context without the qualifiers.