Optimal Cluster Network Architecture
Corosync runs on an isolated network with redundant links and healthy latency margins.
ā” Failover & Join Storm Analysis ($O(N^2)$ Burst)
Simulates the instantaneous UDP packet microburst triggered when all 5 nodes simultaneously broadcast memb_join messages during a node failover or ring reform.
Steady-State Traffic Breakdown
Derived Protocol Timers & Timeouts
Mathematical Derivation
Scalability Curves Across Node Count
Bandwidth (TX per Node & Cluster)
Packet Rate & Effective Token Timeout
Reference Metric Matrix (N = 2 to 64 Nodes)
Membership Transition Peak Load (Join / Re-formation)
When a node fails or joins, the ring transitions from steady-state to active re-formation:
every node floods all peers every 50 ms with memb_join messages until consensus is achieved. This represents the absolute maximum peak load and is the only term scaling quadratically $O(N^2)$.
Architecture Recommendations & Risk Mitigation
Protocol Mechanics & Architectural Deep Dive
Large Cluster Scaling Dynamics (>30ā40 Nodes): Root Cause Analysis
Structural Limitations of Totem SRP & kronosnet at Scale:
Beyond 30 to 40 nodes, Corosync clusters encounter severe structural failure modes driven by protocol physics, kernel socket buffering, and CPU scheduling delays rather than baseline bandwidth:
Beyond 30 to 40 nodes, Corosync clusters encounter severe structural failure modes driven by protocol physics, kernel socket buffering, and CPU scheduling delays rather than baseline bandwidth:
- Quadratic Join Flooding ($O(N^2)$ Burst): During cluster re-formation, all $N$ nodes broadcast
memb_joinmessages to all $N-1$ peers every 50 ms. At $N=40$, this generates 31,200 packets/sec; at $N=64$, it exceeds 80,000 packets/sec. These microbursts overwhelm kernel UDP socket buffers (net.core.rmem_max) → silent packet loss → perpetual ring re-formation loops ("Ring Flapping") → cascading node fencing. - Cumulative Token Ring Pass Delay ($O(N)$): The token passes sequentially node-by-node. If a single node experiences a minor CPU stall (e.g. ZFS ARC reclamation, FUSE lock contention in
pmxcfs), token rotation halts for the entire cluster. At 40 nodes, a 100 ms stall on Node 17 triggers a token timeout on Node 18. - Kernel Socket Buffer Saturation: The Linux default UDP socket buffer (
rmem_default= 212 KB) holds only ~140 MTU-sized packets. Simultaneous join responses from 40 nodes cause instant buffer overflows.
Recommended Hardening Steps for Large Clusters (>32 Nodes):
- Configure
send_join: 80incorosync.conf(adds randomized staggering to prevent packet avalanches). - Elevate Linux socket buffers:
net.core.rmem_max = 16777216(16 MB). - Set
retransmits_before_loss: 10(increases tolerance before triggering fencing). - Architectural Recommendation: Partition large monolithic clusters into smaller federated clusters (≤ 16ā20 nodes) managed via Proxmox Datacenter Manager (PDM).
VM Live Migration & Storage Replication vs. Corosync Traffic
Fundamental Rule: Corosync never transfers VM RAM state or virtual disk data!
A common misconception in cluster design is confusing Corosync traffic with workload traffic:
- Corosync 3 & pmxcfs: Transmits cluster state, configuration files (
/etc/pve/), and quorum heartbeats. Traffic is lightweight (< 1 Mbit/s) but hyper-sensitive to latency and network jitter. - VM Live Migration: Streams virtual machine memory (RAM) directly from host to host over QEMU TCP sockets. This creates bursts of 10 to 100 Gbit/s. If unthrottled live migrations share a physical link with Corosync, bufferbloat and jitter spikes (> 5ā10 ms) cause Corosync packet loss and unintended node fencing.
- Storage Replication (pvesr / ZFS): Periodically syncs ZFS datasets via SSH, creating similar saturation risks on shared links.
Operational Churn Impact (VM Provisioning, Restarts, HA)
The Proxmox Cluster File System (pmxcfs) uses Corosync CPG (Closed Process Group) to keep /etc/pve/ synchronized across all nodes:
- Baseline Polling (pvestatd): Broadcasts Node, VM, Container, and Storage telemetry every 10 seconds.
- VM Lifecycle Operations: Creating, cloning, or modifying a VM writes a new
.conffile, triggering CPG broadcast bursts to all nodes. - HA State Locking: The HA CRM/LRM managers perform lock acquisitions in
/etc/pve/ha/. Node failovers generate intensive CPG lock contention.
Packet Structure & Encryption Overhead (Code Verification)
| Structure | Size | Field Composition | Source |
|---|---|---|---|
totem_message_header | 13 B | magic 2, version 1, type 1, encapsulated 1, nodeid 4, target_nodeid 4 | totem.h |
orf_token | 57 + 16Ā·r B | hdr 13, seq 4, token_seq 4, aru 4, aru_addr 4, ring_id 12, backlog 4, fcc 4, retrans 4, rtr 4 | totemsrp.c |
mcast (Data Header) | 45 B | hdr 13, system_from 4, seq 4, this_seqno 4, ring_id 12, node_id 4, guarantee 4 | totemsrp.c |
memb_join | 33 + 4Ā·N B | hdr 13, system_from 4, proc_entries 4, failed_entries 4, ring_seq 8 + 4 B/Node | totemsrp.c |
| knet PING / PONG | 26 B | link 1, timespec 16, seq 2, timed 1 (+ 6 Fixheader) | onwire_v1.c |
Proxmox default encryption (aes256-CBC + sha256) adds 49 to 64 Bytes of crypto overhead per packet.
Totem Configuration Formulas (exec/totemconfig.c)
token_timeout += (member_count - 2) * token_coefficient // applies when member_count > 2
token_retransmit = (int)(token_timeout / (retransmits_before_loss + 0.2))
hold = (int)(token_retransmit * 0.8 - 1000/HZ) // HZ=100 → -10 ms
consensus = (int)(1.2 * token_timeout)
knet_ping_timeout = token_timeout / knet_pong_count
knet_ping_interval = token_timeout / (knet_pong_count * 2)
Starting with Proxmox VE 9.1+, new clusters default to token_coefficient: 125 (down from 650) to prevent excessive timeout scaling in larger clusters.