We’ve used Proxmox VE for years, and it’s fair to say we’re rather fond of it. It’s a complete open-source platform for managing server virtualization, and its enterprise-grade features give us the flexibility we rely on for day-to-day operations. Its granularity allows us to drill down right into any issue when one arises.
On 5 September 2026, we were particularly glad of that visibility because every node in one of our Proxmox clusters unexpectedly rebooted at roughly the same time.
Yep, you read that right. The entire cluster went down.
This was a cluster built specifically to withstand losing an individual node, so we needed to understand what had happened.
Our immediate priority was to restore service. We disabled High Availability (HA) and brought the hosted services back online in a few hours, then continued investigating what caused the incident.
Finding out why took us through Proxmox HA, Corosync, KNET, Linux networking and eventually the receive queues on a 25 GbE network interface. Along the way, we found several things that looked like possible causes, and quite a few of them were genuine problems.
What eventually became clear was how one network event had worked its way through several layers of the system until cluster communication itself was affected.
It also exposed a weakness in how we had separated cluster communication from customer traffic, and we’re now changing the design as a result.
We think the investigation and its conclusion are worth sharing, both because of what happened and because of what it showed us about the way failures can propagate through a high-availability environment.
Why did the whole cluster reboot?
Each physical server is a node acting as a hypervisor, supplying the processor time, memory and network access its virtual machines need. Several nodes working together form a cluster.
Our first suspicion was high availability (HA) and fencing.
HA watches the nodes and manages recovery when hardware or services fail. Before the cluster can act, though, its members have to agree on who is still present.
They do that through quorum. A majority of votes is required before a group of nodes is allowed to make decisions, which prevents two disconnected parts of a cluster from both believing they are in charge.
Fencing deals with a node that can no longer participate reliably. On an HA node with an active watchdog, losing quorum for too long can cause the watchdog to expire and force a reboot. This prevents the node from continuing to run a virtual machine that another node may be about to start elsewhere.
One node rebooting itself under those conditions can therefore be the system working exactly as designed.
Normally, we would expect a failure to look something like this:
Healthy cluster
|
+-- One node becomes unhealthy
|
+-- The affected node is excluded or fenced
|
`-- The remaining cluster keeps quorum
and continues operatingOur cluster had ample redundancy to lose the affected hypervisor. In this case, however, every node rebooted, which meant the failure clearly did not look like one node simply disappearing.
Corosync was behaving strangely
The next place we looked was Corosync. Corosync is how the nodes exchange status information and agree on cluster membership, while KNET carries those messages and monitors connectivity between them.
The logs showed unstable Corosync communication around one particular node. KNET was reporting connectivity failures, and different cluster members did not always appear to agree about who could communicate with whom.
We were seeing both packet loss, where some network traffic never arrives, and asymmetric connectivity, where one node can hear another but communication does not necessarily work the same way in the opposite direction.
Cluster software finds this sort of partial failure much harder to handle than a clean disconnection. If a server disappears completely, the situation is fairly clear. If it remains present and continues exchanging some traffic while other messages are lost, different members of the cluster can end up with different pictures of what is happening.
And this is where we disappeared down the rabbit hole.
We couldn’t see the forest for all the trees
From there, the investigation moved out of the cluster software and further into Linux network processing.
We worked through Corosync and KNET, then connection tracking, interrupt distribution, NUMA topology, softirqs and the way the network card was distributing incoming traffic.
The frustrating part was that every layer gave us something interesting to look at and consider as a possible root cause.
We could reproduce significant packet loss between cluster members. The network card was reporting large numbers of RX misses and receive-ring discards. Increasing the receive ring size helped, but it did not make the problem disappear.
During one test, a single receive queue discarded more than 500,000 packets in ten seconds.
At that point we knew where packets were disappearing, but we still didn’t understand what was putting so much pressure on the interface.
Eventually we looked at the switch port graphs.
Two figures immediately stood out:
- Around 22 Gbit/s of inbound traffic on an interface rated for 25 GbE.
- One workload on the hypervisor receiving close to two million packets per second at times.
Oh.
Quite a few of the things we had been seeing started to make more sense.
Ironically, the network was fine
There was a slightly amusing reason this wasn’t immediately obvious: the rest of our network didn’t particularly care.
The traffic crossed our switches and routers without causing a wider network event. The network simply delivered it to the destination, exactly as it was designed to do.
The difficulty was at the destination.
The hypervisor’s 25 GbE interface and host networking stack were being pushed hard enough that packets started being discarded. Corosync communication was using the same physical interface and some of the same underlying resources, so cluster traffic was being affected too.
Once the problematic traffic was removed, the relationship between the load on the interface and the Corosync instability became much easier to see.
The chain of events was the important part
A network interface struggling under very heavy traffic is not, on its own, a particularly surprising technical finding.
What mattered to us was how the problem had travelled through the system:
Extreme traffic to one workload
↓
Hypervisor interface comes under heavy load
↓
Packets begin to drop
↓
Corosync communication degrades
↓
Cluster members lose a consistent view
of each other
↓
HA / watchdog behaviour follows
↓
Every cluster node rebootsWe are still working through the exact HA and watchdog sequence that resulted in every node rebooting, so we do not want to claim more certainty about that final part than we have.
By that point, though, we already knew enough to identify the shared dependency underneath it. Traffic affecting a single customer virtual machine had been able to interfere with the communication the cluster relied on to understand its own state.
For us, that was the important realization from the incident. A problem on the customer-facing network had been allowed to reach the cluster control plane.
Why the VLAN could not protect Corosync
Customer traffic and cluster traffic were already separated using virtual local area networks (VLANs).
From a network configuration point of view, the two types of traffic were separate. Physically, however, they still depended on the same network interface, the same receive queues and some of the same processing capacity.
The incident made the difference between logical separation and failure-domain separation very clear.
A VLAN can keep two types of traffic logically distinct while they share equipment. But if the physical interface underneath both VLANs starts dropping packets, the VLAN itself cannot protect Corosync.
We had separated the traffic, but we had not completely separated the ways in which the two networks could fail.
That is what we’re changing.
Our planned system improvements
We’re implementing several changes as a result of the incident.
Corosync is getting its own physical networks
The most important change is that we’re moving Corosync onto separate physical networks, with dedicated interfaces and separate network infrastructure.
The design will also include fallback Corosync networks, giving the cluster another independent communication path if the primary one fails.
Corosync does not need enormous amounts of bandwidth. What it needs is reliable packet delivery, predictable latency and a path that is not affected by whatever happens on the customer-facing network.
The point of the change is therefore not simply to add more capacity, but to remove the shared dependency.
Customer traffic
|
`-- Customer-facing network
Cluster communication
|
+-- Dedicated Corosync network
|
`-- Independent fallback pathIf a customer-facing interface is saturated, Corosync should continue communicating over infrastructure that is not affected by it. If the primary Corosync network fails, the fallback network should allow the cluster to keep talking.
That is the failure model we want.
The alerts were there, but we missed the connection
There is another, slightly humbling part of the story.
Our monitoring caught the unusual traffic and raised alerts at the time. During an active incident involving cluster reboots, Corosync errors, connectivity problems and packet loss, however, there was a lot competing for attention.
We naturally focused on the most serious symptom and started trying to understand why an entire HA cluster had just rebooted.
Looking back, the network alerts were already pointing us towards an important part of the problem.
The answer is not simply to generate more alerts. When an incident is already noisy, more independent alerts can make it harder to understand which events are actually connected.
We’re therefore changing how we correlate and prioritize the information we already collect. If a hypervisor suddenly sees extreme traffic at the same time that Corosync starts reporting connectivity problems, those events should reach the engineer as related signals rather than as two separate alerts competing for attention.
The monitoring had seen the trees. We just hadn’t seen the forest yet.
We’re testing the ugly failures
The incident also reminded us that infrastructure failures do not always present themselves in obvious ways. Sometimes failures are weird, unpredictable and messy. Just like people sometimes are.
It is relatively straightforward to test what happens when a server loses power or someone unplugs a network cable. In those cases, the failure is clear, and you can verify that the cluster responds the way you expect.
What happened here was much less tidy. A link can stay up while quietly dropping packets, communication can become asymmetric, and a host can be overloaded without ever actually going offline.
Those conditions are harder to read, but they are also exactly the sort of conditions a high-availability design needs to cope with.
So we’ll be deliberately testing scenarios involving:
- Packet loss while network links remain connected
- Asymmetric connectivity between cluster members
- Saturation of customer-facing interfaces
- Failure of the primary Corosync network
- Fallback between independent Corosync paths
High availability shouldn’t only work when the failure is a server going offline or a cable being unplugged. Heavy traffic affecting one workload should stay contained, and the cluster should continue communicating even when a failure is partial, asymmetric or difficult to read.
Why we’re sharing the investigation
We could have quietly fixed the problem and moved on, or published the usual summary saying that an infrastructure incident occurred, the cause was identified and corrective measures were being implemented.
That wouldn’t tell anyone very much.
Running infrastructure means dealing with hardware that fails and software that occasionally surprises you. Now and then, systems each behave sensibly on their own but still combine into something nobody predicted.
What matters afterwards is understanding what happened, finding the weakness it exposed and changing the design so that the same sequence cannot happen again.
For us, the principle coming out of this incident is fairly simple:
Customer-plane failure should not become control-plane failure.
We’re changing the architecture accordingly.
And yes, we’re still rather fond of Proxmox
The investigation also reinforced one of the reasons we’ve used Proxmox for so long.
Its granularity allowed us to start with one fairly simple question, why did the entire cluster reboot, and keep drilling down through Proxmox HA, Corosync and KNET, into Linux networking and eventually all the way to individual receive queues on the network interface.
That kind of visibility is valuable when you’re responsible for infrastructure that other businesses depend on.
We learned something useful about our own architecture, and it’s going straight into how we build, test and improve our clusters.
There is also one considerably simpler lesson from the whole experience.
If every node in a Proxmox cluster has mysteriously rebooted, and you’ve spent several hours investigating Corosync, KNET, NUMA topology, Linux softirqs and NIC receive queues…
maybe look at the switch port graph.
We certainly will next time. 🙂
Technical Team at ServeTheWorld