The symptom that made no sense
The first time we saw it, we didn’t trust the dashboard. A distributed training job was scheduled and healthy — every pod was running, no crashes, no OOMKills, nothing in the logs. And yet more than half the GPUs we were paying for sat idle, and training never actually started. Every health check was green, and nothing was computing.
A metaphor that finally made it click
We kept reaching for a way to explain it, and this is the one that stuck. Imagine a sold-out concert hall. Every musician is in their seat, instruments tuned, the hall lit. But the conductor was shown to the wrong wing of the building — and the fire-safety doors, doing exactly their job, sealed that wing off. The result is a full, expensive hall and complete silence.
That is a GPU cluster in this failure mode. The conductor is the training coordinator. The musicians are the GPU workers, who can only play in sync, through the conductor — the same way distributed training synchronizes gradients through a coordinator. And the fire doors are the network policy: a correct, intentional safety rule that nobody told the scheduler about.
The root cause: two correct systems, collectively wrong
Kubernetes scheduling is topology-agnostic by design. It places pods based on available resources — CPU, memory, GPU count — without reasoning about which availability zone a pod lands in. Kubeflow inherits that assumption.
Cilium, on the other hand, is topology-aware. Operators use CiliumNetworkPolicy to draw zone boundaries for blast-radius isolation, compliance, cost control, or to protect an expensive dedicated GPU pool. That’s good security hygiene — and Cilium can’t reschedule a pod that has already been placed.
Put those two together and you get a class of problem where each system makes an individually correct decision, and together they’re wrong. The coordinator lands in one zone, the GPU workers in another, and the network silently blocks the connection between them. No one holds the single map that shows both the pod placement and the locked doors at once.
It’s a spectrum, not a single bug
What surprised us most is that the same root cause shows up three different ways in production:
- Hard block. A zone-boundary network policy denies the connection outright. Workers can’t reach the coordinator and training never starts. You notice in seconds.
- Cross-zone latency. With no hard block, just distance, gradient synchronization (NCCL AllReduce) pays cross-zone round-trip time on every step. Throughput quietly drops 30–60% with no error at all. You notice in hours, if you’re paying attention.
- Cross-AZ egress cost. The traffic crosses availability zones and shows up only on the cloud bill as inter-AZ data transfer. You notice in days, on the invoice.
The demo we built reproduces the first case because it fails cleanly and visibly. But the second and third are the ones that quietly drain budgets in real clusters.
The fix
The surprise is how little you have to change. We didn’t touch Cilium — the locked doors stay locked, because the security policy was never the problem. And we didn’t patch Kubernetes or Kubeflow. We simply gave the scheduler the one piece of information it was missing: keep the whole training group together, in a zone whose network path is open.
In practice that’s a few lines of Kubernetes-native YAML on the workload spec: nodeAffinity to pin the group to the GPU zone, topologySpreadConstraints to co-locate the coordinator and workers, and a toleration so the coordinator can actually land on the GPU-tainted nodes. Once every communicating pod shares a zone, the traffic is intra-zone — the hard block disappears, the latency disappears, and the cross-AZ egress disappears. One fix, all three symptoms.
If you don’t want to hard-code a zone name, podAffinity with a zone topology key achieves the same thing by relationship: place the workers in whatever zone the coordinator landed in. Same idea, portable across clusters.
In our lab, GPU utilization went from around 40% to around 85% the moment the group was co-located.
What we took away
Your Container Network Interface (CNI) has opinions about topology that your scheduler can’t see. Topology-aware network policies are silent to the Kubernetes scheduler — and that silence is where the failure lives.
The fix is Kubernetes-native. No CNI change, no framework patch — just topology spread constraints and affinity in the workload spec.
Instrument before you need it. GPU utilization and pod-zone metrics in Prometheus and Grafana exposed this for us in seconds; without them it can take days.
The pattern generalizes. Any topology-aware CNI plus any distributed ML framework can hit the same wall. It isn’t specific to Cilium or Kubeflow.
Try it yourself
We packaged the whole thing as a reproducible lab — a kind cluster with GPU and CPU zones, the Cilium policy, Prometheus recording rules, a Grafana dashboard, and before/after demo scripts:
Repo: https://github.com/ram2valar/kubeflow-cilium-lab
Talk recording (KubeCon + CloudNativeCon India 2026): https://www.youtube.com/watch?v=BG9XGouyM9c
Spin it up, watch the GPUs go idle, apply the fix, and watch them come back.
Facts Only
* A distributed training job was scheduled and healthy, with pods running and no reported errors in logs.
* More than half of paid GPUs sat idle, and training did not start.
* The failure mode involves a mismatch between Kubernetes scheduling (topology-agnostic) and Cilium network policies (topology-aware).
* In one scenario, a zone-boundary network policy outright denies connections, preventing training startup.
* In another scenario, cross-zone communication results in latency during gradient synchronization, causing throughput drops of 30–60%.
* A third scenario results in cross-AZ data transfer appearing as inter-AZ egress costs on cloud bills.
* The proposed fix involves using Kubernetes features like nodeAffinity and topologySpreadConstraints in the workload specification to ensure co-location.
* Co-location resulted in GPU utilization increasing from approximately 40% to around 85%.
* The root cause is identified as two correct systems making a collectively wrong decision regarding topology awareness.
Executive Summary
A distributed training job could appear healthy on a dashboard while failing to execute due to misalignment between system components. This failure mode is illustrated by an analogy where a scheduler (conductor) and network policies (fire doors) operate correctly in isolation but create a systemic blockage. The core issue stems from the decoupling of topology awareness: Kubernetes scheduling is topology-agnostic, while tools like Cilium implement topology-aware networking rules that can restrict communication between pods across different zones. This creates a situation where separate, correct decisions lead to an incorrect collective outcome.
The failure manifests in three distinct ways depending on what is measured: hard network blocks prevent any training from starting, cross-zone latency causes silent throughput degradation during synchronization, and cross-AZ egress results in unexpected cloud billing costs. The proposed solution bypasses framework patches by addressing the scheduler's missing context directly through Kubernetes-native workload specifications, specifically using node affinity and topology spread constraints to ensure all related components reside within a connected zone.
Full Take
The narrative exposes a critical gap between the operational assumptions embedded in orchestration layers and the topological realities enforced by networking infrastructure. The system failure does not originate from a single buggy component but from an emergent property of decoupled, independently correct subsystems—the scheduler operating on resource constraints and the CNI enforcing security boundaries based on spatial awareness. This points to a systemic vulnerability where topology-aware controls, designed for granular security and cost management (Cilium), operate outside the visibility space of the high-level orchestrator (Kubernetes scheduling). The resulting failure is not merely an operational bug but a manifestation of unstated, context-dependent dependencies that cascade into economic and performance consequences.
The implication is that abstracting away physical topology in favor of purely logical placement risks creating silent failures that are difficult to diagnose because the error lies in the assumed environmental constraints rather than an explicit configuration error. The proposed solution reinforces agency by pushing the fix back into the domain where context is explicitly defined—the workload specification—demonstrating that true resilience requires binding disparate layers through shared, explicit topology knowledge rather than relying on implicit assumptions across independent technology stacks. What are the systemic implications for trust in abstract infrastructure layers when topological awareness is delegated externally? How can systems design mandate a unified view of topology to prevent these emergent, silent failures from becoming the default operational state?
Sentinel — Human
LIKELY_HUMAN (confidence: 0.15)