In distributed inference, large language model workloads are split across multiple GPUs on separate nodes. When a GPU fails during inference, the serving system has to detect the failure, reconfigure, and resume, as quickly as possible. The vLLM community is actively building this capability through multiple approaches, from elastic scale-down to masked degraded operation to fault-tolerant collective backends. All of them depend on the same foundation: the ability to detect that a GPU is gone and to recover communication state without the dead participant.

We tested that foundation. Using NCCL 2.29's elastic primitives, we ran fault recovery scenarios across five nodes on an OpenShift cluster, a multi-node, TCP-only environment where vLLM will actually need to recover from failures. The primitives worked: shrinking a communication group after failures and running collective operations with survivors produced correct results across a variety of configurations, including simultaneous and sequential failures.

The more important finding was about failure detection. When we killed a pod on one node, the other nodes didn't notice; we found GPU communication sockets still in ESTABLISHED state minutes after the remote pod was destroyed. This is a gap between how Kubernetes tears down pods and how TCP detects connection loss, and it affects every approach to GPU fault tolerance on Kubernetes, not just the one we tested. We'll show the /proc/net/tcp evidence, walk through the diagnosis, and cover the fix options from per-socket keepalive to what platform-level changes would eliminate the problem entirely.

This talk covers what we learned testing GPU fault recovery in the conditions it will actually face: real failures, across real nodes, on a real Kubernetes cluster. We'll discuss how these findings apply to vLLM's fault tolerance roadmap and what anyone running distributed GPU workloads on Kubernetes should know.