BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//pretalx//pretalx.devconf.info//devconf-us-2026//talk//B77P98
BEGIN:VTIMEZONE
TZID:EST
BEGIN:STANDARD
DTSTART:20001029T030000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10;UNTIL=20061029T070000Z
TZNAME:EST
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
END:STANDARD
BEGIN:STANDARD
DTSTART:20071104T030000
RRULE:FREQ=YEARLY;BYDAY=1SU;BYMONTH=11
TZNAME:EST
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:20000402T030000
RRULE:FREQ=YEARLY;BYDAY=1SU;BYMONTH=4;UNTIL=20060402T080000Z
TZNAME:EDT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
END:DAYLIGHT
BEGIN:DAYLIGHT
DTSTART:20070311T030000
RRULE:FREQ=YEARLY;BYDAY=2SU;BYMONTH=3
TZNAME:EDT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
UID:pretalx-devconf-us-2026-B77P98@pretalx.devconf.info
DTSTART;TZID=EST:20260925T135000
DTEND;TZID=EST:20260925T142500
DESCRIPTION:In distributed inference\, large language model workloads are s
 plit across multiple GPUs on separate nodes. When a GPU fails during infer
 ence\, the serving system has to detect the failure\, reconfigure\, and re
 sume\, as quickly as possible. The vLLM community is actively building thi
 s capability through multiple approaches\, from elastic scale-down to mask
 ed degraded operation to fault-tolerant collective backends. All of them d
 epend on the same foundation: the ability to detect that a GPU is gone and
  to recover communication state without the dead participant.\n\n  We test
 ed that foundation. Using NCCL 2.29's elastic primitives\, we ran fault re
 covery scenarios across five nodes on an OpenShift cluster\, a multi-node\
 , TCP-only environment where vLLM will actually need to recover from failu
 res. The primitives worked: shrinking a communication group after failures
  and running collective operations with survivors produced correct results
  across a variety of configurations\, including simultaneous and sequentia
 l failures.\n\n The more important finding was about failure detection. Wh
 en we killed a pod on one node\, the other nodes didn't notice\; we found 
 GPU communication sockets still in ESTABLISHED state minutes after the rem
 ote pod was destroyed. This is a gap between how Kubernetes tears down pod
 s and how TCP detects connection loss\, and it affects every approach to G
 PU fault tolerance on Kubernetes\, not just the one we tested. We'll show 
 the /proc/net/tcp evidence\, walk through the diagnosis\, and cover the fi
 x options from per-socket keepalive to what platform-level changes would e
 liminate the problem entirely.\n\n  This talk covers what we learned testi
 ng GPU fault recovery in the conditions it will actually face: real failur
 es\, across real nodes\, on a real Kubernetes cluster. We'll discuss how t
 hese findings apply to vLLM's fault tolerance roadmap and what anyone runn
 ing distributed GPU workloads on Kubernetes should know.
DTSTAMP:20260727T174705Z
LOCATION:Ladd Room (Capacity 170)
SUMMARY:When GPUs Die Mid-Inference: Cross-Node Fault Tolerance for vLLM Se
 rving on Kubernetes - Sean McGovern
URL:https://pretalx.devconf.info/devconf-us-2026/talk/B77P98/
END:VEVENT
END:VCALENDAR