LLM inference using vLLM on TPU and GKE DevConf.US 2026

LLM inference using vLLM on TPU and GKE
.ical
2026-09-25 10:20–10:35, Ladd Room (Capacity 170)

The explosive demand for Large Language Models has created a global GPU bottleneck, forcing platform engineers to seek alternative hardware for production inference. Enter the vLLM engine on Google Cloud TPUs. Traditionally optimized for GPUs, vLLM now brings its industry-leading PagedAttention and continuous batching capabilities to the TPU ecosystem, offering a high-throughput, cost-effective alternative for serving models like Llama-3.3-70B-Instruct.

In this session, we dive into the architecture of running vLLM on Google Kubernetes Engine (GKE). We will demonstrate the practicalities of deploying vLLM on TPU v6e slices and utilizing GKE for seamless orchestration.

What level of experience should the audience have to best understand your session?: Intermediate - attendees should be familiar with the subject

Tahmid Muttaki

Tahmid is a software engineer at RedHat, focusing on productizing one of the most popular open source llm inference engines, vLLM. His work involves setting up cicd infrastructures with many different hardware accelerators to build and test vLLM across a variety of platforms.

LLM inference using vLLM on TPU and GKE .ical 2026-09-25 10:20–10:35, Ladd Room (Capacity 170)

LLM inference using vLLM on TPU and GKE
.ical
2026-09-25 10:20–10:35, Ladd Room (Capacity 170)