The explosive demand for Large Language Models has created a global GPU bottleneck, forcing platform engineers to seek alternative hardware for production inference. Enter the vLLM engine on Google Cloud TPUs. Traditionally optimized for GPUs, vLLM now brings its industry-leading PagedAttention and continuous batching capabilities to the TPU ecosystem, offering a high-throughput, cost-effective alternative for serving models like Llama-3.3-70B-Instruct.

In this session, we dive into the architecture of running vLLM on Google Kubernetes Engine (GKE). We will demonstrate the practicalities of deploying vLLM on TPU v6e slices and utilizing GKE for seamless orchestration.