What if you could combine the speed of a small model with the quality of a large one? Speculative decoding makes this possible by using a lightweight draft model to propose tokens that a larger verifier checks in parallel, delivering 2-5x inference speedups with mathematically identical quality output.

This talk introduces the vLLM project's Speculators library: an end-to-end framework for training, packaging, and deploying speculative decoding models. We will start with the intuition behind speculative decoding, walk through the full Speculators training pipeline, and finish with a live demonstration of serving a speculative decoding model to speed up inference.

Along the way, we will dive into the engineering challenges we solved to make this work at scale. Extracting training data from large language models generates hundreds of megabytes per request, which need to be transferred out of vLLM. Therefore we built a novel hidden states extraction system that repurposes vLLM's KV Connector infrastructure to efficiently stream internal model representations to the training processes, all while preserving vLLM's tensor parallelism and paged attention optimizations. We will share what we learned scaling this across multi-GPU setups and across speculative decoding algorithms like Eagle-3 and DFlash.

Attendees will leave with a practical understanding of how speculative decoding works, how to train their own draft models using Speculators, and how to deploy them for accelerated inference in vLLM.