2025-06-13 –, D105 (capacity 300)
Large language models (LLMs) are pushing the boundaries of artificial intelligence, but their deployment is often hampered by memory bottlenecks arising from the ever-growing size of key-value (KV) caches. Traditional LLM serving systems struggle with inefficient memory utilization and limited scalability. Inspired by the concept of virtual memory paging in operating systems, PagedAttention offers a transformative solution. This novel technique partitions the KV cache into smaller, non-contiguous blocks, enabling dynamic allocation, efficient retrieval, and flexible reuse of memory. By decoupling the physical layout of the cache from the logical structure, PagedAttention minimizes memory fragmentation and overhead.
This approach, integrated within the vLLM framework, an open-source, high-performance LLM serving framework developed at UC Berkeley, yields significant performance gains.Designed to address memory bottlenecks in traditional LLM serving methods, vLLM leverages PagedAttention for efficient KV cache management, optimizing batch processing and eliminating redundant computations. As a result, PagedAttention achieves up to 30× higher throughput compared to traditional LLM serving methods like Hugging Face Transformers, Orca, and NVIDIA’s FasterTransformer. It also reduces KV cache waste to approximately 4%, ensuring near-optimal memory usage and enabling larger batch processing by minimizing memory overhead.
Furthermore, vLLM seamlessly supports advanced sampling techniques, including beam search, without compromising latency. While challenges such as the overhead of managing lookup tables and the potential for increased latency in certain scenarios exist, ongoing research is addressing these limitations. For example, optimized data structures and prefetching strategies can mitigate lookup overhead. Despite these challenges, PagedAttention represents a major advancement in LLM inference, unlocking the potential for scalable and efficient deployment, even on resource-constrained hardware. This breakthrough paves the way for wider adoption of LLMs and empowers researchers to explore even larger and more complex models.
Intermediate - attendees should be familiar with the subject
Rahul Belokar is a seasoned Senior Data Engineer at Red Hat, bringing over eight years of industry experience in software development, data engineering, and technical support. With a strong background in data-driven solutions, Rahul has played a crucial role in optimizing and enhancing data workflows, ensuring efficiency, scalability, and reliability in enterprise environments.
Rahul’s professional journey began in technical support, where he spent five years troubleshooting complex technical challenges and delivering high-quality solutions to customers. His problem-solving abilities, deep technical expertise, and customer-centric approach laid the foundation for his transition into software engineering and later into data engineering. Over the past three years, he has focused on data processing, automation, and analytics, leveraging his skills to transform raw data into meaningful insights.
I am Sagar Aivale, currently working as Senior Data Engineer with the UXE DATA Foundations team, previously a Senior Software Engineer for the Rules Acceleration and Automation team. With over 7+ years at RedHat, I enjoy collaborating with fellow Red Hat colleagues and learning about their work. Outside of work, I love spending time with friends and family.