Are you deploying Large Language Models and struggling to squeeze every ounce of performance out of your costly GPU infrastructure? While engines like vLLM are industry standards, using their default configurations for your unique complex workload shape - involving long prefill, extreme sequence lengths, or high concurrency - can often leave performance on the table. Traditionally, tuning these parameters meant relying on manual tuning or black-box Bayesian Optimization. But what if the tuning could reason about why a configuration failed?

This session introduces the next evolution of llm-tuna, an open-source framework for automated LLM serving optimization. Building on our research published in the ACM (link), we introduce an agentic feedback loop where an AI agent manages a persistent Performance Knowledge Base. Instead of just navigating a parameter space, the agent proactively:

1. Reads and Interprets: Analyzes vLLM logs and engine metrics to identify memory pressure or compute bottlenecks.
2. Profiles and Diagnoses: Triggers low-level PyTorch profiling and kernel-level analysis to understand why certain batch sizes or CUDA graph settings underperform.
3. Learns and Adapts: Grows its knowledge base with every experiment, allowing it to bypass inefficient search paths and suggest optimizations that pure statistical models might miss and accelerate the search for newer users of the framework.

We will showcase the architecture—leveraging a kubernetes cluster for distributed, isolated trials—and demonstrate how this "Expert-in-the-Loop" approach outperforms standard baselines. This study will show how the progression improves over traditional statistical approaches. Attendees will see real-world case studies on models like Qwen2.5 and the massive DeepSeek-R1-671B on H100/H200 clusters.

Key Takeaways:

• The Agentic Edge: How a reasoning agent identifies bottlenecks that Bayesian Optimization ignores.
• Closing the Loop: Integrating PyTorch profiling and log analysis into the automated tuning cycle.
• Building a Performance Moat: Leveraging a persistent, research-backed knowledge base to accelerate deployments across diverse GPU architectures (H100/H200).
• Multi-Objective Optimization: Balancing the trade-offs between prefill latency, decode throughput, and VRAM efficiency.