Large Language Models are increasingly constrained by inference architecture rather than training. As workloads diversify, traditional monolithic serving stacks struggle with mixed latency targets, long context prompts, GPU memory pressure, and inefficient batching. This talk presents two emerging patterns treating LLM inference as a distributed systems challenge.

Dynamic Prefill Decode (PD) separates the compute-intensive prefill phase from the memory-bound decode phase, scaling each across specialized GPU pools to improve utilization and reduce tail latency.
Semantic Routing replaces static load balancing with inference-aware decisions, selecting the right model and infrastructure tier per request.

We explore tools like vLLM, Ray Serve, KServe, and TensorRT-LLM.
The session includes a live demo showing semantic routing, model selection, latency differences, and cost tradeoffs across multiple models.