Large language model deployments commonly serve latency-sensitive online requests alongside latency-insensitive workloads such as document summarization, classification, embedding generation, and model evaluation. Running both workload types without proper coordination can cause batch traffic to consume inference capacity and negatively impact interactive users.
This lightning talk introduces llm-d-async, an asynchronous processing component for llm-d. It explains how requests are placed in queues, processed through worker pools, controlled by dispatch gates, and forwarded to the inference gateway. The talk will show how llm-d-async enables batch workloads to use available inference capacity while protecting interactive traffic.