Commercial LLM-based assistants such as ChatGPT or Claude can call tools, query data, and power agents out of the box. In enterprise settings, these products are delivered through managed external services, creating vendor lock-in and data governance risks. Small self-hosted language models offer a compelling alternative, but require a different approach to build reliable AI assistants.

In this session, we present the design and implementation of an AI assistant powering Red Hat AIPCC Productization’s Release Dashboard. The assistant queries internal data, calls internal tools, and synthesizes answers without relying on external LLM providers. We show how decomposing assistant behavior into a four-layer architecture of action classification, tool selection, argument generation, and answer synthesis makes small, self-hosted language models viable for building reliable AI assistants. We conclude by discussing the metrics and feedback signals used to observe and improve the system.