Red Hat AI Inference maintains its own version of vLLM with various hardware support, container packaging, and bug fixing on top of the open-source code. Between each release we need to ensure the model performs at a baseline performance of the open-source vLLM version. This project covers automated comparison between the upstream and packaged versions, catching regressions before the release goes into production.