90% of GenAI systems fail in production because teams struggle to evaluate and resolve failures. Observability tools like OpenTelemetry and MLflow log massive traces, but hide critical reasoning failures. They tell you where a request traveled, but not why an LLM agent lost its way. This session moves beyond basic observability into the realm of failure mode analysis for agentic systems. Using a real world AI system as a case study, we will dissect why traditional monitoring fails to catch silent agent failures like context window saturation and tool-calling loops.

Attendees will walk away with a practical toolkit for:
1) Defining a Failure Taxonomy: Moving from vague bad outputs to structured categories like retrieval gaps or reasoning drift
2) Building a Tracing-to-Resolution Pipeline: Leveraging OpenTelemetry and MLflow traces to map anomalies directly to code-level fixes, eliminating the guesswork of manual prompt engineering
3) Human-in-the-Loop Evaluations: Scaling quality control by using LLMs to surface high-risk anomalies for critical human review

Learn to stop operating blind and build resilient, production-grade AI systems that your teams can actually trust.