Your alert fires at 3am. An SRE investigates, finds the root cause, applies a known fix, and goes back to sleep. Next week, the same alert fires again. Same investigation. Same fix. Same person woken up.

What if the cluster could do this itself — investigate the alert, reason about the root cause, select the right remediation, execute it, and produce an audit trail that proves it made the right call?

This talk explores the engineering challenges of building closed-loop Kubernetes self-healing with Agents as the reasoning layer. We'll dig into the hard problems:

• Trust boundaries: How do you let an Agent investigate a production cluster without exposing secrets? How do you scope RBAC so the AI agent can read what it needs and nothing more?
• Auditability: How do you make "AI decided to scale your HPA" auditable enough for SOC2? What does an immutable, tamper-evident audit trail look like for autonomous remediation?
• Governance: How do you keep a human in the loop without turning it into another page? When should the AI just fix it, and when should it ask?
• Fleet-scale credentials: How do you architect ephemeral, zero-trust credentials when the same pipeline needs to remediate across a fleet of clusters using ACM and AAP?

Live demo

We'll trigger a real alert on a Kind cluster and watch the full pipeline execute without human intervention:

1. Signal → AlertManager fires
2. Investigation → an AI Agent queries the Kubernetes API, reads logs, describes pods, builds a root cause analysis
3. Decision → an AI Agent selects the right remediation workflow based on the RCA
4. Execution → K8s Jobs/Tekton/Ansible applies the fix
5. Audit → Immutable event records what happened and why

No slides-only theory. Bring your skepticism.