AI Without Metrics Is Guesswork
If you cannot see prompt latency, token spend, and error rates, you cannot operate an AI feature. Treat model calls like any other dependency.
What to Log
Request IDs, model name, token counts, cache hits, retrieval scores, and user-visible failures. Redact PII before logs leave the trust boundary.
Budgets and Alerts
Set per-feature cost ceilings and p95 latency SLOs. Alert when spend spikes or when fallback rates climb — that is often a silent UX regression.
Feedback Loops
Capture thumbs-up/down and sampled transcripts into an evaluation dataset. Continuous improvement requires continuous measurement.