GenAI Observability: Standardizing LLM Serving Telemetry with OpenTelemetry

Operating LLM ServingRuntimes (vLLM, KServe) on Kubernetes and OpenShift clusters without standardized telemetry leads to blind spots during traffic spikes.

The stabilization of OpenTelemetry Semantic Conventions for Generative AI establishes a vendor-neutral standard to monitor, trace, and optimize model serving performance in production.

---

1. The Limits of Traditional Application Metrics on LLMs

For standard HTTP services, overall request latency is typically sufficient. In streaming LLM architectures, aggregate duration hides critical bottlenecks:

• Time to First Token (TTFT): Measures the prompt prefill duration and scheduling latency. TTFT spikes indicate GPU compute saturation or excessive queuing.
• Inter-Token Latency (ITL / Time per Output Token): Measures token-by-token decode speed, directly tied to GPU VRAM memory bandwidth.
• Token Usage & Quota Tracking: Granular accounting of prompt and completion tokens is essential for multi-tenant rate limiting and OOM prevention.

---

2. OpenTelemetry GenAI Standard in Production

Integrating these conventions into cloud-native observability pipelines (Prometheus, Vector, OpenShift Observability):

• Standardized Metrics: gen_ai.server.time_to_first_token, gen_ai.server.inter_token_latency, gen_ai.client.token.usage.
• Unified Distributed Tracing: Seamless span correlation across prompt prefill, vector database retrieval (RAG), and client application queries.
• SLO-Driven Autoscaling: Feeding OpenTelemetry SLIs into KEDA and Horizontal Pod Autoscalers to scale GPU workers proactively before queue saturation.

---

3. Production Skills in Aperta Scientia Curriculums

• AS300 (AI Platform Engineer): Hands-on instrumentation of vLLM and KServe runtimes, Grafana dashboarding, and correlation with bare-metal NVIDIA GPU metrics.
• AS200 (DevOps Platform Engineer): Full observability stack integration (Prometheus, Thanos, Grafana) and SLO-based automated alerting.

---

4. Technical Sources & References

• OpenTelemetry GenAI Semantic Conventions: OpenTelemetry Official Specification
• vLLM Metrics & Prometheus Integration: vLLM Production Metrics
• KServe Observability & Telemetry: KServe Telemetry Guide