How is LLM observability different from traditional APM?

Traditional APM assumes deterministic code - the same input yields the same output, and correctness is binary. LLM calls are probabilistic, quality is a spectrum, costs vary per request with token usage, and failures include hallucinations and prompt injection rather than just exceptions. LLM observability adds prompt/response capture, token and cost tracking, and quality evaluation on top of standard tracing.

What should you monitor in an LLM application?

Latency (including time to first token for streaming), token usage and cost per request and per feature, error and refusal rates, output quality signals (evaluation scores, user feedback, thumbs-down rates), retrieval quality in RAG systems, and safety events such as PII leakage or prompt-injection attempts.

Does OpenTelemetry support LLM observability?

Yes - OpenTelemetry's generative AI semantic conventions define standard attributes for model calls (model name, token counts, etc.), and instrumentation libraries like OpenLIT and OpenLLMetry emit LLM traces as standard OTLP. That means any OTLP backend, including OpenObserve, can serve as an LLM observability platform.

What is LLM Observability?

LLM observability is the practice of monitoring LLM-powered applications — tracing prompts and responses, tracking token costs and latency, and evaluating output quality — to keep AI features reliable, safe, and affordable.

AI Observability Last updated: July 8, 2026

LLM observability is observability for applications built on large language models: capturing the prompts, responses, tool calls, token counts, latencies, and quality signals of every model interaction so teams can debug, optimize, and trust their AI features. It extends distributed tracing to a world where the “service” is probabilistic and every call has a price tag.

Why LLM apps need their own observability

LLM-powered systems break the assumptions of classic monitoring:

Non-determinism — the same prompt can produce different outputs; “correct” is a spectrum, not a status code
New failure modes — hallucinations, refusals, prompt injection, context overflow — none of which throw exceptions
Per-request economics — cost scales with tokens, so one chatty prompt template or runaway agent loop becomes a real bill
Pipeline complexity — RAG retrieval, tool-calling agents, and multi-model chains mean one user question triggers a tree of model and non-model operations

What to capture

Traces — every model call as a span: model, prompt, response, token counts, tool invocations, retrieval steps; agent frameworks produce deep trees worth tracing end-to-end
Cost & usage — tokens and dollars per request, feature, tenant, and model, with alerts on anomalies (see monitoring OpenAI API costs)
Performance — latency and time-to-first-token, provider error and rate-limit rates
Quality & safety — evaluation scores, user feedback, guardrail triggers, PII redaction of captured prompts

The open-standards approach

OpenTelemetry’s generative AI semantic conventions standardize LLM telemetry, and libraries like OpenLIT, OpenLLMetry, and framework integrations (LangChain, LlamaIndex) emit model calls as ordinary OTLP traces. That keeps LLM telemetry in the same platform as the rest of your stack — an AI feature’s slowness might be the vector database, not the model, and only correlated telemetry shows that.

LLM observability in OpenObserve

OpenObserve provides LLM observability on OpenTelemetry standards, with integrations for major model providers and agent frameworks, unified with your logs, metrics, and traces. See LLM monitoring best practices and the tools comparison to go deeper.

Frequently asked questions

: Traditional APM assumes deterministic code - the same input yields the same output, and correctness is binary. LLM calls are probabilistic, quality is a spectrum, costs vary per request with token usage, and failures include hallucinations and prompt injection rather than just exceptions. LLM observability adds prompt/response capture, token and cost tracking, and quality evaluation on top of standard tracing.
: Latency (including time to first token for streaming), token usage and cost per request and per feature, error and refusal rates, output quality signals (evaluation scores, user feedback, thumbs-down rates), retrieval quality in RAG systems, and safety events such as PII leakage or prompt-injection attempts.
: Yes - OpenTelemetry's generative AI semantic conventions define standard attributes for model calls (model name, token counts, etc.), and instrumentation libraries like OpenLIT and OpenLLMetry emit LLM traces as standard OTLP. That means any OTLP backend, including OpenObserve, can serve as an LLM observability platform.

Keep reading

Back to Glossary