LLM Observability: Tracing, Sampling, and Catching Regressions Before Users Do

Classical APM is not enough

Traditional observability was built for deterministic systems. A 200 means success, a 500 means failure, latency is a clean number. LLM systems break all three assumptions. A 200 can contain garbage output. A "success" can violate your schema. Latency has TTFT, ITL, and queue time that all matter differently. Cost varies per request. And the same prompt can produce different outputs across calls.

This is Part 5 of the AI Engineer Series. Evals from Part 4 tell you whether changes are good. Observability tells you what is actually happening in production. Both are mandatory. Neither replaces the other.

What to capture per call

Every LLM invocation should produce a structured event with enough detail to reproduce it later. The minimum schema:

• Identity: request ID, parent span ID, user ID (hashed), feature name, environment, experiment arm if any
• Inputs: full prompt (template name + version + variables), system prompt, model name, temperature, top-p, max_tokens, tools schema
• Outputs: full response, finish reason, tool calls made, validation pass/fail
• Performance: TTFT, ITL, total duration, queue time
• Cost: input tokens (cached and uncached separately), cache write tokens, output tokens, computed cost
• Quality signals: schema validation, user feedback (thumbs up/down), downstream signals (did the user edit the output? did they retry?)

This is more than you probably log today, but each field earns its place. The first time you have to debug a production incident, you will be grateful you logged the prompt version.

Trace structure: spans for everything

A user-visible interaction often involves multiple LLM calls, retrieval steps, tool uses. Model this as a trace with hierarchical spans:

request "answer_question" [4.2s, $0.018]
├── retrieve_context [380ms]
│   ├── embed_query [45ms]
│   ├── vector_search [220ms]
│   └── rerank [115ms]
├── generate_answer [3.6s, $0.018]
│   ├── prompt_construction [12ms]
│   └── llm_call (claude-sonnet-4-6) [3.5s]
└── validate_output [25ms]

This structure tells you where time and money go at a glance. The retrieve span took 380ms? Fine. It took 2.5s? That is your bug.

OpenTelemetry GenAI conventions

The OpenTelemetry GenAI Special Interest Group has been working since April 2024 to standardize attribute names, span types, and metrics for LLM workloads. As of mid-2026, the conventions are still in experimental status, but major vendors (Datadog, Honeycomb, New Relic, Langfuse, Arize Phoenix) already support them. Adopt the standard. It will outlast any vendor's proprietary schema.

The key attribute names you should be emitting:

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer(__name__)

def call_llm_traced(prompt: str, model: str = "claude-sonnet-4-6"):
    with tracer.start_as_current_span(f"chat {model}") as span:
        # GenAI semantic conventions
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.system", "anthropic")
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("gen_ai.request.max_tokens", 1024)
        span.set_attribute("gen_ai.request.temperature", 0.2)
        
        try:
            response = client.messages.create(
                model=model,
                max_tokens=1024,
                messages=[{"role": "user", "content": prompt}],
            )
            
            usage = response.usage
            span.set_attribute("gen_ai.usage.input_tokens", usage.input_tokens)
            span.set_attribute("gen_ai.usage.output_tokens", usage.output_tokens)
            span.set_attribute(
                "gen_ai.usage.cache_read.input_tokens",
                getattr(usage, "cache_read_input_tokens", 0),
            )
            span.set_attribute("gen_ai.response.finish_reasons", [response.stop_reason])
            
            return response
        except Exception as e:
            span.set_status(Status(StatusCode.ERROR, str(e)))
            span.set_attribute("error.type", type(e).__name__)
            raise

Span names follow the pattern {operation} {model}, like chat claude-sonnet-4-6. The naming matters because GenAI-aware backends recognize this pattern and unlock richer dashboards automatically.

Sensitive data handling

You are logging full prompts and responses. These often contain user data, personal information, sometimes credit card numbers users type in by mistake. Three strategies, in order of preference:

• Redact at the SDK layer. Scrub emails, phone numbers, credit cards before they leave your service. Microsoft Presidio is the standard open-source option.
• Tag and segregate. Mark sensitive traces and route them to a more secure storage tier with stricter access controls and shorter retention.
• Sample down. Log full prompts and outputs for 1-5% of traffic, metadata only for the rest.

The OpenTelemetry GenAI conventions explicitly mark content capture as opt-in for this reason. Default to off in production until you have a redaction story.

Sampling strategies

At scale, you cannot store every full trace. Layer your sampling:

• Head sampling. Random fixed rate for general visibility, often 10%.
• Error-biased sampling. 100% of failed calls (schema failures, API errors, judge-flagged outputs).
• Latency-biased sampling. 100% of p99 outliers. The fastest way to find your worst bugs.
• User-feedback-biased. 100% of thumbs-down responses. The user told you it was bad. Listen.

The implementation is a tail-based sampler that decides after the trace completes whether to keep it.

The four metrics that matter most

Build dashboards around these, not the 50 cosmetic ones you can dream up:

• p95 end-to-end latency. The number most users care about.
• Cost per request, broken down by feature. The number your finance team will ask about.
• Schema-failure rate. Sudden jumps mean a prompt or model regression.
• Online eval score on a rolling window. The early warning system for quality drift.

Set SLOs on all four. Alerts fire when they breach. This is the difference between "we hope it is working" and "we know when it stops working."

Catch token-usage drift

The most common silent regression in LLM systems is output length creeping up. You shipped a new prompt that produces slightly more verbose answers. Cost rises 30% in a week. The eval scores barely move, so nothing catches it.

The defense is simple: alert on the average output token count per feature when it changes by more than 20% week-over-week. The alert is annoying. The bug it catches will pay for the noise.

Tool choice

The tooling landscape in 2026:

• Langfuse: open-source, native OpenTelemetry support, strong for self-hosting
• Arize Phoenix: open-source, built-in evaluation and experiment tracking
• Braintrust: hosted, integrated with the eval workflow from Part 4
• Datadog, Honeycomb, New Relic: native GenAI semantic convention support, good for teams already using them
• Helicone: proxy-based, lowest-friction integration

Pick one. The discipline of having observability matters far more than which tool you use. The OpenTelemetry standard means you can switch later without rewriting your instrumentation.

What to build this week

Wire observability into the harness you have been building:

• Add OpenTelemetry instrumentation following the GenAI semantic conventions
• Pick one backend (Langfuse self-hosted is the cheapest path) and point your traces at it
• Build a dashboard with the four metrics: p95 latency, cost per request by feature, schema-failure rate, eval score
• Implement tail-based sampling: keep 10% of normal traces, 100% of errors, 100% of p99 outliers
• Deliberately ship a broken prompt that increases output length 50%. Verify your token-drift alert fires.

That last step is the test. Observability is not real until you have validated that it actually catches regressions.

What is next

Part 6 starts the performance and cost batch. We will look at latency and throughput engineering: TTFT versus ITL, continuous batching, streaming UX, and the numbers that decide whether your AI feels alive or sluggish.

Previous in the series

• Part 1: Harness Engineering
• Part 2: Context Engineering & Retrieval Quality
• Part 3: Structured Outputs and Fallback Chains
• Part 4: Evals

References

1. OpenTelemetry GenAI semantic conventions. The standard you should be emitting, even if your backend does not require it yet.
2. Langfuse documentation. Open-source LLM observability with native OpenTelemetry support.
3. Arize Phoenix documentation. Open-source observability with built-in evals.
4. Microsoft Presidio. The standard open-source PII detection and redaction library.
5. Honeycomb engineering blog. The principles of observability translate directly from classical systems to LLMs.
6. OpenTelemetry GenAI spans spec. The detailed attribute reference for LLM client spans.