Best LLM Observability Tools for Production Apps

Contents

Seven views, four tool categories, one standards fight — and the alert list that actually wakes someone

Your LLM app can look perfectly healthy and fail quietly. Every request returns a 200, latency looks normal, and meanwhile the model invents refund policies and your token bill triples. Your uptime dashboard shrugs.

That gap explains why LLM observability exists. Regular monitoring tells you the app runs. LLM observability tells you whether the app answers well, what each answer costs, and why a bad one went wrong.

This guide covers what to monitor, which kinds of tools exist, how the standards fight affects your choice, and which tools fit which team. FYI: I researched this in October 2026, and the space changes fast, so verify details against each vendor's docs — the tooling layer of the LLMOps production guide.

What Production Observability Must Show

Before you compare tools, decide what you need to see. Most production teams need these seven views:

• Full traces: every prompt, retrieved document, tool call, and response, linked in one tree.
• Token and cost tracking: spend by feature, model, and user.
• Latency breakdowns: time to first token and total time, split by step.
• Quality signals: online evals, judge scores, and user feedback attached to traces.
• Errors and retries: timeouts, rate limits, malformed outputs, and fallbacks.
• Safety flags: policy violations, prompt injection attempts, and sensitive data leaks.
• Session context: the whole conversation, since many failures appear only at turn five.

If a tool can't show most of these, it won't save you at 2 a.m.

The Four Kinds of Tools

People compare tools from different categories as if they were rivals. They aren't. Sort them first.

1. LLM-Native Platforms

These tools center on traces, evals, prompts, and datasets. Examples include Langfuse, LangSmith, Braintrust, and Arize Phoenix. They understand LLM concepts out of the box, like prompt versions and judge scores.

• Strengths: deep LLM features, fast setup, eval workflows.
• Trade-offs: another tool to run, and another place your telemetry lives.

Langfuse leads the open-source side, and ClickHouse acquired it in January 2026. LangSmith fits LangChain and LangGraph apps best. Braintrust leans toward evaluation-driven releases. Phoenix speaks OpenTelemetry natively and runs free when you self-host — the full profiles live in the LLMOps platforms comparison.

2. APM Incumbents With LLM Features

If you already pay for a general observability platform, check whether it covers LLMs. Datadog, for example, offers LLM observability and accepts OpenTelemetry traces that follow the GenAI conventions from version 1.37 onward.

• Strengths: one place for app, infrastructure, and LLM telemetry. Existing alerting and on-call workflows carry over.
• Trade-offs: eval and prompt-management features usually run shallower than in LLM-native tools, and costs can climb with volume.

3. Gateways and Proxies

A gateway sits between your app and the model provider, so you capture every call without much code. That convenience attracts teams that want observability today with no SDK rewrite.

Helicone made this approach popular, but Mintlify acquired it in March 2026, and it now runs in maintenance mode with security and bug fixes only. IMO, that makes it a poor pick for a new multi-year project, so evaluate other gateways or SDK-based options.

• Strengths: quick setup, caching, and routing features.
• Trade-offs: the proxy joins your request path, which adds a dependency and a latency hop. Proxies also see less of your internal app logic than SDK tracing does.

4. Your Own OpenTelemetry Stack

Some teams send OpenTelemetry traces to a general backend they run themselves. Several vendors and open-source projects accept OTLP traces from instrumented LLM frameworks.

• Strengths: full control and portability. No lock-in.
• Trade-offs: you build the dashboards, eval workflows, and prompt tooling yourself.

The Standards Fight You Should Know About

Here's the part most comparison posts skip. Three conventions compete to define how LLM traces look:

• OpenTelemetry GenAI semantic conventions: the community standard, with attributes like gen_ai.usage.input_tokens.
• OpenInference: the convention set Arize maintains and Phoenix uses.
• OpenLLMetry: a third option from Traceloop that mixes newer and legacy attributes.

These conventions matter because they decide how portable your traces stay. Instrument once with a standard, and you can switch backends later. Instrument with a proprietary format, and you inherit a migration project.

The OpenTelemetry GenAI Spec Is Still Moving

Plenty of blog posts claim these conventions became stable. The primary sources say otherwise, so treat them as evolving. Here's what changed recently:

• Version 1.37 reshaped how LLM behavior appears in standard spans and attributes, and many tools now target it.
• gen_ai.provider.name replaced gen_ai.system in that version, yet older instrumentation still emits the old name. Match both in your queries during migration.
• In June 2026, the GenAI conventions moved out of the main semantic-conventions repository into a dedicated GenAI repository.

So what should you do? Prefer OpenTelemetry-compatible tools, pin your instrumentation versions, and expect to update attribute names occasionally — portability is the prize.

A Minimal Instrumentation Sketch

This example shows the idea with the standard attribute names. The model and provider names are placeholders:

from opentelemetry import trace

tracer = trace.get_tracer("my-llm-app")

with tracer.start_as_current_span("chat model-large-v2") as span:
    span.set_attribute("gen_ai.operation.name", "chat")
    span.set_attribute("gen_ai.provider.name", "my-provider")   # was gen_ai.system
    span.set_attribute("gen_ai.request.model", "model-large-v2")
    resp = call_llm(prompt)
    span.set_attribute("gen_ai.usage.input_tokens", resp.usage.input_tokens)
    span.set_attribute("gen_ai.usage.output_tokens", resp.usage.output_tokens)

Figure 1: Traces explain, dashboards summarize — the tool you pick decides which one you get at 2 a.m.

Which Tool Fits Which Team

Match the tool to your situation, not to a ranking:

• You use LangChain or LangGraph: start with LangSmith.
• You need self-hosting and open source: look at Langfuse or Phoenix.
• You gate releases on evals: consider Braintrust or another eval-first platform — the gates the CI/CD guide builds.
• You already run Datadog or a similar APM: test its LLM features first, and add an LLM-native tool only if gaps appear.
• You want maximum portability: instrument with OpenTelemetry and keep your backend swappable.
• You want zero-code visibility fast: try a gateway-style tool, but check its long-term roadmap.

Note that vendor-written comparison posts rank their authors first almost every time. Read them like you'd read a restaurant's review of itself.

What to Alert On

Dashboards don't wake anyone up, alerts do. Start with these:

• Cost per request or daily spend above a threshold — the numbers the cost tracking setup already computes.
• Error and retry rate spikes.
• Latency regressions, especially time to first token.
• Quality drops in online eval scores or thumbs-down rate — the signal the feedback loop captures at the button.
• Refusal rate changes after a deploy.
• Safety flags crossing a set limit.

ALERTS = {  # tie every rule to the deploy marker, not just the clock
    "cost_per_request": {"window": "1h",  "threshold": 0.012, "since_deploy": True},
    "error_rate":       {"window": "15m", "threshold": 0.02},
    "ttft_p95_ms":      {"baseline": "prev_7d", "delta": 0.30},
    "thumbs_down_rate": {"window": "24h", "vs": "prev_7d", "threshold": 1.5},
}

Tie alerts to deploys. When a prompt change lands and cost per request jumps, you want to know within the hour.

Protect Privacy and Control Cost

Observability data holds sensitive material. Prompts and responses often include personal information, so handle them with care:

• Redact or hash personal data before it leaves your app.
• Limit access to raw prompts with role-based controls.
• Set retention periods and delete old traces on schedule.
• Sample high-volume traffic and keep every trace that errors or gets flagged.

Check whether your self-hosted option includes the access controls your compliance team requires. Reports say some open-source tools reserve features like audit logs for paid tiers.

Mistakes I See Teams Make

• Watching only infrastructure metrics. The app stays "healthy" while answers rot.
• Logging everything forever. Storage bills and privacy risk both balloon.
• Locking into proprietary instrumentation. Switching tools later hurts.
• Skipping user feedback signals. Thumbs-downs and regenerate clicks reveal problems before evals do.
• Never reading real traces. Dashboards summarize, and traces explain.

Common Pitfalls

• Buying by ranking — the vendor's own comparison finishes first; sort by category and constraint instead.
• Category confusion — an APM, an LLM-native suite, and a proxy answer different questions; they aren't interchangeable.
• Proprietary instrumentation — every non-standard attribute is a future migration ticket.
• Spec stability myths — the GenAI conventions still move; pin versions and match old and new attribute names.
• Alerts without deploy markers — cost and quality jumps must correlate to releases, not just timestamps.
• Unbounded retention — log-forever turns into storage bills plus privacy exposure.
• Ignoring access tiers — confirm audit logs and RBAC exist in the tier you're actually buying.
• Trace-free on-call — dashboards summarize; when an alert fires, someone must open the real trace.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What must LLM observability show in production?

Seven views. Full traces: every prompt, retrieved document, tool call, and response linked in one tree. Token and cost tracking: spend by feature, model, and user. Latency breakdowns: time to first token and total time, split by step. Quality signals: online evals, judge scores, and user feedback attached to traces. Errors and retries: timeouts, rate limits, malformed outputs, fallbacks. Safety flags: policy violations, prompt injection attempts, sensitive data leaks. Session context: the whole conversation, since many failures only appear at turn five. Regular monitoring tells you the app runs; observability tells you whether it answers well, what each answer costs, and why a bad one went wrong. If a tool can't show most of these, it won't save you at 2 a.m.

What are the four kinds of LLM observability tools?

Sort the categories before comparing — they aren't rivals. LLM-native platforms (Langfuse, LangSmith, Braintrust, Arize Phoenix): deep LLM concepts out of the box, fast setup, eval workflows, but another tool to run and another home for telemetry. APM incumbents with LLM features (Datadog offers LLM observability and accepts OpenTelemetry GenAI-convention traces from version 1.37 onward): one place for app, infrastructure, and LLM telemetry with existing alerting, but shallower eval and prompt-management features and volume-driven cost. Gateways and proxies: capture every call with no SDK rewrite plus caching and routing, but the proxy joins your request path (a dependency and a latency hop) and sees less internal app logic — Helicone, the popular one, is maintenance-mode-only after Mintlify's March 2026 acquisition. Your own OpenTelemetry stack: full control and portability, but you build dashboards, evals, and prompt tooling yourself.

What is the OpenTelemetry GenAI standards situation?

Three conventions compete to define how LLM traces look: OpenTelemetry GenAI semantic conventions (the community standard, with attributes like gen_ai.usage.input_tokens), OpenInference (maintained by Arize, used by Phoenix), and OpenLLMetry (Traceloop's option, which mixes newer and legacy attributes). They decide how portable your traces stay — instrument once with a standard and you can switch backends later; instrument with a proprietary format and you inherit a migration project. The spec is still moving despite blog posts claiming stability: version 1.37 reshaped how LLM behavior appears in spans, gen_ai.provider.name replaced gen_ai.system (match both during migration, since old instrumentation still emits the old name), and in June 2026 the GenAI conventions moved out of the main semantic-conventions repository into a dedicated GenAI repository. Prefer OTel-compatible tools, pin instrumentation versions, and expect occasional attribute renames.

How do you set up LLM observability alerts and privacy?

Alert on what wakes someone up: cost per request or daily spend above threshold, error and retry rate spikes, latency regressions (especially time to first token), quality drops in online eval scores or thumbs-down rate, refusal-rate changes after a deploy, and safety flags crossing a limit. Tie alerts to deploys — when a prompt change lands and cost per request jumps, you want to know within the hour. For privacy: prompts and responses often carry personal data, so redact or hash it before it leaves your app, restrict raw-prompt access with role-based controls, set retention periods and delete old traces on schedule, and sample high-volume traffic while keeping every trace that errors or gets flagged. Check that your self-hosted tier includes the access controls your compliance team requires — reports say some open-source tools reserve audit logs for paid tiers. Then review ten real traces weekly with your team.

Wrapping This Up

Production LLM apps fail in ways uptime monitors never notice. Pick tools from the right category for your team, favor open standards for portability, and watch cost, latency, and quality together.

Start with one feature, one trace view, and one alert. When the next bad answer shows up, you'll find it in minutes instead of in a customer complaint.

Related Articles

• Observability for AI Agents: Tracing with LangSmith and Langfuse
• Best LLMOps Platforms Compared
• LLM Cost Tracking: Attribute Spend by User, Feature, and Model
• What Is LLMOps? Complete Guide to Running LLM Apps in Production