Observability for AI Agents: Tracing with LangSmith and Langfuse

Contents

Figure 1: One trace per run, one span per step — replay the chain instead of guessing

Nested spans, token costs, session context, and the flushing habit that keeps dashboards from staying empty

Your agent just told a customer their order shipped. It hadn't. Now you need to know which step went wrong, and print() statements won't tell you.

Agents are loops: an LLM call, a tool call, another LLM call, and so on. When something breaks, the bug hides somewhere in that chain. Tracing records the chain so you can replay it. This article shows how to set it up with two popular tools, LangSmith and Langfuse, using the same small agent for both. If you're still wiring up your first loop, start with the build-an-agent-from-scratch guide in this series.

Why Agents Need Tracing (Logs Aren't Enough)

A normal web request does one thing. An agent run does a variable number of things, and the model decides how many. Flat logs turn that into a pile of timestamps you have to stitch together by hand.

A trace fixes this with a tree:

• Trace: one complete agent run, from user question to final answer.
• Span (or run): one step inside it, such as an LLM call or a tool call.
• Nesting: child spans sit under their parent, so you see what called what.

From that tree you get latency per step, token usage and cost, errors, and the exact inputs and outputs. That last one is usually where the bug turns up.

The Demo Agent

Both examples trace the same support agent. It has one tool (lookup_order) and a mock LLM, a deterministic stand-in that needs no API key. Swap in your real model call and the tracing code stays the same.

# agent_core.py
import json, re

ORDERS = {"A100": {"status": "shipped", "total": 42.00},
          "A200": {"status": "delivered", "total": 99.50}}

def lookup_order(order_id: str) -> dict:
    if order_id not in ORDERS:
        raise ValueError(f"unknown order {order_id}")
    return {"order_id": order_id, **ORDERS[order_id]}

def mock_llm(messages: list[dict]) -> dict:
    last = messages[-1]
    n_in = sum(len(m["content"].split()) for m in messages) * 2
    if last["role"] == "user":
        m = re.search(r"\b([A-Z]\d{3})\b", last["content"])
        if m:
            return {"tool_call": {"name": "lookup_order", "args": {"order_id": m.group(1)}},
                    "usage": {"input_tokens": n_in, "output_tokens": 14}}
        return {"text": "Which order number are you asking about?",
                "usage": {"input_tokens": n_in, "output_tokens": 9}}
    data = json.loads(last["content"])
    if "error" in data:
        return {"text": "I couldn't find that order. Could you double-check the number?",
                "usage": {"input_tokens": n_in, "output_tokens": 15}}
    return {"text": f"Order {data['order_id']} is {data['status']}. Total: ${data['total']:.2f}.",
            "usage": {"input_tokens": n_in, "output_tokens": 18}}

Notice the agent loop has a hard step cap (four iterations). Tracing shows you a runaway loop, but a cap stops it. Use both — the same defense the guardrails guide applies to budgets.

The shape we want in the UI looks like this:

support_agent
├── mock_llm        (decides to call the tool)
├── lookup_order    (the tool)
└── mock_llm        (writes the answer)

Tracing with LangSmith

LangSmith is LangChain's tracing platform, but it doesn't require LangChain. The Python SDK traces plain functions.

Setup.

pip install langsmith

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="<your key>"
export LANGSMITH_PROJECT="agent-demo"

Also set LANGSMITH_ENDPOINT if you use a non-default region or self-hosted instance, and LANGSMITH_WORKSPACE_ID if your key spans several workspaces.

The code. The @traceable decorator turns a function into a run. Setting run_type tells the UI how to display it: "llm", "tool", or "chain".

# agent_langsmith.py
import asyncio, json

import langsmith as ls
from langsmith import Client, traceable

import agent_core as core


@traceable(run_type="tool", name="lookup_order")
def lookup_order(order_id: str) -> dict:
    return core.lookup_order(order_id)


@traceable(run_type="llm", name="mock_llm")
def call_llm(messages: list[dict]) -> dict:
    return core.mock_llm(messages)


@traceable(run_type="chain", name="support_agent")
def support_agent(question: str) -> str:
    messages = [{"role": "user", "content": question}]
    for _ in range(4):                                  # hard step cap
        reply = call_llm(messages)
        if "text" in reply:
            return reply["text"]
        try:
            result = lookup_order(**reply["tool_call"]["args"])
        except ValueError as e:                         # let the model see the failure
            result = {"error": str(e)}
        messages.append({"role": "tool", "content": json.dumps(result)})
    return "Step limit reached."


async def main() -> None:
    client = Client()
    try:
        questions = ["Where is order A100?", "Where is order Z999?"]
        with ls.trace("support_batch", "chain", project_name="agent-demo",
                      inputs={"questions": questions}) as rt:
            answers = [support_agent(q) for q in questions]
            rt.end(outputs={"answers": answers})
        for a in answers:
            print(a)
    finally:
        await client.flush()                            # short-lived script: send before exit


if __name__ == "__main__":
    asyncio.run(main())

Two things worth explaining:

• Nesting is automatic. support_agent calls call_llm and lookup_order, so those runs land underneath it. No IDs to pass around.
• ls.trace() groups arbitrary code. Here it wraps both questions in one parent run called support_batch. Call rt.end(outputs=...) to attach results.

Don't forget to flush.

LangSmith sends traces in the background. In a long-running server, that's fine. In a short script, the process may exit before the upload finishes, and you'll stare at an empty project wondering what you did wrong. The docs show await client.flush(), which is why main() is async. Call it in a finally block so errors don't skip it.

Wrapping an OpenAI client.

If you call OpenAI directly, wrap_openai traces the client's calls as LLM runs, with model and token usage captured for you. Wrap it once at startup:

from langsmith.wrappers import wrap_openai
from openai import OpenAI

client = OpenAI()
client = wrap_openai(client)

That's one line instead of hand-writing every LLM span — the best deal in the whole SDK.

Tracing with Langfuse

Langfuse is open source and can be self-hosted, which matters if prompts contain customer data. This article uses the v4 Python SDK, a rewrite from earlier this year. Older tutorials often show the v2 trace() API, which is deprecated. If a snippet you find online looks very different from below, check its version first.

Setup.

pip install langfuse

export LANGFUSE_PUBLIC_KEY="<your public key>"
export LANGFUSE_SECRET_KEY="<your secret key>"
export LANGFUSE_BASE_URL="https://cloud.langfuse.com"   # or your self-hosted URL

The code. Langfuse's decorator is @observe. Marking a function with as_type="generation" tells Langfuse it's an LLM call, which unlocks model and usage fields.

# agent_langfuse.py
import json

from langfuse import get_client, observe, propagate_attributes

import agent_core as core

langfuse = get_client()


@observe(name="lookup_order")
def lookup_order(order_id: str) -> dict:
    return core.lookup_order(order_id)


@observe(name="mock_llm", as_type="generation")
def call_llm(messages: list[dict]) -> dict:
    reply = core.mock_llm(messages)
    langfuse.update_current_generation(model="mock-model", usage_details=reply["usage"])
    return reply


@observe(name="support_agent")
def support_agent(question: str, user_id: str, session_id: str) -> str:
    with propagate_attributes(user_id=user_id, session_id=session_id,
                              tags=["support"], metadata={"app": "demo"}):
        messages = [{"role": "user", "content": question}]
        for _ in range(4):                              # hard step cap
            reply = call_llm(messages)
            if "text" in reply:
                return reply["text"]
            try:
                result = lookup_order(**reply["tool_call"]["args"])
            except ValueError as e:                     # let the model see the failure
                result = {"error": str(e)}
            messages.append({"role": "tool", "content": json.dumps(result)})
        return "Step limit reached."


if __name__ == "__main__":
    try:
        print(support_agent("Where is order A100?", user_id="user-1", session_id="sess-1"))
        print(support_agent("Where is order Z999?", user_id="user-1", session_id="sess-1"))
    finally:
        langfuse.flush()                                # short-lived script: send before exit

What propagate_attributes does.

It stamps user ID, session ID, tags and metadata onto every span created inside the with block. That's how you later answer "show me everything user-1 did in this conversation."

Mind the limits. Metadata is a flat dictionary of strings, each at most 200 characters. User and session IDs are capped at 200 characters too. Stuff a whole prompt in there and Langfuse will not be amused.

Flush, again.

Same lesson as LangSmith. flush() sends pending data, and shutdown() flushes and closes the client. The SDK also registers an exit hook, but calling flush() yourself costs one line and removes the guesswork.

Controlling what gets captured.

Decorators record function inputs and outputs by default. Convenient, until a function receives a credit card number. You have options:

• Langfuse: pass capture_input=False or capture_output=False to @observe. Set LANGFUSE_OBSERVE_DECORATOR_IO_CAPTURE_ENABLED=false to turn off decorator capture globally.
• LangSmith: check the SDK's options for hiding or masking inputs and outputs before you trace anything sensitive.
• Either tool: self-hosting keeps data on your own infrastructure.

Decide this before launch. Scrubbing production traces afterwards is miserable.

LangSmith vs. Langfuse: Which One?

Both give you nested traces, token tracking and a UI for digging in. The practical differences:

	LangSmith	Langfuse
Decorator	@traceable	@observe
LLM span type	run_type="llm"	as_type="generation"
Open source / self-host	Check current plans	Yes
Best fit	LangChain / LangGraph stacks	Framework-agnostic, self-hosted

If you already live in LangChain, LangSmith is the lowest-friction path — the same stack the LangGraph tutorial in this series uses. If data residency or open source matters, Langfuse is hard to beat. Neither is wrong, and the code above shows how little you'd rewrite to switch.

Habits That Make Traces Useful

• Name every span. "lookup_order" beats "function_1" at 2 a.m.
• Tag by environment and version. Separate dev noise from production.
• Attach user and session IDs. One bad conversation is findable only if you can group by session.
• Watch the tool-error rate. A tool that fails a lot is usually a bad tool description, not a bad model.
• Alert on step counts. A run with 30 steps is a loop, not a conversation.
• Check how errors show up. Don't assume an exception in a tool appears in the UI the way you expect. Trigger one on purpose (the Z999 order above) and look.

Traces are also the raw material of the evaluation guide's trace-level review: categorize failures, then turn them into test cases.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

Why aren't flat logs enough to debug an AI agent?

A normal web request does one thing; an agent run does a variable number of things, and the model decides how many. Flat logs turn that into a pile of timestamps you have to stitch together by hand. A trace fixes this with a tree: one trace per agent run, spans for each step (LLM call, tool call), and nesting so child spans sit under their parent. From that tree you get latency per step, token usage and cost, errors, and the exact inputs and outputs — which is usually where the bug turns up.

What is the difference between LangSmith and Langfuse?

Both give you nested traces, token tracking, and a UI for digging in. LangSmith uses the @traceable decorator and run_type='llm' for LLM spans, and its wrap_openai helper traces OpenAI calls automatically — lowest friction if you already live in LangChain or LangGraph. Langfuse is open source and self-hostable, uses @observe with as_type='generation', and is framework-agnostic — hard to beat when data residency or open source matters. The tracing code barely changes between them.

Why do I have to flush traces before a script exits?

Both SDKs send traces in the background. In a long-running server that's fine, but in a short script the process may exit before the upload finishes, and you'll stare at an empty project wondering what you did wrong. Call await client.flush() for LangSmith (which is why the demo's main() is async) and langfuse.flush() for Langfuse, in a finally block so errors don't skip it. Langfuse's shutdown() flushes and closes the client.

How do I keep sensitive data out of tracing tools?

Decide before launch, because scrubbing production traces afterwards is miserable. Decorators record function inputs and outputs by default, so in Langfuse pass capture_input=False or capture_output=False to @observe, or set LANGFUSE_OBSERVE_DECORATOR_IO_CAPTURE_ENABLED=false globally; for LangSmith, check the SDK's options for hiding or masking inputs and outputs before tracing anything sensitive. Either tool can be self-hosted, which keeps data on your own infrastructure.

Conclusion

Tracing turns "the agent did something weird" into a tree you can read. Decorate your tool and LLM functions, nest them under one agent span, attach user and session context, and flush before your script exits. That covers 90% of what you'll need.

Pick one tool, run the demo with real keys, and break something on purpose. Then open the trace and find it. That's the fastest way to learn what these dashboards are good for.

Related Articles

• Build Your First AI Agent in Python from Scratch (No Framework)
• Evaluating AI Agents: Metrics, Benchmarks, and Test Strategies
• AI Agent Guardrails and Safety: Prevent Runaway Actions
• LangGraph Tutorial: Build Stateful Multi-Step Agents