LLM Observability Tools Compared (2026)

March 28, 2026 / Updated May 21, 2026 / Arvid Andersson

Once you ship LLM features to production, you need visibility into what's happening: what prompts are being sent, how much each call costs, whether quality is holding up, and where failures occur. LLM observability tools fill this gap. This post compares the tools in the category and what each does well.

Looking for a side-by-side table with filters? See the Observability & Analytics pricing comparison.

Why LLM observability matters

Traditional application monitoring tracks request times and error rates. LLM applications add new dimensions: token costs that can spike unexpectedly, prompt quality that degrades over time, hallucinations that are hard to catch without systematic evaluation, and complex chains where a failure in one step cascades. LLM observability tools capture the full trace of each request, from prompt to completion, and give you the data to debug, optimize, and evaluate.

Open-source options

Langfuse is one of the more widely adopted open-source options, with ~27k GitHub stars as of May 2026. It provides tracing, prompt management, evaluation, and cost tracking, with integrations across most LLM frameworks (LangChain, LlamaIndex, OpenAI SDK). Langfuse joined ClickHouse in January 2026 (announced as an acquisition, no funding figures disclosed). The team has committed to no planned changes to the MIT license or self-hosting, which matters for teams with strict data residency requirements.

Opik (by Comet) is an Apache-2.0 option with built-in evaluation metrics for hallucination, moderation, and relevance. It includes an agent optimizer SDK for automated prompt tuning. Backed by Comet, an established MLOps company.

Lunary is self-hostable (Kubernetes or Docker) and focused on chatbot and RAG evaluation, with paid tiers starting at $20/user/month. Good for smaller teams that want observability without a large budget.

Platform-integrated tools

LangSmith is built by the LangChain team and integrates tightly with the LangChain ecosystem. If your application already uses LangChain, LangSmith gives you tracing, evaluation datasets, and prompt playgrounds with minimal setup. The trade-off is vendor lock-in to the LangChain stack.

Braintrust takes a different angle, focusing on evaluation and experimentation. It treats LLM outputs as data that should be scored, compared, and iterated on. After raising an $80M Series B in February 2026 (customers include Vercel, Notion, Coursera, Dropbox, Replit, Graphite, and Navan), the team built Brainstore, a custom database designed for querying complex AI traces from multi-step agents. The Loop agent generates better prompts, scorers, and datasets automatically. A good fit for teams that want rigorous, data-driven prompt optimization rather than just logging.

Proxy-based monitoring

Helicone works as a proxy that sits between your application and the LLM provider. You change your API base URL to point at Helicone, and it logs every request with zero code changes. This approach is appealing for teams that want observability without modifying their application code. Helicone tracks costs, latency, and usage patterns across providers.

Agent-focused observability

As AI agents become more common, tools like AgentOps specialize in agent-specific observability. AgentOps records every LLM call, tool invocation, and decision point in an agent session, letting you replay runs step-by-step. It integrates with frameworks like CrewAI, AutoGen, and OpenAI Agents SDK. If you're building autonomous agents rather than simple LLM chains, agent-specific tooling gives you better visibility into multi-step reasoning and tool use.

HoneyHive covers similar ground with distributed tracing and drift detection for production agents, plus flexible deployment options (SaaS, hybrid, or self-hosted). Useful for teams that need on-premises deployment alongside agent observability.

ML platforms with LLM support

Weights & Biases started as an experiment tracking platform for ML training and has expanded into LLM observability with its Weave product. If your team already uses W&B for model training, adding LLM tracing keeps everything in one platform. Arize AI takes a similar approach, combining traditional ML monitoring with LLM-specific features like prompt tracing and embedding drift detection. Their open-source Phoenix library provides local tracing and evaluation without sending data to a cloud service.

AI gateways

Portkey combines an AI gateway with observability. It sits between your application and LLM providers, handling request routing, fallbacks, and load balancing while logging every request. The Gateway 2.0 release (in pre-release as of May 2026) is merging previously enterprise-only features into the open-source core, including circuit breakers, semantic caching, budget limits, and real-time metrics. Portkey advertises 10B+ tokens processed daily and supports a wide range of LLMs with 50+ guardrails built in. They have also added an MCP Gateway for governing AI agents across enterprise tools. Useful for teams running multiple LLM providers in production who want both reliability and observability in one layer.

Evaluation and testing

LangWatch bridges evaluation and observability with agent simulation. It runs thousands of synthetic conversations across scenarios, languages, and edge cases, then feeds results back into prompt optimization via DSPy integration. Netherlands-based and open-source. Useful for teams that want to test agent behavior systematically before shipping, not just monitor after the fact.

Two notes on the wider ecosystem: Humanloop, previously listed here for prompt management and A/B testing, was acquired by Anthropic in 2025 and the Humanloop platform is being sunset. Separately, Traceloop, whose open-source OpenLLMetry library provides OpenTelemetry instrumentation used by several tools on this page, was acquired by ServiceNow in March 2026.

Guardrails and safety

Catching bad outputs before they reach users requires more than logging. Guardrails sit on top of LLM calls to block hallucinations, PII leaks, prompt injection attempts, or policy violations at request time.

Patronus AI focuses on automated evaluation with LLM-as-judge approaches, useful when you want to score outputs at scale without writing custom evaluators. Giskard takes a pre-deployment angle, scanning models for vulnerabilities and quality issues before they reach production. Portkey's gateway includes 50+ built-in guardrails, which fits teams that already want a gateway layer. Future AGI bundles guardrails with simulation and evaluation in one platform.

The trade-off across this category is breadth versus depth: dedicated safety tools (Patronus, Giskard) tend to go deeper on a narrower problem, while platforms that combine guardrails with tracing or gateway features reduce integration cost but cover less ground per area.

Comparison

How to choose

The category is broad enough that the right pick depends on what you optimize for. A few starting points:

• Open-source and self-hosted: Langfuse for general-purpose tracing, Arize Phoenix for OpenTelemetry-native, Opik for built-in evaluation metrics.
• Already on LangChain: LangSmith is the path of least resistance.
• Simplest possible setup: Helicone's proxy gets you logging with a one-line base URL change.
• Prompt quality and structured evals: Braintrust for scoring and comparison workflows, Patronus for LLM-as-judge.
• Building agents: AgentOps for session replay and time-travel debugging, HoneyHive for production agent monitoring with flexible deployment.
• Already using W&B or Arize for ML: Their LLM features keep everything in one platform.
• AI gateway plus observability: Portkey for the open-source gateway with routing and logging in one layer.
• Budget-friendly chatbots and RAG: Lunary starts at $20/user/month.
• Agent simulation before shipping: LangWatch for synthetic conversation testing with DSPy integration.
• Guardrails, evals, and simulation under one roof: Future AGI bundles all of these with a free tier that covers prototype-scale usage.

Most of these tools offer free tiers, so the practical approach is to try two or three with your actual workload before committing.

Frequently asked questions