LLM observability and evaluation
How to use
LangSmith.
Tracing agent and LLM runs, investigating failures, comparing prompts and models on datasets, collecting feedback, monitoring production quality, and promoting tested versions.
LangSmith is a LLM application platform for tracing, observability, offline and online evaluation, prompt versioning, feedback, and agent deployment. Tracing agent and LLM runs, investigating failures, comparing prompts and models on datasets, collecting feedback, monitoring production quality, and promoting tested versions. This guide covers the whole path in one place: official access, a first session that produces something reviewable, the checks that make output trustworthy, and the permissions worth limiting before you connect real work.
AI should make the work easier to inspect. If the workflow removes the source, the owner, or the review step, redesign the workflow.
Access & setup
Find, install, and sign in to LangSmith
Get into the official LangSmith experience with the right account and a setup you understand.
- 01
Start at https://www.langchain.com/langsmith and confirm the domain before entering account or payment information.
- 02
Availability: LangSmith is available as a browser-based cloud service with SDK and OpenTelemetry integrations, plus enterprise self-hosted observability, evaluation, and deployment options.
- 03
A LangSmith account and scoped API key or a maintained self-hosted installation, explicit tracing and retention policy, redaction controls, representative evaluation data, and access-separated workspaces.
- 04
Sign in with the account you intend to keep using, then review plan, data, notification, and permission settings.
- 05
Run one low-risk test task before connecting sensitive files, repositories, or workspace data.
- Use official download pages.
- Review permissions during setup.
- Keep installers and applications updated.
First session
Your first useful LangSmith session
Learn the interaction loop using a small task with a clear outcome.
- 01
Define the application boundary, owners, sensitive fields, environments, trace sampling, retention, access, quality metrics, cost limits, and incident response before enabling tracing.
- 02
Instrument a non-production environment with a scoped key, redact secrets and personal data, attach version metadata, and confirm exactly which inputs, outputs, tools, and errors are captured.
- 03
Create a curated dataset from authorized cases, define code, human, and carefully calibrated model evaluators, then compare a baseline and candidate under identical conditions.
- 04
Inspect disagreements and failure clusters rather than averages alone, set release thresholds, test deletion and tracing disablement, and promote only a documented approved version.
- State the outcome before the background.
- Provide the real source material.
- Review the result before expanding the task.
Quality control
Check the quality of LangSmith output
Establish that an autonomous run did the right thing, not merely that it finished.
- 01
Define what the run should achieve and what it must never touch before granting it a single tool.
- 02
Read the full execution trace: which tools were called, with what arguments, and in what order.
- 03
Verify the side effects directly in the target system rather than trusting the agent's own report of success.
- 04
Confirm failures surfaced as failures — a silent retry loop or a swallowed error is more dangerous than a crash.
- 05
Re-run the same task and compare: an agent that behaves differently across identical runs is not yet production-ready.
- Verify side effects in the system of record, not in the agent's summary.
- Require human approval for any irreversible or outward-facing action.
- Log every tool call so a run can be reconstructed afterwards.
Privacy & permissions
Use LangSmith safely
Bound what an autonomous system can reach before you let it run unattended.
- 01
Enumerate every tool, credential, and system the agent can reach, and remove the ones it does not need.
- 02
Require explicit human approval for irreversible actions: sending, publishing, paying, deleting, or deploying.
- 03
Run against non-production data until behaviour is predictable across repeated runs.
- 04
Set hard limits on spend, iterations, and runtime so a failure loop cannot run unbounded.
- 05
Treat anything the agent reads from the web or a document as data, never as instructions it may follow.
- Tracing may capture prompts, responses, tool arguments, retrieved documents, personal data, secrets, or proprietary content. Retention differs by tier and datasets can outlive source traces. LLM-as-judge scores are model opinions and require calibration against human decisions. Use least-privilege keys and workspaces, conditional tracing, redaction, explicit retention, deletion tests, evaluator versioning, spend limits, and human review before treating a dashboard score as a release decision.
- Follow your organisation's approved-use policy.
- Never treat fluent output as authorization to act.
Core workflows
Step-by-step ways to use LangSmith for the work it does best.
Each workflow below has its own outcome, steps and review checkpoints.
Turn representative failures into a versioned dataset and calibrated release gate instead of relying on anecdotal prompt testing.
↓ 02 WorkflowOperate privacy-aware LangSmith production tracingCapture enough agent behavior to debug incidents while minimizing sensitive content, retention, access, and evaluator cost.
↓Workflow 01
Create a release evaluation in LangSmith
Turn representative failures into a versioned dataset and calibrated release gate instead of relying on anecdotal prompt testing.
Outcome: Compare a baseline and candidate with evidence strong enough to support a documented deployment decision.
- 01
Define the decision, user segments, critical tasks, prohibited failures, quality dimensions, latency and cost limits, evaluator owners, thresholds, and rollback rule.
- 02
Build a versioned dataset from authorized synthetic, expert, and production-derived cases, removing secrets and personal data and preserving difficult counterexamples.
- 03
Add deterministic code checks, structured human rubrics, and calibrated model judges, recording judge model, prompt, repetitions, known bias, and disagreement policy.
- 04
Run baseline and candidate with identical data, concurrency, caching, and metadata, then inspect item-level regressions, confidence, variance, failure clusters, latency, and cost.
- 05
Require owner sign-off on thresholds and exceptions, attach the experiment to the release record, canary the candidate, and feed verified production failures back into the dataset.
What good use looks like
- Version data and evaluators together.
- Inspect failures beyond averages.
- Calibrate model judges against humans.
Workflow 02
Operate privacy-aware LangSmith production tracing
Capture enough agent behavior to debug incidents while minimizing sensitive content, retention, access, and evaluator cost.
Outcome: Maintain useful production observability without creating an uncontrolled secondary store of prompts, documents, or credentials.
- 01
Map every field that tracing can capture, classify sensitivity, choose cloud or self-hosted boundaries, and define sampling, retention, deletion, access, incident, and legal policies.
- 02
Use separate scoped keys and projects per environment, redact before export, disable tracing conditionally for sensitive tenants, and attach only non-secret version and routing metadata.
- 03
Sample normal traffic and fully capture approved error classes, validate parent-child spans and tool outcomes, and alert on safety, schema, latency, cost, and failure signals.
- 04
Restrict workspace and dataset access, review online evaluator filters and spend, test trace and dataset deletion, and verify that tracing can be disabled during an incident.
- 05
Audit access, sampling, retention, redaction, evaluator versions, and unresolved traces regularly, exporting only necessary evidence and documenting every policy change.
What good use looks like
- Redact before traces leave the app.
- Separate environments and tenants.
- Test deletion and tracing shutdown.
Official references
Check the current product documentation.
Features, plan limits, availability, and data controls change. These official pages are the starting points used for this guide.
- LangSmith observability quickstart ↗
- LangSmith evaluation ↗
- Data storage and privacy ↗
- Data purging for compliance ↗