Tracing - Scorecard Docs

New to Scorecard? Head straight to the Tracing Quickstart or jump into our ready-to-run Google Colab notebook to see traces in under 5 minutes.

LLM observability means knowing exactly what happened in every generation: latency, token usage, prompts, completions, cost, and errors. Scorecard’s Tracing captures all of this automatically and visualises it in the Scorecard UI. Send traces through SDK wrappers, environment variables, OpenLLMetry, an LLM proxy, or direct OpenTelemetry.

Why Tracing matters

Debug long or failing requests in seconds.
Audit prompts & completions for compliance and safety.
Attribute quality and cost back to specific services or users.
Feed production traffic into evaluations.

If you call it something else

Observability / AI spans / request logs: We capture standard OpenTelemetry traces and spans for LLM calls and related operations.
Agent runs / tools / function calls: These appear as nested spans in the trace tree, with inputs/outputs when available.
Prompt/Completion pairs: Extracted from common keys (openinference.*, ai.prompt / ai.response, gen_ai.*) so they can be turned into testcases and scored.

Instrumentation Methods

Choose the method that fits your stack. All methods send traces to Scorecard’s OpenTelemetry endpoint.

Agent Frameworks
SDK Wrappers
Traceloop
LLM Proxy
OpenTelemetry

For frameworks with built-in OpenTelemetry support (Claude Agent SDK, OpenAI Agents SDK). No code changes—just set environment variables.

export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <your-scorecard-api-key>"
export ENABLE_BETA_TRACING_DETAILED=1
export BETA_TRACING_ENDPOINT="https://tracing.scorecard.io/otel"
export OTEL_LOG_USER_PROMPTS=1
export OTEL_LOG_TOOL_DETAILS=1
export OTEL_LOG_TOOL_CONTENT=1
export OTEL_RESOURCE_ATTRIBUTES="scorecard.project_id=<your-project-id>"

# No code changes required — just set the env vars above and run your app.
import anyio
from claude_agent_sdk import (
    AssistantMessage,
    TextBlock,
    query,
)

async def main():
    async for message in query(prompt="What is 2 + 2?"):
        if isinstance(message, AssistantMessage):
            for block in message.content:
                if isinstance(block, TextBlock):
                    print(f"Claude: {block.text}")

anyio.run(main)

// No code changes required — just set the env vars above and run your app.
import {
  AssistantMessage,
  TextBlock,
  query,
} from "@anthropic-ai/claude-agent-sdk";

async function main() {
  const stream = query({
    prompt: "What is 2 + 2?",
  });

  for await (const message of stream) {
    if (message instanceof AssistantMessage) {
      for (const block of message.content) {
        if (block instanceof TextBlock) {
          console.log("Claude:", block.text);
        }
      }
    }
  }
}

main();

See the Claude Agent SDK Tracing guide for details.

One-line wrapper for OpenAI, Anthropic, or Vercel AI SDK clients. Install the scorecard-ai package and wrap your client—no other changes needed.

pip install 'scorecard-ai[otel]'

npm install scorecard-ai

OpenAI

from scorecard_ai import wrap
from openai import OpenAI
import os

openai = wrap(
    OpenAI(api_key=os.getenv("OPENAI_API_KEY")),
)

response = openai.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is the capital of France?"},
    ],
)

print(response.choices[0].message.content)

import { wrap } from 'scorecard-ai';
import OpenAI from 'openai';

const openai = wrap(
  new OpenAI({
    apiKey: process.env['OPENAI_API_KEY'],
  }),
);

const response = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [
    { role: 'system', content: 'You are a helpful assistant.' },
    { role: 'user', content: 'What is the capital of France?' },
  ],
});

console.log(response.choices[0]?.message?.content);

Anthropic

from scorecard_ai import wrap
from anthropic import Anthropic
import os

claude = wrap(
    Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY")),
)

response = claude.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    messages=[{"role": "user", "content": "What is the capital of France?"}],
)

text_content = next(
    (block for block in response.content if block.type == "text"), None
)
print(text_content.text if text_content else "No response")

import { wrap } from 'scorecard-ai';
import Anthropic from '@anthropic-ai/sdk';

const claude = wrap(
  new Anthropic({
    apiKey: process.env['ANTHROPIC_API_KEY'],
  }),
);

const response = await claude.messages.create({
  model: 'claude-sonnet-4-20250514',
  max_tokens: 1024,
  messages: [{ role: 'user', content: 'What is the capital of France?' }],
});

const textContent = response.content.find((block) => block.type === 'text');
console.log(textContent?.text);

Vercel AI SDK (TypeScript only)

TypeScript

import { openai } from '@ai-sdk/openai';
import * as ai from 'ai';
import { wrapAISDK } from 'scorecard-ai';

const aiSDK = wrapAISDK(ai);

const { text } = await aiSDK.generateText({
  model: openai('gpt-4o'),
  prompt: 'What is the capital of France?',
});

console.log(text);

See the Tracing Quickstart and Vercel AI SDK wrapper for full setup details.

Universal OpenTelemetry-based tracing via OpenLLMetry. Supports LangChain, LlamaIndex, CrewAI, and 20+ providers.

pip install traceloop-sdk

npm install @traceloop/node-server-sdk

Set these environment variables:

export TRACELOOP_BASE_URL="https://tracing.scorecard.io/otel"
export TRACELOOP_HEADERS="Authorization=Bearer%20<your-scorecard-api-key>"

from traceloop.sdk import Traceloop
from traceloop.sdk.decorators import workflow
from traceloop.sdk.instruments import Instruments
from openai import OpenAI

Traceloop.init(disable_batch=True, instruments={Instruments.OPENAI})

client = OpenAI()

@workflow(name="simple_chat")
def chat():
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "Hello!"}]
    )
    return response.choices[0].message.content

result = chat()
print(result)

import * as traceloop from "@traceloop/node-server-sdk";
import OpenAI from "openai";

traceloop.initialize({
  disableBatch: true,
  instrumentModules: { openAI: OpenAI },
});

const openai = new OpenAI();

async function chat() {
  return await traceloop.withWorkflow({ name: "simple_chat" }, async () => {
    const response = await openai.chat.completions.create({
      model: "gpt-4o",
      messages: [{ role: "user", content: "Hello!" }],
    });
    return response.choices[0].message.content;
  });
}

const result = await chat();
console.log(result);

LangChain — add opentelemetry-instrumentation-langchain and use Instruments.LANGCHAIN:

Python

from traceloop.sdk import Traceloop
from traceloop.sdk.instruments import Instruments

# IMPORTANT: Initialize Traceloop before importing any LangChain modules.
Traceloop.init(
    disable_batch=True,
    instruments={Instruments.LANGCHAIN}
)

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful assistant."),
    ("user", "{input}")
])

model = ChatOpenAI(model="gpt-4o-mini")
chain = prompt | model

response = chain.invoke({"input": "What is the capital of France?"})
print(response.content)

LlamaIndex — use Instruments.LLAMA_INDEX:

Python

from traceloop.sdk import Traceloop
from traceloop.sdk.instruments import Instruments
from llama_index.llms.openai import OpenAI
from llama_index.core.llms import ChatMessage

Traceloop.init(
    disable_batch=True,
    instruments={Instruments.LLAMA_INDEX, Instruments.OPENAI}
)

llm = OpenAI(model="gpt-4o")
response = llm.chat([ChatMessage(role="user", content="Hello!")])

print(response.message.content)

CrewAI — use Instruments.CREWAI:

Python

from traceloop.sdk import Traceloop
from traceloop.sdk.instruments import Instruments
from crewai import Agent, Task, Crew

Traceloop.init(
    disable_batch=True,
    instruments={Instruments.CREWAI, Instruments.OPENAI}
)

researcher = Agent(
    role="Researcher",
    goal="Research and summarize topics",
    backstory="You are an expert researcher.",
)

task = Task(
    description="What is the capital of France?",
    expected_output="A brief answer",
    agent=researcher,
)

crew = Crew(agents=[researcher], tasks=[task])
result = crew.kickoff()

print(result)

See the LangChain Quickstart and supported frameworks and providers below.

Change your baseURL to https://llm.scorecard.io. No SDK, no dependencies—just a URL change.

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url="https://llm.scorecard.io",
    default_headers={
        "x-scorecard-api-key": os.environ["SCORECARD_API_KEY"],
    },
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
)

print(response.choices[0].message.content)

import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  baseURL: 'https://llm.scorecard.io',
  defaultHeaders: {
    'x-scorecard-api-key': process.env.SCORECARD_API_KEY,
  },
});

const response = await client.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'What is the capital of France?' }],
});

console.log(response.choices[0].message.content);

See the Tracing Quickstart for full setup details.

Send traces directly using any OpenTelemetry SDK (Python, TypeScript, Java, Go, .NET, Rust).

Endpoint: https://tracing.scorecard.io/otel/v1/traces
Auth: Authorization: Bearer <SCORECARD_API_KEY>

pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-grpc

npm install @opentelemetry/api @opentelemetry/sdk-trace-node @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-http @opentelemetry/resources @opentelemetry/semantic-conventions

from anthropic import Anthropic
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.semconv.resource import ResourceAttributes
import os

resource = Resource.create({
    ResourceAttributes.SERVICE_NAME: "my-app",
})

trace_provider = TracerProvider(resource=resource)
trace_provider.add_span_processor(BatchSpanProcessor(
    OTLPSpanExporter(
        endpoint="https://tracing.scorecard.io/otel",
        headers={"Authorization": f"Bearer {os.environ['SCORECARD_API_KEY']}"}
    )
))
trace.set_tracer_provider(trace_provider)

tracer = trace.get_tracer(__name__)
client = Anthropic()

@tracer.start_as_current_span("chat")
def chat(prompt: str) -> str:
    span = trace.get_current_span()
    span.set_attributes({
        "gen_ai.system": "anthropic",
        "gen_ai.request.model": "claude-sonnet-4-20250514",
    })

    message = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )

    return message.content[0].text

result = chat("What is the capital of France?")
print(result)

trace.get_tracer_provider().force_flush()

import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { Resource } from "@opentelemetry/resources";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { ATTR_SERVICE_NAME } from "@opentelemetry/semantic-conventions";
import { trace, SpanStatusCode } from "@opentelemetry/api";
import OpenAI from "openai";

const exporter = new OTLPTraceExporter({
  url: "https://tracing.scorecard.io/otel/v1/traces",
  headers: {
    Authorization: `Bearer ${process.env.SCORECARD_API_KEY}`,
  },
});

const tracerProvider = new NodeTracerProvider({
  resource: Resource.default().merge(
    new Resource({ [ATTR_SERVICE_NAME]: "my-app" })
  ),
});

tracerProvider.addSpanProcessor(new BatchSpanProcessor(exporter));
tracerProvider.register();

const tracer = trace.getTracer("my-app");
const openai = new OpenAI();

async function chat(prompt: string) {
  return tracer.startActiveSpan("chat", async (span) => {
    span.setAttribute("gen_ai.system", "openai");
    span.setAttribute("gen_ai.request.model", "gpt-4o");

    const completion = await openai.chat.completions.create({
      model: "gpt-4o",
      messages: [{ role: "user", content: prompt }],
    });

    span.setStatus({ code: SpanStatusCode.OK });
    span.end();
    return completion.choices[0].message.content;
  });
}

const result = await chat("What is the capital of France?");
console.log(result);

await tracerProvider.forceFlush();

View Traces in the Records Page

Ingested traces appear on the Records page alongside records from other sources (API, Playground, Kickoff). Each row in the table shows: Each row shows the record’s inputs, outputs, status, source, trace ID, and metric scores. You can search by inputs, outputs, and expected values, or filter by status, source, trace.id, run.id, testcase.id, metric, metadata, time range, and more. Use Quick Filters for presets like “My Recent Records”. Customise visible columns via the Edit Table modal. Open a record to see its trace detail panel, which breaks the trace into four tabs: Conversation, Timeline, Chat, and Debug. For chat-based traces, the Conversation tab replays the trace as user messages, assistant responses, and tool calls: The Debug tab pairs the full span tree with a trace overview (duration, cost, tokens, and model):

Trace Grouping

When running batch operations or multi-step workflows, you can group related traces into a single run using the scorecard.tracing_group_id span attribute. This makes it easier to track and analyze workflows that span multiple LLM calls.

How it works

Add the scorecard.tracing_group_id attribute to your spans with a shared identifier. Scorecard automatically groups spans with the same group ID into a single run.

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

# Use the same group_id for all related operations
group_id = "batch-job-123"

with tracer.start_as_current_span("process_document") as span:
    span.set_attribute("scorecard.tracing_group_id", group_id)
    # Your LLM call here

with tracer.start_as_current_span("summarize_results") as span:
    span.set_attribute("scorecard.tracing_group_id", group_id)
    # Another LLM call in the same workflow

import { trace } from '@opentelemetry/api';

const tracer = trace.getTracer('my-service');

// Use the same groupId for all related operations
const groupId = 'batch-job-123';

const span1 = tracer.startSpan('process_document');
span1.setAttribute('scorecard.tracing_group_id', groupId);
// Your LLM call here
span1.end();

const span2 = tracer.startSpan('summarize_results');
span2.setAttribute('scorecard.tracing_group_id', groupId);
// Another LLM call in the same workflow
span2.end();

Use cases

Batch processing: Group all documents processed in a single batch job
Multi-step agents: Track all LLM calls within an agent’s execution
Workflows: Link related operations across different services
A/B testing: Group traces by experiment variant for comparison

Traces with the same scorecard.tracing_group_id will appear together in the same run, making it easy to analyze aggregate metrics across related operations.

AI-Specific Error Detection

Scorecard’s tracing detects AI-specific behavioral issues that standard observability misses, not just technical failures. It analyzes every interaction for both technical errors and behavioral problems.

Silent Failure Detection

A “silent failure” is when your AI responds, but incorrectly. These often go unnoticed without AI-specific observability. Scorecard automatically detects behavioral errors including off-topic responses, workflow interruptions, safety violations, hallucinations, and context loss. Technical errors like rate limits, timeouts, and API failures are captured automatically through standard trace error recording. AI applications also face issues that error logs don’t capture: semantic drift, safety policy violations, factual accuracy problems, and task completion failures.

Custom Error Detection

Create custom metrics in Scorecard’s UI to catch application-specific issues. An AI-powered metric can analyze trace data for things like off-topic responses, safety violations, or task completion failures, then automatically flag the traces that match.

Supported Frameworks & Providers

Scorecard traces LLM applications built with popular open-source frameworks through OpenLLMetry. OpenLLMetry provides automatic instrumentation for:

Application Frameworks

CrewAI – Multi-agent collaboration
Haystack – Search and question-answering pipelines
LangChain – Chains, agents, and tool calls
Langflow – Visual workflow builder
LangGraph – Multi-step workflows and state machines
LiteLLM – Unified interface for 100+ LLMs
LlamaIndex – RAG pipelines and document retrieval
OpenAI Agents SDK – Assistants API and function calling
Vercel AI SDK – Full-stack AI applications

Scorecard is featured as a recommended observability provider in the official Vercel AI SDK documentation and OpenAI Agents Python README.

LLM Providers

Aleph Alpha
Anthropic
AWS Bedrock
AWS SageMaker
Azure OpenAI
Cohere
Google Gemini
Google Vertex AI
Groq
HuggingFace
IBM Watsonx AI
Mistral AI
Ollama
OpenAI
Replicate
Together AI
and more

Vector Databases

Chroma
LanceDB
Marqo
Milvus
Pinecone
Qdrant
Weaviate

For the complete list of supported integrations, see the OpenLLMetry repository. All integrations are built on OpenTelemetry standards and maintained by the community.

Custom Providers

For frameworks or providers not listed above, you can use:

HTTP Instrumentation: OpenLLMetry’s instrument_http() for HTTP-based APIs
Manual Spans: Emit custom OpenTelemetry spans for proprietary systems

See the OpenLLMetry documentation for manual instrumentation guides.

Use cases

Production observability for LLM quality and safety
Debugging slow/failed requests with full span context
Auditing prompts/completions for compliance
Attributing token cost and latency to services/cohorts
Building evaluation datasets from real traffic (Trace to Testcase)

Next steps

Follow the Tracing Quickstart to send your first trace.
Set up Claude Agent SDK Tracing with zero code changes.
Instrument LangChain chains and agents.
Use the Vercel AI SDK wrapper for full-stack AI apps.
Open the Colab notebook for an interactive tour.

Happy tracing! 🚀

​Why Tracing matters

​If you call it something else

​Instrumentation Methods

​View Traces in the Records Page

​Trace Grouping

​How it works

​Use cases

​AI-Specific Error Detection

​Silent Failure Detection

​Custom Error Detection

​Supported Frameworks & Providers

​Application Frameworks

​LLM Providers

​Vector Databases

​Custom Providers

​Use cases

​Next steps

Why Tracing matters

If you call it something else

Instrumentation Methods

View Traces in the Records Page

Trace Grouping

How it works

Use cases

AI-Specific Error Detection

Silent Failure Detection

Custom Error Detection

Supported Frameworks & Providers

Application Frameworks

LLM Providers

Vector Databases

Custom Providers

Use cases

Next steps