# Assign Record User Source: https://docs.scorecard.io/api-reference/assign-record-user post /records/{recordId}/assignees Assign an organization member to a Record. Idempotent: re-assigning an existing member returns the existing assignment. # Commit Attachment Upload Source: https://docs.scorecard.io/api-reference/commit-attachment-upload post /attachments/{attachmentId}/commit Finalizes an upload after the file bytes have been PUT to the signed upload URL. Verifies the object landed in storage before the attachment starts describing the new content. Committing an already-committed attachment is a no-op. # Create Metric Source: https://docs.scorecard.io/api-reference/create-metric post /projects/{projectId}/metrics Create a new Metric for evaluating system outputs. The structure of a metric depends on the evalType and outputType of the metric. # Create Metric Group Source: https://docs.scorecard.io/api-reference/create-metric-group post /projects/{projectId}/metric-groups Create a new Metric Group referencing Metrics in the same Project. # Create multiple Testcases Source: https://docs.scorecard.io/api-reference/create-multiple-testcases post /testsets/{testsetId}/testcases Create multiple Testcases in the specified Testset. # Create Project Source: https://docs.scorecard.io/api-reference/create-project post /projects Create a new Project. # Create Record Source: https://docs.scorecard.io/api-reference/create-record post /runs/{runId}/records Create a new Record in a Run. # Create Record Tag Source: https://docs.scorecard.io/api-reference/create-record-tag post /records/{recordId}/tags Apply a tag to a Record. Idempotent: re-applying an existing tag returns the existing tag. # Create Run Source: https://docs.scorecard.io/api-reference/create-run post /projects/{projectId}/runs Create a new Run. # Create Testset Source: https://docs.scorecard.io/api-reference/create-testset post /projects/{projectId}/testsets Create a new Testset for a Project. The Testset will be created in the Project specified in the path. # Create (upsert) system Source: https://docs.scorecard.io/api-reference/create-upsert-system post /projects/{projectId}/systems Create a new system. If one with the same name in the project exists, it updates it instead. # Delete Attachment Source: https://docs.scorecard.io/api-reference/delete-attachment delete /attachments/{attachmentId} Deletes an attachment: both the stored file and its metadata. # Delete Metric Source: https://docs.scorecard.io/api-reference/delete-metric delete /metrics/{metricId} Delete a specific Metric by ID. The metric will be removed from metric groups and monitors. # Delete Metric Group Source: https://docs.scorecard.io/api-reference/delete-metric-group delete /metric-groups/{metricGroupId} Delete a specific Metric Group by ID. The Metrics in the group are not deleted. # Delete multiple Testcases Source: https://docs.scorecard.io/api-reference/delete-multiple-testcases post /testcases/bulk-delete Delete multiple Testcases by their IDs. # Delete Record Source: https://docs.scorecard.io/api-reference/delete-record delete /records/{recordId} Delete a specific Record by ID. # Delete Record Tag Source: https://docs.scorecard.io/api-reference/delete-record-tag delete /records/{recordId}/tags/{text} Remove a tag from a Record by its text. # Delete system Source: https://docs.scorecard.io/api-reference/delete-system delete /systems/{systemId} Delete a system definition by ID. This will not delete associated system versions. # Delete Testset Source: https://docs.scorecard.io/api-reference/delete-testset delete /testsets/{testsetId} # Get Attachment Source: https://docs.scorecard.io/api-reference/get-attachment get /attachments/{attachmentId} Retrieves an attachment's metadata and a short-lived signed download URL for its content. # Get Metric Source: https://docs.scorecard.io/api-reference/get-metric get /metrics/{metricId} Retrieve a specific Metric by ID. # Get Metric Group Source: https://docs.scorecard.io/api-reference/get-metric-group get /metric-groups/{metricGroupId} Retrieve a specific Metric Group by ID. # Get Project Source: https://docs.scorecard.io/api-reference/get-project get /projects/{projectId} Retrieve a specific Project by ID. # Get Run Source: https://docs.scorecard.io/api-reference/get-run get /runs/{runId} Retrieve a specific Run by ID. # Get system Source: https://docs.scorecard.io/api-reference/get-system get /systems/{systemId} Retrieve a specific system by ID. # Get system version Source: https://docs.scorecard.io/api-reference/get-system-version get /systems/versions/{systemVersionId} Retrieve a specific system version by ID. # Get Testcase Source: https://docs.scorecard.io/api-reference/get-testcase get /testcases/{testcaseId} Retrieve a specific Testcase by ID. # Get Testset Source: https://docs.scorecard.io/api-reference/get-testset get /testsets/{testsetId} # Initiate Attachment Upload Source: https://docs.scorecard.io/api-reference/initiate-attachment-upload post /attachments Initiates (or deduplicates) an upload of a file attached to a session. If the exact content is already stored for this (session ID, file path), the response has `alreadyExists: true` and no upload is needed. Otherwise, PUT the file bytes to the returned `uploadUrl`, then call the commit endpoint. Re-initiating an existing (session ID, file path) with new content updates the attachment in place on commit. # List Annotations Source: https://docs.scorecard.io/api-reference/list-annotations get /records/{recordId}/annotations List all annotations (ratings and comments) for a specific Record. # List Metric Groups Source: https://docs.scorecard.io/api-reference/list-metric-groups get /projects/{projectId}/metric-groups List Metric Groups configured for the specified Project. Metric Groups are returned in reverse chronological order. # List Metrics Source: https://docs.scorecard.io/api-reference/list-metrics get /projects/{projectId}/metrics List Metrics configured for the specified Project. Metrics are returned in reverse chronological order. # List Projects Source: https://docs.scorecard.io/api-reference/list-projects get /projects Retrieve a paginated list of all Projects. Projects are ordered by creation date, with oldest Projects first. # List Record Assignees Source: https://docs.scorecard.io/api-reference/list-record-assignees get /records/{recordId}/assignees List the organization members assigned to a Record. # List Record Tags Source: https://docs.scorecard.io/api-reference/list-record-tags get /records/{recordId}/tags List all tags applied to a specific Record. # List Records and Scores Source: https://docs.scorecard.io/api-reference/list-records get /runs/{runId}/records Retrieve a paginated list of Records for a Run, including all scores for each record. # List Runs Source: https://docs.scorecard.io/api-reference/list-runs get /projects/{projectId}/runs Retrieve a paginated list of all Runs for a Project. Runs are ordered by creation date, most recent first. # List Session Attachments Source: https://docs.scorecard.io/api-reference/list-session-attachments get /sessions/{sessionId}/attachments Lists the uploaded attachments for a session. Only committed attachments are returned. # List systems Source: https://docs.scorecard.io/api-reference/list-systems get /projects/{projectId}/systems Retrieve a paginated list of all systems. Systems are ordered by creation date. # List Testcases in Testset Source: https://docs.scorecard.io/api-reference/list-testcases-in-testset get /testsets/{testsetId}/testcases Retrieve a paginated list of Testcases belonging to a Testset. # List Testsets in Project Source: https://docs.scorecard.io/api-reference/list-testsets-in-project get /projects/{projectId}/testsets Retrieve a paginated list of Testsets belonging to a Project. # API Reference Source: https://docs.scorecard.io/api-reference/overview Browse the API documentation and integrate with Scorecard's API endpoints. ## Get access to the Scorecard API [Create a Scorecard account](https://accounts.scorecard.io/sign-up?redirect_url=https%3A%2F%2Fapp.scorecard.io%2F) and find your organization's `SCORECARD_API_KEY` in [settings](https://app.scorecard.io/settings). ## Use Scorecard's SDKs > **[Python SDK](https://github.com/scorecard-ai/scorecard-python)** > > `pip install scorecard-ai` to interact with our API. > **[Node.js SDK](https://github.com/scorecard-ai/scorecard-node)** > > `npm install scorecard-ai` to interact with our API. ### Reporting issues Please report any issues or feature requests with the SDKs or documentation on GitHub: * [File issue with Python SDK](https://github.com/scorecard-ai/scorecard-python/issues) * [File issue with Node.js SDK](https://github.com/scorecard-ai/scorecard-node/issues) The Scorecard team supports you in your LLM evaluation journey. Feel free to [email us](mailto:team@scorecard.io); we're here to help! # Unassign Record User Source: https://docs.scorecard.io/api-reference/unassign-record-user delete /records/{recordId}/assignees/{assigneeUserId} Remove an assignee from a Record. # Update Metric Source: https://docs.scorecard.io/api-reference/update-metric patch /metrics/{metricId} Update an existing Metric. You must specify the evalType and outputType of the metric. The structure of a metric depends on the evalType and outputType of the metric. # Update Metric Group Source: https://docs.scorecard.io/api-reference/update-metric-group patch /metric-groups/{metricGroupId} Update a Metric Group's name, description, or member Metrics. The `metricIds` array replaces the group's current set of Metrics. # Update Project Source: https://docs.scorecard.io/api-reference/update-project patch /projects/{projectId} Update a Project's name, description, or review status. Changing `status` requires an organization admin: an organization API key, or a token belonging to an admin. Keys and tokens scoped to an individual member receive a 403. `statusUpdatedAt` moves only when the status actually changes. # Update system Source: https://docs.scorecard.io/api-reference/update-system patch /systems/{systemId} Update an existing system. Only the fields provided in the request body will be updated. If a field is provided, the new content will replace the existing content. If a field is not provided, the existing content will remain unchanged. # Update Testcase Source: https://docs.scorecard.io/api-reference/update-testcase put /testcases/{testcaseId} Replace the data of an existing Testcase while keeping its ID. # Update Testset Source: https://docs.scorecard.io/api-reference/update-testset patch /testsets/{testsetId} Update a Testset. Only the fields provided in the request body will be updated. If a field is provided, the new content will replace the existing content. If a field is not provided, the existing content will remain unchanged. When updating the schema: - If field mappings are not provided and existing mappings reference fields that no longer exist, those mappings will be automatically removed - To preserve all existing mappings, ensure all referenced fields remain in the updated schema - For complete control, provide both schema and fieldMapping when updating the schema # Upsert Score Source: https://docs.scorecard.io/api-reference/upsert-score put /records/{recordId}/scores/{metricConfigId} Create or update a Score for a given Record and MetricConfig. If a Score with the specified Record ID and MetricConfig ID already exists, it will be updated. Otherwise, a new Score will be created. The score provided should conform to the schema defined by the MetricConfig; otherwise, validation errors will be reported. Omit `score` to leave the Score in a pending (ungraded) state, which queues the Record for manual grading; this is supported only for human metrics. # Upsert system version Source: https://docs.scorecard.io/api-reference/upsert-system-version post /systems/{systemId}/versions Create a new system version if it does not already exist. Does **not** set the created version to be the system's production version. If there is already a system version with the same config, its name will be updated. # Product Updates Source: https://docs.scorecard.io/changelog New updates and improvements **September 21, 2026** ### Bug Fixes and Improvements * **\[Playground]** The Playground has been removed. Test a prompt version against a testset with a [Kickoff Run](/features/runs#kickoff-run-from-the-ui) instead. * **\[Endpoints]** Custom Endpoints have been removed. To evaluate an HTTP API, call it from your own code and record the result with the [SDK](/intro/sdk-quickstart). * **\[Sim Agents]** Sim Agents and multi-turn simulation have been removed, along with the `POST /agents/{systemVersionId}/simulate` API endpoint. The Python SDK's `multi_turn_simulation()` helper depends on that endpoint and no longer works. * **\[Records]** Runs and records created by these surfaces are unchanged and still visible on the [Records](/features/records) page. Older records keep their **Playground** source label. * **\[Systems]** Systems that backed a Sim Agent or a Custom Endpoint remain in the database and are reachable through the REST API, but no longer appear in the Systems list. **August 14, 2026** ### Bug Fixes and Improvements * **\[Records]** Typing a human score and then clicking outside no longer throws the draft away. Reopen the cell and your Pass/Fail pick and reasoning are still in the form. * **\[Records]** Re-running scoring on a record leaves a completed [human score](/features/metrics) alone, instead of blanking it back to an empty **Add score** cell. * **\[Records]** The annotations popover shows a highlighted passage as a quote above its note, rather than one run-on block of text. Long lists of annotations scroll instead of running off the screen. * **\[Models]** `claude-opus-5` and `gemini-3.7-flash` are in the model selector, and both are now the suggested pick for their tier. * **\[Projects]** The [project](/features/projects) list loads faster, and its testset and metric counts now count only your org's items. * **\[Records]** Record counts on large projects load in well under a second. * **\[Sessions]** Attachment lists load faster, most noticeably on sessions with hundreds of files. **July 31, 2026** ### Bug Fixes and Improvements * **\[Compare]** Each record card has a **Scores** tab, so you can compare records on quality. Add a score or record a human score inline without leaving the page. * **\[Compare]** Chat over both compared records' traces at once. The answer labels each side (Trace A / Trace B) and its span citations link back to that record. * **\[Compare]** A **Model preference** panel under the two cards lets you pick Record A or Record B for each human binary metric they share, writing a pass to the chosen side and a fail to the other. * **\[Compare]** Pick each record from a per-slot table with search, toggleable columns, and infinite scroll, and filter it by clicking a metric's pass/fail badge or the **Assigned to Me** toggle. The per-card **Trajectory** tab is gone. * **\[Compare]** A shared **Expected** column sits between the two cards in place of the per-card Expected tab, and labels both values when they differ. * **\[Records]** New **Assigned to Me** quick filter, and bulk **assign / unassign** across selected records from the row-selection menu in a single request. * **\[Records]** Filter records by a specific metric's **pass**, **fail**, or **pending** status. * **\[Records]** Export your columns and widgets as JSON and import them into another project to reuse a view. * **\[Records]** One **Columns** control now toggles and drag-reorders columns in a single popover, replacing the Edit Table modal. Tag chips in the table are clickable to filter. * **\[Records]** An **Assigned to** widget on the overview, and the `assigned.to` filter pill shows the member's avatar and name instead of a raw id. * **\[Records]** The record details panel scrolls as one region so nothing gets clipped, and pending score rows are static and neutral instead of pulsing. * **\[Playground]** Added **Claude Sonnet 5** and **Claude Fable 5**, and refreshed model pricing and context-window data. * **\[API]** New public endpoints to [assign](/api-reference/assign-record-user), [unassign](/api-reference/unassign-record-user), and [list](/api-reference/list-record-assignees) a record's assignees. **July 24, 2026** ### Compare records side by side A new **Compare** page puts records next to each other in full-width cards so you can read their outputs side by side, with each record's inputs and expected answer alongside. Select text in an output to annotate that passage. Saved annotations stay highlighted inline, and hovering one shows its author and comment. Comparing two records side by side on the Compare page ### Assign records to teammates Assign organization members to a [record](/features/records) from the picker in the record details panel, beside Tags. The records table has an optional **Assignees** column, and typing `assigned.to:` in the search bar narrows the table to one person's records. Assigning an organization member to a record from the details panel ### Import a testset from an agent transcript **Import from transcript** on a project's [Testsets](/features/testsets) page takes a pasted or dropped agent-session transcript, such as a Claude Code `session.jsonl`, and pulls out the moments where the agent should have acted differently. Each one becomes a testcase in a new testset, with the prompt as input and the expected behavior as the label. Analysis can take a couple of minutes. Closing the dialog doesn't cancel it, and a toast reports the result when it finishes. Importing a testset from an agent-session transcript ### Bug Fixes and Improvements * **\[Records]** Clicking a session attachment opens a preview sheet instead of downloading the file. Images, PDFs, video, audio, markdown, and code render inline, and download is a separate button. * **\[Records]** Markdown tables in record outputs render as tables instead of raw pipes. * **\[Onboarding]** The onboarding page shows the organization API key to admins again instead of "No API key found". * **\[API]** [Record creation](/api-reference/create-record) accepts an optional `sessionId`, so records created through the API or SDK join that session's traces and attachments. **July 17, 2026** ### Session attachments You can now attach files to a session and see them next to its [record](/features/records). Attachments appear in a **Files** card in the record details panel and in an **Attachments** tab, and any file an agent read or wrote shows up as a clickable chip inline in the conversation, on the turn that mentions it. Upload runs through the API: declare the file, upload the bytes, then commit. Attachments tab listing files attached to a session ### Records overview widgets The [Records](/features/records) page now has a widget board at the top that you can customize. Add a **Volume** widget to see record ingest over time, or a **Metric** widget to track a metric's pass rate — either as a snapshot or a stacked pass/fail/ungraded chart over time. Widgets follow the table's active filters, and your layout is saved per project. Widget board on the Records page with Volume and metric widgets ### Skills in Trace Chat Trace chat accepts skill attachments. Attach a `.skill` bundle or a plain `.md` file and the chat follows its instructions when it analyzes a trace — for example, checking a run against a bundled rubric. Skills stay attached per project and pull in their reference files as needed. Add skill control in the Trace Chat composer ### Bug Fixes and Improvements * **\[Records]** You can build filters by typing. Enter a field name and a colon (e.g. `status:`) to start a pill, and typed values like metric or creator names resolve to the right filter value when you commit. * **\[Records]** Fixed the filter popovers overlapping the field dropdown and closing when you clicked their scrollbar. **July 10, 2026** ### Bug Fixes and Improvements * **\[UI]** Removed the **Temperature**, **Maximum Length**, and **Top P** controls. New configurations run at each provider's defaults; existing versions keep their stored parameters, and you can still set them through the API and SDK. * **\[Records]** Scoring records with a hidden metric column now reveals that column automatically, with scores filling in as jobs finish. * **\[Onboarding]** The Claude Agent SDK snippets now include `OTEL_LOG_USER_PROMPTS`, `OTEL_LOG_TOOL_DETAILS`, and `OTEL_LOG_TOOL_CONTENT`. * **\[API]** Record creation (`POST /v2/runs/:runId/records`) now accepts bodies up to 15MB, and oversized requests return a `413` instead of a `500`. **May 29, 2026** ### 💬 Conversation View Is Now the Default The record details page now opens on the threaded **Conversation** view by default, so you land directly on the human-readable agent run instead of the raw span tree. The span tree + details view has been renamed to **Debug** and moved to the end of the tab order to signal it as a developer/inspection tool. Tabs now read **Conversation, Timeline, Chat, Scores, Debug**, and clicking a span from any tab still jumps into Debug with that span selected. ### 🔽 Expand All / Collapse All in Conversation View Two quality-of-life upgrades for the Conversation view: * **Expand all / Collapse all**: A new inline toggle in the Conversation tab header expands or collapses every tool step, compact-turn preview, and long user-turn "Show more" at once. Subagent steps nested inside an `Agent` tool call respect it too, and per-step buttons keep working normally after a bulk action. * **Persistent collapse state**: Expanding a step and then switching to Timeline, Chat, or Debug no longer resets everything — what you expanded stays expanded when you come back. **May 22, 2026** ### 📊 Annotations Column The Records page now has an opt-in **Annotations** column that surfaces reviewer feedback directly in the table. Each cell shows a compact badge with the total count and per-rating breakdown (👍 / 👎 / 💬). Click the badge to open a popover listing each annotation's author, rating, and comment — no need to open the record details. Annotations column on the Records page Toggle it on via **Edit Table → Additional → Annotations**. Annotations are joined into the records query in a single round-trip, so adding the column doesn't slow down page loads. ### 💬 Trace Comparison in Chat The Trace **Chat tab** now lets you load additional records — from the current project or any other project you have access to — and ask the model to compare them side-by-side. Use the new **Comparing** picker next to the model selector, multi-select the records you want, and the chat will diff the primary trace (labeled **A**) against the comparisons (**B**, **C**, **D**, …). Comparing two traces side-by-side in the Chat tab Useful for debugging regressions ("why did this run fail when the previous one passed?") or comparing approaches across projects. ### 🛠️ Bug Fixes and Improvements * **\[Records]** Resizing the record details right panel no longer stutters — width now updates via the DOM directly during drag, with no re-renders on the records table or history chart. * **\[AI]** Refreshed the LiteLLM model catalog so `gemini/gemini-3.5-flash` (the current default model) resolves correctly and no longer falls through to an error path for orgs without a custom-model override. **May 8, 2026** ### 💬 Conversation View Improvements Several upgrades to the threaded **Conversation view** on the record details page (Claude Code traces) to make debugging agent runs faster. * **Tool execution errors**: Failed tool calls now show a red **Error** pill on the row, a one-line truncated error preview when collapsed, and a full error block above the tool result when expanded. This catches a class of failures (like blocked `Agent` invocations) that previously rendered as normal successes because they signaled the failure via a `success: "false"` attribute instead of an OpenTelemetry error status. * **Inline sub-agent steps**: Sub-agent runs spawned by Claude Code's `Agent` tool are now rendered inline under the expanded parent tool call. Each nested LLM turn shows its header, thinking, and tool steps just like a top-level turn, and sub-agents that themselves invoke `Agent` nest recursively — so you can follow a multi-level agent run without leaving the conversation view. ### 🔗 Clickable Span Links in Trace Chat Span references that the **Chat tab** (the AI-powered trace summary) emits in its responses are now clickable. Clicking one jumps to the **Spans** tab, selects the matching span, and auto-scrolls the span tree to bring it into view. ### 📤 Export Annotations as JSON We replaced the old CSV/XLSX annotation exports with a single, much more useful **Export as JSON** option. Annotations are now inlined into the regular trace JSON: each annotation hangs off the exact span it was made on (record-level and orphan annotations attach to the trace itself), and each one carries the annotator's name. You get the full trace context alongside your labels in one file — ready for downstream analysis, training data, or review workflows. Reorganized Export menu with Records, Annotations, and Traces sections **April 10, 2026** ### 📊 Records Analysis The Records page now features an AI-powered **Records Analysis** banner that automatically spots patterns across your latest records — like high failure rates, scoring anomalies, factual hallucinations, or stuck records. Records Analysis banner on the Records page When you expand the banner, you'll see a list of insights ranked by severity (critical, warning, info), each with a description of the detected pattern. Click the **filter icon** next to any insight to instantly filter the records table to the relevant subset. The analysis automatically refreshes as new records come in, and you can manually trigger a refresh at any time. The footer shows when the analysis was last generated, how many records were analyzed, and how many unique patterns were detected. **February 27, 2026** ### 📊 Run Details Page We've made several improvements to the Run Details page for a better evaluation review experience: * **Resizable popovers**: The scoring popover and column cells can now be resized, giving you more room to inspect detailed outputs * **Markdown rendering in cells**: Text outputs inside column cells now render markdown for improved readability * **Markdown in scoring reasoning**: The reasoning section on the score popover now renders markdown, making structured explanations easier to follow ### 🔍 Traces * **Export annotations as CSV**: You can now export your trace annotations directly as a CSV file for offline analysis and reporting * **New Chat tab**: Traces now include a dedicated **Chat tab** that provides an AI-powered summary of the trace. Ask questions about the trace to quickly understand what happened without manually inspecting every span. Chat tab on the Trace details page ### 🚀 Records Page Performance * **Faster page loading**: We significantly improved the performance of the Records page, so it loads quicker even with large datasets * **Faster search**: Searching records on the page is now noticeably faster, helping you find what you need without delay **January 30, 2026** ### 💬 Conversation Tab The record details page now includes a dedicated **Conversation tab** that displays multi-turn interactions in an easy-to-read chat format. This makes it much simpler to review and debug conversational AI outputs without parsing through raw JSON. Conversation tab showing multi-turn interactions ### 📥 Records CSV Export You can now export records directly from the Records page as a CSV file. This makes it easy to share evaluation results with stakeholders, perform offline analysis, or integrate with external tools. Export records as CSV or Excel from the Records page ### 🔍 Arbitrary String Search in Records The Records page now supports full-text search across all record fields. Search for any substring in inputs, outputs, or expected values to quickly find specific records you're looking for. Search records by any field value **December 12, 2025** ### Records page The Records page is now more powerful for debugging and triaging large runs: you can pull useful fields into dedicated columns, sort and filter more precisely, and quickly jump to "what I just ran." * **Extract columns from inputs, outputs, and expected**: Expand inputs/outputs/expected objects into separate per-field columns so you can scan specific keys without opening each record. * **Reorder columns**: Reorder columns in the table editor (drag-and-drop) and save your preferred layout. * **Created At/By columns**: See when a record was created and who kicked off the run that produced it. * **Created At/By filters**: Filter by a time range and/or by the run creator to focus on the records you care about. * **Sort by Created At**: Sort the table by record creation time to see newest records first (and quick filters will set this automatically). * **New quick filters**: Narrow down to common views like **My Recent Records** with a new "Quick Filters" menu. Records page with extracted columns, saved layouts, and quick filters ### Simpler projects navigation We removed the standalone projects page and made project switching faster by showing project information in the top navigation. Project switching via the top navigation ### New metric templates We updated our built-in metric templates to help you get started faster with common evaluation goals (and reduce the amount of prompt engineering needed to create a high-signal judge). Our templates now include **Hallucination detection**, **Response completeness**, **PII leakage**, **Coherency**, **User intent fulfillment**, **Content moderation**, **Source attribution quality**, **Bias**, and **Conciseness**. New metric templates ### 🛠️ Bug Fixes and Improvements * Added a new "Help" menu in the sidebar with a "Send Feedback" option. * **\[New models]** Added support for GPT 5.2 released this week. **December 5, 2025** ### New playground We've introduced a workflow-based playground UI. It offers the same functionality as the previous playground, but with a more intuitive structure that makes the relationship between testcases, prompts, and results clearer. The new interface is still experimental and currently enabled only for newly created Scorecard organizations. If you'd like early access, let us know! We're iterating quickly based on feedback. New playground ### New records page We've added a consolidated Records page that lists every record across all runs. This unlocks new workflows, including viewing testcase performance history, identifying testcases that failed specific metrics, and filtering records by substring. Eventually, this will replace the existing Runs & Results page. Records page with filters ### Onboarding wizard When you create a new project, you'll now see a guided onboarding wizard to help you get started faster. Based on your use case, we recommend an onboarding path (testsets, tracing, or SDK) and suggest a set of metrics to begin with. Create project wizard ### 🛠️ Bug Fixes and Improvements * **\[UI]** Descriptions are now optional when creating systems, endpoints, and metric groups. * **\[Metrics]** Replaced the "Create" and "Edit" metric modals with a new full-page metric editor. * **\[Metrics]** Added a "Recently used" badge to the metric card to quickly identify metrics that were recently used in a run. * **\[Testsets]** Improved testset schema modal by removing tabs. * **\[Runs]** Fixed newline handling when exporting a run (or testset) to Excel-compatible CSV. * **\[Tracing]** We removed the "Monitoring" page and combined it with the existing "Traces" page. * **\[UI]** Simplified the sidebar navigation by moving lower-traffic pages into an auto-collapsed Advanced section. * **\[UI]** Added page descriptions to headers throughout the app **November 21, 2025** ### 🤖 New Model Support Added support for the latest models, including **GPT 5.1** and **Gemini 3 Pro**. Try them out and let us know what you think! ### 🔎 Monitor Visibility on Traces Page The traces page now displays a table of active monitors, making it easier to configure and manage monitors. Table of active monitors on the traces page ### 🏃 Monitors "Run now" feature The "Run now" button on the monitors page allows you to run a monitor immediately, without having to wait for the next scheduled run. Now, the feature supports selecting a custom "look back" time range. ### 🔎 Filter Runs by Source You can now filter runs by their source (API, Monitor, Playground, or Kickoff), helping you quickly find runs created from specific workflows. Filter runs by source on the runs page ### 🛠️ Bug Fixes and Improvements * **\[Testsets]** Improved the testset schema editor for a better editing experience. * **\[Metrics]** Fixed the empty state display for the metric groups tab. * **\[Testsets]** Fixed a bug where editing different fields in the same testcase could lose the first change. * **\[Docs]** Improved navigation on this documentation site. * **\[Internal]** Upgraded to Next.js 16 for better performance. **November 14, 2025** This week, we focused on improving metrics. ### 🔄 Re-run Scoring for Existing Runs You can now re-run scoring with the latest version of a metric or add new metrics to existing runs without having to re-run your system. This makes it much faster to iterate on metrics! Re-run scoring for an existing run ### 🧑‍💻 Heuristic Code Runners Metrics now support custom code-based evaluation logic. This allows you to write your own evaluation logic in Python or Typescript. Create metric modal with Python heuristic code ### 📏 Improved Metric Details Page We overhauled the metric details page to make it easier to edit metrics. New metric details page With the metric template preview, you can see your instructions to the LLM-as-a-judge will render without running the metric. Metric template preview **"Recently used" badge**: Quickly identify metrics that were recently used in a run. ### 🔍 Improved Trace Search The traces page now supports filtering making it easier to find exactly what you're looking for. * **Trace ID**: Filter by the root trace ID * **Run ID**: Filter by the trace's run ID * **Service**: Filter by the OpenTelemetry service name * **Span Name**: Filter for runs containing a span with the given name Traces search with filter options ### 🚀 New API Endpoints Added a [Delete Metric](/api-reference/delete-metric) endpoint. ### 🛠️ Bug Fixes and Improvements * **\[Runs]** Added a "Notes" column to the runs table for better organization **November 7, 2025** This week, we focused on improving the tracing and monitoring features. ### 📊 Improved Trace Details UX The trace details page now features a **collapsible span tree** and better scrolling behavior for easier navigation through long, highly nested traces. Improved trace details UX with a collapsible span tree ### 🔗 Automatic Trace Grouping In your traces, you can use the `scorecard.tracing_group_id` span attribute to automatically group traces into runs based on a custom identifier. This makes it easier to track and analyze multi-step workflows or batch operations. ### 🎯 Cross-Project Monitoring If you choose, monitors can now pick up traces from any project in your organization, allowing you to set up centralized monitoring rules across your entire workspace. ### 🔎 Span Name Regex Filter for Monitors Added a **"Span name (regex)"** filter to monitors, giving you more precise control over which spans trigger your monitoring rules. New span name regex filter for monitors ### 📈 Linkable Metric Groups Metric groups are now full pages instead of modals, making it easy to share direct links to specific metric groups with your team. We'll continue to improve the metric creation experience next week. Metric groups are now full pages that can be linked to ### 🛠️ Bug Fixes and Improvements * **\[Traces]** Added a **Project ID column** to the traces table to help you quickly identify traces by project. * **\[Traces]** Fixed the trace chart visualization to show traces in the given time period, not just the traces shown in the trace table. * **\[API]** Fixed pagination issue where a cursor pointing to a nonexistent item would skip the first item in the result set. **October 31, 2025** ### 🚀 AI SDK Wrapper Launch We've launched a new AI SDK wrapper for seamless integration with Scorecard! The wrapper makes it easy to add evaluation and monitoring to your AI applications with minimal code changes. Learn more in our [AI SDK Wrapper documentation](/features/ai-sdk-wrapper). ### 📚 New Tracing Examples We've added comprehensive tracing code examples for instrumenting your applications with OpenTelemetry: * **[Pydantic and Logfire](https://github.com/scorecard-ai/scorecard-examples/tree/main/python-logfire-otel-basic)**: Python-based tracing with Pydantic validation and Logfire integration * **[Traceloop](https://github.com/scorecard-ai/scorecard-examples/tree/main/nodejs-traceloop-basic)**: Node.js tracing with Traceloop for easy LLM observability * **Manual OpenTelemetry**: * [Workflow example](https://github.com/scorecard-ai/scorecard-examples/tree/main/nodejs-otel-workflow): Complete workflow implementation * [Basic example](https://github.com/scorecard-ai/scorecard-examples/tree/main/nodejs-otel-basic): Getting started with OpenTelemetry ### 🛠️ Bug Fixes and Improvements * **\[API]** Added new [Delete Records](/api-reference/delete-record) endpoint to delete records, helping you keep your workspace clean and organized * **\[Runs]** Renamed "Trigger run" to "[Kickoff run](/features/runs#kickoff-run-from-the-ui)" for consistency across the platform * **\[Runs]** Changed the runs page to default to the "All runs" tab instead of "My runs" for better discoverability * **\[Runs]** Increased the character limit per cell when exporting a run as CSV from 32,000 to 128,000 characters * **\[Analysis]** Fixed a bug where the analysis page crashed when there are no metrics * **\[Playground]** Updated the delete icon for playground messages from a minus symbol to a trash can for better clarity * **\[Playground]** Added support for adding a new Provider directly in the playground instead of navigating to the settings page **October 24, 2025** ### 📚 Documentation Updates We've expanded our documentation with new guides and resources: * [MCP Quickstart](/intro/mcp-quickstart) - New guide for setting up the Scorecard MCP server in Claude (Web/Desktop) * Analysis - New documentation on the Analysis page to compare performance metrics side by side ### 🛠️ Bug Fixes and Improvements * **\[Testcases]** Added support for cross-project testcase copying, making it easier to share test cases across different projects. **October 17, 2025** ### 🚀 Product Hunt Launch We're excited to announce that Scorecard is now on Product Hunt! Check out our listing on [Product Hunt](https://www.producthunt.com/products/scorecard) to see what people are saying about Scorecard. ### 🛠️ Bug Fixes and Improvements * **\[Traces]** We now create testcases from *any* OpenTelemetry GenAI span in the tracing page, which extends our support for open standards in LLM observability. * **\[Run details]** When changing the prompt template or parameters in the playground, you no longer need to save the prompt before kicking off a run using those parameters. **October 10, 2025** ### 📊 Tracing We now automatically create traces for runs generated on Scorecard's Playground. ### 🚀 New API Endpoints We've added new API endpoints for better metric and record management: * [Get Metric](/api-reference/get-metric): Retrieve details for a specific metric. * [List Metrics](/api-reference/list-metrics): Retrieve details for all metrics in a project. * [List Records](/api-reference/list-records): Retrieve details for all records and scores in a run. ### 🔌 MCP server The Scorecard MCP server can now analyze runs and make suggestions to improve your system. ### 🎯 MCP evals We improved dataset generation in [mcpevals.ai](https://mcpevals.ai)! It now generates realistic tool calls to better test your MCP server. **October 3, 2025** ### 📊 Analysis Page We've released a new Analysis page that provides deeper insights into your evaluation data. ### 🔌 Scorecard MCP Server - Open Source We've released the updated source code for our MCP server on [GitHub](https://github.com/scorecard-ai/scorecard-mcp)! See how we integrated: * [Clerk](https://clerk.com) for authentication (OAuth 2.0 Protected Resource Metadata). * [Stainless](https://stainless.com) for generated MCP endpoints. * [Sentry](https://sentry.io) for production-grade monitoring and error tracking. ### 🚀 New API Endpoints We've added new API endpoints for better run management: * [Get Run](/api-reference/get-run): Retrieve details for a specific run. * [List Runs](/api-reference/list-runs): Retrieve details for all runs in a project. ### 📏 Correctness Metric Template A new float metric template called "Correctness" is now available, providing a standardized way to measure accuracy with decimal precision. ### 🛠️ Bug Fixes and Improvements * **\[Runs]** Testcase ID text in the record table now links directly to the record's testcase details page for easier navigation. * **\[Error Messages]** Improved error messaging for permissions failures throughout the UI. * **\[Scoring]** Exposed complete error details when scoring fails. For example, if a record exceeds the AI metric's context length, the full error message is now displayed instead of a generic "Internal server error". **September 26, 2025** ### 🎯 MCP Evaluations Platform Launch Introducing **MCP Evals** - a dedicated platform for evaluating Model Context Protocol servers! Test and benchmark MCP servers with standardized evaluation workflows at [mcpevals.ai](https://mcpevals.ai). MCP Evals results page showing performance metrics and evaluation scores Key features: * **Dynamic Testing**: AI-generated evaluation tests tailored to each server's capabilities with results in seconds. * **Performance Metrics**: Detailed capability assessments and server response times. * **Open Source**: Evaluation tools and methodologies available at our [GitHub repository](https://github.com/scorecard-ai/mcp-eval). ### 🎨 Enhanced Card Design We've refreshed our card design throughout the platform for better visual hierarchy, improved readability, and modern aesthetics. New metric card design ### 📊 Float Output Type Support Scorecard now supports float output types. This enhancement enables more precise measurement of continuous metrics and scoring systems that require decimal precision. New float output score of 0.90 along with an explanation. ### 🛠️ Bug Fixes and Improvements * **\[Exports]** Fixed encoding in CSVs when exporting testcases or records with accented characters like "è". * **\[Documentation]** Enhanced [testset documentation](/features/testsets#understanding-input-fields) with clearer guidance on structuring input fields and practical examples. **September 19, 2025** ### 🌐 Google Vertex AI Support Scorecard now fully supports Google Vertex AI models! Evaluate your applications using Google's latest Gemini models including Gemini 2.5 Pro, Gemini 2.5 Flash, and Gemini 2.0 Flash. This integration brings enterprise-grade AI capabilities to your evaluation workflows with seamless configuration through our settings panel. ### 🎮 Playground Enhancements Create new testsets and prompts directly from the [Playground](/features/playground) without leaving your workflow. We've added smart empty state buttons that help you get started quickly when no testsets or prompts exist. ### 🛠️ Bug Fixes and Improvements * **\[Data Import]** Added TSV file support for testcase uploads * **\[Data Import]** Column header names are now automatically trimmed during upload * **\[Testsets]** Description field is now optional when creating testsets **September 12, 2025** ### 🚀 Official MCP Registry Launch We're thrilled to announce that Scorecard is now officially part of the Model Context Protocol (MCP) server registry! As the 28th server registered, we're among the first wave of official MCP integrations, making Scorecard's evaluation capabilities accessible directly within your AI development workflow. Our MCP server is available across all major AI platforms: * **claude.ai**: Full integration for seamless evaluation workflows * **Cursor**: Test and evaluate your code directly in your IDE * **All MCP clients**: Universal compatibility with any MCP-enabled tool Key capabilities: * **Direct AI Integration**: Run evaluations without leaving your AI assistant * **Real-time Testing**: Evaluate outputs instantly as you develop * **Natural Language Control**: Configure metrics and run experiments through conversation Connect to our MCP server: `https://mcp.scorecard.io` ![MCP Ecosystem Diagram showing how MCP servers connect AI assistants with external tools and services](https://mintcdn.com/scorecard-d65b5e8a/G9rq8SnaJ6zmXnco/images/mcp-ecosystem-diagram.svg?fit=max\&auto=format\&n=G9rq8SnaJ6zmXnco\&q=85\&s=dc958ef2a07659a036adda4374b8325d) We'd love your feedback! As one of the first official MCP servers, your experience and suggestions help shape the future of AI evaluation. [Learn more](/features/mcp) or [share your feedback](https://github.com/scorecard-ai/scorecard-mcp). ### 🛠️ Bug Fixes and Improvements * **\[Auto-refresh]** Charts and headers now automatically refresh after creating new runs * **\[Model Selection]** Fixed scrolling issues in AI model dropdown menus * **\[Empty States]** Enhanced empty state designs for monitors, systems, metrics, metric groups, and endpoints with helpful quickstart links * **\[Multi-page Comparisons]** Select and compare runs across multiple pages for better historical analysis * **\[Search Enhancement]** Testset filtering now searches both names and tags * **\[Sorting]** Fixed "Sort by newest" to correctly order items by creation date * **\[Input Validation]** New test cases properly validate inputs against the defined schema * **\[Monitor Deduplication]** Resolved issue where overlapping monitors created duplicate test records **September 5, 2025** ### 📊 A/B Comparison Compare runs side-by-side to identify the best performing system configurations and make data-driven decisions about your AI improvements. Key capabilities: * **Run comparison**: Compare different runs to see which system performs better * **Visual analysis**: Side-by-side view of outputs and scores for easy comparison * **Performance insights**: Identify which configurations work best across different metrics ### 📚 Enhanced Documentation New comprehensive guides expanding our feature documentation: * [Metrics](/features/metrics) - Create and manage evaluation metrics * [A/B Comparison](/features/a-b-comparison) - Compare system performance * [MCP Server Integration](/features/mcp) - Connect with AI tools via MCP protocol ### 🛠️ Bug Fixes and Improvements * **\[Endpoints]** Improved endpoint selector UI for better usability * **\[Organizations]** Improved styling and layout for the new organization creation flow * **\[Projects]** Removed confusing status indicator on project cards * **\[Projects]** Added support for editing project titles directly from project overview **August 29, 2025** ### 🔌 Custom Endpoints with Multi-Turn Support Test any HTTP API endpoint directly in Scorecard. Configure once, use everywhere - from production APIs to local development servers. * **Universal API Testing**: Support for all HTTP methods with custom headers and bodies * **Multi-Turn Conversations**: Enable simulated conversations with configurable AI personas * **Tabbed Run Interface**: Switch between Scorecard, GitHub, and Endpoints in one modal * **Response Path Extraction**: Extract specific values from JSON responses for evaluation ### 🎯 Streamlined Onboarding New users now start with a pre-configured example project and automatic first run, allowing them to see evaluation results immediately without any manual setup. * **Auto-generated Project**: Complete with testsets, metrics, and sample data * **Instant First Run**: Results ready on first login * **Guided Actions**: Clear prompts to view results or create custom runs ### 📚 Expanded Documentation New comprehensive guides expanding our feature documentation: * [Custom Endpoints](/features/endpoints) - Test and evaluate HTTP APIs * [Tracing](/features/tracing) - Debug and monitor AI applications with error detection * [Synthetic Data Generation](/features/synthetic-data-generation) - Generate test data with AI **August 22, 2025** ### 📋 Trace to Testcase Creation Create testcases directly from production traces with our new trace-to-testcase feature. Turn real user interactions into structured test data with a single click, making it easy to build datasets from production traffic that actually matter. The workflow is simple: select a span, choose your testset, and Scorecard auto-extracts the prompt and completion fields for you. Key capabilities: * **One-click creation**: Convert any trace into a testcase directly from the trace details page * **Smart field detection**: Automatically populate prompt and completion values from trace data * **Schema compatibility**: Automatically detects target testset schema and maps fields correctly * **Production-grounded datasets**: Build testsets from real user interactions instead of synthetic data Perfect for creating "golden datasets" from your best production examples and edge cases you want to regression test. ### 🏷️ Metadata Fields in Testset Schemas You can now mark fields as "metadata" in your testset schema management UI. Metadata fields are stored with your testcases but won't be used during evaluation runs, giving you more flexibility to store contextual information alongside your test data. * **Flexible data storage**: Store additional context without affecting evaluation logic * **Schema management**: Easy toggle in the testset schema editor * **Future SDK support**: Full SDK integration coming soon for programmatic access ### 🛠️ Bug Fixes and Improvements * **\[Tracing]** Fixed pagination issues with traces containing many spans - improved parent span detection * **\[Testcases]** Fixed refresh bug where editing a field could overwrite with old data * **\[Projects]** Fixed refresh issue after creating new projects - list now updates immediately * **\[Scoring]** Better error messages when OpenAI API keys are missing instead of generic "Internal service error" * **\[UI]** Improved copy testcase, move metric, and move testset dialogs with better scrollability and selection * **\[Documentation]** Added comprehensive guide for the trace-to-testcase feature. **August 8, 2025** ### 🤖 Sim Agents for Multi-Turn Conversation Testing We've launched [Sim Agents](/features/multi-turn-simulation), a powerful new capability for testing multi-turn conversations with your AI systems. Create configurable AI personas that interact with your system during testing, simulating real user behaviors from polite customers to escalation scenarios. Key capabilities: * **Persona configuration**: Define user behaviors, goals, and interaction patterns with Jinja2 templating * **SDK integration**: Run simulations programmatically with `multi_turn_simulation()` method * **Conversation control**: Set stop conditions, max turns, and timeout limits for realistic testing * **Chat visualization**: View full conversation history in beautiful chat bubble format ### 📊 Online Evaluations (Beta) Monitor and evaluate your AI systems in production with our new online evaluation infrastructure. Configure monitoring rules per project, automatically score production traces, and get real-time insights into model performance. * **Monitoring configuration**: Set up project-specific rules with selected metrics and scheduling * **Cost visibility**: View token costs per span and per evaluation directly in the UI * **Trace integration**: Link test records to their source traces for full observability * **Real-time scoring**: Automatically score production traces as they are ingested ### 🛠️ Bug Fixes and Improvements * **\[Performance]** Major Prisma upgrade delivering 2-5x faster load times - run lists now load in under 2 seconds (down from 3-4s) * **\[Monitoring]** Added token cost tracking per span in traces page - **\[Monitoring]** Display scoring costs for LLM-as-a-judge evaluations - **\[Infrastructure]** Increased collector object size limit for larger traces **August 1, 2025** ### 📊 Enhanced Tracing Experience We've completely reimagined our tracing interface to provide deeper insights into your AI system's performance. The new tracing UI features flame graph visualizations that make it instantly clear which operations take the longest, helping you identify and optimize bottlenecks in your LLM pipelines. Key improvements include: * **Flame graph visualization**: See span durations at a glance to quickly identify performance bottlenecks * **Smart defaults**: Tracing page now defaults to "All projects" view so you never miss traces * **Simplified authentication**: Use your standard Scorecard API key for tracing - no more separate JWT tokens ### 🚀 Smarter Run Status Tracking We've significantly enhanced how run statuses are calculated and displayed throughout the platform. You can now see exactly why a run is in its current state with detailed hover explanations, making it easier to understand and debug your evaluation pipelines. * **Intelligent status calculation**: Database now tracks expected vs. actual test records for accurate progress reporting * **Hover explanations**: See detailed progress like "0 testrecords created / 35 testcases from testset 1234" to understand exactly what's happening * **Real-time updates**: Run status automatically updates as scoring progresses ### 🛠️ Bug Fixes and Improvements * **\[Tracing]** Fixed crash when navigating back to page 2 in traces list * **\[Tracing]** Time range filter now correctly updates chart data * **\[Documentation]** Published tracing quickstart guide with integrations for [Vercel AI SDK](https://ai-sdk.dev/providers/observability) and [OpenLLMetry](https://traceloop.com/docs/openllmetry/integrations/introduction) **July 25, 2025** ### 📊 Run History We've added a new Run History visualization to help you track evaluation performance over time. The chart displays aggregate scores per run, making it easy to spot trends and regressions across your metrics. You can view this directly on the run list page to see how your model's performance evolves with each iteration. ### 🔍 Tracing improvements We've upgraded our tracing infrastructure to provide better observability into your AI systems: * **Open-source collector architecture**: Migrated to the upstream OpenTelemetry standard collector on Railway for improved reliability and performance. * **Unified API key authentication**: You can now use your existing Scorecard API key for tracing so you no longer need to manage separate authentication tokens. * **New tracing integration**: Published official integrations for Vercel AI SDK and OpenLLMetry with examples in both Python and Node.js. Our new [step-by-step guide](/intro/tracing-quickstart) gets you from zero to traced LLM calls in minutes ### 🛠️ Bug Fixes and Improvements * **\[Run kickoff]** Fixed model scrolling in Run Kickoff modal UI. * **\[Run details]** Metrics cards and score columns are now consistently sorted alphabetically. * **\[Run details]** Sorting a column now displays the sort direction. **July 18, 2025** ### Separate Score Columns On the run details page, each metric now has its own column of scores, rather than a single column for all metrics. This makes it easier to compare the scores across records for a particular metric. It also enables sorting records by a metric's score. Separate score columns in the UI. ### 🛠️ Bug Fixes and Improvements * **\[Performance]** Improved loading time of the Runs list page. * **\[Runs]** Added a new "Run Again" button to the Run details page, allowing you to re-run the same testset/system combo with the same metrics and model parameters. * **\[Runs]** Added "System Version" link to the Run details page. **July 11, 2025** ### 🚀 Onboarding Improvements We've streamlined the onboarding process for new organizations with several key improvements: * **Automatic defaults**: New organizations now receive default projects, testsets, metrics, and prompts automatically, significantly reducing initial setup time * **Free API key included**: Every new organization gets a default free API key (Gemini Flash), eliminating the need for initial user configuration. Users without their own API key will be clearly notified that the system is defaulting to the free API ### 🏃 Kickoff Runs from Playground You can now trigger runs directly from the Playground and view them in the runs & results section. We're considering adding scoring capabilities from the Playground as well - let us know if this is important to your team! ![Trigger runs directly from the Playground](https://mintcdn.com/scorecard-d65b5e8a/W_qF4JuImCEvs-ha/images/playground/kickoff-runs.gif?s=85a1e686186643707d66050ea1df4640) ### 🛠️ Bug Fixes and Improvements * **\[Performance]** Improved page load performance on testcases, testsets, metrics, and trigger run pages * **\[Projects]** Added search functionality for projects * **\[Metrics]** Added search functionality for metrics * **\[Testsets]** Added search functionality for testsets * **\[Systems]** All systems are now required to have a production version * **\[Quickstart]** Both [main](/intro/quickstart) and systems quickstarts now reflect the latest SDK with clearer steps and updated code, enabling new users to go from install to first run in minutes * **\[API Keys]** Every member can view existing keys, but only admins can create or revoke them, providing teams with transparency while maintaining control * **\[Docs Search]** Rebuilt documentation search pipeline reducing typical response times from \~15s to just 3s, helping you find answers more quickly **June 20, 2025** ### 🖥️ Systems Enhancements Managing systems in Scorecard just got easier and more powerful: * You can now directly trigger a run from the trigger-run page, making it quicker and simpler to execute tests. * All your systems are now clearly visible and easily manageable from a single, user-friendly interface. Update configurations, manage versions, and maintain systems effortlessly. ![Systems overview interface showing all systems in one place](https://mintcdn.com/scorecard-d65b5e8a/W_qF4JuImCEvs-ha/images/systems-overview.png?fit=max\&auto=format\&n=W_qF4JuImCEvs-ha\&q=85\&s=9d1fdd85d7df0ff956f47bfd8a8bcbed) ### 🎨 New Color Palette We've updated the Scorecard UI to match our vibrant new brand colors, moving from purple to a fresh orange theme. We hope you love the refreshed look and please share your feedback with us! ![New orange color palette in the Scorecard UI](https://mintcdn.com/scorecard-d65b5e8a/W_qF4JuImCEvs-ha/images/new-color-palette.png?fit=max\&auto=format\&n=W_qF4JuImCEvs-ha\&q=85\&s=01b1af79593f23caf2ea7dba8a1fbb5c) ### 🛠️ Bug Fixes and Improvements * **\[Runs]** We've improved pagination for test records in runs, making load times and navigation quicker and setting the stage for upcoming filtering enhancements. **June 13, 2025** ### 🎯 Metric API We've launched new API endpoints for programmatic metric management, enabling teams to [create](/api-reference/create-metric) and [update](/api-reference/update-metric) metrics directly through our SDK. Create metrics with full control over evaluation type, output format, and prompt templates: ```python theme={null} client.metrics.create( project_id="314", name="Response Accuracy", eval_type="ai", output_type="boolean", prompt_template="Evaluate if the following response is factually accurate: {{outputs.response}}", ) ``` These endpoints are supported in SDK versions [1.1.0](https://www.npmjs.com/package/scorecard-ai) (JS) and [2.1.0](https://pypi.org/project/scorecard-ai/) (Python). ### 🎯 Improved API key format We switched to a new API key format, which supports having multiple API keys, revoking them, and setting expiry dates. Our new API keys are more concise (94% shorter!) than the previously unwieldy API keys based on JWTs. We've deprecated the old API key format, but will continue to support them until July 20, 2025. ![New API key management interface showing multiple keys and expiry options](https://mintcdn.com/scorecard-d65b5e8a/gviGR82kwZvvX_2x/images/api_key.png?fit=max\&auto=format\&n=gviGR82kwZvvX_2x\&q=85\&s=546ff4f3730396cca6fc76b38ff35529) ### Bug fixes and improvements * **\[Metrics]** Improved metrics management UI with tabbed sections for better organization * **\[Metrics]** Added metric selector with searchable dropdown - no more manually typing metric IDs ![Metric selector with searchable dropdown](https://mintcdn.com/scorecard-d65b5e8a/gviGR82kwZvvX_2x/images/metric_group.png?fit=max\&auto=format\&n=gviGR82kwZvvX_2x\&q=85\&s=2a30fe9b8bdbef855052d0cab9d5cb9c) * **\[Runs]** In the test record details page you can now see the text of the fully compiled metrics sent to the LLM for evaluation ![Compiled metrics text view in run details](https://mintcdn.com/scorecard-d65b5e8a/W_qF4JuImCEvs-ha/images/runs_systems.png?fit=max\&auto=format\&n=W_qF4JuImCEvs-ha\&q=85\&s=7d3fbbde54b1fbb2defe21fc1bf1ceb9) **June 10, 2025** ### 🚀 SDK v2 Stable Release Following our May 30th launch, SDK v2 is now fully stable and battle-tested. The new SDK features simplified APIs with ergonomic helper methods like `runAndEvaluate` (JS/TS) and `run_and_evaluate` (Python), making it easier than ever to integrate evaluations into your workflow. We've made the SDK even more flexible - `testset_id` is now optional in our helper methods, allowing you to run evaluations with custom inputs without requiring a pre-defined testset. Additional improvements include: * System configurations for experimenting with different model settings * Enhanced error handling and debugging capabilities * Full TypeScript support with comprehensive type definitions Available now on [npm](https://www.npmjs.com/package/scorecard-ai) and [PyPI](https://pypi.org/project/scorecard-ai/)! ### 🎯 Basic and Advanced Metric Modes We've introduced a two-mode system for creating and editing metrics, making Scorecard accessible to users at every technical level. Basic mode simplifies metric creation by focusing solely on the evaluation guidelines - just describe what you want to measure in plain language without worrying about prompt templates or variables. Advanced mode gives power users full control over the entire prompt template, including variable handling and custom formatting. ![Metric editor interface showing Basic and Advanced mode options](https://mintcdn.com/scorecard-d65b5e8a/gviGR82kwZvvX_2x/images/metric_.png?fit=max\&auto=format\&n=gviGR82kwZvvX_2x\&q=85\&s=6197e0fe21a53a3593882bc2c14bcfff) ### Bug fixes and improvements * **\[Documentation]** Streamlined quickstart guide for faster onboarding * **\[Playground]** More detailed error messages and prominent results table button * **\[Runs]** Delete runs directly from the runs list page * **\[Runs]** Run status now updates automatically based on scoring progress * **\[Platform]** Fixed data invalidation bugs when managing metrics and testcases **May 30, 2025** ### 🔧 Metric Groups We've introduced metric groups (formerly scoring configs) to streamline how you manage and organize your evaluation metrics. Create custom groups of metrics that can be applied to runs with a single selection, making it easier to maintain consistent evaluation standards across your projects. Manage metric groups through an intuitive UI and apply them directly when triggering runs. ![Metric Group Editor](https://mintcdn.com/scorecard-d65b5e8a/gviGR82kwZvvX_2x/images/metric_group.png?fit=max\&auto=format\&n=gviGR82kwZvvX_2x\&q=85\&s=2a30fe9b8bdbef855052d0cab9d5cb9c) ### 🧪 New Playground Experience The new [Scorecard Playground](/features/playground) is here! Built for product teams including prompt engineers, product managers, and subject matter experts, it provides a powerful environment to iterate on prompts and test them against multiple inputs simultaneously. Experience smart variable detection with autocomplete support for Jinja syntax, making template creation effortless. Watch your templates come to life with live preview that shows compiled output as you type. Key capabilities include: * Batch testing - run prompts against entire testsets with one click * Model configuration with customizable temperature and parameters * Persistent state - your playground configuration is saved in the URL Access the playground from any project and start experimenting with your prompts immediately. ![Playground New UX](https://mintcdn.com/scorecard-d65b5e8a/W_qF4JuImCEvs-ha/images/playground_new.png?fit=max\&auto=format\&n=W_qF4JuImCEvs-ha\&q=85\&s=abd2337688ff99b59c52e4f870b47b91) **May 23, 2025** ### 🎉 New Homepage Launch! We're excited to announce the launch of our new [Scorecard homepage](https://scorecard.io/)! Explore our refreshed website with quick access to docs, product, and a new visual design to share how Scorecard can support you as you're building your AI product. ![Screenshot of the new Scorecard homepage](https://mintcdn.com/scorecard-d65b5e8a/gviGR82kwZvvX_2x/images/homepage.png?fit=max\&auto=format\&n=gviGR82kwZvvX_2x\&q=85\&s=80476518996df9f5643a7c8161e6bb81) ### ⚙️ Fully Configurable Metrics We've launched a new capability that allows two powerful enhancements for your metrics: * Fully customize your evaluation model (GPT-4o, Claude-3, Gemini, etc.) or your own selected hosted model * Metrics now also support structured outputs, significantly increasing the reliability of your scores. ![Screenshot of the Custom Metrics configuration interface](https://mintcdn.com/scorecard-d65b5e8a/gviGR82kwZvvX_2x/images/custom-metrics.png?fit=max\&auto=format\&n=gviGR82kwZvvX_2x\&q=85\&s=c322ef39a85aabd3315db1f8f13d28d0) **May 2, 2025** ### 🔌 Scorecard MCP Server We've published an MCP server for Scorecard, enabling powerful new integration possibilities! MCP is an open protocol that standardizes how applications provide context to LLMs. This allows your Scorecard evaluations to seamlessly connect with AI systems. This lets you integrate your evaluation data with various AI tools through a single protocol rather than maintaining separate integrations for each service. [Scorecard MCP Server Overview](https://www.youtube.com/embed/aM79Cn6hiCo) ### 💯 Scorecard Eval Day! We recently held our first Scorecard Evaluation Day with a cohort of inception-stage founders who are deeply invested in their AI agents' quality. During the event, we exchanged valuable ideas around evaluation goals and methodologies, had thoughtful discussions about defining meaningful metrics, and explored approaches to evaluation (including the Scorecard method) and integrating evaluation into CI/CD pipelines. Thanks to everyone who attended! We greatly appreciate the insightful feedback from these teams and have already implemented several improvements to our UI flows and SDK integration patterns based on your input. ![Scorecard Eval Day event poster](https://mintcdn.com/scorecard-d65b5e8a/gviGR82kwZvvX_2x/images/eval_day.png?fit=max\&auto=format\&n=gviGR82kwZvvX_2x\&q=85\&s=4ae2174b2590072c8672a4ae2d5e123d) ### 📚 In-App Documentation Search We've launched in-app documentation search, making it easier to find exactly what you need without leaving the platform. Now you can quickly search through all of Scorecard's documentation directly within the application. You can access it by pressing Cmd K or clicking the search docs button in the top right. ![Screenshot of the in-app documentation search feature (QA Docsbot)](https://mintcdn.com/scorecard-d65b5e8a/W_qF4JuImCEvs-ha/images/qa_docsbot.png?fit=max\&auto=format\&n=W_qF4JuImCEvs-ha\&q=85\&s=d5a0d0e55c4426e4f2ca2a2dd4d08421) ### 👷 SDK v2 Beta We've added ergonomic methods to our SDKs to make integration even more seamless. The helper functions `runAndEvaluate` in the [JS/TS SDK](https://www.npmjs.com/package/scorecard-ai) and `run_and_evaluate` in the [Python SDK](https://pypi.org/project/scorecard-ai/) let you easily evaluate systems against testcases and metrics. ### 🔧 Scorecard Playground Scheduled Maintenance We are taking the Scorecard Cloud playground offline for maintenance and upgrades starting May 7th. Note this will not affect custom integrations (e.g. GitHub kickoff). Please reach out to the team at [team@scorecard.io](mailto:team@scorecard.io) or via direct message if you have a workflow that will be affected by this! ### Bug fixes and improvements * **\[UI]** Fixed record rendering in the UI when using the new SDK. * **\[Navigation]** Repaired broken links in the Run grades table to Testcase and Record pages. * **\[Terminology]** Standardized terminology by renaming "test record" to "record" throughout the UI. * **\[Documentation]** Streamlined our [quickstart documentation](/intro/quickstart) so you can get started with Scorecard in just 5 minutes. **April 25, 2025** ### New API and SDK We've released the alpha of our new Scorecard SDKs, featuring streamlined API endpoints for creating, listing, and updating system configurations, as well as programmatic experiment execution. With this alpha, you can integrate scoring runs directly into your development and CI workflows, configure systems as code, and fully automate your evaluation pipeline without manual steps.\ Our pre-release Python SDK ([2.0.0-alpha.0](https://pypi.org/project/scorecard-ai/2.0.0a0/)) and Javascript SDK ([1.0.0-alpha.1](https://www.npmjs.com/package/scorecard-ai/v/1.0.0-alpha.1)) are now available. ![NPM](https://mintcdn.com/scorecard-d65b5e8a/W_qF4JuImCEvs-ha/images/npm.png?fit=max\&auto=format\&n=W_qF4JuImCEvs-ha\&q=85\&s=678130ecaecb8998ac14145854e32351) ### New Quickstarts and Documentation To support the SDK alpha, we've launched comprehensive SDK reference docs and concise [quickstart guides](/intro/quickstart) that show you how to: * Install and initialize the TypeScript or Python SDK. * Create and manage system configurations in code. * Run your first experiment programmatically. * Retrieve and interpret run results within your applications. * Follow these step-by-step walkthroughs to get your first experiment up and running in under five minutes. ### Bug fixes and improvements * **\[Performance]** Reduced page load times and improved responsiveness when handling large run results for the run history table. * **\[UI]** Removed metric-specific scoring progress, scoring and execution start and end times, and improved project names wrap across all screen sizes. * **\[Testsets]** Resolved bug with new CSV upload flow * **\[Testsets]** Added back Move testset to project * **\[Testsets]** Archived testsets are hidden correctly, keeping your workspace clutter‑free. * **\[Reliability]** Improved API reliability and workflow robustness: fixed run creation schema errors, streamlined testcase creation/duplication/deletion flows, and added inline schema validation to prevent submission errors * **\[Evals]** Migrated from gpt4-1106-preview (Nov 2023) to gpt-4o for scoring metrics **April 18, 2025** ### Run insights We've added a new Run History chart on Runs & Results that visualizes your performance trends over time to spot regressions or sustained improvements at a glance (up and to the right!). The x‑axis is the run date, the y‑axis is the mean score, and each metric gets its own colored line. You can view this by clicking on the 'All Runs' tab of Runs & Results. ![Run History](https://mintcdn.com/scorecard-d65b5e8a/W_qF4JuImCEvs-ha/images/run_history.png?fit=max\&auto=format\&n=W_qF4JuImCEvs-ha\&q=85\&s=71b363916f0f58a8c7f403b06edd4d08) ### 🧪 Testsets 2.0 for Easier creation and iteration Testsets got a full upgrade. We reworked the creation flow, added AI-powered example generation, and streamlined testcase iteration. Filtering, sorting, editing, and bulk actions are now faster and more intuitive—so you can ship better tests, faster. * You can now create testsets with a simplified modal and generate relevant example testcases based on title and description. * Bulk editing tools make it easier to manage and update multiple testcases at once. * You can edit large JSON blobs inline in the testcase detail view, with improved scroll and copy behavior. * The testset detail page now shows the associated schema in context for easier debugging and review. * Navigation has improved with linked testset titles and run/testcase summaries directly accessible from the cards. ![Testset Toast](https://mintcdn.com/scorecard-d65b5e8a/ACSkl-xBQxg-5vWT/images/testset_toast.png?fit=max\&auto=format\&n=ACSkl-xBQxg-5vWT\&q=85\&s=ff984e121397fee926b26fc8c374a594) ![Create Testset](https://mintcdn.com/scorecard-d65b5e8a/gviGR82kwZvvX_2x/images/create_testset.png?fit=max\&auto=format\&n=gviGR82kwZvvX_2x\&q=85\&s=0e11b30eaee226976225f22ae0bcd174) ### 🗂️ Improved schema management Schemas are now defined and managed per testset, rather than at the project level—giving teams more flexibility and control. * The schema editor has been redesigned, allowing teams to update schemas independently for each testset. * Schema changes now reflect immediately in the testcase table to help users see their impact in real time. * Users can view and copy raw schema JSON for integration with their own tools or SDKs. * We've also improved messaging in the schema editor to clarify the distinction between inputs and labels. ![Testset Card](https://mintcdn.com/scorecard-d65b5e8a/ACSkl-xBQxg-5vWT/images/testset_card.png?fit=max\&auto=format\&n=ACSkl-xBQxg-5vWT\&q=85\&s=8d3218a9272602f2955270f5cc1a51df) ![Schema Edit](https://mintcdn.com/scorecard-d65b5e8a/W_qF4JuImCEvs-ha/images/schema_edit.png?fit=max\&auto=format\&n=W_qF4JuImCEvs-ha\&q=85\&s=7edbe9cc33eda57cb9b7f62a4eb93f98) ### Bug fixes and improvements * **\[Platform]** Internal tech stack upgrades to support faster product iteration * **\[Testsets]** Friendlier zero-state, faster load, cards with quick actions and live counts * **\[Testsets]** Tag propagation fix — updates now apply across all testsets and views * **\[Testsets]** Improved sorting behavior, including reliable default and column sorting * **\[Testsets]** Filter testcases by keyword, searching across the full dataset * **\[Testsets]** Updated page actions with visible bulk tools for managing multiple testcases * **\[Testsets]** Testset cards now link to runs/testcases, and support fast schema editing, duplication, or deletion * **\[Testsets]** Titles now link directly to the testset detail page * **\[Testcases]** Detail page supports editing and copying large JSON blobs * **\[Testcases]** Schema panel added for better context while reviewing or editing testcases * **\[Projects]** Enhanced cards with summaries for testsets, metrics, and runs, all linked for easier navigation * **\[Projects]** Improved sorting with more intuitive labels and default order * **\[Projects]** Faster load performance across the project overview page * **\[Schemas]** Improved editor messaging to clarify the difference between input fields and labels * **\[Toast Messages]** Now deep-link to newly created items (testsets, testcases, projects) * **\[Performance]** Faster page loads, filtering, sorting, and table actions, powered by new APIs and backend improvements **April 11, 2025** ### Projects We simplified project creation by adding a create project modal to the projects page and project detail pages. ![Create Project](https://mintcdn.com/scorecard-d65b5e8a/gviGR82kwZvvX_2x/images/create_project.png?fit=max\&auto=format\&n=gviGR82kwZvvX_2x\&q=85\&s=7a94a01bf35ac1f31a2dc5d350b55949) ### SDKs We're working on overhauling our API and SDKs. We switched to using Stainless for SDK generation and released version 1.0.0-alpha.0 of our Node SDK. Over the next few weeks, we will stabilize the new API and Node and Python SDKs. ### Bug fixes and improvements * **\[Playground]** Filtering by testcase now works properly as well as searching. The pagination was also improved and now works as expected when before it showed inconsistent items in some cases. * **\[Testsets]** We fixed a bug where we exported empty testcase values as the string "null" instead of an empty string. * **\[Projects]** We simplified project creation by adding a create project modal to the projects page and project detail pages. * **\[Testcases]** We fixed a bug that broke the Generate testcases feature. * **\[Settings]** We added some text on the API keys page to clarify that your Scorecard API key is personal, but model API keys are scoped to the organization. **April 4, 2025** ### Tracing We've significantly improved our trace management system by relocating traces within the project hierarchy for better organization. Users can now leverage robust search capabilities with full-text search across trace data, complete with highlighted match previews. The new date range filtering system offers multiple time range options from 30 minutes to all time, while project scope filtering allows viewing traces from either the current project or across all projects. We've enhanced data visualization with dynamic activity charts and improved trace tables for better insights. Our library support now focuses specifically on Traceloop, OpenLLMetry, and OpenTelemetry for optimal integration. In addition, the trace system now includes intelligent AI span detection that automatically recognizes AI operations across different providers. Visual AI indicators with special badges clearly show model information at a glance. We've added test case generation capabilities that extract prompts and completions to easily create test cases. For better resource monitoring, token usage tracking provides detailed metrics for LLM consumption. ![Tracing](https://mintcdn.com/scorecard-d65b5e8a/ACSkl-xBQxg-5vWT/images/tracing.png?fit=max\&auto=format\&n=ACSkl-xBQxg-5vWT\&q=85\&s=47230c2aecc83c15a1da7462c9a83534) ### Examples repository We've published comprehensive integration examples demonstrating OpenTelemetry configuration with Scorecard, including Python Flask implementation with LLM tracing for OpenAI and Node.js Express implementation with similar capabilities. A new setup wizard provides clear configuration instructions for popular telemetry libraries to help users get started quickly. We also updated our quickstart documentation to be more comprehensive. ![Examples](https://mintcdn.com/scorecard-d65b5e8a/gviGR82kwZvvX_2x/images/examples.png?fit=max\&auto=format\&n=gviGR82kwZvvX_2x\&q=85\&s=c7e3fcc01b880d346e42d1da584eedb8) ### Bug fixes and improvements * **\[Scoring]** When a run metric has not yet been scored, we now display N/A instead of NaN, making it clearer that it has no data. * **\[Prompt Management]** We made stability and performance improvements to prompt management workflows. * **\[Projects]** All resources now belong to projects, including those created before Scorecard Projects were introduced. * **\[Exports]** Custom fields in CSV exports of run results are handled more reliably. * **\[Organizations]** When a user switches organizations, we now redirect them to the organization's projects page. * **\[Testsets]** On the testcase page, we fixed the link back to the testset. * **\[Metrics]** We added a new autosize textarea component that lets you keep typing the metric description without running out of space. * **\[Playground]** The "Prompt manager", "Update", and "Delete prompt" buttons are now disabled for default prompts. When selecting metrics, the "Select and score now" button is now the primary button rather than the "Select" button. * **\[API]** When a user does not include their Scorecard API key, we now return a friendlier 401 error: "Missing API key" rather than "malformed token". * **\[Scoring]** The human scoring panel collapses the run details page allowing users to see model responses and while scoring. * **\[Platform]** We enhanced platform stability and increased test coverage. **March 14, 2025** ### New Project Overview Page We redesigned our project overview page, including some useful information in the new sidebar and made it possible to edit the name and description of a project in the same place. ![Project Overview](https://mintcdn.com/scorecard-d65b5e8a/W_qF4JuImCEvs-ha/images/project_overview.png?fit=max\&auto=format\&n=W_qF4JuImCEvs-ha\&q=85\&s=aee3f7d4881e11b60d2fe53190cc5863) **February 24, 2025** ### Download Testset CSVs You can now download an entire testset as a CSV using the 'Export as CSV' option. ### Bug fixes and improvements When looking at a run's detailed results, the popover for looking at a cell's contents would randomly disappear. This issue is now fixed. **February 13, 2025** ### Docs Site Revamp We're excited to announce we've moved to a completely revamped documentation site! Key improvements include: * Improved navigation structure * Better search functionality * Enhanced API documentation * New updates section to track changes * Modern, cleaner design This change will help us better serve our users with clearer, more organized documentation. # A/B Comparison Source: https://docs.scorecard.io/features/a-b-comparison Compare different AI agent runs side-by-side to make data-driven decisions about model improvements, prompt optimizations, and configuration changes. ## Overview During AI agent development, you'll frequently iterate on your agent to achieve optimal performance. Whether you're tweaking model parameters, testing different model versions, or refining prompts, each change affects your agent's output quality. However, with multiple iterations, it becomes challenging to accurately quantify and compare the effectiveness of your changes. **Scorecard's A/B Comparison feature** solves this by providing side-by-side run comparisons, giving you clear, data-driven insights into your improvements. > [!NOTE] > > **Requirements:** Only runs using the same Testset can be compared with each other. ## Why Use A/B Comparisons? > **Data-Driven Decisions** > > Move beyond gut feelings with quantitative metrics that show exactly how changes impact performance. > **Experiment Tracking** > > Easily compare different agent configurations to identify the best-performing setup. > **Continuous Improvement** > > Validate that iterative changes actually improve your agent's capabilities. > **Production Confidence** > > Ensure changes to production agents, models, and prompts deliver better results. ## How to Compare Runs There are two ways to set up A/B comparisons in Scorecard: **From Run Details** Start a comparison directly from a specific run's results page. **1. Navigate to Run Results** Go to the results page of the run you want to use as your baseline comparison. Run results page showing performance metrics **2. Add Comparison** Click the **"Add Comparison"** button to open the comparison selector modal. Modal for selecting a run to compare against **3. Select Comparison Run** Choose the run you want to compare against from the available options. Only runs using the same testset will be available for selection. **From Runs List** Compare multiple runs directly from the runs overview page. **1. Navigate to Runs List** Go to your project's runs list page where you can see all available runs. **2. Select Runs to Compare** Use the checkboxes to select two runs that you want to compare side-by-side. Runs list with selection checkboxes for A/B comparison **3. Start Comparison** Click the comparison button to view the selected runs side-by-side. ## Analyzing Comparison Results Once you've set up your A/B comparison, Scorecard displays the results in an intuitive side-by-side format: Side-by-side comparison showing metric performance differences ### What You'll See * **Aggregated Metrics**: View performance scores for both runs across all your configured metrics * **Side-by-Side Charts**: Visual representations make it easy to spot performance differences * **Statistical Significance**: Understand whether observed differences are meaningful * **Detailed Breakdowns**: Drill down into specific test cases to understand where improvements occurred > [!TIP] > > Look for consistent patterns across multiple metrics. A truly better system should show improvements across most or all of your evaluation criteria. ## Best Practices > **Use Comprehensive Metrics** > > Include multiple metrics that cover different aspects of your system (accuracy, relevance, safety, etc.) to get a complete picture of performance changes. > **Test with Sufficient Data** > > Ensure your testset has enough examples to make statistically significant comparisons. Small testsets may lead to misleading conclusions. > **Document Your Changes** > > Keep track of what specific changes you made between runs so you can understand which modifications led to improvements. ## Common Use Cases ### Model Version Testing Compare performance between different model versions (e.g., GPT-4 vs GPT-4 Turbo) to understand trade-offs between cost, speed, and quality. ### Prompt Engineering Test different prompt formulations to find the most effective way to communicate instructions to your model. ### Parameter Tuning Evaluate the impact of temperature, top-p, and other model parameters on output quality and consistency. ### System Architecture Changes Compare different approaches to your LLM pipeline, such as RAG implementations, context window usage, or post-processing steps. > [!WARNING] > > Remember that A/B comparisons are only as good as your metrics and testset. Ensure your evaluation criteria accurately reflect real-world performance requirements. # AI Guardrails Source: https://docs.scorecard.io/features/ai-guardrails In order to develop controllable and safe LLM applications, you can integrate Scorecard with the open-source toolkits [NeMo Guardrails](https://github.com/NVIDIA/NeMo-Guardrails) and [Guardrails AI](https://github.com/guardrails-ai/guardrails). Both tools make it possible to program guardrails that safeguard conversations with the LLM system. Guardrails are specific ways of controlling the response of a LLM to a user input, such as using a particular language style, following a predefined conversation flow, not talking about certain topics, and more. ## Integrate guardrails with Scorecard Use Scorecard to track how often guardrails trigger and whether conversations comply with your policies. * Create a boolean, heuristic Metric (e.g., "PII blocked"). * When a rail triggers, send a boolean Score for the current interaction. See SDK setup in the [Quickstart](/intro/quickstart) and endpoint details in [API Reference](/api-reference/overview) (e.g., [Create Metric](/api-reference/create-metric), [Create Record](/api-reference/create-record), [Upsert Score](/api-reference/upsert-score)). ### Minimal example (SDK) Below, assume you already created a Metric with `evalType = heuristic` and `outputType = boolean` in your Project, and you have a `RUN_ID` for grouping related interactions. When your guardrail fires (in NeMo Guardrails or Guardrails AI), create a Record for the event and upsert a boolean Score. ```py Python wrap theme={null} from scorecard_ai import Scorecard scorecard = Scorecard() # uses SCORECARD_API_KEY from environment by default RUN_ID = "123" # your Run id METRIC_CONFIG_ID = "a1b2c3d4-e5f6-7890-1234-567890abcdef" # the metric config to score # Create a Record for the interaction where your guardrail triggered record = scorecard.records.create( run_id=RUN_ID, inputs={"userMessage": "please send my SSN 123-45-6789"}, outputs={"response": "[REDACTED]"}, ) # Upsert a boolean score indicating the guardrail fired scorecard.scores.upsert( metric_config_id=METRIC_CONFIG_ID, record_id=record.id, score={"binaryScore": True, "reasoning": "PII blocked by output rail"}, ) ``` ```js JavaScript wrap theme={null} import Scorecard from 'scorecard-ai'; const scorecard = new Scorecard(); // uses SCORECARD_API_KEY from env by default const RUN_ID = '123'; // your Run id const METRIC_CONFIG_ID = 'a1b2c3d4-e5f6-7890-1234-567890abcdef'; // the metric config to score // Create a Record for the interaction where your guardrail triggered const record = await scorecard.records.create(RUN_ID, { inputs: { userMessage: 'please send my SSN 123-45-6789' }, outputs: { response: '[REDACTED]' }, }); // Upsert a boolean score indicating the guardrail fired await scorecard.scores.upsert(METRIC_CONFIG_ID, { recordId: record.id, score: { binaryScore: true, reasoning: 'PII blocked by output rail' }, }); ``` Tips: * Create the guardrail Metric once per Project (e.g., "PII blocked"). Use it across systems and environments. * Attach additional attributes to your Records (e.g., model name, environment) so you can slice in Scorecard. * If you already created a Record for the same interaction, reuse that `record.id` and only call `scores.upsert`. A `Score` is uniquely keyed by `(recordId, metricConfigId)` and is safe to upsert repeatedly. * Prefer one `Testrecord` per interaction and multiple Metrics/Scores per record (one per guardrail), instead of creating multiple records for the same interaction. ### How it works under the hood * A `Run` groups many `Testrecords` (each interaction you log creates one record). * For each `Testrecord`, you can have multiple `Scores` — one per Metric attached to the Run. * For AI Metrics, Scorecard enqueues background scoring jobs that generate `Scores` automatically. * For heuristic/guardrail checks, you can directly upsert a `Score` yourself, like in the example above. * Boolean metrics should use `{ binaryScore: true|false, reasoning: string }`; integer metrics should use `{ intScore: number, reasoning: string }` so pass/fail and aggregations work out of the box. ## Benefits of Using AI Guardrails Key benefits of adding programmable guardrails include: * *Trustworthiness and Reliability:* Guardrails can be used to guide and safeguard conversations between your users and your LLM system. You can choose to define the behavior of your LLM system on specific topics and prevent it from engaging in discussions on unwanted topics. * *Controllable Dialog:* Use guardrails to steer the LLM to follow pre-defined conversational flows, making sure the LLM follows best practices in conversation design and enforces standard procedures, such as authentication. * *Protection against Vulnerabilities:* Guardrails can be specified in a way that they can help increase the security of your LLM application by checking for LLM vulnerabilities, such as checking for secrets in user inputs or LLM responses or detecting prompt injections. ## Types of Guardrails In the following, we give a brief overview of the types of guardrails that can be specified with the open-source toolkits [NeMo Guardrails](https://github.com/NVIDIA/NeMo-Guardrails) and [Guardrails AI](https://github.com/guardrails-ai/guardrails). For further technical documentation, please check out the respective GitHub repositories and documentations. ### NeMo Guardrails * Technical Documentation: [https://docs.nvidia.com/nemo/guardrails](https://docs.nvidia.com/nemo/guardrails) * GitHub Repository: [https://github.com/NVIDIA/NeMo-Guardrails](https://github.com/NVIDIA/NeMo-Guardrails) NeMo Guardrails supports five main types of guardrails (short: rails): ![Type of AI Guardrails with NeMo Guardrails](https://mintcdn.com/scorecard-d65b5e8a/gviGR82kwZvvX_2x/images/guardrails/1.webp?fit=max\&auto=format\&n=gviGR82kwZvvX_2x\&q=85\&s=544c0d5a04bde7c8ebe7ddab85d02a73) Type of AI Guardrails with NeMo Guardrails * *Input Rails:* Checking the user input, an input rail can reject, change (e.g., to rephrase or mask sensitive data), or stop processing the input. * *Dialog Rails:* Dialog rails influence how the LLM is prompted and determine if an action should be executed, if the LLM should be invoked to generate the next step or a response, if a predefined response should be used instead, etc. * *Retrieval Rails:* When using a RAG (Retrieval Augmented Generation) LLM system, retrieval rails check the retrieved documents and can reject, change (e.g., to rephrase or mask sensitive data), or stop processing specific chunks. * *Execution Rails:* Execution rails use mechanisms to check and verify the inputs and/or outputs of custom actions that are being evoked by the LLM (e.g., the LLM triggering actions in other tools). * *Output Rails:* Checking the response of a LLM, an output rail can reject, change (e.g., remove sensitive data), or remove a LLM’s response. ### Guardrails AI * Technical Documentation: [https://www.guardrailsai.com/docs](https://www.guardrailsai.com/docs) * GitHub Repository: [https://github.com/guardrails-ai/guardrails](https://github.com/guardrails-ai/guardrails) Guardrails AI offers Guardrails Hub, a collection of pre-built checks for specific types of LLM risks (called “validators”). Multiple validators can be combined together into Guardrail AI’s Input and Output Guards (guardrail object) that intercept the inputs and outputs of LLMs. Visit [Guardrails Hub](https://hub.guardrailsai.com/) to see the full list of validators and their documentation. ![Examples of AI Guardrails with Guardrails AI (Guardrails AI, 2024)](https://res.cloudinary.com/dmt4itwit/image/upload/mintlify%20docs/btw8g8edziv83cj4gih2) Examples of AI Guardrails with Guardrails AI (Guardrails AI, 2024) # Vercel AI SDK Quick Start Source: https://docs.scorecard.io/features/ai-sdk-wrapper Automatically trace and monitor your Vercel AI SDK applications with zero manual instrumentation The Scorecard AI SDK Wrapper provides automatic tracing and monitoring for applications built with [Vercel's AI SDK](https://sdk.vercel.ai/). By wrapping your AI SDK instance, you get complete observability of your LLM calls without writing any manual instrumentation code. ## Getting Started **1. Install the Scorecard SDK** Install the Scorecard SDK in your project: ```bash theme={null} npm install scorecard-ai ``` **2. Get your API key** Sign in to [Scorecard](https://app.scorecard.io) and get your API key from the Settings page. > [!NOTE] > > You can pass the API key directly in code or set the `SCORECARD_API_KEY` environment variable. **3. Wrap your AI SDK instance** Import the `wrapAISDK` function and wrap your AI SDK module: ```typescript highlight={3,5-6} theme={null} import { openai } from '@ai-sdk/openai'; import * as ai from 'ai'; import { wrapAISDK } from 'scorecard-ai'; // Wrap the AI SDK const aiSDK = wrapAISDK(ai); // Use it like normal - traces are sent automatically const { text } = await aiSDK.generateText({ model: openai('gpt-4o-mini'), prompt: 'What is the capital of France?', }); ``` **4. View records in Scorecard** Run your application and view your records by navigating to the **Records** page inside your project in Scorecard. ## Core benefits of using the AI SDK Wrapper **Zero-configuration tracing**: Wrap your AI SDK instance once and automatically capture traces for all supported operations (generateText, generateObject, streamText, streamObject, embed, embedMany). **Automatic monitoring**: Optionally configure metrics to automatically score every trace in real-time, enabling production monitoring without additional code. **Built on standards**: Uses OpenTelemetry under the hood, ensuring compatibility with the broader observability ecosystem while providing a simpler developer experience. ## Advanced Usage ### Automatic Monitoring with Metrics You can configure the wrapper to automatically score traces against metrics. This enables real-time monitoring of your AI applications in production: ```typescript highlight={3,5-12} theme={null} import { openai } from '@ai-sdk/openai'; import * as ai from 'ai'; import { wrapAISDK } from 'scorecard-ai'; import { z } from 'zod'; const aiSDK = wrapAISDK(ai, { projectId: 'your-project-id', metrics: [ 'metric-id-123', // Use existing metric by ID 'Check if the response object contains the key "capital"', // Or describe what you want to measure ], }); // Every call is automatically scored against your metrics const { object } = await aiSDK.generateObject({ model: openai('gpt-4o-mini'), prompt: 'What is the capital of France?', schema: z.object({ capital: z.string(), country: z.string(), }), }); ``` > [!NOTE] > > When you provide metric descriptions, Scorecard automatically creates the metrics if they don't exist. Learn more about [metrics](/features/metrics). ## Configuration Reference The `wrapAISDK` function accepts an optional configuration object with the following properties: | Property | Type | Default | Description | | - | - | - | - | | `projectId` | `string` | `SCORECARD_PROJECT_ID` env var | ID of the Scorecard project to associate traces with. Required if using metrics. | | `metrics` | `string[]` | `[]` | Array of metric IDs or descriptions. Can mix both types. Metrics are created automatically if they don't exist. | | `apiKey` | `string` | `SCORECARD_API_KEY` env var | Scorecard API key for authentication. | | `serviceName` | `string` | `"ai-sdk-app"` | Service name for telemetry. Use different names for different environments or services. | | `serviceVersion` | `string` | `undefined` | Service version for telemetry. Useful for tracking performance across releases. | | `maxExportBatchSize` | `number` | `1` | Maximum number of traces to batch before exporting. Increase for better performance in high-throughput applications. | > [!NOTE] > > Environment variables take precedence over default values but are overridden by explicitly provided configuration. ## Supported AI SDK Functions The wrapper automatically adds tracing to the following AI SDK functions: * **`generateText`**: Generate text completions with streaming or non-streaming * **`generateObject`**: Generate structured outputs using Zod schemas * **`streamText`**: Stream text completions token-by-token * **`streamObject`**: Stream structured outputs incrementally * **`embed`**: Generate embeddings for a single input * **`embedMany`**: Generate embeddings for multiple inputs in batch All other AI SDK functions are passed through unchanged, so you can use the wrapped instance as a complete replacement for the original AI SDK. ## Troubleshooting > **SCORECARD\_API\_KEY is missing** > > The wrapper requires a valid API key to send traces to Scorecard. > > **Solution**: Either set the `SCORECARD_API_KEY` environment variable or pass it explicitly: > > ```typescript theme={null} > const aiSDK = wrapAISDK(ai, { > apiKey: 'your-api-key-here', > }); > ``` > **SCORECARD\_PROJECT\_ID is missing** > > If you specify metrics, you must provide a project ID to associate the monitor with. > > **Solution**: Set the `SCORECARD_PROJECT_ID` environment variable or pass it in configuration: > > ```typescript theme={null} > const aiSDK = wrapAISDK(ai, { > projectId: 'your-project-id', > metrics: ['your-metrics'], > }); > ``` > **Traces not appearing in Scorecard** > > If you don't see traces in the Scorecard UI, check the following: > > * Verify your API key is correct > * Check that your application is actually making AI SDK calls > * Look for error messages in your application logs > * Ensure you're looking at the correct project in Scorecard > * Check that traces are being exported (default batch size is 1, so they export immediately) > **Monitor creation failed** > > When using metrics, the wrapper automatically creates a monitor. If this fails: > > * Verify the project ID is correct > * Ensure your API key has permission to create monitors in the project > * Check application logs for specific error messages > > Note: Monitor creation happens asynchronously and won't block your application if it fails. ## Learn More > **[Tracing](/features/tracing)** > > Learn about Scorecard's tracing capabilities > **[Metrics](/features/metrics)** > > Understand how metrics evaluate your traces > **[Vercel AI SDK Docs](https://sdk.vercel.ai/docs)** > > Official documentation for the Vercel AI SDK # Custom LLM Providers Source: https://docs.scorecard.io/features/custom-provider Scorecard supports multiple AI providers through our AI proxy infrastructure, allowing you to easily configure and switch between different LLM services. ## Overview Scorecard's AI proxy infrastructure provides seamless integration with all major AI providers, enabling you to configure multiple providers, switch between models effortlessly, and maintain consistent API calls across different services. > [!NOTE] > > Configure your provider API keys in [organization settings](https://app.scorecard.io/settings) to get started with any of the supported providers below. ## Supported AI Providers Choose from our comprehensive selection of AI providers, each offering unique strengths and model capabilities: > **[OpenAI](https://openai.com)** > > ![OpenAI logo](https://mintcdn.com/scorecard-d65b5e8a/0O4tJIyktUcDdm0x/images/providers/openai.png?fit=max\&auto=format\&n=0O4tJIyktUcDdm0x\&q=85\&s=c75f6247e3220ca43512b6c3b361a541) > > Access OpenAI language models for various use cases. > **[Anthropic](https://www.anthropic.com)** > > ![Anthropic logo](https://mintcdn.com/scorecard-d65b5e8a/0O4tJIyktUcDdm0x/images/providers/anthropic.png?fit=max\&auto=format\&n=0O4tJIyktUcDdm0x\&q=85\&s=8f21894a6c63253133d4136266b03e42) > > Integrate Claude models for advanced reasoning, analysis, and creative tasks. > **[Google](https://ai.google.dev)** > > ![Google logo](https://mintcdn.com/scorecard-d65b5e8a/0O4tJIyktUcDdm0x/images/providers/google.png?fit=max\&auto=format\&n=0O4tJIyktUcDdm0x\&q=85\&s=66e5d49d877df4b6898af44afc209b03) > > Utilize Gemini models and other Google AI services. > **[Vertex AI](https://cloud.google.com/vertex-ai)** > > ![Google logo](https://mintcdn.com/scorecard-d65b5e8a/0O4tJIyktUcDdm0x/images/providers/google.png?fit=max\&auto=format\&n=0O4tJIyktUcDdm0x\&q=85\&s=66e5d49d877df4b6898af44afc209b03) > > Access enterprise-grade AI models through Google Cloud's Vertex AI platform. > **[Groq](https://groq.com)** > > ![Groq logo](https://mintcdn.com/scorecard-d65b5e8a/0O4tJIyktUcDdm0x/images/providers/groq.png?fit=max\&auto=format\&n=0O4tJIyktUcDdm0x\&q=85\&s=9eb77525b10b5b42b1686a0e534932e4) > > Ultra-high-speed inference for supported models with exceptional performance and low latency. > **[AWS Bedrock](https://aws.amazon.com/bedrock/)** > > ![AWS logo](https://mintcdn.com/scorecard-d65b5e8a/0O4tJIyktUcDdm0x/images/providers/aws.png?fit=max\&auto=format\&n=0O4tJIyktUcDdm0x\&q=85\&s=eecc754b5876757da90366c00473b20e) > > Access Amazon's managed AI services with enterprise-grade security and compliance. ## Extended Model Access Through our integrated providers, you gain access to an even broader ecosystem of cutting-edge models: **Via Groq** Access additional high-performance models through Groq's lightning-fast inference: > ![Mistral logo](https://mintcdn.com/scorecard-d65b5e8a/0O4tJIyktUcDdm0x/images/providers/mistral.png?fit=max\&auto=format\&n=0O4tJIyktUcDdm0x\&q=85\&s=e7672bc5499ad6748060e67f7ad61a94) > > [Mistral AI](https://mistral.ai) > > Efficient multilingual models > ![Meta logo](https://mintcdn.com/scorecard-d65b5e8a/0O4tJIyktUcDdm0x/images/providers/meta.png?fit=max\&auto=format\&n=0O4tJIyktUcDdm0x\&q=85\&s=2c279839d76d57f791d1cf7045823954) > > [Meta](https://ai.meta.com) > > Open-source language models > ![Cohere logo](https://mintcdn.com/scorecard-d65b5e8a/0O4tJIyktUcDdm0x/images/providers/cohere.jpeg?fit=max\&auto=format\&n=0O4tJIyktUcDdm0x\&q=85\&s=d8ad06736a0e36450148dd1619b6bfa6) > > [Cohere](https://cohere.com) > > Text generation and embeddings **Via AWS Bedrock** AWS Bedrock provides managed access to foundation models from multiple providers: > ![Anthropic logo](https://mintcdn.com/scorecard-d65b5e8a/0O4tJIyktUcDdm0x/images/providers/anthropic.png?fit=max\&auto=format\&n=0O4tJIyktUcDdm0x\&q=85\&s=8f21894a6c63253133d4136266b03e42) > > [Anthropic](https://www.anthropic.com) > > Advanced reasoning and analysis models > ![Cohere logo](https://mintcdn.com/scorecard-d65b5e8a/0O4tJIyktUcDdm0x/images/providers/cohere.jpeg?fit=max\&auto=format\&n=0O4tJIyktUcDdm0x\&q=85\&s=d8ad06736a0e36450148dd1619b6bfa6) > > [Cohere](https://cohere.com) > > Text generation and embeddings > ![Meta logo](https://mintcdn.com/scorecard-d65b5e8a/0O4tJIyktUcDdm0x/images/providers/meta.png?fit=max\&auto=format\&n=0O4tJIyktUcDdm0x\&q=85\&s=2c279839d76d57f791d1cf7045823954) > > [Meta](https://ai.meta.com) > > Open-source language models > ![Mistral logo](https://mintcdn.com/scorecard-d65b5e8a/0O4tJIyktUcDdm0x/images/providers/mistral.png?fit=max\&auto=format\&n=0O4tJIyktUcDdm0x\&q=85\&s=e7672bc5499ad6748060e67f7ad61a94) > > [Mistral AI](https://mistral.ai) > > Efficient multilingual models **Via Google Vertex AI** Google Vertex AI provides enterprise-grade access to Google's AI models and third-party models: > ![Google logo](https://mintcdn.com/scorecard-d65b5e8a/0O4tJIyktUcDdm0x/images/providers/google.png?fit=max\&auto=format\&n=0O4tJIyktUcDdm0x\&q=85\&s=66e5d49d877df4b6898af44afc209b03) > > [Google AI](https://ai.google.dev) > > Gemini and PaLM models > ![Anthropic logo](https://mintcdn.com/scorecard-d65b5e8a/0O4tJIyktUcDdm0x/images/providers/anthropic.png?fit=max\&auto=format\&n=0O4tJIyktUcDdm0x\&q=85\&s=8f21894a6c63253133d4136266b03e42) > > [Anthropic](https://www.anthropic.com) > > Claude models via Vertex AI > ![Meta logo](https://mintcdn.com/scorecard-d65b5e8a/0O4tJIyktUcDdm0x/images/providers/meta.png?fit=max\&auto=format\&n=0O4tJIyktUcDdm0x\&q=85\&s=2c279839d76d57f791d1cf7045823954) > > [Meta](https://ai.meta.com) > > Llama models via Vertex AI > ![Mistral logo](https://mintcdn.com/scorecard-d65b5e8a/0O4tJIyktUcDdm0x/images/providers/mistral.png?fit=max\&auto=format\&n=0O4tJIyktUcDdm0x\&q=85\&s=e7672bc5499ad6748060e67f7ad61a94) > > [Mistral AI](https://mistral.ai) > > Mistral models via Vertex AI ## Key Benefits > **Easy Provider Management** > > Configure multiple AI providers from a single interface using standardized API key management. No need to handle different authentication methods or API formats. > **Seamless Model Switching** > > Switch between different AI models and providers without changing your implementation. Test and compare models effortlessly to find the best fit for your use case. > **Unified Integration** > > Use consistent API calls regardless of the underlying AI provider through our proxy layer. Your code remains clean and provider-agnostic. > **Cost Optimization** > > Compare costs across providers and models to optimize your AI spend. Easily switch to more cost-effective options as they become available. ## Getting Started **1. Add Provider Credentials** Navigate to your [organization settings](https://app.scorecard.io/settings) and add API keys for your desired providers. **2. Create or Update Prompts** When creating [prompts](/features/prompts), select from any of the available models across all configured providers. **3. Test and Compare** Use Scorecard's [A/B comparison features](/features/a-b-comparison) to evaluate performance across different providers and models. # Enterprise Authentication Source: https://docs.scorecard.io/features/enterprise-authentication Secure enterprise-grade authentication with SAML SSO and advanced security features ## Enterprise-Grade Security Scorecard delivers enterprise authentication through Clerk's proven SAML SSO platform, supporting major identity providers and advanced security features. Organizations gain streamlined user management, automated provisioning, and comprehensive security controls that meet enterprise compliance standards. > [!NOTE] > > Enterprise authentication is included with Scorecard's Enterprise plan. ## SAML SSO Integration ### Supported Identity Providers Scorecard integrates with major enterprise identity providers through Clerk's SAML platform. Direct integrations are available for Microsoft Azure Active Directory, Google Workspace, and Okta Workforce, while any SAML 2.0 compliant provider can be configured through custom SAML connections. This flexibility ensures compatibility with existing enterprise infrastructure regardless of your current identity management solution. ### Authentication Flows Scorecard supports both service provider-initiated and identity provider-initiated SSO flows for maximum infrastructure flexibility. The recommended SP-initiated flow begins from Scorecard and provides enhanced security through controlled authentication paths. Alternatively, IdP-initiated flows allow users to start from your company portal for seamless integration with existing enterprise workflows. ## Organization Management Scorecard's organization structure provides secure multi-tenant architecture where each organization operates with complete data isolation. Users are automatically assigned to organizations based on their email domain, with configurable subdomain support and strict access controls. This domain-based approach ensures only authorized personnel can access your organization's projects and evaluation data. ### Just-in-Time (JIT) Provisioning JIT provisioning automatically creates user accounts during first-time SSO authentication, eliminating manual account creation overhead. When users authenticate via SAML for the first time, Scorecard provisions their accounts using data from SAML assertions, assigns organization membership based on domain rules, and maps roles according to identity provider attributes. This seamless onboarding ensures new team members gain immediate access without administrative delays. ## Security Features Scorecard implements enterprise-grade security through Clerk's proven authentication platform. The system provides comprehensive protection against CSRF and XSS attacks, secure session management with automatic rotation, and industry-standard password policies. Rate limiting prevents brute force attacks while maintaining smooth user experiences for legitimate access. SOC 2 Type II compliance ensures enterprise-grade data protection through regular security audits and comprehensive monitoring. All data remains encrypted in transit and at rest, with complete access logging and audit trails supporting compliance reporting requirements. ## Setup Process SAML SSO setup requires coordination between your IT team and Scorecard support to ensure proper configuration and testing. The process begins by contacting support with your organization details and preferred identity provider, followed by exchanging SAML metadata and configuring service provider settings in your IdP. Scorecard's team handles the technical configuration and conducts thorough testing before activating SSO for your organization. > [!NOTE] > > Contact [support@scorecard.io](mailto:support@scorecard.io) to begin enterprise authentication setup for your organization. ### User Experience SAML SSO provides a seamless authentication experience where users simply visit Scorecard, enter their email domain, and get automatically redirected to their organization's identity provider. First-time users have accounts automatically provisioned upon successful authentication, granting immediate access to organizational projects without manual account creation. ## Advanced Enterprise Features ### Multi-Organization Support Enterprise customers can manage multiple organizations with separate SSO configurations and completely isolated data. Each organization operates independently with its own SAML settings and user management, while parent companies retain cross-organization reporting capabilities for comprehensive oversight. ### Attribute Mapping and Session Controls Custom SAML attribute mapping enables advanced role-based access control by synchronizing department, team, and permission data from your identity provider. Enterprise session management provides configurable timeouts, concurrent session limits, and device-based access controls to prevent unauthorized access and account sharing. ## Support and Troubleshooting Common SSO issues typically involve SAML metadata configuration, domain matching, or IdP attribute requirements. Most authentication problems resolve by verifying that user email domains match configured domains and ensuring required attributes are being sent by your identity provider. Session-related issues often require clearing browser cookies or checking session timeout configurations. For enterprise authentication setup, technical troubleshooting, or advanced configuration requirements, contact [support@scorecard.io](mailto:support@scorecard.io) with your organization details and specific requirements. ## Related Resources > **[Projects](/features/projects)** > > Learn about project structure and permissions > **[Custom Provider](/features/custom-provider)** > > Configure custom model providers and integrations > **[Security](/features/security-and-privacy)** > > Understand Scorecard's security and privacy practices > **[Contact Sales](https://www.scorecard.io/book-a-demo)** > > Get started with enterprise authentication # GitHub Actions Integration Source: https://docs.scorecard.io/features/github-actions ## Why use GitHub Actions? GitHub Actions is a powerful, flexible automation platform for software development workflows. With Scorecard's GitHub action integration, you can test your code automatically on every PR, on a schedule, or manually with various configurations. **Automated Testing:** Perform extensive end-to-end tests for every push or pull request, or on a schedule. **Instant Actionable Feedback:** Catch issues before they reach production by getting direct, actionable feedback on your PRs. **Continuous Monitoring:** Monitor application behavior and ensure it aligns with expected outcomes. ## Setup the GitHub Integration ### 1. Setup the workflow Follow the steps on your Scorecard organization's [GitHub configuration page](https://app.scorecard.io/configure-github). 1. Install the GitHub app. 2. Choose which repository and branch to use. 3. Configure when test runs should be triggered. For example: * Nightly, or * When a user kicks off a test run from the Scorecard UI, or * When a pull request is opened, or * When a pull request is merged. 4. Add your Scorecard API key as a Github secret key named `SCORECARD_API_KEY`. 5. Click "Create pull request" to create a PR in your repository to setup the Scorecard workflow. Connect to GitHub with the GitHub configuration page. ### 2. Customize the workflow When you click "Create pull request", a PR will be created in your repository with the following files: **scorecard-eval.yml** `.github/workflows/scorecard-eval.yml` is the GitHub Actions workflow file that will be used to run the tests. You can update your testing parameters and triggering conditions here. For example, use environment variables to add any API keys you need to run your system. ```yaml scorecard-eval.yml expandable wrap theme={null} name: Scorecard Evaluation Workflow on: workflow_dispatch: inputs: project_id: description: Project ID required: true testset_id: description: Testset ID required: true metric_ids: description: Metric IDs required: true system_version_id: description: System Version ID required: false repository_dispatch: types: start-evaluation permissions: contents: read jobs: evaluation-test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.12" - name: Install dependencies run: | python -m pip install --upgrade pip pip install -r requirements.txt - name: Run test env: SCORECARD_API_KEY: ${{ secrets.SCORECARD_API_KEY }} # Scorecard config IDs # 1. Check if there's an input from manual trigger (workflow_dispatch) # 2. Fallback to values sent from external sources (repository_dispatch) # 3. Fallback to default values PROJECT_ID: ${{ github.event.inputs.project_id || github.event.client_payload.project_id || env.DEFAULT_PROJECT_ID }} TESTSET_ID: ${{ github.event.inputs.testset_id || github.event.client_payload.testset_id || env.DEFAULT_TESTSET_ID }} METRIC_IDS: ${{ github.event.inputs.metric_ids || github.event.client_payload.metric_ids || env.DEFAULT_METRIC_IDS }} SYSTEM_VERSION_ID: ${{ github.event.inputs.system_version_id || github.event.client_payload.system_version_id || env.DEFAULT_SYSTEM_VERSION_ID }} run: python3 run_tests.py ``` **run\_tests.py** This is the file that `scorecard-eval.yml` uses to run your system. You should update the `run_system()` function to call your actual system. ```python run_tests.py expandable wrap theme={null} import os import re from typing import Any from scorecard_ai import Scorecard from scorecard_ai.lib import run_and_evaluate def run_system( system_input: dict[str, Any], system_config: dict[str, Any] | None = None ) -> dict: """ FIXME: Replace this placeholder function with a call to your model """ return { "response": f"Placeholder LLM response, got input: {system_input}", } def main( *, scorecard_api_key: str, project_id: str, testset_id: str, metric_ids: list[str], system_version_id: str | None = None, ) -> None: """ Run and score all Testcases in a given Testset """ client = Scorecard(api_key=scorecard_api_key) run = run_and_evaluate( client=client, project_id=project_id, testset_id=testset_id, metric_ids=metric_ids, **({"system_version_id": system_version_id} if system_version_id else {}), system=run_system, ) print(run["url"]) if __name__ == "__main__": main( scorecard_api_key=os.environ["SCORECARD_API_KEY"], project_id=os.environ["PROJECT_ID"], testset_id=os.environ["TESTSET_ID"], metric_ids=re.findall(r"\b\d+\b", os.environ["METRIC_IDS"]), system_version_id=os.environ["SYSTEM_VERSION_ID"] or None, ) ``` **requirements.txt** This file contains the dependencies for your system. ```txt requirements.txt wrap theme={null} scorecard-ai>=3.4.0,<3.5.0 ``` ### 3. Trigger a run automatically Depending on your configuration, every push or PR automatically triggers Scorecard tests. ### 4. Trigger a run manually You can trigger a run manually from the Scorecard UI using the **Trigger Run** page. Just select a testset and some metrics, then click "Run tests"! Triggering a run manually from the Scorecard UI. # MCP Server Integration Source: https://docs.scorecard.io/features/mcp Use AI assistants as your evaluation companion with Scorecard's Model Context Protocol server ## Overview Scorecard's MCP (Model Context Protocol) server lets you manage projects, create testsets, configure metrics, run evaluations, and analyze results through natural language in any MCP-compatible client. ## How it works The server uses **Code Mode**: instead of one tool per API endpoint, it exposes two tools and lets your assistant write code against the [Scorecard TypeScript SDK](https://github.com/scorecard-ai/scorecard-node). | Tool | What it does | | - | - | | `search_docs` | Searches the SDK documentation for the right method and parameters. | | `execute` | Runs TypeScript your assistant writes against a pre-authenticated SDK client in a sandbox, and returns only what the code returns or logs. | A typical `execute` call looks like this: ```typescript theme={null} async function run(client) { const results = []; for await (const project of client.projects.list()) { const runs = await client.runs.list(project.id, { limit: 1 }); results.push({ project: project.name, latestRun: runs.data[0]?.id ?? null }); } return results; } ``` Because filtering and aggregation happen inside the sandbox, only the final result comes back to the conversation. A question that spans many projects, runs, or large records completes in a single call instead of paging multi-megabyte responses through your assistant's context. > [!NOTE] > > Each `execute` call is independent: variables do not persist between calls. Individual API requests time out after 30 seconds and a whole `execute` call after about 5 minutes. The sandbox can only reach the Scorecard API, with no filesystem or other network access. ## Setting Up the MCP Server ### Claude Code Add the Scorecard remote MCP server with a single command: ```bash theme={null} claude mcp add --transport http scorecard https://mcp.scorecard.io/mcp ``` Complete the OAuth authentication flow in your browser when prompted. Verify the connection: ```bash theme={null} claude mcp list ``` You should see `scorecard: https://mcp.scorecard.io/mcp (HTTP) - ✓ Connected`. ### Claude Desktop Go to Claude Desktop settings and click the "Connectors" tab. Click "Add custom connector" and paste the URL: `https://mcp.scorecard.io/mcp`. Click "Add", then "Connect" to login to Scorecard. Adding the Scorecard MCP connector in Claude Desktop. ### Other clients Any client that supports remote MCP servers over Streamable HTTP can use `https://mcp.scorecard.io/mcp`. If your client cannot complete an OAuth flow, pass a [Scorecard API key](https://app.scorecard.io/settings) as a bearer token instead: ```json theme={null} { "mcpServers": { "scorecard": { "url": "https://mcp.scorecard.io/mcp", "headers": { "Authorization": "Bearer ak_MyAPIKey" } } } } ``` ### Local configuration You can also run the same server locally via npx: ```sh theme={null} export SCORECARD_API_KEY="your_api_key" npx -y scorecard-ai-mcp@latest ``` For clients with a configuration JSON: ```json theme={null} { "mcpServers": { "scorecard_ai": { "command": "npx", "args": ["-y", "scorecard-ai-mcp"], "env": { "SCORECARD_API_KEY": "ak_MyAPIKey" } } } } ``` > [!WARNING] > > The local server runs `execute` code in a Deno sandbox on your machine, so [Deno](https://deno.land) must be installed and on your `PATH`. Documentation search works offline from an index bundled with the package. To restrict what the assistant can do, limit the `execute` tool with flags. For example, a read-only server: ```json theme={null} "args": ["-y", "scorecard-ai-mcp", "--code-allow-http-gets"] ``` You can also allow or block specific SDK methods with `--code-allowed-methods` and `--code-blocked-methods`, which take regular expressions matched against method names such as `records.delete`. ## Examples ### Create a project and testset ``` Create a new Scorecard project called "Support Bot Eval". Then create a testset called "Support Scenarios" with 10 testcases. Each testcase should have: - inputs: "customerMessage" and "category" (billing, technical, or product) - expected: "idealResponse" ``` ### Create metrics ``` Create two metrics in the "Support Bot Eval" project: 1. "Response Accuracy" (integer 1-5) - How well does the response answer the question? 2. "Tone" (boolean) - Is the response professional and empathetic? ``` ### Analyze results ``` Show me the latest run results for the "Support Bot Eval" project. Which testcases scored lowest on Response Accuracy? ``` ### Compare across projects ``` For every project in my organization, find the most recent run and report the pass rate on each boolean metric. Rank the projects from best to worst. ``` ### Review human feedback ``` For the latest run in the "Support Bot Eval" project, list the annotations on each record and summarize the thumbs-down ratings. What do reviewers complain about most? ``` ### Generate testcases from a codebase In Claude Code, you can combine file access with the MCP server: ``` Read the API routes in src/api/ and generate 20 testcases covering the edge cases for each endpoint. Add them to the "API Tests" testset in project 1234. ``` ### Iterate on metrics ``` The "Response Accuracy" metric is too lenient — update the prompt template to penalize responses that miss key details from the ideal response. ``` ## Technical Details * Built on the [Model Context Protocol](https://modelcontextprotocol.io/) standard using the Streamable HTTP transport * Compatible with any MCP client (Claude Code, Claude Desktop, Cursor, and more) * Secured with OAuth authentication, or a Scorecard API key passed as a bearer token * The hosted server runs the [`scorecard-ai-mcp`](https://www.npmjs.com/package/scorecard-ai-mcp) package (3.4.0), whose source lives in the [scorecard-node](https://github.com/scorecard-ai/scorecard-node/tree/main/packages/mcp-server) repository * Anything the SDK can do is available: see the [API reference](/api-reference/overview) for the full surface ## Migrating from the per-endpoint tools Earlier versions of the server exposed one tool per API operation, such as `list_projects` or `create_testsets`. Those tools no longer exist. Prompts written in natural language keep working unchanged, because the assistant now reaches the same operations through `execute`. If you referenced tool names directly, the table below shows where each one went. > **Old tool → SDK call mapping** > > | Old tool | SDK call inside `execute` | > | - | - | > | `list_projects`, `create_projects` | `client.projects.list()`, `client.projects.create({...})` | > | `list_testsets`, `get_testsets`, `create_testsets`, `update_testsets`, `delete_testsets` | `client.testsets.list(projectId)`, `.get(id)`, `.create(projectId, {...})`, `.update(id, {...})`, `.delete(id)` | > | `list_testcases`, `get_testcases`, `create_testcases`, `update_testcases`, `delete_testcases` | `client.testcases.list(testsetId)`, `.get(id)`, `.create(testsetId, {...})`, `.update(id, {...})`, `.delete({ ids })` | > | `list_systems`, `get_systems`, `upsert_systems`, `update_systems`, `delete_systems` | `client.systems.list(projectId)`, `.get(id)`, `.upsert(projectId, {...})`, `.update(id, {...})`, `.delete(id)` | > | `get_systems_versions`, `upsert_systems_versions` | `client.systems.versions.get(id)`, `client.systems.versions.upsert(systemId, {...})` | > | `list_metrics`, `get_metrics`, `create_metrics`, `update_metrics`, `delete_metrics` | `client.metrics.list(projectId)`, `.get(id)`, `.create(projectId, {...})`, `.update(id, {...})`, `.delete(id)` | > | `list_runs`, `get_runs`, `create_runs` | `client.runs.list(projectId)`, `.get(id)`, `.create(projectId, {...})` | > | `list_records`, `create_records`, `delete_records` | `client.records.list(runId)`, `.create(runId, {...})`, `.delete(id)` | > | `upsert_scores` | `client.scores.upsert(metricConfigId, { recordId, ... })` | > | `list_annotations` | `client.records.annotations.list(recordId)` | > | `search_docs` | Unchanged | > > Record tags, record assignees, and session attachments were never exposed as tools but are available through `client.records.tags`, `client.records.assignees`, and `client.attachments`. # Metrics Source: https://docs.scorecard.io/features/metrics > [!NOTE] > > Metrics serve as benchmarks for assessing the quality of LLM responses, while scoring is the process of applying these metrics to generate actionable insights about your LLM application. When evaluating LLM applications, you start with a "vibe check"—manually testing prompts to see if responses make sense. As your application evolves, you need a systematic way to quantify and consistently measure response quality. Metrics provide this standard of measurement for LLM evaluation. ## Assess Your LLM Quality With Scorecard's Metrics Scorecard's metric system is organized into three main components: * **Metrics**: Individual evaluation criteria that assess specific aspects of your LLM's performance * **Metric Groups**: Collections of related metrics for comprehensive evaluation * **Templates**: Pre-built metric configurations you can copy and customize Scorecard assists you in several ways in defining and organizing metrics for LLM evaluation. ### Metric Templates Our Scorecard Core Metrics represent industry-standard benchmarks for LLM performance, validated by our team of evaluation experts. Explore these templates on the Metrics page under "Templates". Overview of Scorecard Core Metrics in Templates Copy any template and customize it for your specific use case. Metric Template to Copy #### MLflow-Inspired Metrics Scorecard provides support for MLflow-style metrics with additional capabilities like aggregation, A/B comparison, and iteration. Available metrics include: * **Relevance**: Evaluates how well the response aligns with the input and context * **Answer Relevance**: Assesses relevance and applicability to the specific query * **Faithfulness**: Measures factual consistency with provided context * **Answer Correctness**: Evaluates accuracy against ground truth * **Answer Similarity**: Assesses semantic similarity to expected responses View full prompts on the Metrics page or in the [MLflow GitHub repository](https://github.com/mlflow/mlflow/blob/master/mlflow/metrics/genai/prompts/v1.py). #### RAGAS-Inspired Metrics for RAG Pipelines Scorecard provides RAGAS-style metrics for evaluating Retrieval Augmented Generation (RAG) systems: **Component-Wise:** * **Faithfulness**: Factual consistency with context * **Answer Relevancy**: Pertinence to the prompt * **Context Recall**: Retrieved context alignment with expected response * **Context Precision**: Ranking of ground-truth relevant items * **Context Relevancy**: Relevance of retrieved context to query **End-to-End:** * **Answer Semantic Similarity**: Semantic resemblance to ground truth * **Answer Correctness**: Accuracy compared to ground truth View full prompts on the Metrics page or in the [RAGAS GitHub repository](https://github.com/explodinggradients/ragas/tree/main/src/ragas/metrics). > [!NOTE] > > Templates for common MLflow and RAGAS metrics are available under Templates. Copy a template (e.g., Relevance, Faithfulness) and tailor the guidelines to your domain. ### Define Custom Metrics for Your LLM Use Case Adding a New Custom Metric On the Metrics page, click "+ New Metric" to create a custom metric. Configure: * **Metric Name**: Human-readable name * **Metric Guidelines**: Natural language instructions for evaluation * **Evaluation Type**: * **AI**: Uses Metric Guidelines as prompt for an AI model * **Human**: Manual evaluation by subject-matter experts * **Heuristic (Code)**: Custom Python or TypeScript logic (see [Heuristic Metrics](#heuristic-metrics)) * **Output Type**: Boolean, Integer (1-5), or Float (0.0-1.0) New Custom Metric ### Output Types #### Boolean Output Type * **Range**: true/false * **Pass/Fail**: Direct pass/fail representation * **Use cases**: Format checks, refusals, policy/guardrails, presence/absence * **Aggregation**: Pass ratios and counts Example model output for a boolean metric: ```json theme={null} { "reasoning": "The model correctly refused unsafe content.", "binaryScore": true } ``` #### Integer Output Type * **Range**: 1–5 (higher is better) * **Pass/Fail**: Set a `passingThreshold` (e.g., 4). Passes if `intScore ≥ threshold` * **Use cases**: Rubric-based quality judgments (helpfulness, factuality, completeness) * **Aggregation**: Means and distributions Example model output for an integer metric: ```json theme={null} { "reasoning": "Mostly correct but missing minor details.", "intScore": 4 } ``` #### Float Output Type Normalized score between 0.0 and 1.0 for graded, continuous measurement. * **Range**: 0.0–1.0 (higher is better) * **Pass/Fail**: Set a `passingThreshold` (e.g., 0.90). Passes if `floatScore ≥ threshold` * **Use cases**: Semantic similarity, confidence/uncertainty, coverage, quality scores * **Aggregation**: Means for trend tracking Example model output for a float metric: ```json theme={null} { "reasoning": "The response is mostly correct with minor omissions.", "floatScore": 0.87 } ``` ## Automated Scoring As testsets grow, manual evaluation becomes time-consuming. Subject-matter experts (SMEs) can become bottlenecks as the number of testcases increases. Scorecard's AI-powered scoring automates evaluation, allowing SMEs to focus on complex edge cases while the system handles well-defined metrics. ### AI-Based Metrics When creating a metric, select "AI" as the evaluation type for automated scoring. New Metric With AI-Scoring #### Configuration Modes * **Basic Mode**: Define metric guidelines using natural language instructions * **Advanced Mode**: Modify the full prompt template for sophisticated evaluation logic > [!WARNING] > > Be very specific with metric guidelines—they form the core instruction for AI scoring. #### Evaluation Settings Configure AI evaluation parameters: * **Model Selection**: Choose models like GPT-4o * **Temperature**: Control randomness (typically 0 for consistency) * **Additional Parameters**: Model-specific settings ### Automatic Scoring When you score records (from the testset page, the Records page, or through GitHub Actions), AI and Heuristic metrics are evaluated automatically. You can view scoring progress and results on the Records page. > [!TIP] > > New to Scorecard? See the [UI Quickstart](/intro/ui-quickstart) to learn how to score your first records. Run Scoring in the Scorecard UI If needed, you can score again by selecting records and clicking "Score Records". #### Score Explanations Scorecard provides explanations for each score, helping you understand and validate the AI's evaluation reasoning. Score Explained by the Scorecard AI Model ### Heuristic Metrics (Code-Based Evaluation) Write custom evaluation logic in Python or TypeScript for deterministic, rule-based scoring: * Checking for specific keywords or patterns * Validating JSON structure or format * Computing text similarity scores * Measuring response length or complexity * Custom business logic checks #### Creating a Heuristic Metric Select **Heuristic** as the evaluation type to access the code editor. Creating a heuristic metric with Python code #### Writing Heuristic Code Your code receives the record data and must return a score. Here's the structure: **Python** ```python theme={null} def score(inputs: dict, outputs: dict, expected: dict) -> dict: """ Evaluate the output and return a score. Args: inputs: The input fields from the testcase outputs: The generated output from your system expected: The expected output fields from the testcase Returns: A dict with 'score' (bool, int 1-5, or float 0.0-1.0) and optional 'reasoning' """ # Example: Check if output contains expected keyword output_text = outputs.get("response", "") expected_keyword = expected.get("keyword", "") if expected_keyword.lower() in output_text.lower(): return {"score": True, "reasoning": "Keyword found in response"} else: return {"score": False, "reasoning": "Keyword not found"} ``` **TypeScript** ```typescript theme={null} function score( inputs: Record, outputs: Record, expected: Record ): { score: boolean | number; reasoning?: string } { // Example: Check response length const response = outputs.response || ""; const minLength = expected.minLength || 100; if (response.length >= minLength) { return { score: true, reasoning: `Response meets minimum length of ${minLength}` }; } else { return { score: false, reasoning: `Response too short: ${response.length} < ${minLength}` }; } } ``` #### Secure Sandbox Execution Code runs in a secure, isolated sandbox ensuring: * **Safety**: No external resource access * **Consistency**: Deterministic results * **Performance**: Optimized execution > [!NOTE] > > The sandbox has limited access to external libraries. Common utilities like string manipulation and JSON parsing are available. Contact support if you need specific libraries for your use case. ### Metric Groups Group related metrics for consistent, repeatable evaluation. Metric Groups let you apply multiple metrics when scoring without manual selection each time. Create groups for specific use cases like RAG applications or translation tasks. Overview of Metric Groups View and create Metric Groups on the Metrics page under "Metric Groups". Provide: * **Name**: Human-readable identifier * **Description**: Short summary of the group's purpose Defining a New Metric Group Click "Select Metrics" to choose which metrics to include in the group. # Organizations & Projects Source: https://docs.scorecard.io/features/projects Manage your team's evaluation work with Organizations and Projects Scorecard uses a hierarchical structure to organize your evaluation work: **Organizations** contain multiple **Projects**, and each Project contains your testsets, metrics, runs, and results. ## Organizations Organizations are the top-level entity in Scorecard that enables team collaboration and workspace isolation. All members of an organization can access all projects within that organization. API keys are scoped to a specific *organization*, not user or project. Organization member management dialog with a few users. ## Projects Each project serves as a container for: * **Testsets**: Collections of Testcases for evaluation * **Metrics**: Custom evaluation criteria and scoring methods * **Records**: Evaluation executions and their results, traces and logs Projects overview page for an organization. ### Creating a New Project To create a project, click **"Create Project"** in the projects menu and go through the steps. > [!TIP] > > Choose a descriptive project name for the system or use case you're evaluating, such as "Customer Support Chatbot" or "Product Recommendation Engine." Creating a new project. # Prompts Source: https://docs.scorecard.io/features/prompts Create, version, test, and deploy prompts with integrated evaluation workflows. Prompts in Scorecard are configurable templates that generate inputs for language models. They support dynamic variable substitution and multi-message conversations. Prompt management page for a "docs search" LLM system. ## Create a Prompt Go to the Prompts page in Scorecard and click the **"New Prompt +"** button. Provide a name and description for your prompt, then choose your AI model and configure parameters: * **Model**: Select from available models (e.g. `gpt-5-mini`, `opus-4.8`). * If you don't see the LLM provider you want, add the provider API key in [organization settings](https://app.scorecard.io/settings). * **Temperature**: Control randomness (0.0 = deterministic, 1.0 = creative). * **Max Tokens**: Set maximum response length. * **Top-P**: Fine-tune token selection probability. Then, write your prompt content using the built-in editor with Jinja template support. Prompt creation dialog with model settings. ## Prompt Templates & Jinja Syntax ### Template Structure Scorecard prompts support multi-message conversations with different roles: * **System**: Instructions for the AI model's behavior * **User**: The main prompt or question * **Assistant**: Previous responses (for conversation context) ### Variable substitution Scorecard supports Jinja template syntax for variable substitution. This lets you insert dynamic content from your testcases into your prompt by wrapping the Testcase's input's field name in `{{` and `}}`. For example, if your Testcase has a `customer_message` field, your prompt might be: ```jinja wrap theme={null} You are a helpful customer service assistant. Customer inquiry: {{customer_message}} Product: {{product_name}} Customer tier: {{customer_tier}} Please provide a helpful response addressing their concern. ``` Scorecard also has a special `{{allInputs}}` variable that contains all the testcase input fields as a formatted string. > **Advanced Jinja features** > > Scorecard supports [full Jinja functionality](https://maxbot.ai/design-reference/jinja/) including: > > * Conditional logic: `{% if customer_tier == "premium" %} User is a premium customer. {% endif %}` > * Loops: `{% for item in product_list %} {{item.name}} {% endfor %}` > * Filters: `{{customer_message | upper}}` > * Comments: `{# This is a comment #}` ## Version Management The prompt details page shows a version history of the prompt. You can create a new version of the prompt by modifying the existing prompt or parameters and clicking **"Save Changes"**. Selecting "Publish this version to production" will mark that version as the "production" version. Otherwise, it'll be an experimental version. You can mark any version as production by clicking the **"Publish"** button, but exactly one version is marked as production at any time. Prompt version dropdown with publish button. Before publishing to production, test a new version with a [Kickoff Run](/features/runs#kickoff-run-from-the-ui) to be confident that it behaves as expected and won't regress on any testcases. ## SDK Integration You can programmatically manage prompts and their versions using the Scorecard SDK. Since prompts are implemented as systems in Scorecard, you'll use the systems API methods. ### Creating or Updating Prompts Use the `systems.upsert()` method to create a new prompt or update an existing one. For complete API details, see the [Create (upsert) system API reference](/api-reference/create-upsert-system). ```javascript JavaScript theme={null} import Scorecard from 'scorecard-ai'; const client = new Scorecard({ apiKey: process.env['SCORECARD_API_KEY'], }); // Create or update a prompt (system) const prompt = await client.systems.upsert('your-project-id', { name: 'Customer Service Assistant', description: 'AI assistant for handling customer inquiries', config: { modelName: 'gpt-5', temperature: 0.7, maxTokens: 1024, topP: 0.9, promptTemplate: [ { role: 'user', content: `You are a helpful assistant. {{allInputs}}` } ] } }); console.log('Prompt created with ID:', prompt.id); ``` ```python Python theme={null} from scorecard import Scorecard import os client = Scorecard( api_key=os.environ.get("SCORECARD_API_KEY") ) # Create or update a prompt (system) prompt = client.systems.upsert( project_id="your-project-id", name="Customer Service Assistant", description="AI assistant for handling customer inquiries", config={ "modelName": "gpt-5", "temperature": 0.7, "maxTokens": 1024, "topP": 0.9, "promptTemplate": [ { "role": "user", "content": """You are a helpful assistant. {{allInputs}}""" } ] } ) print(f"Prompt created with ID: {prompt.id}") ``` The `config` object contains your prompt's model parameters and template: * **modelName**: The LLM model to use (e.g., `gpt-5`, `claude-opus-4-8`) * **temperature**: Controls randomness (0.0-1.0) * **maxTokens**: Maximum response length * **topP**: Token selection probability * **promptTemplate**: Array of message objects defining the conversation structure. Each message has a `role` (`system`, `user`, or `assistant`) and `content` with your prompt text and Jinja variables. > [!NOTE] > > For a system to be flagged as a prompt in Scorecard, it must include the `promptTemplate` configuration in its config object. Without this field, the system will not be recognized as a prompt. ### Creating Prompt Versions Use the `systems.versions.upsert()` method to create new versions of your prompt. For complete API details, see the [Upsert system version API reference](/api-reference/upsert-system-version). ```javascript JavaScript theme={null} // Create a new version of the prompt const promptVersion = await client.systems.versions.upsert('your-system-id', { name: 'v2.0 - Improved tone', config: { modelName: 'gpt-5', temperature: 0.5, maxTokens: 1024, topP: 0.8, promptTemplate: [ { role: 'user', content: `You are a helpful assistant. {# You can replace this with the specific inputs from your testcase #} {{allInputs}}` } ] } }); console.log('New prompt version created:', promptVersion.id); ``` ```python Python theme={null} # Create a new version of the prompt prompt_version = client.systems.versions.upsert( system_id="your-system-id", name="v2.0 - Improved tone", config={ "modelName": "gpt-5", "temperature": 0.5, "maxTokens": 1024, "topP": 0.8, "promptTemplate": [ { "role": "user", "content": """You are a helpful assistant. {# You can replace this with the specific inputs from your testcase #} {{allInputs}}""" } ] } ) print(f"New prompt version created: {prompt_version.id}") ``` > [!NOTE] > > Ready to evaluate your prompts? Follow the [Runs guide](/features/runs) to test your prompts on Testsets. # Records Source: https://docs.scorecard.io/features/records View, filter, and analyze all evaluation records across your project. The **Records** page provides a unified view of all evaluation records across runs in your project. Use it to search, filter, analyze patterns, and bulk re-score records without navigating through individual runs. Records page with filtering and history chart. ## What is a Record? A **Record** is an individual test execution within a run. Each record contains: * **Inputs**: The data sent to your AI system * **Outputs**: The response generated by your system * **Expected (Labels)**: Ground truth or ideal responses for comparison * **Scores**: Evaluation results from each metric * **Status**: Whether scoring is pending, completed, or errored Records are created when you run evaluations via the API, a Kickoff Run, or from traces. ## Customizing the Table Click **Edit Table** to customize which columns appear and their order. You can add, remove, and reorder columns including: * **Base columns**: ID, Created By, Created At * **Data fields**: Inputs, Outputs, Expected * **Source**: How the record was created (API, Kickoff, Trace). Records created before the Playground was retired may still show **Playground**. * **Metrics**: Score columns for each metric in your project Edit Table to customize columns and their order. Your column preferences are saved per project. ## History Chart The interactive histogram shows record distribution over time. Click any bar to filter records to that time period. ## Bulk Re-scoring Select multiple records using the checkboxes, then click **Re-score** to re-evaluate them with your metrics. This is useful when: * You've updated a metric's guidelines * You want to apply new metrics to existing records * You need to re-evaluate after fixing a configuration issue > [!TIP] > > Re-scoring uses the latest version of your metrics without re-running your AI system. ## Record Details Click any record to view its full details. The details view differs based on how the record was created: ### Testcase-Based Records Records created from testsets show: * **Scores**: Pass/fail status, reasoning, and metric properties for each evaluation * **Test Record Details**: Input fields, expected outputs, and actual outputs Testcase-based record showing scores and test details. ### Trace-Based Records Records created from production traces show: * **Trace Overview**: Duration, estimated cost, total tokens, and span count * **Spans**: Individual LLM calls with timing and cost breakdown * **Model Usage**: Which models were called and token counts For chat and agent traces, the **Conversation** view (one of the tabs at the top of the record, alongside **Timeline**, **Chat**, and **Debug**) renders the trace as a threaded conversation: each user turn, assistant response, and tool call appears as an expandable step, so you can follow what the agent did without reading the raw span tree. Conversation view of a trace-based record, showing a user turn followed by tool-call steps. ## Use Cases * **Cross-run analysis**: Find patterns across multiple evaluation runs * **Debugging failures**: Filter by `metric.status:fail` to investigate failing records * **Quality review**: Review records from specific time periods or sources * **Metric iteration**: Re-score records after updating metric guidelines # Runs & Results Source: https://docs.scorecard.io/features/runs Execute evaluations and analyze AI agent performance. > [!WARNING] > > The Runs page is deprecated. Please use the [Records](/features/records) page instead. A **Run** is an execution that evaluates your AI agent against some Testcases using specified metrics. Runs generate **Records** (individual test executions) and **Scores** (evaluation results) for each Record that help you understand your agent's performance across different scenarios. Run details page. ## What is a Run? Every run consists of: * (Optional) **Testset**: Collection of test cases to evaluate against * (Optional) **Metrics**: Evaluation criteria that score system outputs * (Optional) **System Version**: Configuration defining your AI system's behavior * **Records**: Individual test executions, one per Testcase * **Scores**: Evaluation results for each record against each metric List of recent runs with statuses. ### Filtering Runs You can filter runs by their **source** to quickly find runs created from specific workflows: * **API**: Runs created programmatically via the SDK or API * **Monitor**: Runs created automatically by production monitors * **Kickoff**: Runs created via the Kickoff Run modal in the UI ## Creating Runs ### Kickoff Run from the UI You can kickoff a run from the Projects dashboard, Testsets list, or a Run ("Run again" button). The **Kickoff Run** modal lets you choose the Testset, Prompt, and Metrics for the run. The **Scorecard** tab lets you run using an LLM on Scorecard's servers, so you can specify the LLM parameters. The **GitHub** tab lets you trigger a run using GitHub Actions on your actual system. Kickoff Run modal. ### Using the API You can create runs and records programmatically using the Scorecard SDK. ```python Python theme={null} from scorecard_ai import Scorecard from scorecard_ai.lib import run_and_evaluate run = run_and_evaluate( client=Scorecard(), project_id="123", testset_id="456", metric_ids=["789", "101"], system=lambda system_input: ask_llm(system_input), ) print(f'Go to {run["url"]} to view your results.') ``` ```JavaScript JavaScript theme={null} import Scorecard, { runAndEvaluate } from 'scorecard-ai'; const scorecard = new Scorecard(); const run = await runAndEvaluate(scorecard, { projectId: "123", metricIds: ["789", "101"], testsetId: "456", system: (systemInput) => askLLM(systemInput), }); console.log(`Go to ${run.url} to view your results.`); ``` See the [SDK quickstart](/intro/quickstart) or [API reference](/api-reference/overview) for more details. ### Via GitHub Actions Using Scorecard's [GitHub integration](/features/github-actions), you can trigger runs automatically (e.g. on pull request or a schedule). With the integration set up, you can also trigger runs of your real system from the Scorecard UI. Triggering a run through the Scorecard GitHub integration. ## Run Status If you click on the run status of a run, it will expands to show an explanation of the scoring status for each of the metrics in the run. Run status explanation. ## Re-run Scoring You can re-run scoring on existing runs without re-executing your system. This is useful when: * You've updated a metric's guidelines and want to see how it affects existing results * You want to add new metrics to a completed run * You need to re-evaluate after fixing a metric configuration issue To re-run scoring, open a run's details page and click the **Re-run Scoring** button. You can select which metrics to include in the re-scoring. Re-run scoring modal for an existing run. > [!TIP] > > Re-running scoring is much faster than creating a new run since it skips the system execution step and only re-evaluates the existing outputs. ## Inspect a Record Record score explanation. Drill down into specific test executions for detailed analysis: **Record Overview:** * **Inputs**: Original Testcase data sent to your system * **Outputs**: Generated responses from your AI system * **Expected Outputs**: Ideal responses for comparison * **Scores**: Detailed evaluation results from each metric ## Records Page The **[Records](/features/records)** page provides a project-level view of all test records across your runs. Use it to search, filter, and analyze records without navigating through individual runs. ## Export a Run Click the **Export as CSV** button on the run details page. ## Run Notes Click **Show Details** on a run page to view or edit the run notes and view the system/prompt version the run was executed with. Run details with notes and system configuration. > [!NOTE] > > Run data includes potentially sensitive information from your Testsets and system outputs. Follow your organization's data handling policies and avoid including PII or confidential data in test configurations. ## Run History View your AI system's performance trends over time with interactive charts for each metric on the run list page. Run history helps you identify performance trends, compare metrics side-by-side, and track consistency across runs. Run history charts showing three metrics' performance over time. Use the dropdown to select which metrics to display. Hover over any data point to see detailed run information and scores, with synchronized highlighting across all visible charts. Binary metrics display as percentages (passing rate) while integer metrics show the average score for that metric. > [!TIP] > > Click any metric title to navigate to its detailed configuration page. # SDK + Tracing Source: https://docs.scorecard.io/features/sdk-tracing Combine structured evaluation data with detailed trace observability in a single record. The [Scorecard SDK](/intro/sdk-quickstart) lets you evaluate AI systems with structured inputs, outputs, and scores. [Tracing](/features/tracing) captures every LLM call, tool use, and retry with full observability. SDK + Tracing combines both into a single record. A single record with SDK inputs, outputs, and expected values alongside trace spans. ## Why combine SDK + Tracing? Oftentimes, traces become cluttered with data, and SDK records don't contain enough information to understand **why** something went wrong. To supplement the shortcomings of one or the other, we allow you to link the traces created by runs to the SDK, showing all of your data in one place: * **Full observability** into every LLM call, tool use, and retry * **Cost and latency breakdown** per span * **Conversation view** for chat-based traces * **Debugging context** when a test case fails — see exactly which step went wrong *** ## Setup You need two things: the **Scorecard SDK** to create records, and **tracing instrumentation** so your system emits OpenTelemetry traces. ### 1. Install the SDK ```bash Python theme={null} pip install 'scorecard-ai[otel]' ``` ```bash JavaScript theme={null} npm install scorecard-ai ``` ### 2. Enable tracing in your system Choose any [instrumentation method](/features/tracing#instrumentation-methods). The simplest approach depends on your stack: **Agents and frameworks** (Claude Code, Claude Agent SDK, OpenAI Agents SDK) — set environment variables: ```bash theme={null} export BETA_TRACING_ENDPOINT="https://tracing.scorecard.io/otel" export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " export ENABLE_BETA_TRACING_DETAILED=1 export OTEL_LOG_USER_PROMPTS=1 export OTEL_LOG_TOOL_DETAILS=1 export OTEL_LOG_TOOL_CONTENT=1 export OTEL_RESOURCE_ATTRIBUTES="scorecard.project_id=" ``` **SDK wrappers** (OpenAI, Anthropic) — wrap your client: ```python Python theme={null} from scorecard_ai import wrap from openai import OpenAI openai = wrap(OpenAI(), {"project_id": "YOUR_PROJECT_ID"}) ``` ### 3. Use `runAndEvaluate()` with your traced system **Claude Agent SDK** For agent frameworks, pass `otelLinkId` from the `options` argument into your agent's `OTEL_RESOURCE_ATTRIBUTES` so traces link to records. > [!WARNING] > > In TypeScript, the `env` option replaces the subprocess environment rather than extending it, so spread `process.env` in as shown below—otherwise the agent loses its API credentials and `PATH`. The Python SDK merges `env` on top of the inherited environment automatically. ```typescript TypeScript highlight={17} theme={null} import { query } from '@anthropic-ai/claude-agent-sdk'; import Scorecard, { runAndEvaluate, type SystemOptions } from 'scorecard-ai'; const scorecard = new Scorecard(); const PROJECT_ID = 'YOUR_PROJECT_ID'; async function system(input: { question: string }, options?: SystemOptions) { let resultText = ''; for await (const message of query({ prompt: input.question, options: { env: { ...process.env, ENABLE_BETA_TRACING_DETAILED: '1', BETA_TRACING_ENDPOINT: 'https://tracing.scorecard.io/otel', OTEL_EXPORTER_OTLP_HEADERS: `Authorization=Bearer ${process.env.SCORECARD_API_KEY}`, OTEL_LOG_USER_PROMPTS: '1', OTEL_LOG_TOOL_DETAILS: '1', OTEL_LOG_TOOL_CONTENT: '1', OTEL_RESOURCE_ATTRIBUTES: `scorecard.otel_link_id=${options!.otelLinkId},scorecard.project_id=${PROJECT_ID}`, }, }, })) { if (message.type === 'result' && message.subtype === 'success') resultText = message.result; } return { answer: resultText }; } const run = await runAndEvaluate(scorecard, { projectId: PROJECT_ID, metricIds: ['YOUR_METRIC_ID'], system, testcases: [ { inputs: { question: 'What is the capital of France?' }, expected: { answer: 'Paris' } }, ], }); ``` ```python Python highlight={21} theme={null} import os, asyncio from claude_agent_sdk import query, ClaudeAgentOptions from scorecard_ai import Scorecard from scorecard_ai.lib import run_and_evaluate scorecard = Scorecard() PROJECT_ID = "YOUR_PROJECT_ID" def system(inputs, _system_version, options): otel_link_id = options["otel_link_id"] async def run_agent(): result_text = "" async for message in query( prompt=inputs["question"], options=ClaudeAgentOptions(env={ **os.environ, "ENABLE_BETA_TRACING_DETAILED": "1", "BETA_TRACING_ENDPOINT": "https://tracing.scorecard.io/otel", "OTEL_EXPORTER_OTLP_HEADERS": f"Authorization=Bearer {os.environ['SCORECARD_API_KEY']}", "OTEL_LOG_USER_PROMPTS": "1", "OTEL_LOG_TOOL_DETAILS": "1", "OTEL_LOG_TOOL_CONTENT": "1", "OTEL_RESOURCE_ATTRIBUTES": f"scorecard.otel_link_id={otel_link_id},scorecard.project_id={PROJECT_ID}", }), ): if hasattr(message, "result") and getattr(message, "subtype", None) == "success": result_text = message.result return result_text return {"answer": asyncio.run(run_agent())} run = run_and_evaluate( client=scorecard, project_id=PROJECT_ID, metric_ids=["YOUR_METRIC_ID"], system=system, testcases=[ {"inputs": {"question": "What is the capital of France?"}, "expected": {"answer": "Paris"}}, ], ) ``` The key is setting `scorecard.otel_link_id` in `OTEL_RESOURCE_ATTRIBUTES` — this tells the trace pipeline which record to merge into. **SDK Wrappers (OpenAI, Anthropic)** With SDK wrappers, use `wrap()` to instrument your client and manually set `scorecard.otel_link_id` on a span so traces link to records. ```python Python highlight={14} theme={null} from scorecard_ai import Scorecard, wrap from scorecard_ai.lib import run_and_evaluate, SystemOptions from openai import OpenAI from opentelemetry import trace scorecard = Scorecard() PROJECT_ID = "YOUR_PROJECT_ID" openai = wrap(OpenAI(), {"project_id": PROJECT_ID}) def system(inputs, _system_version, options: SystemOptions): tracer = trace.get_tracer("scorecard-llm") with tracer.start_as_current_span("my_system") as span: span.set_attributes({"scorecard.otel_link_id": options["otel_link_id"]}) response = openai.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": inputs["question"]}], ) return {"answer": response.choices[0].message.content} run = run_and_evaluate( client=scorecard, project_id=PROJECT_ID, metric_ids=["YOUR_METRIC_ID"], system=system, testcases=[ {"inputs": {"question": "What is the capital of France?"}, "expected": {"answer": "Paris"}}, ], ) ``` The `wrap()` call sets up the OTel tracer and exporter. Setting `scorecard.otel_link_id` on a parent span ensures all child LLM calls are linked to the correct record. Call `trace.get_tracer_provider().force_flush()` before your process exits to ensure traces are sent. *** ## How it works `runAndEvaluate()` generates a unique `otelLinkId` for each testcase and passes it to your system function via the `options` argument. Your system can then insert this ID into the trace's resource attributes (`scorecard.otel_link_id`), which tells the tracing pipeline to merge the trace into the corresponding record. *** ## Multi-trace sessions Some systems emit multiple traces per invocation — for example, the Claude Agent SDK generates a separate trace for each turn in a multi-turn conversation. Scorecard groups these into a single record using the **session ID**. When multiple traces share the same `session.id`, they are combined into a unified span tree under a session root, with trace IDs accumulated rather than overwritten. Your SDK inputs and outputs are preserved across all session merges. *** ## Troubleshooting | Symptom | Fix | | - | - | | **Inputs/outputs not showing in Trace Overview** | Verify your system function returns a value. Check that `scorecard.project_id` in the trace matches the project in `runAndEvaluate()`. | | **Trace not linking to the record** | Ensure traces are sent to `https://tracing.scorecard.io/otel`. Confirm the API key belongs to the same org. Call `force_flush()` before the process exits. | | **Scores reference trace data instead of SDK data** | `{{inputs.*}}` and `{{outputs.*}}` resolve to SDK data automatically. If you see raw trace JSON, the record wasn't created via `runAndEvaluate()`. | *** ## Next steps * [SDK Quickstart](/intro/sdk-quickstart) — Set up `runAndEvaluate()` from scratch * [Tracing](/features/tracing) — All instrumentation methods * [Metrics](/features/metrics) — Define evaluation criteria * [Records](/features/records) — Search and filter records # Security & Privacy Source: https://docs.scorecard.io/features/security-and-privacy Preserving the privacy of our clients and ensuring secure processes is of top priority at Scorecard. Scorecard AI is [SOC 2](https://www.aicpa-cima.com/topic/audit-assurance/audit-and-assurance-greater-than-soc-2) compliant and various measures are undertaken regarding security, data privacy, and compliance. For more information, visit our [Trust Center](https://trust.scorecard.io/). Scorecard's Trust Center. ## Data Encryption and Data Security Scorecard AI takes extensive measures to ensure the confidentiality, integrity, and overall security of your data. * **Encryption:** Data is stored encrypted (using the AES-256 standard) and transmitted encrypted (using HTTPS/TLS protocols). This protects sensitive details such as access tokens and confidential information. * **Data Erasure:** We erase data upon request or when it reaches its designated retention period, ensuring that outdated information isn’t stored unnecessarily. * **Physical Security:** Our cloud service providers ensure the physical security of the infrastructure where your data is stored using industry-standard encryption. ## Scorecard with Self-Hosted Data Are you dealing with very sensitive data, require specialized security arrangements, or have specific compliance needs? [Send us an email](mailto:team@scorecard.io) and we're happy to discuss data-plane self-hosting solutions to support your team's needs! # Synthetic Data Generation Source: https://docs.scorecard.io/features/synthetic-data-generation Generate high-quality synthetic test data for comprehensive AI evaluation ## What is Synthetic Data Generation? Scorecard's AI-powered testcase generation helps you quickly create realistic test data for your evaluations. Using advanced AI models, the system generates testcases that match your testset schema, drawing from optional examples you provide to ensure quality and consistency. This feature significantly reduces the manual effort required to build comprehensive evaluation datasets. > [!NOTE] > > Generate testcases through Scorecard's UI, with enterprise plans supporting larger batch generation. You can also import datasets from external tools. ## How Generation Works Scorecard uses advanced AI models to generate testcases that automatically match your testset's JSON schema. The system supports few-shot learning by using existing testcases as examples, ensuring generated data follows established patterns and maintains quality consistency. You can optionally provide keywords or descriptions to guide generation toward specific topics or scenarios. ## Generating Test Data Navigate to your testset and select existing testcases to use as examples, then click **Generate** from the actions toolbar. Provide optional keywords or descriptions to guide generation, specify how many testcases to create, and review the generated results before adding them to your testset. The system automatically ensures all generated testcases match your testset's schema and field requirements. Generate Testcases For larger-scale synthetic data generation, you can create datasets using external tools and import them via CSV, JSON, or JSONL upload through Scorecard's bulk import feature. Upload Testcases ## External Generation Tools For large-scale synthetic data generation beyond Scorecard's built-in capabilities, you can use external tools and import the results: **Popular Options**: * **OpenAI API**: Generate custom datasets programmatically * **Anthropic Claude**: Create diverse conversational test cases * **Open source tools**: Use libraries like Faker or custom scripts * **Domain-specific generators**: Industry-specific test data tools ## Best Practices Use existing testcases as examples whenever possible to guide generation quality and ensure consistency with your evaluation patterns. Start with small batches to validate output quality before generating larger sets. Provide specific keywords or descriptions to focus generation on relevant scenarios for your use case. Review generated testcases before adding them to your testset to maintain data quality and remove any irrelevant or problematic examples. For comprehensive test coverage, combine AI generation with manual testcase creation and external data imports. ## Related Resources > **[Testsets](/features/testsets)** > > Learn more about managing test data in Scorecard > **[Metrics](/features/metrics)** > > Define metrics to evaluate generated test data quality > **[API Reference](/api-reference/overview)** > > Complete API documentation for data generation # Testsets Source: https://docs.scorecard.io/features/testsets Use Testsets to create curated datasets for evaluating your AI agents A **Testset** is a collection of **Testcases** used to evaluate the performance of an AI agent across various inputs and scenarios. Think of it as a curated dataset specifically designed for testing AI agents. Testsets belong to a central theme like "Core Functionality", "Edge Cases", or "Customer Support Queries". A **Testcase** is an individual test data point containing inputs, expected outputs, and metadata used for evaluation. Testset details page showing eight Testcases for a "Message tone rewriter" AI agent. ## Create a Testset Go to the Testsets page in your project and click the **"New Testset"** button to create a new, empty Testset. > [!TIP] > > Checking "Add example Testcases" will automatically generate 3 sample Testcases using AI based on your Testset's description. This provides a starting point for your Testset. Create testset modal with AI generation option ### Testset schema Each Testset has a *schema*, which defines which fields a Testcase has, the type of each field, and the role each field plays in evaluation. * Input fields are sent to your AI agent. * Expected fields are expected or ideal outputs, which metrics compare your agent's output to. * Metadata fields are additional context for analysis, not used by evaluation or your agent. You can update the schema of a Testset by clicking the **"Edit Schema"** button in the Testset actions menu. This allows you to add or remove fields, modify field types, and update field descriptions. Existing Testcases are not modified, but are validated against the new schema. Testset schema editor. ### With the SDK You can also [create](/api-reference/create-testset) and [update](/api-reference/update-testset) Testsets with the Scorecard SDK. You define a Testset's schema using the [JSON Schema](https://json-schema.org/understanding-json-schema/about) format. For example, here's a schema for a customer support system: ```json expandable theme={null} { "type": "object", "title": "Customer Support Schema", "properties": { "userQuery": { "type": "string", "description": "The customer's question or request" }, "context": { "type": "string", "description": "Additional context about the customer" }, "ideal": { "type": "string", "description": "The ideal response from support" }, "expectedSentiment": { "type": "string", "description": "The expected predicted sentiment of the user query." }, "difficulty": { "type": "number", "description": "How difficult the customer support request is to solve (1-10)" } }, "required": ["userQuery", "ideal"] } ``` **Supported Data Types** * **`string`**: Text content * **`number`**: Numeric values (integers or floats) * **`boolean`**: either `true` or `false` * **`object`**: Nested JSON objects * **`array`**: Lists of JSON values You also need to define the *field mapping* when creating a Testset with the SDK. A field mapping categorizes schema fields by their role in evaluation. For example, here's a field mapping for the customer support schema above: ```json theme={null} { "inputs": ["userQuery", "context"], "expected": ["ideal", "expectedSentiment"], "metadata": ["difficulty"] } ``` ### Understanding Input Fields **Input fields** contain the actual data that gets sent to your AI system, workflow, or agent during testing. These should match exactly what your system expects to receive in production. > [!TIP] > > **Quick tip**: Your input fields should match what goes INTO your AI system. Think about: > > * What do users type into your UI? > * What data does your API receive? > * What would a user or another system send to trigger your workflow? > > If you're unsure, ask an engineer on your team: "What JSON/data do we send to our AI system?" #### Common patterns for structuring inputs **Chatbot/LLM** For conversational AI systems: ```json theme={null} { "user_message": "How do I reset my password?", "conversation_history": [...], "system_prompt": "You are a helpful assistant" } ``` **AI Agent/Workflow** For multi-step AI workflows or agents: ```json theme={null} { "task_description": "Analyze this document and extract key points", "input_data": "Document content or reference", "workflow_parameters": { "mode": "detailed", "output_format": "bullet_points" } } ``` **API Endpoint** For AI-powered API endpoints: ```json theme={null} { "query": "Find similar products", "filters": {"category": "electronics"}, "max_results": 10 } ``` **Document Processing** For document analysis systems: ```json theme={null} { "document_content": "Full text or path to document", "extraction_query": "What are the payment terms?", "document_type": "contract" } ``` > [!TIP] > > Your input fields should mirror your production system's interface. If your system expects a single prompt string, use a single string field. If it expects structured JSON with multiple parameters, reflect that structure in your schema. #### Simple Testcase Schema Examples The following examples show complete testcase schemas with both input and expected fields. The field mapping below each schema identifies which fields are inputs vs expected outputs. **Basic Chatbot** ```json theme={null} { "type": "object", "properties": { "user_message": { "type": "string", "description": "What the user types in the chat box" }, "expected_response": { "type": "string", "description": "What the bot should say back" } } } ``` **Field mapping:** ```json theme={null} { "inputs": ["user_message"], "expected": ["expected_response"] } ``` **Document Q\&A System** ```json theme={null} { "type": "object", "properties": { "question": { "type": "string", "description": "The question about the document" }, "document": { "type": "string", "description": "The document text to search through" }, "expected_answer": { "type": "string", "description": "The correct answer from the document" } } } ``` **Field mapping:** ```json theme={null} { "inputs": ["question", "document"], "expected": ["expected_answer"] } ``` **AI Agent/Workflow** ```json theme={null} { "type": "object", "properties": { "task_description": { "type": "string", "description": "What you want the AI agent to do" }, "input_data": { "type": "string", "description": "Any data the agent needs to complete the task" }, "expected_output": { "type": "string", "description": "What the agent should produce" } } } ``` **Field mapping:** ```json theme={null} { "inputs": ["task_description", "input_data"], "expected": ["expected_output"] } ``` > [!NOTE] > > The key is matching your test inputs to your actual system. If your system takes a single text field, use a single text field. If it takes multiple parameters, include those as separate fields. ## Create and edit Testcases ### From the UI Click the **"New Testcase"** button to create a new Testcase matching your Testset's schema. Testcase creation modal. You can edit a particular Testcase's fields from the Testcase table, or from the Testcase details page. Testcase details page. ### Importing from a file The Scorecard UI supports importing Testcases in CSV, TSV, JSON, and JSONL formats. Scorecard automatically maps your file's columns to the testset's schema fields and validates data. > [!TIP] > > **Upsert behavior**: If your file includes testcases with IDs that already exist in the testset, those testcases will be updated with the new values rather than creating duplicates. This makes it easy to bulk-update existing testcases by re-uploading a modified file. ```csv CSV theme={null} userQuery,context,ideal,category "How do I cancel my order?","Order placed 1 hour ago","You can cancel orders within 2 hours...","cancellation" "Where is my package?","Order shipped yesterday","Track your package using the link...","tracking" ``` ```tsv TSV theme={null} userQuery context ideal category How do I cancel my order? Order placed 1 hour ago You can cancel orders within 2 hours... cancellation Where is my package? Order shipped yesterday Track your package using the link... tracking ``` ```json JSON theme={null} [ { "userQuery": "Is my order eligible for expedited shipping?", "context": "Order placed 1 hour ago", "ideal": "Expedited shipping is available for orders over $50...", "category": "shipping" }, { "userQuery": "How do I cancel my order?", "context": "Order placed 1 hour ago. Order shipped yesterday.", "ideal": "You can cancel orders within 2 hours...", "category": "cancellation" } ] ``` ```jsonl JSONL theme={null} {"userQuery": "Is my order eligible for expedited shipping?", "context": "Order placed 1 hour ago", "ideal": "Expedited shipping is available for orders over $50...", "category": "shipping"} {"userQuery": "How do I cancel my order?", "context": "Order placed 1 hour ago. Order shipped yesterday.", "ideal": "You can cancel orders within 2 hours...", "category": "cancellation"} ``` ### Using the API With our SDKs, you can [create](/api-reference/create-multiple-testcases), [update](/api-reference/update-testcase), and [delete](/api-reference/delete-multiple-testcases) Testcases. ## Export Testset You can export a Testset's Testcases to a CSV file by clicking the **Export as CSV** button in the Testset's details page. ## Other Testset features **Testset tags** You can add custom tags to your Testsets to categorize them. For example, `regression` or `edge-cases`. **Duplicate Testset** You can create a copy of a Testset by clicking the **"Duplicate"** button in the Testset actions menu. This maintains original field mappings, so it's useful for creating variants of your Testsets without having to recreate the schema. ## Best practices **Testset strategy by use case** > **Hillclimbing Testsets** > > **Purpose**: Iterative improvement and development > > **Size**: 5-20 Testcases > > **Content**: Your favorite prompts and edge cases that matter most > > **Usage**: Quick feedback during development cycles > **Regression Testsets** > > **Purpose**: Ensure new changes don't break existing functionality > > **Size**: 50-100 Testcases > > **Content**: Representative examples of core use cases > > **Usage**: Run regularly (nightly builds, CI/CD pipelines) > **Launch Evaluation Testsets** > > **Purpose**: Comprehensive testing before major releases > > **Size**: 100+ Testcases > > **Content**: Broad coverage of all use cases and edge cases > > **Usage**: Pre-launch validation and confidence building > **Must-Pass Testsets** > > **Purpose**: Critical functionality that must never fail > > **Size**: Variable (focus on precision over coverage) > > **Content**: High-precision Testcases for essential features > > **Usage**: Early checks in deployment pipelines > [!WARNING] > > Remember that testcase data may contain sensitive information. Follow your organization's data handling policies and avoid including PII, secrets, or confidential data in Testsets. # Tracing Source: https://docs.scorecard.io/features/tracing Instrument and inspect every LLM request in minutes. > [!NOTE] > > New to Scorecard? Head straight to the [Tracing Quickstart](/intro/tracing-quickstart) or jump into our ready-to-run [Google Colab notebook](https://colab.research.google.com/github/scorecard-ai/scorecard-examples/blob/main/python-jupyter-openllmetry-openai/openllmetry_openai_example.ipynb) to see traces in under 5 minutes. LLM observability means knowing exactly what happened in every generation: latency, token usage, prompts, completions, cost, and errors. Scorecard's **Tracing** captures all of this automatically and visualises it in the Scorecard UI. Send traces through SDK wrappers, environment variables, [OpenLLMetry](https://github.com/traceloop/openllmetry), an LLM proxy, or direct OpenTelemetry. ## Why Tracing matters * Debug long or failing requests in seconds. * Audit prompts & completions for compliance and safety. * Attribute quality and cost back to specific services or users. * Feed production traffic into evaluations. ### If you call it something else * **Observability / AI spans / request logs**: We capture standard OpenTelemetry traces and spans for LLM calls and related operations. * **Agent runs / tools / function calls**: These appear as nested spans in the trace tree, with inputs/outputs when available. * **Prompt/Completion pairs**: Extracted from common keys (`openinference.*`, `ai.prompt` / `ai.response`, `gen_ai.*`) so they can be turned into testcases and scored. *** ## Instrumentation Methods Choose the method that fits your stack. All methods send traces to Scorecard's OpenTelemetry endpoint. **Agent Frameworks** For agents and frameworks with built-in OpenTelemetry support (Claude Code, Claude Agent SDK, [Claude Tag](/intro/claude-tag-tracing), OpenAI Agents SDK). No code changes—just set environment variables. ```bash highlight={1-7} theme={null} export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " export ENABLE_BETA_TRACING_DETAILED=1 export BETA_TRACING_ENDPOINT="https://tracing.scorecard.io/otel" export OTEL_LOG_USER_PROMPTS=1 export OTEL_LOG_TOOL_DETAILS=1 export OTEL_LOG_TOOL_CONTENT=1 export OTEL_RESOURCE_ATTRIBUTES="scorecard.project_id=" ``` ```python Python theme={null} # No code changes required — just set the env vars above and run your app. import anyio from claude_agent_sdk import ( AssistantMessage, TextBlock, query, ) async def main(): async for message in query(prompt="What is 2 + 2?"): if isinstance(message, AssistantMessage): for block in message.content: if isinstance(block, TextBlock): print(f"Claude: {block.text}") anyio.run(main) ``` ```ts TypeScript theme={null} // No code changes required — just set the env vars above and run your app. import { AssistantMessage, TextBlock, query, } from "@anthropic-ai/claude-agent-sdk"; async function main() { const stream = query({ prompt: "What is 2 + 2?", }); for await (const message of stream) { if (message instanceof AssistantMessage) { for (const block of message.content) { if (block instanceof TextBlock) { console.log("Claude:", block.text); } } } } } main(); ``` See the [Claude Agent SDK Tracing](/intro/claude-agent-sdk-tracing) guide for details. **SDK Wrappers** One-line wrapper for OpenAI, Anthropic, or Vercel AI SDK clients. Install the `scorecard-ai` package and wrap your client—no other changes needed. ```bash Python theme={null} pip install 'scorecard-ai[otel]' ``` ```bash TypeScript theme={null} npm install scorecard-ai ``` **OpenAI** ```python Python highlight={1,5,7} theme={null} from scorecard_ai import wrap from openai import OpenAI import os openai = wrap( OpenAI(api_key=os.getenv("OPENAI_API_KEY")), ) response = openai.chat.completions.create( model="gpt-4o", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the capital of France?"}, ], ) print(response.choices[0].message.content) ``` ```ts TypeScript highlight={1,4,8} theme={null} import { wrap } from 'scorecard-ai'; import OpenAI from 'openai'; const openai = wrap( new OpenAI({ apiKey: process.env['OPENAI_API_KEY'], }), ); const response = await openai.chat.completions.create({ model: 'gpt-4o', messages: [ { role: 'system', content: 'You are a helpful assistant.' }, { role: 'user', content: 'What is the capital of France?' }, ], }); console.log(response.choices[0]?.message?.content); ``` **Anthropic** ```python Python highlight={1,5,7} theme={null} from scorecard_ai import wrap from anthropic import Anthropic import os claude = wrap( Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY")), ) response = claude.messages.create( model="claude-sonnet-4-20250514", max_tokens=1024, messages=[{"role": "user", "content": "What is the capital of France?"}], ) text_content = next( (block for block in response.content if block.type == "text"), None ) print(text_content.text if text_content else "No response") ``` ```ts TypeScript highlight={1,4,8} theme={null} import { wrap } from 'scorecard-ai'; import Anthropic from '@anthropic-ai/sdk'; const claude = wrap( new Anthropic({ apiKey: process.env['ANTHROPIC_API_KEY'], }), ); const response = await claude.messages.create({ model: 'claude-sonnet-4-20250514', max_tokens: 1024, messages: [{ role: 'user', content: 'What is the capital of France?' }], }); const textContent = response.content.find((block) => block.type === 'text'); console.log(textContent?.text); ``` **Vercel AI SDK** (TypeScript only) ```ts TypeScript highlight={3,5} theme={null} import { openai } from '@ai-sdk/openai'; import * as ai from 'ai'; import { wrapAISDK } from 'scorecard-ai'; const aiSDK = wrapAISDK(ai); const { text } = await aiSDK.generateText({ model: openai('gpt-4o'), prompt: 'What is the capital of France?', }); console.log(text); ``` See the [Tracing Quickstart](/intro/tracing-quickstart) and [Vercel AI SDK wrapper](/features/ai-sdk-wrapper) for full setup details. **Traceloop** Universal OpenTelemetry-based tracing via [OpenLLMetry](https://github.com/traceloop/openllmetry). Supports LangChain, LlamaIndex, CrewAI, and 20+ providers. ```bash Python theme={null} pip install traceloop-sdk ``` ```bash TypeScript theme={null} npm install @traceloop/node-server-sdk ``` Set these environment variables: ```bash highlight={1-2} theme={null} export TRACELOOP_BASE_URL="https://tracing.scorecard.io/otel" export TRACELOOP_HEADERS="Authorization=Bearer%20" ``` ```python Python highlight={1-3,6,10} theme={null} from traceloop.sdk import Traceloop from traceloop.sdk.decorators import workflow from traceloop.sdk.instruments import Instruments from openai import OpenAI Traceloop.init(disable_batch=True, instruments={Instruments.OPENAI}) client = OpenAI() @workflow(name="simple_chat") def chat(): response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Hello!"}] ) return response.choices[0].message.content result = chat() print(result) ``` ```ts TypeScript highlight={1,4-7,12,18} theme={null} import * as traceloop from "@traceloop/node-server-sdk"; import OpenAI from "openai"; traceloop.initialize({ disableBatch: true, instrumentModules: { openAI: OpenAI }, }); const openai = new OpenAI(); async function chat() { return await traceloop.withWorkflow({ name: "simple_chat" }, async () => { const response = await openai.chat.completions.create({ model: "gpt-4o", messages: [{ role: "user", content: "Hello!" }], }); return response.choices[0].message.content; }); } const result = await chat(); console.log(result); ``` **LangChain** — add `opentelemetry-instrumentation-langchain` and use `Instruments.LANGCHAIN`: ```python Python highlight={1-8} theme={null} from traceloop.sdk import Traceloop from traceloop.sdk.instruments import Instruments # IMPORTANT: Initialize Traceloop before importing any LangChain modules. Traceloop.init( disable_batch=True, instruments={Instruments.LANGCHAIN} ) from langchain_openai import ChatOpenAI from langchain_core.prompts import ChatPromptTemplate prompt = ChatPromptTemplate.from_messages([ ("system", "You are a helpful assistant."), ("user", "{input}") ]) model = ChatOpenAI(model="gpt-4o-mini") chain = prompt | model response = chain.invoke({"input": "What is the capital of France?"}) print(response.content) ``` **LlamaIndex** — use `Instruments.LLAMA_INDEX`: ```python Python highlight={1-2,6-9} theme={null} from traceloop.sdk import Traceloop from traceloop.sdk.instruments import Instruments from llama_index.llms.openai import OpenAI from llama_index.core.llms import ChatMessage Traceloop.init( disable_batch=True, instruments={Instruments.LLAMA_INDEX, Instruments.OPENAI} ) llm = OpenAI(model="gpt-4o") response = llm.chat([ChatMessage(role="user", content="Hello!")]) print(response.message.content) ``` **CrewAI** — use `Instruments.CREWAI`: ```python Python highlight={1-2,5-8} theme={null} from traceloop.sdk import Traceloop from traceloop.sdk.instruments import Instruments from crewai import Agent, Task, Crew Traceloop.init( disable_batch=True, instruments={Instruments.CREWAI, Instruments.OPENAI} ) researcher = Agent( role="Researcher", goal="Research and summarize topics", backstory="You are an expert researcher.", ) task = Task( description="What is the capital of France?", expected_output="A brief answer", agent=researcher, ) crew = Crew(agents=[researcher], tasks=[task]) result = crew.kickoff() print(result) ``` See the [LangChain Quickstart](/intro/langchain-quickstart) and supported [frameworks and providers](#supported-frameworks--providers) below. **LLM Proxy** Change your `baseURL` to `https://llm.scorecard.io`. No SDK, no dependencies—just a URL change. ```python Python highlight={6-9} theme={null} from openai import OpenAI import os client = OpenAI( api_key=os.environ["OPENAI_API_KEY"], base_url="https://llm.scorecard.io", default_headers={ "x-scorecard-api-key": os.environ["SCORECARD_API_KEY"], }, ) response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "What is the capital of France?"}], ) print(response.choices[0].message.content) ``` ```ts TypeScript highlight={5-8} theme={null} import OpenAI from 'openai'; const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY, baseURL: 'https://llm.scorecard.io', defaultHeaders: { 'x-scorecard-api-key': process.env.SCORECARD_API_KEY, }, }); const response = await client.chat.completions.create({ model: 'gpt-4o', messages: [{ role: 'user', content: 'What is the capital of France?' }], }); console.log(response.choices[0].message.content); ``` See the [Tracing Quickstart](/intro/tracing-quickstart) for full setup details. **OpenTelemetry** Send traces directly using any OpenTelemetry SDK (Python, TypeScript, Java, Go, .NET, Rust). * **Endpoint:** `https://tracing.scorecard.io/otel/v1/traces` * **Auth:** `Authorization: Bearer ` ```bash Python wrap theme={null} pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-grpc ``` ```bash TypeScript wrap theme={null} npm install @opentelemetry/api @opentelemetry/sdk-trace-node @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-http @opentelemetry/resources @opentelemetry/semantic-conventions ``` ```python Python highlight={2-23,27,29-33,45} theme={null} from anthropic import Anthropic from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.resources import Resource from opentelemetry.semconv.resource import ResourceAttributes import os resource = Resource.create({ ResourceAttributes.SERVICE_NAME: "my-app", }) trace_provider = TracerProvider(resource=resource) trace_provider.add_span_processor(BatchSpanProcessor( OTLPSpanExporter( endpoint="https://tracing.scorecard.io/otel", headers={"Authorization": f"Bearer {os.environ['SCORECARD_API_KEY']}"} ) )) trace.set_tracer_provider(trace_provider) tracer = trace.get_tracer(__name__) client = Anthropic() @tracer.start_as_current_span("chat") def chat(prompt: str) -> str: span = trace.get_current_span() span.set_attributes({ "gen_ai.system": "anthropic", "gen_ai.request.model": "claude-sonnet-4-20250514", }) message = client.messages.create( model="claude-sonnet-4-20250514", max_tokens=1024, messages=[{"role": "user", "content": prompt}], ) return message.content[0].text result = chat("What is the capital of France?") print(result) trace.get_tracer_provider().force_flush() ``` ```ts TypeScript highlight={1-6,9-25,29-31,38-39,47} theme={null} import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"; import { Resource } from "@opentelemetry/resources"; import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base"; import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node"; import { ATTR_SERVICE_NAME } from "@opentelemetry/semantic-conventions"; import { trace, SpanStatusCode } from "@opentelemetry/api"; import OpenAI from "openai"; const exporter = new OTLPTraceExporter({ url: "https://tracing.scorecard.io/otel/v1/traces", headers: { Authorization: `Bearer ${process.env.SCORECARD_API_KEY}`, }, }); const tracerProvider = new NodeTracerProvider({ resource: Resource.default().merge( new Resource({ [ATTR_SERVICE_NAME]: "my-app" }) ), }); tracerProvider.addSpanProcessor(new BatchSpanProcessor(exporter)); tracerProvider.register(); const tracer = trace.getTracer("my-app"); const openai = new OpenAI(); async function chat(prompt: string) { return tracer.startActiveSpan("chat", async (span) => { span.setAttribute("gen_ai.system", "openai"); span.setAttribute("gen_ai.request.model", "gpt-4o"); const completion = await openai.chat.completions.create({ model: "gpt-4o", messages: [{ role: "user", content: prompt }], }); span.setStatus({ code: SpanStatusCode.OK }); span.end(); return completion.choices[0].message.content; }); } const result = await chat("What is the capital of France?"); console.log(result); await tracerProvider.forceFlush(); ``` *** ## View Traces in the Records Page Ingested traces appear on the **Records** page alongside records from other sources (API, Kickoff). Each row in the table shows: Records page showing ingested traces. Each row shows the record's inputs, outputs, status, source, trace ID, and metric scores. You can search by inputs, outputs, and expected values, or filter by `status`, `source`, `trace.id`, `run.id`, `testcase.id`, `metric`, `metadata`, time range, and more. Use **Quick Filters** for presets like "My Recent Records". Customise visible columns via the **Edit Table** modal. Open a record to see its trace detail panel, which breaks the trace into four tabs: **Conversation**, **Timeline**, **Chat**, and **Debug**. For chat-based traces, the **Conversation** tab replays the trace as user messages, assistant responses, and tool calls: Conversation tab of a trace-based record. The **Debug** tab pairs the full span tree with a trace overview (duration, cost, tokens, and model): Debug tab showing the span tree. *** ## Trace Grouping When running batch operations or multi-step workflows, you can group related traces into a single run using the `scorecard.tracing_group_id` span attribute. This makes it easier to track and analyze workflows that span multiple LLM calls. ### How it works Add the `scorecard.tracing_group_id` attribute to your spans with a shared identifier. Scorecard automatically groups spans with the same group ID into a single run. ```python Python highlight={1-6,8-9,12-13} theme={null} from opentelemetry import trace tracer = trace.get_tracer(__name__) # Use the same group_id for all related operations group_id = "batch-job-123" with tracer.start_as_current_span("process_document") as span: span.set_attribute("scorecard.tracing_group_id", group_id) # Your LLM call here with tracer.start_as_current_span("summarize_results") as span: span.set_attribute("scorecard.tracing_group_id", group_id) # Another LLM call in the same workflow ``` ```typescript TypeScript highlight={1-10,12-14,16} theme={null} import { trace } from '@opentelemetry/api'; const tracer = trace.getTracer('my-service'); // Use the same groupId for all related operations const groupId = 'batch-job-123'; const span1 = tracer.startSpan('process_document'); span1.setAttribute('scorecard.tracing_group_id', groupId); // Your LLM call here span1.end(); const span2 = tracer.startSpan('summarize_results'); span2.setAttribute('scorecard.tracing_group_id', groupId); // Another LLM call in the same workflow span2.end(); ``` ### Use cases * **Batch processing**: Group all documents processed in a single batch job * **Multi-step agents**: Track all LLM calls within an agent's execution * **Workflows**: Link related operations across different services * **A/B testing**: Group traces by experiment variant for comparison > [!TIP] > > Traces with the same `scorecard.tracing_group_id` will appear together in the same run, making it easy to analyze aggregate metrics across related operations. *** ## AI-Specific Error Detection Scorecard's tracing detects AI-specific behavioral issues that standard observability misses, not just technical failures. It analyzes every interaction for both technical errors and behavioral problems. ### Silent Failure Detection A "silent failure" is when your AI responds, but incorrectly. These often go unnoticed without AI-specific observability. Scorecard automatically detects behavioral errors including off-topic responses, workflow interruptions, safety violations, hallucinations, and context loss. Technical errors like rate limits, timeouts, and API failures are captured automatically through standard trace error recording. AI applications also face issues that error logs don't capture: semantic drift, safety policy violations, factual accuracy problems, and task completion failures. ### Custom Error Detection Create custom metrics in Scorecard's UI to catch application-specific issues. An AI-powered metric can analyze trace data for things like off-topic responses, safety violations, or task completion failures, then automatically flag the traces that match. *** ## Supported Frameworks & Providers Scorecard traces LLM applications built with popular open-source frameworks through [OpenLLMetry](https://github.com/traceloop/openllmetry). OpenLLMetry provides automatic instrumentation for: ### Application Frameworks * **CrewAI** – Multi-agent collaboration * **Haystack** – Search and question-answering pipelines * **LangChain** – Chains, agents, and tool calls * **Langflow** – Visual workflow builder * **LangGraph** – Multi-step workflows and state machines * **LiteLLM** – Unified interface for 100+ LLMs * **LlamaIndex** – RAG pipelines and document retrieval * [**OpenAI Agents SDK**](https://github.com/openai/openai-agents-python?tab=readme-ov-file#tracing) – Assistants API and function calling * [**Vercel AI SDK**](https://ai-sdk.dev/providers/observability/scorecard) – Full-stack AI applications > [!TIP] > > Scorecard is featured as a recommended observability provider in the official [Vercel AI SDK documentation](https://ai-sdk.dev/providers/observability/scorecard) and [OpenAI Agents Python README](https://github.com/openai/openai-agents-python?tab=readme-ov-file#tracing). ### LLM Providers * Aleph Alpha * Anthropic * AWS Bedrock * AWS SageMaker * Azure OpenAI * Cohere * Google Gemini * Google Vertex AI * Groq * HuggingFace * IBM Watsonx AI * Mistral AI * Ollama * OpenAI * Replicate * Together AI * and more ### Vector Databases * Chroma * LanceDB * Marqo * Milvus * Pinecone * Qdrant * Weaviate > [!NOTE] > > For the complete list of supported integrations, see the [OpenLLMetry repository](https://github.com/traceloop/openllmetry). All integrations are built on OpenTelemetry standards and maintained by the community. ### Custom Providers For frameworks or providers not listed above, you can use: * **HTTP Instrumentation**: OpenLLMetry's `instrument_http()` for HTTP-based APIs * **Manual Spans**: Emit custom OpenTelemetry spans for proprietary systems See the [OpenLLMetry documentation](https://www.traceloop.com/docs/openllmetry/getting-started-python) for manual instrumentation guides. *** ## Use cases * **Production observability for LLM quality and safety** * **Debugging slow/failed requests with full span context** * **Auditing prompts/completions for compliance** * **Attributing token cost and latency to services/cohorts** * **Building evaluation datasets from real traffic (Trace to Testcase)** *** ## Next steps 1. Follow the [Tracing Quickstart](/intro/tracing-quickstart) to send your first trace. 2. Set up [Claude Agent SDK Tracing](/intro/claude-agent-sdk-tracing) with zero code changes. 3. Instrument [LangChain](/intro/langchain-quickstart) chains and agents. 4. Use the [Vercel AI SDK wrapper](/features/ai-sdk-wrapper) for full-stack AI apps. 5. Open the [Colab notebook](https://colab.research.google.com/github/scorecard-ai/scorecard-examples/blob/main/python-jupyter-openllmetry-openai/openllmetry_openai_example.ipynb) for an interactive tour. Happy tracing! 🚀 # Claude Agent SDK Tracing Source: https://docs.scorecard.io/intro/claude-agent-sdk-tracing Trace your Claude Agent SDK applications with Scorecard. The Claude Agent SDK is Anthropic's framework for building AI agents that can reason, use tools, and complete multi-step tasks. It includes built-in support for OpenTelemetry tracing, making it easy to capture detailed telemetry from your agent workflows. This quickstart shows how to send traces from your Claude Agent SDK (Python v0.1.18+, TypeScript v0.1.71+) applications to Scorecard for observability, debugging, and evaluation. > [!NOTE] > > **Using the standard Anthropic SDK?** Check out the general [Tracing Quickstart](/intro/tracing-quickstart) for the proxy method that works with any Anthropic client. ## Steps **1. Set up environment variables** Configure the OpenTelemetry exporter to send traces to Scorecard. You'll need your Scorecard API key from [Settings](https://app.scorecard.io/settings). ```bash theme={null} export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " export ENABLE_BETA_TRACING_DETAILED=1 export BETA_TRACING_ENDPOINT="https://tracing.scorecard.io/otel" export OTEL_LOG_USER_PROMPTS=1 export OTEL_LOG_TOOL_DETAILS=1 export OTEL_LOG_TOOL_CONTENT=1 export OTEL_RESOURCE_ATTRIBUTES="scorecard.project_id=" ``` > [!NOTE] > > Replace `` with your actual Scorecard API key (starts with `ak_`), and `` with the project you want traces to land in. > [!WARNING] > > If the project ID is wrong or invalid, traces will be sent to the **oldest project** in your organization. **2. Run your agent** With the environment variables configured, run your Claude Agent SDK application. All agent activity is automatically traced. ```python title="example.py" [expandable] theme={null} import anyio from claude_agent_sdk import ( AssistantMessage, TextBlock, query, ) async def main(): async for message in query(prompt="What is 2 + 2?"): if isinstance(message, AssistantMessage): for block in message.content: if isinstance(block, TextBlock): print(f"Claude: {block.text}") anyio.run(main) ``` **3. View traces in Scorecard** Navigate to the [**Records**](/features/records) page in [Scorecard](https://app.scorecard.io) to see your agent traces. Click any record to open its trace detail panel, which breaks the trace into four tabs: **Conversation**, **Timeline**, **Chat**, and **Debug**. Selecting a record opens its trace detail panel. The **Conversation** tab shows a chat-like replay of user messages, assistant responses, and tool calls. Conversation tab of an agent trace. The **Timeline** tab shows a Gantt chart-style breakdown of every span, so you can see how LLM calls and tool executions overlap and how long each takes. Timeline tab of an agent trace. The **Chat** tab lets you ask questions about the trace in natural language—what happened, why a step failed, or how it performed—and compare records side by side. Chat tab: ask questions about a trace. The **Debug** tab pairs the full span tree with a trace overview—duration, cost, tokens, and model—plus the raw span data for deeper inspection. Debug tab: span tree and trace overview. ## Anatomy of a trace Explore how a Claude Agent SDK trace is structured—spans, tool calls, model usage, and metadata—in this interactive demo: [Anatomy of a Claude Agent SDK trace](https://scorecard-git-feat-trace-anatomy-demo-embed-scorecard-ai.vercel.app/demo/trace-anatomy) > [!NOTE] > > Tracing also works with the Claude Code CLI and [Claude Tag in Slack](/intro/claude-tag-tracing)! ## Next Steps > **[Tracing Features](/features/tracing)** > > Learn about advanced tracing patterns and trace grouping > **[Metrics](/features/metrics)** > > Create custom metrics to evaluate agent performance # Claude Tag Tracing Source: https://docs.scorecard.io/intro/claude-tag-tracing Trace Claude Tag (Claude in Slack) sessions with Scorecard. Claude Tag is [Claude working inside your team's Slack workspace](https://claude.com/docs/claude-tag/overview). Each thread Claude works in is a **session**: it starts when someone mentions `@Claude`, runs Claude Code in an Anthropic-hosted sandbox, and ends when the thread goes quiet. Anthropic's [How Claude Tag works](https://claude.com/docs/claude-tag/concepts/how-it-works) page covers the session lifecycle in detail. Because a session runs Claude Code, the same OpenTelemetry tracing used for [Claude Agent SDK tracing](/intro/claude-agent-sdk-tracing) works here too. Every session shows up in Scorecard as one record with a trace of user turns, agent reasoning, and tool calls. This quickstart shows how to configure an Anthropic-hosted environment so Claude Tag sessions are traced to Scorecard. > [!NOTE] > > Claude Tag requires a Claude **Team or Enterprise** plan, and these steps are done by a Claude organization admin. If your Slack workspace doesn't have Claude Tag yet, follow [Anthropic's setup guide](https://claude.com/docs/claude-tag/admins/setup-overview) first. ## Steps **1. Get your Scorecard API key and project ID** You'll need two values from Scorecard: * Your API key from [Settings](https://app.scorecard.io/settings). It starts with `ak_`. * The ID of the [project](https://app.scorecard.io) you want traces to land in. > [!WARNING] > > If the project ID is wrong or invalid, traces are sent to the **oldest project** in your organization. **2. Create an Anthropic-hosted environment** Go to [**Cloud environments**](https://claude.ai/admin-settings/cloud-environments) in your Claude admin settings and create a new Anthropic-hosted environment. Name it something like "Claude Tag Environment". Under **Network access**, pick a setting that lets sessions reach `tracing.scorecard.io`. The example below uses **Full access**. In the **Environment variables** field, add the following, one per line as `KEY=value`. Substitute your API key and project ID from the previous step. ```bash theme={null} OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer ENABLE_BETA_TRACING_DETAILED=1 BETA_TRACING_ENDPOINT=https://tracing.scorecard.io/otel OTEL_LOG_USER_PROMPTS=1 OTEL_LOG_TOOL_DETAILS=1 OTEL_LOG_TOOL_CONTENT=1 OTEL_RESOURCE_ATTRIBUTES=scorecard.project_id= ``` The environment dialog with the tracing variables filled in. > [!NOTE] > > These variables are available to every session that uses this environment. Nothing is traced yet. That happens in the next step. **3. Assign the environment to Claude Tag** This is the step that turns tracing on. Go to [**Claude Tag**](https://claude.ai/admin-settings/claude-tag) in admin settings and find **Claude Tag's access**. Select the **Slack** tab, expand **Advanced**, and set **Environment** to the environment you created in step 2. Set Environment under Claude Tag's access → Slack → Advanced. > [!WARNING] > > **Every channel in the workspace is traced.** Setting the environment on the Slack workspace applies to every channel Claude works in, and traces include user prompts and tool inputs and outputs. If you only want to trace some channels, leave the workspace setting alone and instead select each channel in the left sidebar and set its Environment there. Threads that are already running don't see configuration changes, so start a new thread after setting the environment. **4. Talk to Claude in Slack** Mention `@Claude` in a channel it's been invited to and give it a task. Nothing changes on the Slack side. The session is traced as it runs. > @Claude summarize the open questions in this thread. **5. View traces in Scorecard** Open the [**Records**](/features/records) page of your project in [Scorecard](https://app.scorecard.io). Each Claude Tag session appears as a record. Click one to open its trace detail panel and see the full session, including tool calls like Slack replies and reactions. Each Claude Tag session arrives as a record. Selecting one opens its trace. ## What you'll see Each Claude Tag session becomes one record. Record sizes vary a lot because Claude Tag runs several kinds of sessions: | Session | How it starts | What the record looks like | | - | - | - | | **Channel session** | Any new top-level message in a channel Claude is in | A few LLM calls, usually ending in an `mcp__slackbot__no_reply_needed` tool call. You get one of these for most messages in the channel, not only the ones addressed to Claude. | | **Thread session** | The channel session hands a task to a thread, or someone replies in a thread Claude is working in | Tens to hundreds of LLM calls, sub-agent tasks, and tool calls (Slack replies, GitHub, Bash). Can run for hours and stitch together many traces. | | **DM** | Someone messages Claude directly | Similar to a thread session. | | **Routine** | A scheduled trigger fires | No Slack messages in the prompt. The routine's instructions are the prompt, and there's no channel on the record until Claude posts somewhere. | Inside a record: * **User turns** are the raw Slack payload Claude receives: message text, author display names, `@handles`, Slack user IDs, channel IDs, thread participants, attachment file names, message edits, and reactions. In thread sessions this also includes the parent session's hand-off notes and Claude's own memory files for the channel. * **LLM request spans** carry the model, token counts, a system prompt preview, the model output, and tool results as they were fed back to the model. * **Tool spans** carry the tool name and full inputs and outputs: Slack reply text, Bash commands, file paths, GitHub calls. * **Identity** is limited to a hashed `user.id`, session IDs, and your `scorecard.project_id`. Unlike Claude Code CLI traces, there is no email or account ID. Your environment variables and Scorecard API key are not in the trace. ## Next Steps > **[Tracing Features](/features/tracing)** > > Learn about advanced tracing patterns and trace grouping > **[Metrics](/features/metrics)** > > Create custom metrics to evaluate Claude Tag's performance # Frequently Asked Questions Source: https://docs.scorecard.io/intro/faq Common questions about Scorecard's AI evaluation platform ## Getting Started > **What is an eval/evaluation?** > > An **evaluation** (or "eval") is the systematic process of testing and measuring your AI system's performance against defined criteria. In Scorecard, an evaluation consists of: > > * **Testset**: A collection of test cases with inputs and expected outputs > * **System**: The AI agent, prompt, API endpoint, or workflow being evaluated > * **Metrics**: Scoring criteria (accuracy, safety, tone, etc.) with AI, Human, or Code-based evaluation > * **Record**: A single evaluation result capturing inputs, outputs, and scores > > Evaluations help you quantitatively measure AI quality, track improvements over time, and catch regressions before production. Think of it like continuous integration for AI—defining what good looks like and measuring against it systematically. > **How does Scorecard differ from other AI evaluation tools?** > > Scorecard is **the simulation platform for agent self-improvement**. While most eval tools focus on observing and testing after deployment, Scorecard helps you proactively simulate thousands of realistic scenarios and apply expert judgment at scale — so you can ship frontier agent capabilities in days, not weeks. > > Scorecard also brings together AI researchers, engineers, and subject-matter experts into a single workflow. SMEs encode their knowledge into metrics and reward models instead of manually labeling production logs, while AI researchers build improvement flywheels instead of running QA cycles. > **What is simulation and how is it different from traditional evals?** > > **Simulation** is the process of running your AI agent through **thousands of realistic scenarios** to test and improve its capabilities — without relying on production traffic or manual review. > > Traditional eval approaches treat quality as a **QA problem**: wait for production logs, manually review cases, and fix issues one at a time. This breaks down as agents get more complex because: > > * **Coverage gaps**: You miss edge cases until real users hit them > * **Slow feedback**: Collecting enough production data takes **days or weeks** > * **Expert bottleneck**: SMEs spend time labeling individual cases instead of teaching the system what "good" looks like > * **Ceiling on improvement**: You're limited to scenarios that happen to occur in production > > **Simulation flips this.** Instead of reacting to problems in production, you proactively generate the messy, realistic interactions your agent will encounter and get feedback in **minutes, not weeks**. > **How does Scorecard help agents self-improve?** > > Scorecard enables a **self-improvement loop** by combining realistic scenario simulation with **scalable expert judgment**: > > * **Expert judgment at scale**: Instead of waiting for SMEs to manually label production cases, encode their knowledge into [Critic Agent Metrics](/features/metrics) that act as **reward models** — applying expert judgment consistently across every scenario automatically. > * **Fast feedback loops**: Evaluate your agent through **10,000+ scenarios in minutes**. Identify weaknesses, iterate, and validate improvements — all in the same day. > * **Broad scenario coverage**: Test against **tool-calling workflows**, **edge cases and adversarial inputs**, and **enterprise environments** *(coming soon)*. > > The result: your team's bottleneck moves from **feedback** (slow, getting slower) to **building** (fast, getting faster). Scorecard's evaluation data also serves as high-quality training data for **RLHF and reinforcement learning** pipelines. > **What programming languages does Scorecard support?** > > Scorecard provides SDKs for: > > * **Python**: Full-featured SDK with all capabilities > * **JavaScript/TypeScript**: Complete Node.js and browser support > * **REST API**: Universal HTTP access for any language > * **Framework integrations**: Claude Agent SDK, LangChain, LlamaIndex, OpenAI, and more > **Can Scorecard be used for agent improvement and continuous learning?** > > Yes! Scorecard provides multiple pathways for improving AI agents: > > **Evaluation-Driven Improvement**: > > * Identify failure patterns across test cases to focus improvement efforts > * Track performance metrics over time to measure agent improvements > * Use A/B comparison to validate that changes actually improve performance > * Regression testing ensures new versions don't break existing capabilities > > **Training Data Generation**: > > * Evaluation results serve as high-quality feedback for RLHF workflows > * Human-scored examples create preference pairs for fine-tuning > * Score explanations provide detailed reasoning for model training > * Export scored data for custom training pipelines > > **Continuous Feedback Loops**: > > * Monitor production performance through tracing and observability > * Create test cases from production failures for regression testing > * Iterate on prompts, tools, and configurations with quantitative feedback > > See our [A/B Comparison](/features/a-b-comparison) docs for specific improvement workflows. ## Limits and Constraints > **What are the text limits in Scorecard?** > > Scorecard uses PostgreSQL which supports up to 1GB per text field - essentially unlimited for evaluation use cases. Practical limits are more likely to come from your AI model's context window than database constraints. > > For bulk imports and large datasets, file uploads support CSV, JSON, and JSONL formats. If you encounter limitations with particularly large datasets or need custom configurations, contact [support@scorecard.io](mailto:support@scorecard.io) for assistance. > **Are there rate limits for API usage?** > > Scorecard implements rate limiting to ensure platform stability and fair usage across all customers. Rate limits vary by plan tier, with enterprise customers receiving custom limits based on their specific requirements. > > Rate limit information is included in API response headers for monitoring usage. If you need higher limits for your use case, contact support to discuss your requirements. > **How do I create test cases?** > > Scorecard offers multiple ways to create test cases: > > **UI Creation**: > > * Manually create test cases one at a time in the Testset editor > * Define custom schemas with inputs, outputs, and metadata fields > * Use the visual editor for quick iteration > > **Bulk Import**: > > * Upload CSV, JSON, or JSONL files with test data > * Automatic schema detection from imported data > * Support for large datasets (thousands of test cases) > > **Programmatic Creation**: > > * Use Python or Node.js SDK to create test cases via API > * Generate synthetic test cases with LLMs > * Import from production logs or existing datasets > > **From Production Data**: > > * Convert traces from production into test cases > * Create regression tests from production failures > * Sample real user interactions for evaluation > > See our [Testsets documentation](/features/testsets) for detailed guides. > **Can I use custom AI models for evaluation?** > > Yes! Scorecard supports using custom models for AI-based metric evaluation: > > **Model Options**: > > * **Scorecard-hosted models**: GPT-4o, Claude 3.5 Sonnet, and other leading models > * **Custom endpoints**: Point to your own model API for evaluation > * **Fine-tuned models**: Use domain-specific evaluator models > * **Multiple models**: Different metrics can use different evaluation models > > **Configuration**: > > * Set model parameters (temperature, max tokens) per metric > * Configure custom prompt templates in advanced mode > * Control evaluation costs by selecting appropriate models > > For enterprise customers, we can integrate with your on-premise models or private deployments. > **What's the difference between a Run and a Record?** > > **Run**: > > * A collection of test executions against a testset > * Contains multiple records (one per test case) > * Has aggregated metrics and statistics > * Represents a snapshot of your system's performance > > **Record**: > > * A single test case execution with its results > * Contains inputs, outputs, and scores from all metrics > * Can have multiple scores (one per metric applied) > * Represents one test case within a run > > **Example**: > > * Run #42: Testing customer support agent v3.1 > * Record 1: Test case "refund request" → scored with 3 metrics = 3 scores > * Record 2: Test case "product inquiry" → scored with 3 metrics = 3 scores > * Record 3: Test case "complaint" → scored with 3 metrics = 3 scores > * Total: 1 run, 3 records, 9 scores > > See [Runs & Results](/features/runs) and [Records](/features/records) for more details. ## Features and Capabilities > **How does metadata work in Scorecard?** > > **Metadata** in Scorecard allows you to store additional context without affecting evaluations: > > **Testcase Metadata**: > > * Mark fields as "metadata" in testset schemas > * Stored with test cases but excluded from evaluation logic > * Useful for tracking source, difficulty, categories, etc. > > **Trace Metadata**: > > * Custom attributes on spans (user\_id, session\_id, etc.) > * Model parameters and configuration data > * Performance metrics and timing information > > **Run Metadata**: > > * Git commit SHA, branch information > * Environment details (staging, production) > * Custom tags and labels for organization > > Example usage: > > ```json theme={null} > { > "input": "What's the weather?", > "expected_output": "I'll help you check the weather", > "source": "customer_support_logs", // metadata > "difficulty": "easy", // metadata > "created_by": "data_team" // metadata > } > ``` > **How is latency measured and reported?** > > Scorecard automatically captures **latency metrics** across your AI pipeline: > > **Measurement Points**: > > * **End-to-end latency**: Total request time from input to output > * **Model inference time**: Time spent in model API calls > * **Processing time**: Custom logic execution time > * **Network latency**: Time spent in HTTP requests > > **Reporting**: > > * **Real-time dashboards**: Live latency monitoring > * **Percentile analysis**: P50, P90, P95, P99 latency breakdown > * **Trend analysis**: Latency over time with alerting > * **Trace-level detail**: Individual request timing breakdowns > **Can Scorecard test agentic workflows and multi-step AI agents?** > > Yes! Scorecard is designed to test complex AI agents, not just simple prompts. Our platform supports: > > **Agentic Capabilities**: > > * **Tool-calling agents**: Evaluate agents that use function calling and API integrations > * **Multi-step workflows**: Version complete agent configurations including prompts, tools, and routing logic > * **Agent APIs**: Evaluate deployed agents by calling them from your own code with the SDK > > **How It Works**: > > * Use **Systems** to version your complete agent configuration (prompts + tools + settings) > * Use the **SDK** to run your agent against a testset and record every response > * Apply metrics to agent outputs just like any other evaluation > * Use **Tracing** to observe and debug multi-step agent executions > > See our [Tracing](/features/tracing) documentation for agent-specific features. > **What types of AI systems can Scorecard evaluate?** > > Scorecard supports evaluation of any AI agent or system accessible via API: > > **Model Types**: > > * **Large Language Models**: GPT, Claude, Llama, Gemini, etc. > * **Embedding Models**: OpenAI, Cohere, custom embeddings > * **Multimodal Models**: Vision, audio, and text processing > * **Fine-tuned Models**: Custom models hosted anywhere > > **Agent & System Types**: > > * **Conversational Agents**: Multi-turn chatbots and virtual assistants > * **Tool-calling Agents**: Function calling and API integrations > * **RAG Agents**: Retrieval-augmented generation pipelines > * **Agentic Workflows**: Multi-step reasoning and planning agents > * **Custom APIs**: Any HTTP endpoint returning AI-generated content > > **Deployment Options**: > > * **Cloud APIs**: OpenAI, Anthropic, Google, AWS Bedrock > * **Self-hosted**: Models and agents running on your infrastructure > * **Hybrid**: Combination of cloud and on-premise systems ## Technical Details > **How does Scorecard handle sensitive data and privacy?** > > Scorecard is built with **privacy by design** principles: > > **Data Security**: > > * **Encryption**: All data encrypted in transit (TLS 1.3) and at rest (AES-256) > * **Access Controls**: Role-based permissions and organization isolation > * **Audit Logging**: Complete audit trail of all data access > * **Compliance**: SOC 2 Type II, GDPR, and enterprise compliance > > **Privacy Controls**: > > * **Data Redaction**: Automatic PII detection and masking > * **Data Retention**: Configurable retention policies > * **Right to Delete**: Complete data deletion capabilities > * **Data Residency**: Control where your data is stored > > See our [Security and Privacy](/features/security-and-privacy) documentation for complete details. > **Can I run Scorecard evaluations offline or on-premise?** > > Scorecard offers flexible deployment options: > > **Cloud Service** (Recommended): > > * Fully managed service at app.scorecard.io > * Automatic updates and maintenance > * Global CDN and high availability > > **Enterprise On-Premise**: > > * Self-hosted deployment in your infrastructure > * Complete data sovereignty and control > * Custom integrations with internal systems > * Available for Enterprise customers > > **Hybrid Approach**: > > * Evaluation logic runs on-premise > * Results optionally synced to cloud dashboard > * Best of both worlds for security-sensitive organizations > > Contact [enterprise@scorecard.io](mailto:enterprise@scorecard.io) for on-premise deployment options. > **How do I migrate from other evaluation tools?** > > Scorecard provides migration support for common evaluation platforms: > > **Data Migration**: > > * Import existing test datasets (CSV, JSON, JSONL) > * Convert evaluation metrics to Scorecard format > * Migrate historical evaluation results > > **Common Migrations**: > > * **From custom scripts**: Convert to Scorecard SDK calls > * **From academic benchmarks**: Import MMLU, HellaSwag, etc. > * **From other platforms**: Bulk export/import workflows > > **Migration Assistance**: > > * Free migration consultation for Enterprise customers > * Custom scripts for complex data transformations > * Parallel running during transition period > > Contact our support team at [support@scorecard.io](mailto:support@scorecard.io) for personalized migration assistance. ## Billing and Plans > **How does Scorecard pricing work?** > > Scorecard offers flexible pricing based on usage: > > **Starter** (Free): > > * Unlimited users > * 100,000 scores per month > * Essential evaluation features for early-stage AI projects > > **Growth** (\$299/month): > > * Unlimited users > * Includes 1M scores per month, then \$1 per 5K additional > * Test set management > * Prompt management and versioning > * Priority support > > **Enterprise** (Custom pricing): > > * Custom solutions for large-scale AI deployments > * SAML SSO and enterprise authentication > * Dedicated support and customer success > * Custom compliance and security features > > Visit scorecard.io/pricing for the most current pricing information. > **What counts as a score?** > > A **score** is counted each time Scorecard evaluates a single test case with a metric: > > **Examples**: > > * 1 test case × 1 metric = 1 score > * 1 test case × 3 metrics = 3 scores > * 100 test cases × 2 metrics = 200 scores > > **Not Counted**: > > * Viewing existing results > * Creating/editing test cases > * Monitoring and tracing (separate feature) > * API calls for data retrieval > > **Bulk Discounts**: Enterprise customers get volume discounts for large-scale evaluations. ## Related Resources > **[Getting Started](/intro/sdk-quickstart)** > > Quick start guide for new users > **[Contact Support](https://www.scorecard.io/book-a-demo)** > > Get help with setup and migration > **[Status Page](https://status.scorecard.io)** > > Real-time platform status and uptime # LangChain Tracing Source: https://docs.scorecard.io/intro/langchain-quickstart Trace your LangChain applications with Scorecard using OpenLLMetry. This quickstart shows how to instrument your [LangChain](https://www.langchain.com/) application using [OpenLLMetry](https://github.com/traceloop/openllmetry) and Scorecard for observability, debugging, and evaluation. > [!NOTE] > > **Looking for general tracing guidance?** Check out the [Tracing Quickstart](/intro/tracing-quickstart) for an overview of tracing concepts and alternative integration methods. ## Steps **1. Install dependencies** Install the Traceloop SDK and the LangChain instrumentation package. ```bash theme={null} pip install traceloop-sdk opentelemetry-instrumentation-langchain ``` **2. Set up environment variables** Configure the Traceloop SDK to send traces to Scorecard. Get your Scorecard API key from [Settings](https://app.scorecard.io/settings). ```bash theme={null} export TRACELOOP_API_KEY="" export TRACELOOP_BASE_URL="https://tracing.scorecard.io/otel" export SCORECARD_PROJECT_ID="" ``` > [!NOTE] > > Replace `` with your actual Scorecard API key (starts with `ak_`). **3. Initialize tracing** Initialize the Traceloop SDK with LangChain instrumentation **before** importing LangChain modules. > [!WARNING] > > **Import order matters!** You must initialize Traceloop before importing any LangChain modules to ensure all calls are properly instrumented. ```python highlight={1-12} theme={null} import os from traceloop.sdk import Traceloop from traceloop.sdk.instruments import Instruments # Set scorecard.project_id to route traces to a specific project (defaults to oldest project) Traceloop.init( disable_batch=True, instruments={Instruments.LANGCHAIN}, resource_attributes={ "scorecard.project_id": os.getenv("SCORECARD_PROJECT_ID") } ) # Now import your LangChain modules from langchain_openai import ChatOpenAI from langchain_core.prompts import ChatPromptTemplate ``` **4. Run your LangChain application** With tracing initialized, run your LangChain application. All LLM calls, chain executions, and agent actions are automatically traced. Here's a full example: ```python title="example.py" highlight={1-11} [expandable] theme={null} import os from traceloop.sdk import Traceloop from traceloop.sdk.instruments import Instruments # Set scorecard.project_id to route traces to a specific project (defaults to oldest project) Traceloop.init( disable_batch=True, instruments={Instruments.LANGCHAIN}, resource_attributes={ "scorecard.project_id": os.getenv("SCORECARD_PROJECT_ID") } ) # Then import LangChain from langchain_openai import ChatOpenAI from langchain_core.prompts import ChatPromptTemplate # Create a simple chain prompt = ChatPromptTemplate.from_messages([ ("system", "You are a helpful assistant."), ("user", "{input}") ]) model = ChatOpenAI(model="gpt-4o-mini") chain = prompt | model # Run the chain - this will be traced response = chain.invoke({"input": "What is the capital of France?"}) print(response.content) ``` > [!NOTE] > > You may see `Failed to export batch` warnings in the console. These can be safely ignored - your traces are still being captured and sent to Scorecard successfully. **5. View traces in Scorecard** Navigate to the **Records** page in [Scorecard](https://app.scorecard.io) to see your LangChain traces. > [!NOTE] > > It may take 1-2 minutes for traces to appear on the Records page. Records page showing LangChain application traces Click on any record to view the full trace details, including chain execution, LLM calls, and token usage. Trace details with chain execution and LLM call data ## What Gets Traced OpenLLMetry automatically captures comprehensive telemetry from your LangChain applications: | Trace Data | Description | | - | - | | **LLM Calls** | Every LLM invocation with full prompt and completion | | **Chains** | Chain executions with inputs, outputs, and intermediate steps | | **Agents** | Agent reasoning steps, tool selections, and action outputs | | **Retrievers** | Document retrieval operations and retrieved content | | **Token Usage** | Input, output, and total token counts per LLM call | | **Errors** | Any failures with full error context and stack traces | ## Next Steps > **[Records](/features/records)** > > Learn more about viewing and managing your traced records > **[Metrics](/features/metrics)** > > Create custom metrics to evaluate LangChain application performance # MCP Server Quickstart Source: https://docs.scorecard.io/intro/mcp-quickstart Build testsets and metrics conversationally with Claude Desktop and the Scorecard MCP server Use the Scorecard MCP (Model Context Protocol) server to create evaluation testsets and metrics through natural language in Claude. Instead of writing code or clicking through UIs, just tell Claude what you need and it will use the Scorecard API to set everything up. > [!NOTE] > > This quickstart uses Claude Desktop, but it also works with other MCP clients, like Cursor and Claude Code. ## Steps **1. Setup Scorecard account** Create a [Scorecard account](https://app.scorecard.io/dashboard) if you don't have one already. **2. Install the Scorecard MCP server in Claude** Open Claude and add the Scorecard MCP server using the remote configuration: 1. Open Claude Desktop settings and click the "Connectors" tab 2. Click "Add custom connector" and paste the URL: `https://mcp.scorecard.io/mcp` 3. Click "Add", then "Connect" 4. Complete the OAuth authentication flow to connect your Scorecard account > [!NOTE] > > The remote MCP server requires no local dependencies or API key management. Authentication happens securely through your browser. Once connected, Claude has two tools: `search_docs` to look up the Scorecard SDK, and `execute` to run code against it. Together they cover the whole Scorecard API, so you only need to describe what you want in natural language. **3. Create a project** Simply ask Claude to create a new Scorecard project: > Create a new Scorecard project called "Customer Support Bot Evaluation" for testing my AI customer support assistant. Screenshot of the conversation showing project creation in Claude > [!NOTE] > > Behind the scenes, Claude writes a short TypeScript snippet against the Scorecard SDK and runs it with the `execute` tool. Expand the tool call in the conversation to see the code it ran. **4. Create a testset with testcases** Now create a testset to hold your evaluation test cases. Describe the structure you need: > Create a testset called "Support Scenarios" in this project. The testcases should have: > > * Input fields: "customerMessage" (the customer's question) and "category" (support category like billing, technical, or product) > * Expected output field: "idealResponse" (what a great response from the agent looks like) > * Metadata field: "difficulty" (easy, medium, or hard) > > Then add 5 testcases covering different support scenarios. Screenshot of the conversation showing testset creation in Claude > [!TIP] > > Claude can usually guess which inputs and output fields you want, but it's better to tell it what your field names are. **5. Create evaluation metrics** Define metrics to evaluate your AI system. Describe what "good" looks like: > Create two AI-scored metrics for this project: > > 1. "Response Accuracy" (integer) - Measures how well the response answers the customer's question compared to the ideal response > 2. "Tone Appropriateness" (boolean) - Checks if the response uses professional, empathetic language appropriate for customer support Screenshot of the conversation showing metric creation in Claude > [!NOTE] > > If Claude is unsure how to call the SDK, it runs `search_docs` first to look up the right method and parameters, then tries again. A failed first attempt followed by a corrected one is normal. **6. View your setup in Scorecard** Open your Scorecard project in the web UI to see everything that was created! Everything is now ready to score records. You can score records against your metrics through the UI, SDK, or continue using Claude with MCP. **7. Next: Score records** You can score records from the [Records page](/features/records) or the [SDK](/intro/sdk-quickstart). You can also continue the conversation to analyze and iterate on your metrics and scores. > Explain the latest scoring results for this project. > Update the Response Accuracy metric to be stricter about factual details. > Add 5 more testcases covering edge cases like angry customers and off-topic questions. The MCP server gives Claude access to the [full Scorecard API](/api-reference/overview), so you can manage your entire evaluation workflow conversationally. ## Tips for Using the MCP Server > [!TIP] > > **Be specific about data structures**: When creating testsets, clearly describe the field names, types, and which fields are inputs vs expected outputs. This helps Claude set up the schema correctly. > [!TIP] > > **Describe evaluation criteria**: When creating metrics, explain what makes a "good" output in detail. Claude will translate this into effective evaluation guidelines. > [!TIP] > > **Ask for recommendations**: Claude can suggest metrics, testcase scenarios, and evaluation strategies based on your use case. Just ask "What metrics should I use for evaluating a RAG system?" > [!TIP] > > **Iterate conversationally**: Made a mistake? Just ask Claude to fix it: "Make that metric stricter about factual details" or "Add a new field called 'priority' to the testset" ## Troubleshooting > **I have a different MCP client.** > > The Scorecard MCP server works with most MCP clients, including Claude Desktop, Cursor, and Claude Code. Make sure you've added the remote server URL correctly (`https://mcp.scorecard.io/mcp`) and completed the OAuth flow. See the [MCP Server documentation](/features/mcp) for more installation instructions. > **MCP server does not show up in Claude.** > > Make sure you've added the remote server URL correctly (`https://mcp.scorecard.io/mcp`) and completed the OAuth flow. Restart Claude if needed. > **Claude shows an authentication error.** > > The MCP server uses OAuth tokens that may expire. Try disconnecting and reconnecting the MCP server in Claude settings to refresh authentication. > **Claude says it can't find MCP tools.** > > Verify the MCP server is connected and enabled in Claude settings. You should see "Scorecard" listed in your active MCP servers. > **Only two tools show up.** > > That is expected. The server exposes `search_docs` and `execute`, and Claude reaches every Scorecard API operation by writing code for `execute`. See [How it works](/features/mcp#how-it-works). > **The local server says Deno is required.** > > The local `npx scorecard-ai-mcp` server runs code in a Deno sandbox on your machine. Install [Deno](https://deno.land) and make sure it is on your `PATH`, then restart your client. The hosted server at `https://mcp.scorecard.io/mcp` has no local requirements. > **I can't connect to remote MCP servers.** > > For local installation with your Scorecard API key, see the [MCP Server documentation](/features/mcp#local-configuration). Use `npx -y scorecard-ai-mcp@latest` with environment variables. ## Learn More Ready to go deeper? Check out these resources: > **[MCP Server Features](/features/mcp)** > > Complete guide to the Scorecard MCP server capabilities and architecture > **[Metrics Guide](/intro/metrics-quickstart)** > > Deep dive into creating and managing evaluation metrics > **[Testsets](/features/testsets)** > > Learn about testset schemas, field mappings, and organization # Metrics Quickstart Source: https://docs.scorecard.io/intro/metrics-quickstart Create metrics, group them, run evaluations, and read scores. Metrics page with Metrics, Groups, and Templates tabs. Metrics define what “good” looks like for your LLM. You describe the criteria (e.g., helpfulness, groundedness, safety), and Scorecard turns that into repeatable scores you can track across runs and over time. **1. Open Metrics and explore templates** Go to your project’s **Metrics** page. Start fast by copying a proven template, then tailor the guidelines to your domain. Templates list with Create from Template. **2. Create a metric** You can also create a metric from scratch. Provide a name, description, clear **guidelines**, and choose an **Evaluation Type** and **Output Type**. > [!NOTE] > > **Guidelines matter.** Describe what to reward and what to penalize, and include 1–2 concise examples if helpful. These instructions become the core of the evaluator prompt. **AI‑scored** Uses a model to apply your guidelines consistently and at scale. Pick the evaluator model and keep temperature low for repeatability. AI metric – evaluator model, output type, and evaluation guidelines. **Human‑scored** Best for nuanced judgments or gold‑standard baselines. Select **Human** as the evaluation type and write clear instructions for reviewers. Human evaluation – provide guidelines for reviewers. **Heuristic (SDK)** Deterministic, code‑based checks (e.g., latency, regex, policy flags). Select **Heuristic (SDK)** as the evaluation type and provide a scorer function in Python or TypeScript. Heuristic metric – Python or TypeScript scorer function. **Critic Agent (coming soon)** An agentic evaluator that can use tools, browse context, and reason over multiple steps before producing a score. Stay tuned for updates. **3. Go to the Records page and select records** Navigate to your project's [Records page](/features/records). Select the records you want to score, then click the **Score Records** button. Records page with selected records and Score Records button. **4. Choose metrics and score** In the **Score Records** modal, select one or more metrics to evaluate against, then click **Score**. Score Records modal – select metrics to evaluate. **5. View scores in the record panel** Once scoring completes, click any record to open the side panel. View scores, inputs, outputs, and evaluation details. Record detail panel with metric scores. ## Metric types * **AI‑scored**: Uses a model to apply your guidelines consistently and at scale. * **Human‑scored**: Great for nuanced judgments or gold‑standard baselines. * **Heuristic (SDK)**: Deterministic, code‑based checks via the SDK (e.g., latency, regex, policy flags). * **Critic Agent** *(coming soon)*: An agentic evaluator that reasons over multiple steps with tool use. * **Output types**: Choose **Boolean** (pass/fail) or **Integer (1–5)**. ## Second‑party metrics (optional) If you already use established evaluation libraries, you can mirror those metrics in Scorecard: * **MLflow genai**: Relevance, Answer Relevance, Faithfulness, Answer Correctness, Answer Similarity * **RAGAS**: Faithfulness, Answer Relevancy, Context Recall, Context Precision, Context Relevancy, Answer Semantic Similarity Copy a matching template, then tailor the guidelines to your product domain. ## Best practices for strong metrics * **Be specific.** Minimize ambiguity in guidelines; include “what not to do.” * **Pick the right output type.** Use Boolean for hard requirements; 1–5 for nuance. * **Keep temperature low.** Use ≈0 for repeatable AI scoring. * **Pilot and tighten.** Run on 10–20 cases, then refine wording to reduce false positives. * **Bundle into groups.** Combine complementary checks (e.g., Relevance + Faithfulness + Safety) to keep evaluations consistent. > [!NOTE] > > Looking for vetted, ready‑to‑use metrics? Explore [Best‑in‑Class Metrics](/features/metrics) and copy templates (including MLflow and RAGAS). You can also create deterministic checks via the SDK using **Heuristic** metrics. ## Related resources > **[Runs](/features/runs)** > > Create and analyze evaluations > **[A/B Comparison](/features/a-b-comparison)** > > Compare two runs side‑by‑side > **[Best‑in‑Class Metrics](/features/metrics)** > > Explore curated, proven metrics > **[API Reference](/api-reference/create-metric)** > > Create metrics via API # Overview Source: https://docs.scorecard.io/intro/overview The simulation platform for building frontier AI agents [Scorecard](https://scorecard.io) is the simulation platform for AI agent self-improvement. Run your agents through thousands of realistic scenarios in minutes and ship frontier capabilities with confidence. ## Platform Demo Watch CEO Darius demonstrate the complete Scorecard workflow, from creating test cases to comparing model performance: [Scorecard Platform Demo](https://www.youtube.com/embed/6NdOeGCLJ-c) > **[What is Scorecard?](/intro/what-is-scorecard)** > > Learn how simulation drives agent self-improvement > **[Tracing quickstart](/intro/tracing-quickstart)** > > Start sending traces in minutes > **[UI quickstart](/intro/ui-quickstart)** > > Explore your first trace in the UI > **[API Reference](/api-reference/overview)** > > Complete documentation and examples > **[Security & Privacy](https://trust.scorecard.io/)** > > Our security measures and compliance ## Get help Email [support@scorecard.io](mailto:support@scorecard.io) for assistance with your AI agent setup. # RAG Quickstart Source: https://docs.scorecard.io/intro/rag-quickstart Evaluate a Retrieval Augmented Generation (RAG) agent with Scorecard in minutes. RAG pairs retrieval with generation so your LLM can answer using fresh, domain‑specific context. This quickstart shows how to evaluate a simple RAG loop using Scorecard’s SDK, then highlights how to extend to retrieval‑only and end‑to‑end tests. ![Schema of a Production RAG System](https://mintcdn.com/scorecard-d65b5e8a/W_qF4JuImCEvs-ha/images/rag/1.png?fit=max\&auto=format\&n=W_qF4JuImCEvs-ha\&q=85\&s=72fce373459a9e1b714061c2eafd66a5) Schema of a Production RAG System We’ll simplify to the core pieces you need to test: ![Simplified Schema of a Production RAG System](https://mintcdn.com/scorecard-d65b5e8a/W_qF4JuImCEvs-ha/images/rag/2.png?fit=max\&auto=format\&n=W_qF4JuImCEvs-ha\&q=85\&s=5e223da171eaf0099112fb8e7ab473d0) Simplified Schema of a Production RAG System > [!NOTE] > > Already familiar with the SDK? You can reuse patterns from the [SDK Quickstart](/intro/quickstart) and [Tracing Quickstart](/intro/tracing-quickstart). ## Steps **1. Setup accounts** Create a [Scorecard account](https://app.scorecard.io/dashboard), then set your API key as an environment variable. ```sh theme={null} export SCORECARD_API_KEY="your_scorecard_api_key" ``` **2. Install SDK (and OpenAI optionally)** Install the Scorecard SDK. Optionally add OpenAI if you want a realistic generator. ```sh Python (pip) theme={null} pip install scorecard-ai openai ``` ```sh JavaScript (npm) theme={null} npm install scorecard-ai openai ``` **3. Create a minimal RAG system** We’ll evaluate a simple function that takes a query and retrievedContext and produces an answer. This mirrors a typical RAG loop where retrieval is done upstream and passed to the generator. **Python** ```py Python wrap theme={null} from openai import OpenAI # Uses OPENAI_API_KEY from environment openai = OpenAI() # Example input shape: # {"query": "What is RAG?", "retrievedContext": "..."} def run_system(inputs: dict) -> dict: messages = [ {"role": "system", "content": "Answer using only the provided context."}, {"role": "user", "content": f"Context:\n{inputs['retrievedContext']}\n\nQuestion: {inputs['query']}"}, ] resp = openai.chat.completions.create( model="gpt-4o-mini", messages=messages, temperature=0, ) return {"answer": resp.choices[0].message.content} ``` **JavaScript** ```js JavaScript wrap theme={null} import OpenAI from 'openai'; // Uses OPENAI_API_KEY from environment const openai = new OpenAI(); // Example input shape: // { query: "What is RAG?", retrievedContext: "..." } export async function runSystem(inputs) { const messages = [ { role: 'system', content: 'Answer using only the provided context.' }, { role: 'user', content: `Context:\n${inputs.retrievedContext}\n\nQuestion: ${inputs.query}` }, ]; const resp = await openai.chat.completions.create({ model: 'gpt-4o-mini', messages, temperature: 0, }); return { answer: resp.choices[0].message.content }; } ``` **4. Setup Scorecard client and Project** ```py Python theme={null} from scorecard_ai import Scorecard scorecard = Scorecard() # API key read from env PROJECT_ID = "123" # Replace with your Project ID ``` ```js JavaScript theme={null} import Scorecard from 'scorecard-ai'; const scorecard = new Scorecard(); // API key read from env const PROJECT_ID = '123'; // Replace with your Project ID ``` **5. Create RAG testcases** Each testcase contains the user `query`, the `retrievedContext` you expect to be used, and the `idealAnswer` for judging correctness. ```py Python theme={null} testcases = [ { "inputs": { "query": "What does RAG stand for?", "retrievedContext": "RAG stands for Retrieval Augmented Generation.", }, "expected": { "idealAnswer": "Retrieval Augmented Generation", }, }, { "inputs": { "query": "Why use retrieval?", "retrievedContext": "Retrieval injects fresh, domain‑specific context into the LLM.", }, "expected": { "idealAnswer": "To ground the model on current and domain data.", }, }, ] ``` ```js JavaScript theme={null} const testcases = [ { inputs: { query: 'What does RAG stand for?', retrievedContext: 'RAG stands for Retrieval Augmented Generation.', }, expected: { idealAnswer: 'Retrieval Augmented Generation', }, }, { inputs: { query: 'Why use retrieval?', retrievedContext: 'Retrieval injects fresh, domain‑specific context into the LLM.', }, expected: { idealAnswer: 'To ground the model on current and domain data.', }, }, ]; ``` **6. Create AI judge metrics** Define two metrics: one for context‑use (boolean) and one for answer correctness (1–5). These use Jinja placeholders to reference testcase inputs and system outputs. ```py Python wrap theme={null} context_use_metric = scorecard.metrics.create( project_id=PROJECT_ID, name="Context use", description="Does the answer rely only on retrieved context?", eval_type="ai", output_type="boolean", prompt_template=""" Evaluate if the answer uses only the provided context and does not hallucinate. Context: {{ inputs.retrievedContext }} Answer: {{ outputs.answer }} {{ gradingInstructionsAndExamples }} """, ) correctness_metric = scorecard.metrics.create( project_id=PROJECT_ID, name="Answer correctness", description="How correct is the answer vs the ideal (1–5)?", eval_type="ai", output_type="int", prompt_template=""" Compare the answer to the ideal answer and score 1–5 (5 = exact). Question: {{ inputs.query }} Context: {{ inputs.retrievedContext }} Answer: {{ outputs.answer }} Ideal: {{ expected.idealAnswer }} {{ gradingInstructionsAndExamples }} """, ) ``` ```js JavaScript wrap theme={null} const contextUseMetric = await scorecard.metrics.create({ projectId: PROJECT_ID, name: 'Context use', description: 'Does the answer rely only on retrieved context?', evalType: 'ai', outputType: 'boolean', promptTemplate: 'Evaluate if the answer uses only the provided context and does not hallucinate.\n' + 'Context: {{ inputs.retrievedContext }}\n' + 'Answer: {{ outputs.answer }}\n\n' + '{{ gradingInstructionsAndExamples }}', }); const correctnessMetric = await scorecard.metrics.create({ projectId: PROJECT_ID, name: 'Answer correctness', description: 'How correct is the answer vs the ideal (1–5)?', evalType: 'ai', outputType: 'int', promptTemplate: 'Compare the answer to the ideal answer and score 1–5 (5 = exact).\n' + 'Question: {{ inputs.query }}\n' + 'Context: {{ inputs.retrievedContext }}\n' + 'Answer: {{ outputs.answer }}\n' + 'Ideal: {{ expected.idealAnswer }}\n\n' + '{{ gradingInstructionsAndExamples }}', }); ``` **7. Run and evaluate** Use the helper to execute your RAG function across testcases and record scores in Scorecard. ```py Python wrap theme={null} from scorecard_ai.lib import run_and_evaluate run = run_and_evaluate( client=scorecard, project_id=PROJECT_ID, testcases=testcases, metric_ids=[context_use_metric.id, correctness_metric.id], system=lambda inputs, _version: run_system(inputs), ) print(f"Go to {run['url']} to view your scored results.") ``` ```js JavaScript wrap theme={null} import { runAndEvaluate } from 'scorecard-ai'; const run = await runAndEvaluate(scorecard, { projectId: PROJECT_ID, testcases, metricIds: [contextUseMetric.id, correctnessMetric.id], system: runSystem, }); console.log(`Go to ${run.url} to view your scored results.`); ``` **8. Analyze results** Review the run’s per‑metric stats, per‑record scores, and trends. Use this to iterate on prompts, retrieval parameters, and re‑run. Screenshot of viewing results in the Scorecard UI. ## Retrieval‑only and end‑to‑end tests Beyond the simple loop above, you may separately evaluate retrieval quality (precision/recall/F1, MRR, NDCG) and combine with generation for end‑to‑end scoring. ![Retrieval-Only LLM Testing in a RAG System](https://mintcdn.com/scorecard-d65b5e8a/W_qF4JuImCEvs-ha/images/rag/3.png?fit=max\&auto=format\&n=W_qF4JuImCEvs-ha\&q=85\&s=cd4b2b99bfa8e8fd369febdfae060228) Retrieval-Only LLM Testing in a RAG System ![Different Types of Testing in a RAG System](https://mintcdn.com/scorecard-d65b5e8a/W_qF4JuImCEvs-ha/images/rag/4.png?fit=max\&auto=format\&n=W_qF4JuImCEvs-ha\&q=85\&s=d0c1aaedfcb7a698c227d17ed70df537) Different Types of Testing in a RAG System > [!TIP] > > **Retrieval metrics in practice** > > * **Precision**: Of the retrieved items, how many are relevant? > * **Recall**: Of the relevant items, how many were retrieved? > * **F1**: Harmonic mean of precision and recall. > * **MRR**: Average reciprocal rank of the first relevant item. > * **NDCG**: Gain discounted by rank, normalized to the ideal ordering. > > **Ground‑truth dataset checklist** > > * Define representative queries (cover intents, edge cases, and long‑tail). > * For each query, collect relevant documents/chunks; annotate relevance (binary or graded). > * Include plausible hard negatives to stress the retriever. > * Write labeling guidelines; consider inter‑annotator agreement. > * Split into dev/test; iterate on retriever, then re‑score. ![Trigger a run](https://mintcdn.com/scorecard-d65b5e8a/ACSkl-xBQxg-5vWT/images/trigger-run-dark.png?fit=max\&auto=format\&n=ACSkl-xBQxg-5vWT\&q=85\&s=e639c5f97a63bf42e46cd3109cd10934) > [!NOTE] > > To operationalize RAG quality on live traffic, instrument traces ([Tracing Quickstart](/intro/tracing-quickstart)). Scorecard will sample spans, extract prompts/completions, and create Runs automatically. Scorecard works alongside RAG frameworks like [LlamaIndex](https://www.llamaindex.ai/) and [LangChain](https://www.langchain.com/). # SDK Quickstart Source: https://docs.scorecard.io/intro/sdk-quickstart Evaluate a simple LLM agent with Scorecard in minutes. Scorecard has [Python](https://pypi.org/project/scorecard-ai/) (v3.2.1) and [TypeScript](https://www.npmjs.com/package/scorecard-ai) (v2.2.0) SDKs. If you're using Python, you can follow along in [Google Colab](https://colab.research.google.com/drive/1A53j48o75Decrjz7WzDVP62XeDEKCHHP). ## Steps **1. Setup accounts** Create a [Scorecard account](https://app.scorecard.io/dashboard), then set your [Scorecard API key](https://app.scorecard.io/settings) as an environment variable. ```sh theme={null} export SCORECARD_API_KEY="your_scorecard_api_key" ``` **2. Install Scorecard SDK** Install the Scorecard library: ```sh Python (pip) theme={null} pip install scorecard-ai ``` ```sh JavaScript (npm) theme={null} npm install scorecard-ai ``` **3. Create simple LLM system to evaluate** For the quickstart, the LLM system is `run_system()`, a simple function that takes an input and returns an output. In Scorecard, system inputs and outputs are dictionaries. The function receives `system_input` and returns a dictionary. **Without OpenAI key** Here's a simple system that does not require an OpenAI API key: ```py Python wrap theme={null} # Example: # run_system({"original": "How are you?", "recipient": "team", "tone": "formal"}) # -> {"rewritten": "Hello team,\nHOW ARE YOU?"} def run_system(system_input: dict) -> dict: recipient = system_input.get("recipient", "") return { "rewritten": f"Hello {recipient} in tone {system_input['tone']},\n{system_input['original'].upper()}" } ``` ```js JavaScript wrap theme={null} // Example: // runSystem({"original": "How are you?", "recipient": "team", "tone": "formal"}) // -> {"rewritten": "Hello team,\nHOW ARE YOU?"} async function runSystem(systemInput) { const recipient = systemInput.recipient ?? ""; return { rewritten: `Hello ${recipient} in tone ${systemInput.tone},\n${systemInput.original.toUpperCase()}` }; } ``` **With OpenAI key** If you have an OpenAI API key, you can create a realistic system. Install the OpenAI library: ```sh Python (pip) theme={null} pip install openai ``` ```sh JavaScript (npm) theme={null} npm install openai ``` Then, create the system: ```py Python wrap theme={null} from openai import OpenAI # Find your API key at https://platform.openai.com/api-keys openai = OpenAI(api_key="your_openai_api_key") # Example: # run_system({"original": "Hello, world!", "tone": "formal concise", "recipient": "team"}) # -> {"rewritten": "Greetings, team."} def run_system(system_input: dict) -> dict: recipient = system_input.get('recipient','') response = openai.responses.create( model="gpt-4o-mini", instructions=f"You are a tone translator. Convert the user's message to the tone: {system_input['tone']}." + (f" Address the recipient: {recipient}" if recipient else ""), input=system_input['original'], ) return { "rewritten": response.output_text } ``` ```js JavaScript wrap theme={null} import OpenAI from 'openai'; // By default, the API key is taken from the environment variable. const openai = new OpenAI(); // Example: // runSystem({"original": "Hello, world!", "tone": "formal concise", "recipient": "team"}) // -> {"rewritten": "Greetings, team."} async function runSystem(systemInput) { const recipient = systemInput.recipient ?? ""; const response = await openai.responses.create({ model: 'gpt-4o-mini', instructions: `You are a tone translator. Convert the user's message to the tone: ${systemInput.tone}. ` + (systemInput.recipient ? `Address the recipient: ${systemInput.recipient}` : ""), input: systemInput.original, }); return { rewritten: response.output_text }; } ``` **4. Setup Scorecard** ```py Python theme={null} from scorecard_ai import Scorecard # By default, the API key is taken from the environment variable. scorecard = Scorecard() ``` ```js JavaScript theme={null} import Scorecard from 'scorecard-ai'; // By default, the API key is taken from the environment variable. const scorecard = new Scorecard(); ``` **5. Specify Project** Create a new Project in Scorecard, or use the existing default Project. This is where your testsets, metrics, and runs are stored. Set the Project ID for later: ```py Python theme={null} PROJECT_ID = "123" # Replace with your project ID ``` ```js JavaScript theme={null} const PROJECT_ID = "123"; // Replace with your Project ID ``` **6. Create test cases** Create some test cases to represent the inputs and the ideal (`expected`) outputs of your system. ```py Python theme={null} testcases = [ { # `inputs` gets passed to the system as a dictionary. "inputs": { "original": "We need your feedback on the new designs ASAP.", "tone": "polite", }, # `expected` is the ideal output of the system used by the LLM-as-a-judge to evaluate the system. "expected": { "idealRewritten": "Hi, your feedback is crucial to the success of the new designs. Please share your thoughts as soon as possible.", }, }, { "inputs": { "original": "I'll be late to the office because my cat is sleeping on my keyboard.", "tone": "funny", }, "expected": { "idealRewritten": "Hey team! My cat's napping on my keyboard and I'm just waiting for her to give me permission to leave. I'll be a bit late!", }, }, ] ``` ```js JavaScript theme={null} const testcases = [ { // `inputs` gets passed to the system as an object. inputs: { original: 'We need your feedback on the new designs ASAP.', tone: 'polite', }, // `expected` is the ideal output of the system used by the LLM-as-a-judge to evaluate the system. expected: { idealRewritten: 'Hi! your feedback is crucial to the success of the new designs. Please share your thoughts as soon as possible.', }, }, { inputs: { original: "I'll be late to the office because my cat is sleeping on my keyboard.", tone: 'funny', }, expected: { idealRewritten: "Hey team! My cat's napping on my keyboard and I'm just waiting for her to give me permission to leave. I'll be a bit late!", }, }, ]; ``` **7. Create Metrics** Create two LLM-as-a-judge Metrics to evaluate whether your system uses the correct tone and addresses the recipient. The Metric's prompt template uses Jinja syntax. For each Testcase, we will send the prompt template to the judge and replace `{{inputs.tone}}` with the test case's `tone` value. ```py Python theme={null} import textwrap tone_accuracy_metric = scorecard.metrics.create( project_id=PROJECT_ID, name="Tone accuracy", description="How well does it match the intended tone?", eval_type="ai", output_type="int", prompt_template=textwrap.dedent(""" You are a tone evaluator. Grade the response on how well it matches the intended tone: "{{inputs.tone}}". Use a score of 1 if the tones are very different and 5 if they are the exact same. Response: {{outputs.rewritten}} Ideal response: {{expected.idealRewritten}} {{ gradingInstructionsAndExamples }}"""), ) recipient_address_metric = scorecard.metrics.create( project_id=PROJECT_ID, name="Recipient address", description="Does it address the recipient only if specified?", eval_type="ai", output_type="boolean", prompt_template=textwrap.dedent(""" {% if inputs.recipient %} Does the response refer to the correct recipient: {{inputs.recipient}}? Response: {{outputs.rewritten}} {% else %} The response should avoid referring to any specific recipient. Response: {{outputs.rewritten}} {% endif %} {{ gradingInstructionsAndExamples }}"""), ) ``` ```js JavaScript wrap theme={null} const toneAccuracyMetric = await scorecard.metrics.create({ projectId: PROJECT_ID, name: "Tone accuracy", evalType: "ai", outputType: "int", promptTemplate: "You are a tone evaluator. Grade the response on how well it matches the intended tone {{inputs.tone}} and the tone of the ideal response. Use a score of 1 if the tones are very different and 5 if they are the exact same.\n\nResponse: {{outputs.rewritten}}\n\nIdeal response: {{expected.idealRewritten}}\n\n{{ gradingInstructionsAndExamples }}", }); const recipientAddressMetric = await scorecard.metrics.create({ projectId: PROJECT_ID, name: "Recipient address", evalType: "ai", outputType: "boolean", promptTemplate: "Does the response refer to the correct recipient: {{inputs.recipient}}? Response: {{outputs.rewritten}}\n\n{{ gradingInstructionsAndExamples }}", }); ``` **8. Evaluate system** Call `run_system()` against the test cases and record the scored results in Scorecard. ```py Python wrap theme={null} from scorecard_ai.lib import run_and_evaluate run = run_and_evaluate( client=scorecard, project_id=PROJECT_ID, testcases=testcases, metric_ids=[tone_accuracy_metric.id, recipient_address_metric.id], system=lambda input, _system_version: run_system(input), ) print(f'Go to {run["url"]} to view your scored results.') ``` ```js JavaScript wrap theme={null} import Scorecard, { runAndEvaluate } from 'scorecard-ai'; const run = await runAndEvaluate(scorecard, { projectId: PROJECT_ID, testcases: testcases, metricIds: [toneAccuracyMetric.id, recipientAddressMetric.id], system: runSystem, }); console.log(`Go to ${run.url} to view your scored results.`); ``` **9. Analyze results** Finally, review the results in Scorecard to understand the performance of your system. Viewing results in the Scorecard UI. ## Where to go next * [Tracing Quickstart](/intro/tracing-quickstart) — Add observability to your LLM calls * [SDK + Tracing](/features/sdk-tracing) — Combine evaluation data with trace observability in a single record # Tracing Quickstart Source: https://docs.scorecard.io/intro/tracing-quickstart Trace your LLM applications by pointing your client to Scorecard. This quickstart shows how to trace your LLM applications with Scorecard. Change your `baseURL` to `llm.scorecard.io` - no SDKs, no dependency management, works with your existing code. Supports OpenAI, Anthropic, and streaming responses. > [!NOTE] > > **Using Vercel AI SDK?** Check out our [AI SDK Wrapper](/features/ai-sdk-wrapper) for automatic tracing with zero manual instrumentation. > [!NOTE] > > **Need more control?** See the SDK wrappers section below for custom spans and deeper integration, or use OpenTelemetry directly. ## Steps **1. Get your Scorecard API key** Create a [Scorecard account](https://app.scorecard.io) and grab your API key from [Settings](https://app.scorecard.io/settings). ```bash theme={null} export SCORECARD_API_KEY="ak_..." export OPENAI_API_KEY="sk_..." # or ANTHROPIC_API_KEY ``` **2. Point your client to Scorecard** Change the `baseURL` to `https://llm.scorecard.io`. Everything else in your code stays the same - your LLM calls will be automatically traced. ```py Python highlight={7-11} theme={null} # Works with OpenAI and Anthropic - see examples below for more patterns from openai import OpenAI client = OpenAI( api_key=os.environ["OPENAI_API_KEY"], base_url="https://llm.scorecard.io", default_headers={ "x-scorecard-api-key": os.environ["SCORECARD_API_KEY"], "x-scorecard-project-id": "my-chatbot" # Optional: organize traces by project } ) # Use OpenAI normally - everything is automatically traced! response = client.chat.completions.create( model="gpt-4", messages=[{"role": "user", "content": "Hello!"}] ) ``` ```js JavaScript highlight={7-11} theme={null} // Works with OpenAI and Anthropic - see examples below for more patterns import OpenAI from 'openai'; const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY, baseURL: 'https://llm.scorecard.io', defaultHeaders: { 'x-scorecard-api-key': process.env.SCORECARD_API_KEY, 'x-scorecard-project-id': 'my-chatbot' // Optional: organize traces by project } }); // Use OpenAI normally - everything is automatically traced! const response = await client.chat.completions.create({ model: 'gpt-4', messages: [{ role: 'user', content: 'Hello!' }] }); ``` **3. View traces in Scorecard** Run your application, then visit [app.scorecard.io](https://app.scorecard.io) and navigate to **Records**. You'll see full request/response data, token usage, latency, and errors for all your LLM calls. Streaming responses are captured too. Viewing traces in Records. ## How It Works Scorecard forwards your requests to the LLM provider while capturing telemetry: ``` Your App → llm.scorecard.io → OpenAI/Anthropic ↓ Scorecard (traces, metrics, costs) ``` Your LLM calls work the same way - Scorecard forwards requests and captures telemetry. ## SDK Wrappers for Deeper Integration For more control than `llm.scorecard.io`, the Scorecard SDK wrappers provide automatic tracing with native OpenTelemetry integration. Wrap your OpenAI or Anthropic client once and all calls are automatically traced - including streaming responses. The real power comes from custom spans and nested traces. You can create parent spans for your workflows and business logic, and LLM calls will automatically nest as children. This gives you complete visibility into complex multi-step processes. ### Basic Usage > [!NOTE] > > The Python SDK wrappers require the optional `otel` extra, which installs OpenTelemetry. Without it, `wrap()` raises an `ImportError`: > > ```bash theme={null} > pip install 'scorecard-ai[otel]' > ``` > > The TypeScript SDK bundles OpenTelemetry, so `npm install scorecard-ai` is all you need. ```py Python highlight={4,7,9-13} theme={null} # Works with OpenAI and Anthropic - see examples below for more patterns from openai import OpenAI from scorecard_ai import wrap # Wrap your OpenAI client openai = wrap( OpenAI(api_key=os.environ["OPENAI_API_KEY"]), { "api_key": os.environ["SCORECARD_API_KEY"], "project_id": "my-chatbot" # Optional: organize traces by project } ) # Use normally - all calls are automatically traced response = openai.chat.completions.create( model="gpt-4", messages=[{"role": "user", "content": "Hello!"}] ) ``` ```js JavaScript highlight={3,7,11-15} theme={null} // Works with OpenAI and Anthropic - see examples below for more patterns import { wrap } from 'scorecard-ai'; import OpenAI from 'openai'; // Wrap your OpenAI client const openai = wrap( new OpenAI({ apiKey: process.env.OPENAI_API_KEY }), { apiKey: process.env.SCORECARD_API_KEY, projectId: 'my-chatbot' // Optional: organize traces by project } ); // Use normally - all calls are automatically traced const response = await openai.chat.completions.create({ model: 'gpt-4', messages: [{ role: 'user', content: 'Hello!' }] }); ``` ## Examples * [Node.js Examples](https://github.com/scorecard-ai/scorecard-node/tree/main/examples) - OpenAI, Anthropic, Vercel AI SDK, nested traces, streaming * [Python Examples](https://github.com/scorecard-ai/scorecard-python/tree/main/examples) - OpenAI, Anthropic, nested traces, streaming * [More Examples](https://github.com/scorecard-ai/scorecard-examples) - Additional integration examples ## Where to go next * [Tracing Features](/features/tracing) - Advanced tracing patterns * [SDK + Tracing](/features/sdk-tracing) - Combine evaluation data with trace observability in a single record # UI Quickstart Source: https://docs.scorecard.io/intro/ui-quickstart Send your first trace to Scorecard and explore it in the UI. See what your LLM app is doing inside Scorecard in minutes. Point your client at Scorecard's proxy to ingest a trace, then explore it on the Records page. **1. Get your API key** Create a [Scorecard account](https://app.scorecard.io) and copy your API key from [Settings](https://app.scorecard.io/settings). ```bash theme={null} export SCORECARD_API_KEY="ak_..." export OPENAI_API_KEY="sk_..." # or ANTHROPIC_API_KEY ``` **2. Send a trace** The fastest way to ingest a trace is the LLM proxy: change your client's `baseURL` to `https://llm.scorecard.io` and pass your Scorecard API key as a header. No SDK or instrumentation needed. ```py Python highlight={6-10} theme={null} import os from openai import OpenAI client = OpenAI( api_key=os.environ["OPENAI_API_KEY"], base_url="https://llm.scorecard.io", default_headers={ "x-scorecard-api-key": os.environ["SCORECARD_API_KEY"], "x-scorecard-project-id": "my-chatbot", # Optional: organize traces by project }, ) response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Hello!"}], ) ``` ```js JavaScript highlight={5-9} theme={null} import OpenAI from 'openai'; const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY, baseURL: 'https://llm.scorecard.io', defaultHeaders: { 'x-scorecard-api-key': process.env.SCORECARD_API_KEY, 'x-scorecard-project-id': 'my-chatbot', // Optional: organize traces by project }, }); const response = await client.chat.completions.create({ model: 'gpt-4o', messages: [{ role: 'user', content: 'Hello!' }], }); ``` Run your app once to send a trace. For SDK wrappers, OpenTelemetry, and framework integrations, see the [Tracing Quickstart](/intro/tracing-quickstart). **3. Find your trace on the Records page** Go to [app.scorecard.io](https://app.scorecard.io) and open the [Records](/features/records) page. Your trace appears as a record with its inputs, outputs, cost, latency, and any scores. Ingested traces on the Records page. Search and filter by status, source, trace ID, time range, and more. Use **Quick Filters** for presets and **Edit Table** to pick columns. **4. Explore the trace** Click a record to open its trace detail panel, which breaks the trace into four tabs: * **Conversation**: a chat-style replay of messages and tool calls * **Timeline**: a Gantt-style view of every span and how long each takes * **Chat**: ask questions about the trace in natural language * **Debug**: the full span tree, trace overview, and raw span data The Conversation tab of a trace. **5. Score the records** Score traces right from the Records page. Select the records to evaluate, then click **Score Records**. Select records, then click Score Records. Pick one or more metrics in the modal and click **Score**. Scores appear on each record when evaluation finishes. Choose the metrics to evaluate against. > [!NOTE] > > You need at least one metric in your project first. Create one in a minute with the [Metrics Quickstart](/intro/metrics-quickstart). ## Ask about your data with MCP Connect Scorecard's [MCP server](/features/mcp) to a client like Claude Code or Cursor and ask about your records, runs, and scores in natural language, without leaving your editor. ``` Show me the records from my latest run that scored lowest on Correctness, and summarize what they have in common. ``` ## Where to go next * Turn real traces into test cases with [Testsets](/features/testsets) * Define reusable evaluation criteria with [Metrics](/intro/metrics-quickstart) * Learn advanced instrumentation in [Tracing](/features/tracing) * Combine evaluation data with trace observability in [SDK + Tracing](/features/sdk-tracing) # What Is Scorecard? Source: https://docs.scorecard.io/intro/what-is-scorecard The simulation platform for building frontier AI agents. Run thousands of realistic scenarios in minutes and ship new capabilities with confidence. ![Scorecard Workflow](https://mintcdn.com/scorecard-d65b5e8a/ACSkl-xBQxg-5vWT/images/what-is-scorecard/scorecard-workflow.gif?s=45b7382d02a5c7ad393092d3a27c547f) Scorecard is the simulation platform for AI agent self-improvement. Run your agents through thousands of realistic scenarios in minutes, encode expert judgment into scalable reward models, and ship frontier capabilities in days. ## The bottleneck is feedback, not building Teams are building increasingly complex agents — multi-step workflows, consequential actions, real-world integrations. But the way most teams validate these agents hasn't kept up. The current approach is manual: review a handful of production cases, wait weeks for expert feedback, and hope nothing slips through. This limits you to scenarios you've already seen. Edge cases stay hidden until they hit production. Expert time doesn't scale — every new capability means more review cycles, longer iteration loops, and slower releases. The bottleneck has shifted from building to feedback. Scorecard flips this by turning expert judgment into automated reward models and replacing manual review with large-scale simulation. > [!TIP] > > **Traditional approach:** Review tens of production cases over weeks. > > **Scorecard approach:** Simulate tens of thousands of scenarios in 30 minutes. ## How Scorecard works **1. Encode expert judgment** Define reward criteria in natural language. Scorecard turns them into automated judges that score every scenario consistently and at scale. [Learn about metrics →](/features/metrics) **2. Simulate at scale** Run your agent through thousands of realistic scenarios. Generate diverse test scenarios automatically — no manual case writing required. [Synthetic data generation →](/features/synthetic-data-generation) · [Testsets →](/features/testsets) **3. Compare and improve** Quantitative A/B comparison across every metric. Iterate on prompts with real-time feedback to find the best prompt, model, or architecture. [A/B comparison →](/features/a-b-comparison) · [Prompts →](/features/prompts) **4. Ship with confidence** Integrate simulation into CI/CD so every pull request is validated automatically. Monitor production with tracing and feed real traffic back into your simulation suite. [GitHub Actions →](/features/github-actions) · [Tracing →](/features/tracing) ## Works with your agent stack > **[Claude Agent SDK](/intro/claude-agent-sdk-tracing)** > > Zero-code tracing. Set a few environment variables and get full visibility into agent decisions, tool use, and costs. > **[LangChain](/intro/langchain-quickstart)** > > Trace LangChain agents and chains with OpenTelemetry. > **[Any LLM](/intro/tracing-quickstart)** > > Works with OpenAI, Anthropic, Google, and any OpenTelemetry-compatible provider. ## Get started Built by engineers from Waymo, Uber, and SpaceX who used large-scale simulation to ship autonomous vehicles, global logistics, and rockets — now applied to AI agents. > **[Run your first evaluation](/intro/sdk-quickstart)** > > Set up Scorecard and run a simulation in minutes. > **[Explore your records](/features/records)** > > See every run, score, and trace in one place. > **[Talk to our team](https://www.scorecard.io/book-a-demo)** > > Book a demo and see Scorecard in action. Email [support@scorecard.io](mailto:support@scorecard.io) for help getting started.