> ## Documentation Index > Fetch the complete documentation index at: https://docs.scorecard.io/llms.txt > Use this file to discover all available pages before exploring further. # Product Updates > New updates and improvements export const DarkLightImage = ({lightSrc, caption, alt, darkSrc = null, width = "1000"}) => { const getAbsoluteUrl = src => { if (src.startsWith('http://') || src.startsWith('https://')) { return src; } const currentUrl = typeof window !== 'undefined' ? window.location.origin : ''; if (currentUrl.includes('.mintlify.app')) { const subdomain = currentUrl.split('.')[0].replace('https://', ''); return `https://mintlify.s3.us-west-1.amazonaws.com/${subdomain}${src.startsWith('/') ? '' : '/'}${src}`; } else if (currentUrl === 'https://docs.scorecard.io') { return `https://mintlify.s3.us-west-1.amazonaws.com/scorecard-d65b5e8a${src.startsWith('/') ? '' : '/'}${src}`; } else { return `${currentUrl}${src.startsWith('/') ? '' : '/'}${src}`; } }; const content = <> {alt}

; if (caption) { return {content}; } else { return content; } }; ### 💬 Conversation View Improvements Several upgrades to the threaded **Conversation view** on the record details page (Claude Code traces) to make debugging agent runs faster. * **Tool execution errors**: Failed tool calls now show a red **Error** pill on the row, a one-line truncated error preview when collapsed, and a full error block above the tool result when expanded. This catches a class of failures (like blocked `Agent` invocations) that previously rendered as normal successes because they signaled the failure via a `success: "false"` attribute instead of an OpenTelemetry error status. * **Inline sub-agent steps**: Sub-agent runs spawned by Claude Code's `Agent` tool are now rendered inline under the expanded parent tool call. Each nested LLM turn shows its header, thinking, and tool steps just like a top-level turn, and sub-agents that themselves invoke `Agent` nest recursively — so you can follow a multi-level agent run without leaving the conversation view. ### 🔗 Clickable Span Links in Trace Chat Span references that the **Chat tab** (the AI-powered trace summary) emits in its responses are now clickable. Clicking one jumps to the **Spans** tab, selects the matching span, and auto-scrolls the span tree to bring it into view. ### 📤 Export Annotations as JSON We replaced the old CSV/XLSX annotation exports with a single, much more useful **Export as JSON** option. Annotations are now inlined into the regular trace JSON: each annotation hangs off the exact span it was made on (record-level and orphan annotations attach to the trace itself), and each one carries the annotator's name. You get the full trace context alongside your labels in one file — ready for downstream analysis, training data, or review workflows. ### 📊 Records Analysis The Records page now features an AI-powered **Records Analysis** banner that automatically spots patterns across your latest records — like high failure rates, scoring anomalies, factual hallucinations, or stuck records. When you expand the banner, you'll see a list of insights ranked by severity (critical, warning, info), each with a description of the detected pattern. Click the **filter icon** next to any insight to instantly filter the records table to the relevant subset. The analysis automatically refreshes as new records come in, and you can manually trigger a refresh at any time. The footer shows when the analysis was last generated, how many records were analyzed, and how many unique patterns were detected. ### 📊 Run Details Page We've made several improvements to the Run Details page for a better evaluation review experience: * **Resizable popovers**: The scoring popover and column cells can now be resized, giving you more room to inspect detailed outputs * **Markdown rendering in cells**: Text outputs inside column cells now render markdown for improved readability * **Markdown in scoring reasoning**: The reasoning section on the score popover now renders markdown, making structured explanations easier to follow ### 🔍 Traces * **Export annotations as CSV**: You can now export your trace annotations directly as a CSV file for offline analysis and reporting * **New Chat tab**: Traces now include a dedicated **Chat tab** that provides an AI-powered summary of the trace. Ask questions about the trace to quickly understand what happened without manually inspecting every span. ### 🚀 Records Page Performance * **Faster page loading**: We significantly improved the performance of the Records page, so it loads quicker even with large datasets * **Faster search**: Searching records on the page is now noticeably faster, helping you find what you need without delay ### 💬 Conversation Tab The record details page now includes a dedicated **Conversation tab** that displays multi-turn interactions in an easy-to-read chat format. This makes it much simpler to review and debug conversational AI outputs without parsing through raw JSON. ### 📥 Records CSV Export You can now export records directly from the Records page as a CSV file. This makes it easy to share evaluation results with stakeholders, perform offline analysis, or integrate with external tools. ### 🔍 Arbitrary String Search in Records The Records page now supports full-text search across all record fields. Search for any substring in inputs, outputs, or expected values to quickly find specific records you're looking for. ### Records page The Records page is now more powerful for debugging and triaging large runs: you can pull useful fields into dedicated columns, sort and filter more precisely, and quickly jump to "what I just ran." * **Extract columns from inputs, outputs, and expected**: Expand inputs/outputs/expected objects into separate per-field columns so you can scan specific keys without opening each record. * **Reorder columns**: Reorder columns in the table editor (drag-and-drop) and save your preferred layout. * **Created At/By columns**: See when a record was created and who kicked off the run that produced it. * **Created At/By filters**: Filter by a time range and/or by the run creator to focus on the records you care about. * **Sort by Created At**: Sort the table by record creation time to see newest records first (and quick filters will set this automatically). * **New quick filters**: Narrow down to common views like **My Recent Records** with a new "Quick Filters" menu. ### Simpler projects navigation We removed the standalone projects page and made project switching faster by showing project information in the top navigation. ### New metric templates We updated our built-in metric templates to help you get started faster with common evaluation goals (and reduce the amount of prompt engineering needed to create a high-signal judge). Our templates now include **Hallucination detection**, **Response completeness**, **PII leakage**, **Coherency**, **User intent fulfillment**, **Content moderation**, **Source attribution quality**, **Bias**, and **Conciseness**. ### 🛠️ Bug Fixes and Improvements * Added a new "Help" menu in the sidebar with a "Send Feedback" option. * **\[New models]** Added support for GPT 5.2 released this week. ### New playground We've introduced a workflow-based playground UI. It offers the same functionality as the previous playground, but with a more intuitive structure that makes the relationship between testcases, prompts, and results clearer. The new interface is still experimental and currently enabled only for newly created Scorecard organizations. If you'd like early access, let us know! We're iterating quickly based on feedback. ### New records page We've added a consolidated Records page that lists every record across all runs. This unlocks new workflows, including viewing testcase performance history, identifying testcases that failed specific metrics, and filtering records by substring. Eventually, this will replace the existing Runs & Results page. ### Onboarding wizard When you create a new project, you'll now see a guided onboarding wizard to help you get started faster. Based on your use case, we recommend an onboarding path (testsets, tracing, or SDK) and suggest a set of metrics to begin with. ### 🛠️ Bug Fixes and Improvements * **\[UI]** Descriptions are now optional when creating systems, endpoints, and metric groups. * **\[Metrics]** Replaced the "Create" and "Edit" metric modals with a new full-page metric editor. * **\[Metrics]** Added a "Recently used" badge to the metric card to quickly identify metrics that were recently used in a run. * **\[Testsets]** Improved testset schema modal by removing tabs. * **\[Runs]** Fixed newline handling when exporting a run (or testset) to Excel-compatible CSV. * **\[Tracing]** We removed the "Monitoring" page and combined it with the existing "Traces" page. * **\[UI]** Simplified the sidebar navigation by moving lower-traffic pages into an auto-collapsed Advanced section. * **\[UI]** Added page descriptions to headers throughout the app ### 🤖 New Model Support Added support for the latest models, including **GPT 5.1** and **Gemini 3 Pro**. Try them out and let us know what you think! ### 🔎 Monitor Visibility on Traces Page The traces page now displays a table of active monitors, making it easier to configure and manage monitors. ### 🏃 Monitors "Run now" feature The "Run now" button on the monitors page allows you to run a monitor immediately, without having to wait for the next scheduled run. Now, the feature supports selecting a custom "look back" time range. ### 🔎 Filter Runs by Source You can now filter runs by their source (API, Monitor, Playground, or Kickoff), helping you quickly find runs created from specific workflows. ### 🛠️ Bug Fixes and Improvements * **\[Testsets]** Improved the testset schema editor for a better editing experience. * **\[Metrics]** Fixed the empty state display for the metric groups tab. * **\[Testsets]** Fixed a bug where editing different fields in the same testcase could lose the first change. * **\[Docs]** Improved navigation on this documentation site. * **\[Internal]** Upgraded to Next.js 16 for better performance. This week, we focused on improving metrics. ### 🔄 Re-run Scoring for Existing Runs You can now re-run scoring with the latest version of a metric or add new metrics to existing runs without having to re-run your system. This makes it much faster to iterate on metrics! ### 🧑‍💻 Heuristic Code Runners Metrics now support custom code-based evaluation logic. This allows you to write your own evaluation logic in Python or Typescript. ### 📏 Improved Metric Details Page We overhauled the metric details page to make it easier to edit metrics. With the metric template preview, you can see your instructions to the LLM-as-a-judge will render without running the metric. **"Recently used" badge**: Quickly identify metrics that were recently used in a run. ### 🔍 Improved Trace Search The traces page now supports filtering making it easier to find exactly what you're looking for. * **Trace ID**: Filter by the root trace ID * **Run ID**: Filter by the trace's run ID * **Service**: Filter by the OpenTelemetry service name * **Span Name**: Filter for runs containing a span with the given name ### 🚀 New API Endpoints Added a [Delete Metric](/api-reference/delete-metric) endpoint. ### 🛠️ Bug Fixes and Improvements * **\[Runs]** Added a "Notes" column to the runs table for better organization This week, we focused on improving the tracing and monitoring features. ### 📊 Improved Trace Details UX The trace details page now features a **collapsible span tree** and better scrolling behavior for easier navigation through long, highly nested traces. ### 🔗 Automatic Trace Grouping In your traces, you can use the `scorecard.tracing_group_id` span attribute to automatically group traces into runs based on a custom identifier. This makes it easier to track and analyze multi-step workflows or batch operations. ### 🎯 Cross-Project Monitoring If you choose, monitors can now pick up traces from any project in your organization, allowing you to set up centralized monitoring rules across your entire workspace. ### 🔎 Span Name Regex Filter for Monitors Added a **"Span name (regex)"** filter to monitors, giving you more precise control over which spans trigger your monitoring rules. ### 📈 Linkable Metric Groups Metric groups are now full pages instead of modals, making it easy to share direct links to specific metric groups with your team. We'll continue to improve the metric creation experience next week. ### 🛠️ Bug Fixes and Improvements * **\[Traces]** Added a **Project ID column** to the traces table to help you quickly identify traces by project. * **\[Traces]** Fixed the trace chart visualization to show traces in the given time period, not just the traces shown in the trace table. * **\[API]** Fixed pagination issue where a cursor pointing to a nonexistent item would skip the first item in the result set. ### 🚀 AI SDK Wrapper Launch We've launched a new AI SDK wrapper for seamless integration with Scorecard! The wrapper makes it easy to add evaluation and monitoring to your AI applications with minimal code changes. Learn more in our [AI SDK Wrapper documentation](/features/ai-sdk-wrapper). ### 📚 New Tracing Examples We've added comprehensive tracing code examples for instrumenting your applications with OpenTelemetry: * **[Pydantic and Logfire](https://github.com/scorecard-ai/scorecard-examples/tree/main/python-logfire-otel-basic)**: Python-based tracing with Pydantic validation and Logfire integration * **[Traceloop](https://github.com/scorecard-ai/scorecard-examples/tree/main/nodejs-traceloop-basic)**: Node.js tracing with Traceloop for easy LLM observability * **Manual OpenTelemetry**: * [Workflow example](https://github.com/scorecard-ai/scorecard-examples/tree/main/nodejs-otel-workflow): Complete workflow implementation * [Basic example](https://github.com/scorecard-ai/scorecard-examples/tree/main/nodejs-otel-basic): Getting started with OpenTelemetry ### 🛠️ Bug Fixes and Improvements * **\[API]** Added new [Delete Records](/api-reference/delete-record) endpoint to delete records, helping you keep your workspace clean and organized * **\[Runs]** Renamed "Trigger run" to "[Kickoff run](/features/runs#kickoff-run-from-the-ui)" for consistency across the platform * **\[Runs]** Changed the runs page to default to the "All runs" tab instead of "My runs" for better discoverability * **\[Runs]** Increased the character limit per cell when exporting a run as CSV from 32,000 to 128,000 characters * **\[Analysis]** Fixed a bug where the analysis page crashed when there are no metrics * **\[Playground]** Updated the delete icon for playground messages from a minus symbol to a trash can for better clarity * **\[Playground]** Added support for adding a new Provider directly in the playground instead of navigating to the settings page ### 📚 Documentation Updates We've expanded our documentation with new guides and resources: * [MCP Quickstart](/intro/mcp-quickstart) - New guide for setting up the Scorecard MCP server in Claude (Web/Desktop) * Analysis - New documentation on the Analysis page to compare performance metrics side by side ### 🛠️ Bug Fixes and Improvements * **\[Testcases]** Added support for cross-project testcase copying, making it easier to share test cases across different projects. ### 🚀 Product Hunt Launch We're excited to announce that Scorecard is now on Product Hunt! Check out our listing on [Product Hunt](https://www.producthunt.com/products/scorecard) to see what people are saying about Scorecard. ### 🛠️ Bug Fixes and Improvements * **\[Traces]** We now create testcases from *any* OpenTelemetry GenAI span in the tracing page, which extends our support for open standards in LLM observability. * **\[Run details]** When changing the prompt template or parameters in the playground, you no longer need to save the prompt before kicking off a run using those parameters. ### 📊 Tracing We now automatically create traces for runs generated on Scorecard's Playground. ### 🚀 New API Endpoints We've added new API endpoints for better metric and record management: * [Get Metric](/api-reference/get-metric): Retrieve details for a specific metric. * [List Metrics](/api-reference/list-metrics): Retrieve details for all metrics in a project. * [List Records](/api-reference/list-records): Retrieve details for all records and scores in a run. ### 🔌 MCP server The Scorecard MCP server can now analyze runs and make suggestions to improve your system. ### 🎯 MCP evals We improved dataset generation in [mcpevals.ai](https://mcpevals.ai)! It now generates realistic tool calls to better test your MCP server. ### 📊 Analysis Page We've released a new Analysis page that provides deeper insights into your evaluation data. ### 🔌 Scorecard MCP Server - Open Source We've released the updated source code for our MCP server on [GitHub](https://github.com/scorecard-ai/scorecard-mcp)! See how we integrated: * [Clerk](https://clerk.com) for authentication (OAuth 2.0 Protected Resource Metadata). * [Stainless](https://stainless.com) for generated MCP endpoints. * [Sentry](https://sentry.io) for production-grade monitoring and error tracking. ### 🚀 New API Endpoints We've added new API endpoints for better run management: * [Get Run](/api-reference/get-run): Retrieve details for a specific run. * [List Runs](/api-reference/list-runs): Retrieve details for all runs in a project. ### 📏 Correctness Metric Template A new float metric template called "Correctness" is now available, providing a standardized way to measure accuracy with decimal precision. ### 🛠️ Bug Fixes and Improvements * **\[Runs]** Testcase ID text in the record table now links directly to the record's testcase details page for easier navigation. * **\[Error Messages]** Improved error messaging for permissions failures throughout the UI. * **\[Scoring]** Exposed complete error details when scoring fails. For example, if a record exceeds the AI metric's context length, the full error message is now displayed instead of a generic "Internal server error". ### 🎯 MCP Evaluations Platform Launch Introducing **MCP Evals** - a dedicated platform for evaluating Model Context Protocol servers! Test and benchmark MCP servers with standardized evaluation workflows at [mcpevals.ai](https://mcpevals.ai). Key features: * **Dynamic Testing**: AI-generated evaluation tests tailored to each server's capabilities with results in seconds. * **Performance Metrics**: Detailed capability assessments and server response times. * **Open Source**: Evaluation tools and methodologies available at our [GitHub repository](https://github.com/scorecard-ai/mcp-eval). ### 🎨 Enhanced Card Design We've refreshed our card design throughout the platform for better visual hierarchy, improved readability, and modern aesthetics. ### 📊 Float Output Type Support Scorecard now supports float output types. This enhancement enables more precise measurement of continuous metrics and scoring systems that require decimal precision. ### 🛠️ Bug Fixes and Improvements * **\[Exports]** Fixed encoding in CSVs when exporting testcases or records with accented characters like "è". * **\[Documentation]** Enhanced [testset documentation](/features/testsets#understanding-input-fields) with clearer guidance on structuring input fields and practical examples. ### 🌐 Google Vertex AI Support Scorecard now fully supports Google Vertex AI models! Evaluate your applications using Google's latest Gemini models including Gemini 2.5 Pro, Gemini 2.5 Flash, and Gemini 2.0 Flash. This integration brings enterprise-grade AI capabilities to your evaluation workflows with seamless configuration through our settings panel. ### 🎮 Playground Enhancements Create new testsets and prompts directly from the [Playground](/features/playground) without leaving your workflow. We've added smart empty state buttons that help you get started quickly when no testsets or prompts exist. ### 🛠️ Bug Fixes and Improvements * **\[Data Import]** Added TSV file support for testcase uploads * **\[Data Import]** Column header names are now automatically trimmed during upload * **\[Testsets]** Description field is now optional when creating testsets ### 🚀 Official MCP Registry Launch We're thrilled to announce that Scorecard is now officially part of the Model Context Protocol (MCP) server registry! As the 28th server registered, we're among the first wave of official MCP integrations, making Scorecard's evaluation capabilities accessible directly within your AI development workflow. Our MCP server is available across all major AI platforms: * **claude.ai**: Full integration for seamless evaluation workflows * **Cursor**: Test and evaluate your code directly in your IDE * **All MCP clients**: Universal compatibility with any MCP-enabled tool Key capabilities: * **Direct AI Integration**: Run evaluations without leaving your AI assistant * **Real-time Testing**: Evaluate outputs instantly as you develop * **Natural Language Control**: Configure metrics and run experiments through conversation Connect to our MCP server: `https://mcp.scorecard.io` MCP Ecosystem Diagram showing how MCP servers connect AI assistants with external tools and services

MCP Ecosystem Diagram showing how MCP servers connect AI assistants with external tools and services

We'd love your feedback! As one of the first official MCP servers, your experience and suggestions help shape the future of AI evaluation. [Learn more](/features/mcp) or [share your feedback](https://github.com/scorecard-ai/scorecard-mcp). ### 🛠️ Bug Fixes and Improvements * **\[Auto-refresh]** Charts and headers now automatically refresh after creating new runs * **\[Model Selection]** Fixed scrolling issues in AI model dropdown menus * **\[Empty States]** Enhanced empty state designs for monitors, systems, metrics, metric groups, and endpoints with helpful quickstart links * **\[Multi-page Comparisons]** Select and compare runs across multiple pages for better historical analysis * **\[Search Enhancement]** Testset filtering now searches both names and tags * **\[Sorting]** Fixed "Sort by newest" to correctly order items by creation date * **\[Input Validation]** New test cases properly validate inputs against the defined schema * **\[Monitor Deduplication]** Resolved issue where overlapping monitors created duplicate test records ### 📊 A/B Comparison Compare runs side-by-side to identify the best performing system configurations and make data-driven decisions about your AI improvements. Key capabilities: * **Run comparison**: Compare different runs to see which system performs better * **Visual analysis**: Side-by-side view of outputs and scores for easy comparison * **Performance insights**: Identify which configurations work best across different metrics ### 📚 Enhanced Documentation New comprehensive guides expanding our feature documentation: * [Metrics](/features/metrics) - Create and manage evaluation metrics * [A/B Comparison](/features/a-b-comparison) - Compare system performance * [MCP Server Integration](/features/mcp) - Connect with AI tools via MCP protocol ### 🛠️ Bug Fixes and Improvements * **\[Endpoints]** Improved endpoint selector UI for better usability * **\[Organizations]** Improved styling and layout for the new organization creation flow * **\[Projects]** Removed confusing status indicator on project cards * **\[Projects]** Added support for editing project titles directly from project overview ### 🔌 Custom Endpoints with Multi-Turn Support Test any HTTP API endpoint directly in Scorecard. Configure once, use everywhere - from production APIs to local development servers. * **Universal API Testing**: Support for all HTTP methods with custom headers and bodies * **Multi-Turn Conversations**: Enable simulated conversations with configurable AI personas * **Tabbed Run Interface**: Switch between Scorecard, GitHub, and Endpoints in one modal * **Response Path Extraction**: Extract specific values from JSON responses for evaluation ### 🎯 Streamlined Onboarding New users now start with a pre-configured example project and automatic first run, allowing them to see evaluation results immediately without any manual setup. * **Auto-generated Project**: Complete with testsets, metrics, and sample data * **Instant First Run**: Results ready on first login * **Guided Actions**: Clear prompts to view results or create custom runs ### 📚 Expanded Documentation New comprehensive guides expanding our feature documentation: * [Custom Endpoints](/features/endpoints) - Test and evaluate HTTP APIs * [Tracing](/features/tracing) - Debug and monitor AI applications with error detection * [Synthetic Data Generation](/features/synthetic-data-generation) - Generate test data with AI ### 📋 Trace to Testcase Creation Create testcases directly from production traces with our new trace-to-testcase feature. Turn real user interactions into structured test data with a single click, making it easy to build datasets from production traffic that actually matter. The workflow is simple: select a span, choose your testset, and Scorecard auto-extracts the prompt and completion fields for you. Key capabilities: * **One-click creation**: Convert any trace into a testcase directly from the trace details page * **Smart field detection**: Automatically populate prompt and completion values from trace data * **Schema compatibility**: Automatically detects target testset schema and maps fields correctly * **Production-grounded datasets**: Build testsets from real user interactions instead of synthetic data Perfect for creating "golden datasets" from your best production examples and edge cases you want to regression test. ### 🏷️ Metadata Fields in Testset Schemas You can now mark fields as "metadata" in your testset schema management UI. Metadata fields are stored with your testcases but won't be used during evaluation runs, giving you more flexibility to store contextual information alongside your test data. * **Flexible data storage**: Store additional context without affecting evaluation logic * **Schema management**: Easy toggle in the testset schema editor * **Future SDK support**: Full SDK integration coming soon for programmatic access ### 🛠️ Bug Fixes and Improvements * **\[Tracing]** Fixed pagination issues with traces containing many spans - improved parent span detection * **\[Testcases]** Fixed refresh bug where editing a field could overwrite with old data * **\[Projects]** Fixed refresh issue after creating new projects - list now updates immediately * **\[Scoring]** Better error messages when OpenAI API keys are missing instead of generic "Internal service error" * **\[UI]** Improved copy testcase, move metric, and move testset dialogs with better scrollability and selection * **\[Documentation]** Added comprehensive guide for the trace-to-testcase feature. ### 🤖 Sim Agents for Multi-Turn Conversation Testing We've launched [Sim Agents](/features/multi-turn-simulation), a powerful new capability for testing multi-turn conversations with your AI systems. Create configurable AI personas that interact with your system during testing, simulating real user behaviors from polite customers to escalation scenarios. Key capabilities: * **Persona configuration**: Define user behaviors, goals, and interaction patterns with Jinja2 templating * **SDK integration**: Run simulations programmatically with `multi_turn_simulation()` method * **Conversation control**: Set stop conditions, max turns, and timeout limits for realistic testing * **Chat visualization**: View full conversation history in beautiful chat bubble format ### 📊 Online Evaluations (Beta) Monitor and evaluate your AI systems in production with our new online evaluation infrastructure. Configure monitoring rules per project, automatically score production traces, and get real-time insights into model performance. * **Monitoring configuration**: Set up project-specific rules with selected metrics and scheduling * **Cost visibility**: View token costs per span and per evaluation directly in the UI * **Trace integration**: Link test records to their source traces for full observability * **Real-time scoring**: Automatically score production traces as they are ingested ### 🛠️ Bug Fixes and Improvements * **\[Performance]** Major Prisma upgrade delivering 2-5x faster load times - run lists now load in under 2 seconds (down from 3-4s) * **\[Monitoring]** Added token cost tracking per span in traces page * **\[Monitoring]** Display scoring costs for LLM-as-a-judge evaluations * **\[Infrastructure]** Increased collector object size limit for larger traces ### 📊 Enhanced Tracing Experience We've completely reimagined our tracing interface to provide deeper insights into your AI system's performance. The new tracing UI features flame graph visualizations that make it instantly clear which operations take the longest, helping you identify and optimize bottlenecks in your LLM pipelines. Key improvements include: * **Flame graph visualization**: See span durations at a glance to quickly identify performance bottlenecks * **Smart defaults**: Tracing page now defaults to "All projects" view so you never miss traces * **Simplified authentication**: Use your standard Scorecard API key for tracing - no more separate JWT tokens ### 🚀 Smarter Run Status Tracking We've significantly enhanced how run statuses are calculated and displayed throughout the platform. You can now see exactly why a run is in its current state with detailed hover explanations, making it easier to understand and debug your evaluation pipelines. * **Intelligent status calculation**: Database now tracks expected vs. actual test records for accurate progress reporting * **Hover explanations**: See detailed progress like "0 testrecords created / 35 testcases from testset 1234" to understand exactly what's happening * **Real-time updates**: Run status automatically updates as scoring progresses ### 🛠️ Bug Fixes and Improvements * **\[Tracing]** Fixed crash when navigating back to page 2 in traces list * **\[Tracing]** Time range filter now correctly updates chart data * **\[Documentation]** Published tracing quickstart guide with integrations for [Vercel AI SDK](https://ai-sdk.dev/providers/observability) and [OpenLLMetry](https://traceloop.com/docs/openllmetry/integrations/introduction) ### 📊 Run History We've added a new Run History visualization to help you track evaluation performance over time. The chart displays aggregate scores per run, making it easy to spot trends and regressions across your metrics. You can view this directly on the run list page to see how your model's performance evolves with each iteration. ### 🔍 Tracing improvements We've upgraded our tracing infrastructure to provide better observability into your AI systems: * **Open-source collector architecture**: Migrated to the upstream OpenTelemetry standard collector on Railway for improved reliability and performance. * **Unified API key authentication**: You can now use your existing Scorecard API key for tracing so you no longer need to manage separate authentication tokens. * **New tracing integration**: Published official integrations for Vercel AI SDK and OpenLLMetry with examples in both Python and Node.js. Our new [step-by-step guide](/intro/tracing-quickstart) gets you from zero to traced LLM calls in minutes ### 🛠️ Bug Fixes and Improvements * **\[Run kickoff]** Fixed model scrolling in Run Kickoff modal UI. * **\[Run details]** Metrics cards and score columns are now consistently sorted alphabetically. * **\[Run details]** Sorting a column now displays the sort direction. ### Separate Score Columns On the run details page, each metric now has its own column of scores, rather than a single column for all metrics. This makes it easier to compare the scores across records for a particular metric. It also enables sorting records by a metric's score. ### 🛠️ Bug Fixes and Improvements * **\[Performance]** Improved loading time of the Runs list page. * **\[Runs]** Added a new "Run Again" button to the Run details page, allowing you to re-run the same testset/system combo with the same metrics and model parameters. * **\[Runs]** Added "System Version" link to the Run details page. ### 🚀 Onboarding Improvements We've streamlined the onboarding process for new organizations with several key improvements: * **Automatic defaults**: New organizations now receive default projects, testsets, metrics, and prompts automatically, significantly reducing initial setup time * **Free API key included**: Every new organization gets a default free API key (Gemini Flash), eliminating the need for initial user configuration. Users without their own API key will be clearly notified that the system is defaulting to the free API ### 🏃 Kickoff Runs from Playground You can now trigger runs directly from the Playground and view them in the runs & results section. We're considering adding scoring capabilities from the Playground as well - let us know if this is important to your team!

Trigger runs directly from the Playground

### 🛠️ Bug Fixes and Improvements * **\[Performance]** Improved page load performance on testcases, testsets, metrics, and trigger run pages * **\[Projects]** Added search functionality for projects * **\[Metrics]** Added search functionality for metrics * **\[Testsets]** Added search functionality for testsets * **\[Systems]** All systems are now required to have a production version * **\[Quickstart]** Both [main](/intro/quickstart) and systems quickstarts now reflect the latest SDK with clearer steps and updated code, enabling new users to go from install to first run in minutes * **\[API Keys]** Every member can view existing keys, but only admins can create or revoke them, providing teams with transparency while maintaining control * **\[Docs Search]** Rebuilt documentation search pipeline reducing typical response times from \~15s to just 3s, helping you find answers more quickly ### 🖥️ Systems Enhancements Managing systems in Scorecard just got easier and more powerful: * You can now directly trigger a run from the trigger-run page, making it quicker and simpler to execute tests. * All your systems are now clearly visible and easily manageable from a single, user-friendly interface. Update configurations, manage versions, and maintain systems effortlessly. Systems overview interface showing all systems in one place

Systems overview interface showing all systems in one place

### 🎨 New Color Palette We've updated the Scorecard UI to match our vibrant new brand colors, moving from purple to a fresh orange theme. We hope you love the refreshed look and please share your feedback with us! New orange color palette in the Scorecard UI

New orange color palette in the Scorecard UI

### 🛠️ Bug Fixes and Improvements * **\[Runs]** We've improved pagination for test records in runs, making load times and navigation quicker and setting the stage for upcoming filtering enhancements. ### 🎯 Metric API We've launched new API endpoints for programmatic metric management, enabling teams to [create](/api-reference/create-metric) and [update](/api-reference/update-metric) metrics directly through our SDK. Create metrics with full control over evaluation type, output format, and prompt templates: ```python theme={null} client.metrics.create( project_id="314", name="Response Accuracy", eval_type="ai", output_type="boolean", prompt_template="Evaluate if the following response is factually accurate: {{outputs.response}}", ) ``` These endpoints are supported in SDK versions [1.1.0](https://www.npmjs.com/package/scorecard-ai) (JS) and [2.1.0](https://pypi.org/project/scorecard-ai/) (Python). ### 🎯 Improved API key format We switched to a new API key format, which supports having multiple API keys, revoking them, and setting expiry dates. Our new API keys are more concise (94% shorter!) than the previously unwieldy API keys based on JWTs. We've deprecated the old API key format, but will continue to support them until July 20, 2025. New API key management interface showing multiple keys and expiry options

New API key management interface showing multiple keys and expiry options

### Bug fixes and improvements * **\[Metrics]** Improved metrics management UI with tabbed sections for better organization * **\[Metrics]** Added metric selector with searchable dropdown - no more manually typing metric IDs

Metric selector with searchable dropdown

* **\[Runs]** In the test record details page you can now see the text of the fully compiled metrics sent to the LLM for evaluation Compiled metrics text view in run details

Compiled metrics text view in run details

### 🚀 SDK v2 Stable Release Following our May 30th launch, SDK v2 is now fully stable and battle-tested. The new SDK features simplified APIs with ergonomic helper methods like `runAndEvaluate` (JS/TS) and `run_and_evaluate` (Python), making it easier than ever to integrate evaluations into your workflow. We've made the SDK even more flexible - `testset_id` is now optional in our helper methods, allowing you to run evaluations with custom inputs without requiring a pre-defined testset. Additional improvements include: * System configurations for experimenting with different model settings * Enhanced error handling and debugging capabilities * Full TypeScript support with comprehensive type definitions Available now on [npm](https://www.npmjs.com/package/scorecard-ai) and [PyPI](https://pypi.org/project/scorecard-ai/)! ### 🎯 Basic and Advanced Metric Modes We've introduced a two-mode system for creating and editing metrics, making Scorecard accessible to users at every technical level. Basic mode simplifies metric creation by focusing solely on the evaluation guidelines - just describe what you want to measure in plain language without worrying about prompt templates or variables. Advanced mode gives power users full control over the entire prompt template, including variable handling and custom formatting. Metric editor interface showing Basic and Advanced mode options

Metric editor interface showing Basic and Advanced mode options

### Bug fixes and improvements * **\[Documentation]** Streamlined quickstart guide for faster onboarding * **\[Playground]** More detailed error messages and prominent results table button * **\[Runs]** Delete runs directly from the runs list page * **\[Runs]** Run status now updates automatically based on scoring progress * **\[Platform]** Fixed data invalidation bugs when managing metrics and testcases ### 🔧 Metric Groups We've introduced metric groups (formerly scoring configs) to streamline how you manage and organize your evaluation metrics. Create custom groups of metrics that can be applied to runs with a single selection, making it easier to maintain consistent evaluation standards across your projects. Manage metric groups through an intuitive UI and apply them directly when triggering runs. Metric Group Editor

### 🧪 New Playground Experience The new [Scorecard Playground](/features/playground) is here! Built for product teams including prompt engineers, product managers, and subject matter experts, it provides a powerful environment to iterate on prompts and test them against multiple inputs simultaneously. Experience smart variable detection with autocomplete support for Jinja syntax, making template creation effortless. Watch your templates come to life with live preview that shows compiled output as you type. Key capabilities include: * Batch testing - run prompts against entire testsets with one click * Model configuration with customizable temperature and parameters * Persistent state - your playground configuration is saved in the URL Access the playground from any project and start experimenting with your prompts immediately. Playground New UX

### 🎉 New Homepage Launch! We're excited to announce the launch of our new [Scorecard homepage](https://scorecard.io/)! Explore our refreshed website with quick access to docs, product, and a new visual design to share how Scorecard can support you as you're building your AI product. Screenshot of the new Scorecard homepage

Screenshot of the new Scorecard homepage

### ⚙️ Fully Configurable Metrics We've launched a new capability that allows two powerful enhancements for your metrics: * Fully customize your evaluation model (GPT-4o, Claude-3, Gemini, etc.) or your own selected hosted model * Metrics now also support structured outputs, significantly increasing the reliability of your scores. Screenshot of the Custom Metrics configuration interface

Screenshot of the Custom Metrics configuration interface

### 🔌 Scorecard MCP Server We've published an MCP server for Scorecard, enabling powerful new integration possibilities! MCP is an open protocol that standardizes how applications provide context to LLMs. This allows your Scorecard evaluations to seamlessly connect with AI systems. This lets you integrate your evaluation data with various AI tools through a single protocol rather than maintaining separate integrations for each service.