Product Updates

​

🚀 Official MCP Registry Launch

• claude.ai: Full integration for seamless evaluation workflows
• Cursor: Test and evaluate your code directly in your IDE
• All MCP clients: Universal compatibility with any MCP-enabled tool

• Direct AI Integration: Run evaluations without leaving your AI assistant
• Real-time Testing: Evaluate outputs instantly as you develop
• Natural Language Control: Configure metrics and run experiments through conversation

​

🛠️ Bug Fixes and Improvements

• [Auto-refresh] Charts and headers now automatically refresh after creating new runs
• [Model Selection] Fixed scrolling issues in AI model dropdown menus
• [Empty States] Enhanced empty state designs for monitors, systems, metrics, metric groups, and endpoints with helpful quickstart links
• [Multi-page Comparisons] Select and compare runs across multiple pages for better historical analysis
• [Search Enhancement] Testset filtering now searches both names and tags
• [Sorting] Fixed “Sort by newest” to correctly order items by creation date
• [Input Validation] New test cases properly validate inputs against the defined schema
• [Monitor Deduplication] Resolved issue where overlapping monitors created duplicate test records

​

📊 A/B Comparison

• Run comparison: Compare different runs to see which system performs better
• Visual analysis: Side-by-side view of outputs and scores for easy comparison
• Performance insights: Identify which configurations work best across different metrics

​

📚 Enhanced Documentation

• Metrics - Create and manage evaluation metrics
• A/B Comparison - Compare system performance
• MCP Server Integration - Connect with AI tools via MCP protocol

​

🛠️ Bug Fixes and Improvements

• [Endpoints] Improved endpoint selector UI for better usability
• [Organizations] Improved styling and layout for the new organization creation flow
• [Projects] Removed confusing status indicator on project cards
• [Projects] Added support for editing project titles directly from project overview

​

🔌 Custom Endpoints with Multi-Turn Support

• Universal API Testing: Support for all HTTP methods with custom headers and bodies
• Multi-Turn Conversations: Enable simulated conversations with configurable AI personas
• Tabbed Run Interface: Switch between Scorecard, GitHub, and Endpoints in one modal
• Response Path Extraction: Extract specific values from JSON responses for evaluation

​

🎯 Streamlined Onboarding

• Auto-generated Project: Complete with testsets, metrics, and sample data
• Instant First Run: Results ready on first login
• Guided Actions: Clear prompts to view results or create custom runs

​

📚 Expanded Documentation

• Custom Endpoints - Test and evaluate HTTP APIs
• Tracing - Debug and monitor AI applications with error detection
• Synthetic Data Generation - Generate test data with AI

​

📋 Trace to Testcase Creation

• One-click creation: Convert any trace into a testcase directly from the trace details page
• Smart field detection: Automatically populate prompt and completion values from trace data
• Schema compatibility: Automatically detects target testset schema and maps fields correctly
• Production-grounded datasets: Build testsets from real user interactions instead of synthetic data

​

🏷️ Metadata Fields in Testset Schemas

• Flexible data storage: Store additional context without affecting evaluation logic
• Schema management: Easy toggle in the testset schema editor
• Future SDK support: Full SDK integration coming soon for programmatic access

​

🛠️ Bug Fixes and Improvements

• [Tracing] Fixed pagination issues with traces containing many spans - improved parent span detection
• [Testcases] Fixed refresh bug where editing a field could overwrite with old data
• [Projects] Fixed refresh issue after creating new projects - list now updates immediately
• [Scoring] Better error messages when OpenAI API keys are missing instead of generic “Internal service error”
• [UI] Improved copy testcase, move metric, and move testset dialogs with better scrollability and selection
• [Documentation] Added comprehensive guides for monitoring quickstart, monitoring features, and trace-to-testcase

​

🤖 Sim Agents for Multi-Turn Conversation Testing

• Persona configuration: Define user behaviors, goals, and interaction patterns with Jinja2 templating
• SDK integration: Run simulations programmatically with multi_turn_simulation() method
• Conversation control: Set stop conditions, max turns, and timeout limits for realistic testing
• Chat visualization: View full conversation history in beautiful chat bubble format

​

📊 Online Evaluations (Beta)

• Monitoring configuration: Set up project-specific rules with selected metrics and scheduling
• Cost visibility: View token costs per span and per evaluation directly in the UI
• Trace integration: Link test records to their source traces for full observability
• Real-time scoring: Automatically score production traces as they are ingested

​

🛠️ Bug Fixes and Improvements

• [Performance] Major Prisma upgrade delivering 2-5x faster load times - run lists now load in under 2 seconds (down from 3-4s)
• [Monitoring] Added token cost tracking per span in traces page

• [Monitoring] Display scoring costs for LLM-as-a-judge evaluations
• [Infrastructure] Increased collector object size limit for larger traces

​

📊 Enhanced Tracing Experience

• Flame graph visualization: See span durations at a glance to quickly identify performance bottlenecks
• Smart defaults: Tracing page now defaults to “All projects” view so you never miss traces
• Simplified authentication: Use your standard Scorecard API key for tracing - no more separate JWT tokens

​

🚀 Smarter Run Status Tracking

• Intelligent status calculation: Database now tracks expected vs. actual test records for accurate progress reporting
• Hover explanations: See detailed progress like “0 testrecords created / 35 testcases from testset 1234” to understand exactly what’s happening
• Real-time updates: Run status automatically updates as scoring progresses

​

🛠️ Bug Fixes and Improvements

• [Tracing] Fixed crash when navigating back to page 2 in traces list
• [Tracing] Time range filter now correctly updates chart data
• [Documentation] Published tracing quickstart guide with integrations for Vercel AI SDK and OpenLLMetry

​

📊 Run History

​

🔍 Tracing improvements

• Open-source collector architecture: Migrated to the upstream OpenTelemetry standard collector on Railway for improved reliability and performance.
• Unified API key authentication: You can now use your existing Scorecard API key for tracing so you no longer need to manage separate authentication tokens.
• New tracing integration: Published official integrations for Vercel AI SDK and OpenLLMetry with examples in both Python and Node.js. Our new step-by-step guide gets you from zero to traced LLM calls in minutes

​

🛠️ Bug Fixes and Improvements

• [Run kickoff] Fixed model scrolling in Run Kickoff modal UI.
• [Run details] Metrics cards and score columns are now consistently sorted alphabetically.
• [Run details] Sorting a column now displays the sort direction.

​

Separate Score Columns

Separate score columns in the UI.

​

🛠️ Bug Fixes and Improvements

• [Performance] Improved loading time of the Runs list page.
• [Runs] Added a new “Run Again” button to the Run details page, allowing you to re-run the same testset/system combo with the same metrics and model parameters.
• [Runs] Added “System Version” link to the Run details page.

​

🚀 Onboarding Improvements

• Automatic defaults: New organizations now receive default projects, testsets, metrics, and prompts automatically, significantly reducing initial setup time
• Free API key included: Every new organization gets a default free API key (Gemini Flash), eliminating the need for initial user configuration. Users without their own API key will be clearly notified that the system is defaulting to the free API

​

🏃 Kickoff Runs from Playground

​

🛠️ Bug Fixes and Improvements

• [Performance] Improved page load performance on testcases, testsets, metrics, and trigger run pages
• [Projects] Added search functionality for projects
• [Metrics] Added search functionality for metrics
• [Testsets] Added search functionality for testsets
• [Systems] All systems are now required to have a production version
• [Quickstart] Both main and systems quickstarts now reflect the latest SDK with clearer steps and updated code, enabling new users to go from install to first run in minutes
• [API Keys] Every member can view existing keys, but only admins can create or revoke them, providing teams with transparency while maintaining control
• [Docs Search] Rebuilt documentation search pipeline reducing typical response times from ~15s to just 3s, helping you find answers more quickly

​

🖥️ Systems Enhancements

• You can now directly trigger a run from the trigger-run page, making it quicker and simpler to execute tests.
• All your systems are now clearly visible and easily manageable from a single, user-friendly interface. Update configurations, manage versions, and maintain systems effortlessly.

​

🎨 New Color Palette

​

🛠️ Bug Fixes and Improvements

• [Runs] We’ve improved pagination for test records in runs, making load times and navigation quicker and setting the stage for upcoming filtering enhancements.

​

🎯 Metric API

client.metrics.create(
    project_id="314",
    name="Response Accuracy",
    eval_type="ai",
    output_type="boolean",
    prompt_template="Evaluate if the following response is factually accurate: {{outputs.response}}",
)

​

🎯 Improved API key format

​

Bug fixes and improvements

• [Metrics] Improved metrics management UI with tabbed sections for better organization
• [Metrics] Added metric selector with searchable dropdown - no more manually typing metric IDs
• [Runs] In the test record details page you can now see the text of the fully compiled metrics sent to the LLM for evaluation

​

🚀 SDK v2 Stable Release

• System configurations for experimenting with different model settings
• Enhanced error handling and debugging capabilities
• Full TypeScript support with comprehensive type definitions

​

🎯 Basic and Advanced Metric Modes

​

Bug fixes and improvements

• [Documentation] Streamlined quickstart guide for faster onboarding
• [Playground] More detailed error messages and prominent results table button
• [Runs] Delete runs directly from the runs list page
• [Runs] Run status now updates automatically based on scoring progress
• [Platform] Fixed data invalidation bugs when managing metrics and testcases

​

🔧 Metric Groups

​

🧪 New Playground Experience

• Batch testing - run prompts against entire testsets with one click
• Model configuration with customizable temperature and parameters
• Persistent state - your playground configuration is saved in the URL

​

🎉 New Homepage Launch!

​

⚙️ Fully Configurable Metrics

• Fully customize your evaluation model (GPT-4o, Claude-3, Gemini, etc.) or your own selected hosted model
• Metrics now also support structured outputs, significantly increasing the reliability of your scores.

​

🔌 Scorecard MCP Server

​

💯 Scorecard Eval Day!

​

📚 In-App Documentation Search

​

👷 SDK v2 Beta

​

🔧 Scorecard Playground Scheduled Maintenance

​

Bug fixes and improvements

• [UI] Fixed record rendering in the UI when using the new SDK.
• [Navigation] Repaired broken links in the Run grades table to Testcase and Record pages.
• [Terminology] Standardized terminology by renaming “test record” to “record” throughout the UI.
• [Documentation] Streamlined our quickstart documentation so you can get started with Scorecard in just 5 minutes.

​

New API and SDK

​

New Quickstarts and Documentation

• Install and initialize the TypeScript or Python SDK.
• Create and manage system configurations in code.
• Run your first experiment programmatically.
• Retrieve and interpret run results within your applications.
• Follow these step-by-step walkthroughs to get your first experiment up and running in under five minutes.

  ​

  Bug fixes and improvements

• [Performance] Reduced page load times and improved responsiveness when handling large run results for the run history table.
• [UI] Removed metric-specific scoring progress, scoring and execution start and end times, and improved project names wrap across all screen sizes.
• [Testsets] Resolved bug with new CSV upload flow
• [Testsets] Added back Move testset to project
• [Testsets] Archived testsets are hidden correctly, keeping your workspace clutter‑free.
• [Reliability] Improved API reliability and workflow robustness: fixed run creation schema errors, streamlined testcase creation/duplication/deletion flows, and added inline schema validation to prevent submission errors
• [Evals] Migrated from gpt4-1106-preview (Nov 2023) to gpt-4o for scoring metrics

​

Run insights

​

🧪 Testsets 2.0 for Easier creation and iteration

• You can now create testsets with a simplified modal and generate relevant example testcases based on title and description.
• Bulk editing tools make it easier to manage and update multiple testcases at once.
• You can edit large JSON blobs inline in the testcase detail view, with improved scroll and copy behavior.
• The testset detail page now shows the associated schema in context for easier debugging and review.
• Navigation has improved with linked testset titles and run/testcase summaries directly accessible from the cards.

​

🗂️ Improved schema management

• The schema editor has been redesigned, allowing teams to update schemas independently for each testset.
• Schema changes now reflect immediately in the testcase table to help users see their impact in real time.
• Users can view and copy raw schema JSON for integration with their own tools or SDKs.
• We’ve also improved messaging in the schema editor to clarify the distinction between inputs and labels.

​

Bug fixes and improvements

• [Platform] Internal tech stack upgrades to support faster product iteration
• [Testsets] Friendlier zero-state, faster load, cards with quick actions and live counts
• [Testsets] Tag propagation fix — updates now apply across all testsets and views
• [Testsets] Improved sorting behavior, including reliable default and column sorting
• [Testsets] Filter testcases by keyword, searching across the full dataset
• [Testsets] Updated page actions with visible bulk tools for managing multiple testcases
• [Testsets] Testset cards now link to runs/testcases, and support fast schema editing, duplication, or deletion
• [Testsets] Titles now link directly to the testset detail page
• [Testcases] Detail page supports editing and copying large JSON blobs
• [Testcases] Schema panel added for better context while reviewing or editing testcases
• [Projects] Enhanced cards with summaries for testsets, metrics, and runs, all linked for easier navigation
• [Projects] Improved sorting with more intuitive labels and default order
• [Projects] Faster load performance across the project overview page
• [Schemas] Improved editor messaging to clarify the difference between input fields and labels
• [Toast Messages] Now deep-link to newly created items (testsets, testcases, projects)
• [Performance] Faster page loads, filtering, sorting, and table actions, powered by new APIs and backend improvements

​

Projects

​

SDKs

​

Bug fixes and improvements

• [Playground] Filtering by testcase now works properly as well as searching. The pagination was also improved and now works as expected when before it showed inconsistent items in some cases.
• [Testsets] We fixed a bug where we exported empty testcase values as the string “null” instead of an empty string.
• [Projects] We simplified project creation by adding a create project modal to the projects page and project detail pages.
• [Testcases] We fixed a bug that broke the Generate testcases feature.
• [Settings] We added some text on the API keys page to clarify that your Scorecard API key is personal, but model API keys are scoped to the organization.

​

Tracing

​

Examples repository

​

Bug fixes and improvements

• [Scoring] When a run metric has not yet been scored, we now display N/A instead of NaN, making it clearer that it has no data.
• [Prompt Management] We made stability and performance improvements to prompt management workflows.
• [Projects] All resources now belong to projects, including those created before Scorecard Projects were introduced.
• [Exports] Custom fields in CSV exports of run results are handled more reliably.
• [Organizations] When a user switches organizations, we now redirect them to the organization’s projects page.
• [Testsets] On the testcase page, we fixed the link back to the testset.
• [Metrics] We added a new autosize textarea component that lets you keep typing the metric description without running out of space.
• [Playground] The “Prompt manager”, “Update”, and “Delete prompt” buttons are now disabled for default prompts. When selecting metrics, the “Select and score now” button is now the primary button rather than the “Select” button.
• [API] When a user does not include their Scorecard API key, we now return a friendlier 401 error: “Missing API key” rather than “malformed token”.
• [Scoring] The human scoring panel collapses the run details page allowing users to see model responses and while scoring.
• [Platform] We enhanced platform stability and increased test coverage.

​

New Project Overview Page

​

Download Testset CSVs

​

Bug fixes and improvements

​

Docs Site Revamp

• Improved navigation structure
• Better search functionality
• Enhanced API documentation
• New updates section to track changes
• Modern, cleaner design