Questions? Contact Us

Here you find guides and resources to help evaluate your LLM applications with Scorecard.

Overview

Scorecard AI Docs

Evaluate a simple LLM system with Scorecard in minutes.

Quickstart

Scorecard is a platform to support product teams and engineers build, evaluate and deploy Large Language Model (LLM) applications to production.

What Is Scorecard?

Let’s walk through the journey of an LLM Developer, from start to finish. Buckle up! 🤠

The LLM Developer's Journey

Technical System Overview

Projects

Prompt Management

Tracing

Preserving the privacy of our clients and ensuring secure processes is of top priority at Scorecard.

Privacy By Design

Automated Scoring

Best In Class Metrics

Imagine these scenarios: The performance of your LLM application drops suddenly, you lose track of which LLM calls you have made, or you do not notice potential security leaks. 

Logging

A Scorecard Testset is a collection of Testcases used to evaluate the performance of an LLM application across a variety of inputs and scenarios.

Testset Management

During the development of LLM applications, it is common practice to iteratively adjust the system to find the optimal setup that produces the best results. 

A/B Comparison

In order to develop controllable and safe LLM applications, you can integrate Scorecard with the open-source toolkits [NeMo Guardrails](https://github.com/NVIDIA/NeMo-Guardrails) and [Guardrails AI](https://github.com/guardrails-ai/guardrails). 

AI Guardrails

Run Inspection

Scorecard leverages the open source AI Proxy LiteLLM to allow you to manage multiple LLM APIs and easily swap out different AI models.

AI Proxy

Welcome to the Scorecard Cookbooks. Here you’ll find code and guides for accomplishing common tasks with the Scorecard API and SDKs.

Cookbooks

Retrieval Augmented Generation (RAG) is the use of retrieval methods (e.g. via search and vector stores) to provide generative models with additional context, or *grounding*.

RAG: Retrieval Augmented Generation

CI/CD With GitHub Actions

Browse the API documentation and integrate with Scorecard's API endpoints.

API Reference

Migrate your Scorecard code from to the new SDK.

SDK Migration Guide

Create Project

Retrieve a paginated list of all Projects. Projects are ordered by creation date, with oldest Projects first.

List Projects

Create a new Testset for a Project. The Testset will be created in the Project specified in the path.

Create Testset

Retrieve a paginated list of Testsets belonging to a Project.

List Testsets in Project

Get Testset by ID

Update a Testset. Only the fields provided in the request body will be updated.
If a field is provided, the new content will replace the existing content.
If a field is not provided, the existing content will remain unchanged.

When updating the schema:
- If field mappings are not provided and existing mappings reference fields that no longer exist, those mappings will be automatically removed
- To preserve all existing mappings, ensure all referenced fields remain in the updated schema
- For complete control, provide both schema and fieldMapping when updating the schema

Update Testset

Delete Testset

Create multiple Testcases in the specified Testset.

Create multiple Testcases

Retrieve a paginated list of Testcases belonging to a Testset.

List Testcases in Testset

Get Testcase

Replace the data of an existing Testcase while keeping its ID.

Update Testcase

Delete multiple Testcases

Create Run

Update Run status

Create Record

Create or update a Score for a given Record and MetricConfig. If a Score with the specified Record ID and MetricConfig ID already exists, it will be updated. Otherwise, a new Score will be created. The score provided should conform to the schema defined by the MetricConfig; otherwise, validation errors will be reported.

Upsert Score

Create a new system definition that specifies the interface contracts for a component you want to evaluate.

A system acts as a template that defines three key contracts through JSON Schemas:
1. Input Schema: What data your system accepts (e.g., user queries, context documents)
2. Output Schema: What data your system produces (e.g., responses, confidence scores)
3. Config Schema: What parameters can be adjusted (e.g., model selection, temperature)

This separation lets you evaluate any system as a black box, focusing on its interface rather than implementation details.

Create system

Retrieve a paginated list of all systems. Systems are ordered by creation date.

List systems

Get system

Update an existing system definition. Only the fields provided in the request body will be updated.
If a field is provided, the new content will replace the existing content.
If a field is not provided, the existing content will remain unchanged.

When updating schemas:
- The system will accept your changes regardless of compatibility with existing configurations
- Schema updates won't invalidate existing evaluations or configurations
- For significant redesigns, creating a new system definition provides a cleaner separation

Update system

Delete a system definition by ID. This will not delete associated system configurations.

Delete system

Create a new configuration for a system.

Each configuration contains specific parameter values that match the system's configSchema - things like model parameters, thresholds, or processing options.
Once created, configurations cannot be modified, ensuring stable reference points for evaluations.

When creating a configuration:
- The 'config' object is validated against the parent system's configSchema
- Configurations with validation errors are still stored, with errors included in the response
- Validation errors indicate fields that don't match the schema but don't prevent creation
- Having validation errors may affect how some evaluation metrics are calculated

Create system configuration

Retrieve a paginated list of configurations for a specific system.

System configurations provide concrete parameter values for a System Under Test, defining exactly how the system should be configured during an evaluation run.

Introduction

Features

How To Use Scorecard

Overview

Quickstart

API Reference

Cookbooks

Guides

Scorecard Overview

Features

Privacy & Security

Service Status

Questions? Contact Us

Introduction

Features

How To Use Scorecard

Quickstart

API Reference

Cookbooks

Guides

Scorecard Overview

Features

Privacy & Security

Service Status

​Questions? Contact Us

Questions? Contact Us