Scorecard Recipes

Welcome to the Scorecard Cookbooks. Here you’ll find code and guides for accomplishing common tasks with the Scorecard API and SDKs.

Cookbooks

Scorecard AI Docs

Here you find guides and resources to help evaluate your LLM applications with Scorecard.

Overview

Scorecard is a platform to support product teams and engineers build, evaluate and deploy Large Language Model (LLM) applications to production.

What Is Scorecard?

Let’s walk through the journey of an LLM Developer, from start to finish. Buckle up! 🤠

The LLM Developer's Journey

Technical System Overview

Projects

Prompt Management

Tracing

Preserving the privacy of our clients and ensuring secure processes is of top priority at Scorecard.

Privacy By Design

Automated Scoring

Best In Class Metrics

Imagine these scenarios: The performance of your LLM application drops suddenly, you lose track of which LLM calls you have made, or you do not notice potential security leaks. 

Logging

A Scorecard Testset is a collection of Testcases used to evaluate the performance of an LLM application across a variety of inputs and scenarios.

Testset Management

During the development of LLM applications, it is common practice to iteratively adjust the system to find the optimal setup that produces the best results. 

A/B Comparison

In order to develop controllable and safe LLM applications, you can integrate Scorecard with the open-source toolkits [NeMo Guardrails](https://github.com/NVIDIA/NeMo-Guardrails) and [Guardrails AI](https://github.com/guardrails-ai/guardrails). 

AI Guardrails

Run Inspection

Scorecard leverages the open source AI Proxy LiteLLM to allow you to manage multiple LLM APIs and easily swap out different AI models.

AI Proxy

Retrieval Augmented Generation (RAG) is the use of retrieval methods (e.g. via search and vector stores) to provide generative models with additional context, or *grounding*.

RAG: Retrieval Augmented Generation

CI/CD With GitHub Actions

API Reference

Retrieve a paginated list of all Projects. Projects are ordered by creation date, with oldest Projects first.

Get projects

Create a new Testset for a Project. The Testset will be created in the Project specified in the path.

Create Testset

Retrieve a paginated list of Testsets belonging to a Project.

Get Testsets in Project

Get Testset by ID

Update a Testset. Only the fields provided in the request body will be updated.
If a field is provided, the new content will replace the existing content.
If a field is not provided, the existing content will remain unchanged.

When updating the schema:
- If field mappings are not provided and existing mappings reference fields that no longer exist, those mappings will be automatically removed
- To preserve all existing mappings, ensure all referenced fields remain in the updated schema
- For complete control, provide both schema and fieldMapping when updating the schema

Update Testset

Delete Testset

Create multiple Testcases in the specified Testset.

Create multiple Testcases

Retrieve a paginated list of Testcases belonging to a Testset.

Get Testcases in Testset

Get Testcase

Replace the data of an existing Testcase while keeping its ID.

Update Testcase

Delete multiple Testcases

Create Run

Update Run status

Create Record

Create a new system definition that specifies the interface contracts for a component you want to evaluate.

A system acts as a template that defines three key contracts through JSON Schemas:
1. Input Schema: What data your system accepts (e.g., user queries, context documents)
2. Output Schema: What data your system produces (e.g., responses, confidence scores)
3. Config Schema: What parameters can be adjusted (e.g., model selection, temperature)

This separation lets you evaluate any system as a black box, focusing on its interface rather than implementation details.

Create system

Retrieve a paginated list of all systems. Systems are ordered by creation date.

Get systems

Get system by ID

Update an existing system definition. Only the fields provided in the request body will be updated.
If a field is provided, the new content will replace the existing content.
If a field is not provided, the existing content will remain unchanged.

When updating schemas:
- The system will accept your changes regardless of compatibility with existing configurations
- Schema updates won't invalidate existing evaluations or configurations
- For significant redesigns, creating a new system definition provides a cleaner separation

Update system

Delete a system definition by ID. This will not delete associated system configurations.

Delete system

Create a new configuration for a system.

Each configuration contains specific parameter values that match the system's configSchema - things like model parameters, thresholds, or processing options.
Once created, configurations cannot be modified, ensuring stable reference points for evaluations.

When creating a configuration:
- The 'config' object is validated against the parent system's configSchema
- Configurations with validation errors are still stored, with errors included in the response
- Validation errors indicate fields that don't match the schema but don't prevent creation
- Having validation errors may affect how some evaluation metrics are calculated

Create system configuration

Retrieve a paginated list of configurations for a specific system.

System configurations provide concrete parameter values for a System Under Test, defining exactly how the system should be configured during an evaluation run.

Introduction

Features

How To Use Scorecard

Cookbooks

Scorecard Recipes

E2E Text Extraction and Vector Search Python Demo

Anthropic Retrieval Augmented Generation (RAG) Example

Create a Testset with Custom Schema

Multi-Message Prompt using Scorecard SDK

Prompt with Custom Variables using Scorecard SDK

Heuristic Scoring Example - Exact String Match

Tracing with Scorecard - Python Demo

Tracing with Scorecard - Node Demo

Prompt Management Example

Using Second-Party Metrics from MLflow

Using Second-Party Metrics from RAGAS

Introduction

Features

How To Use Scorecard

​Scorecard Recipes

E2E Text Extraction and Vector Search Python Demo

Anthropic Retrieval Augmented Generation (RAG) Example

Create a Testset with Custom Schema

Multi-Message Prompt using Scorecard SDK

Prompt with Custom Variables using Scorecard SDK

Heuristic Scoring Example - Exact String Match

Tracing with Scorecard - Python Demo

Tracing with Scorecard - Node Demo

Prompt Management Example

Using Second-Party Metrics from MLflow

Using Second-Party Metrics from RAGAS

Scorecard Recipes