Scorecard Recipes

Welcome to the Scorecard Cookbooks. Here you’ll find code and guides for accomplishing common tasks with the Scorecard API and SDKs.

Cookbooks

Scorecard AI Docs

Here you find guides and resources to help evaluate your LLM applications with Scorecard.

Overview

Scorecard is a platform to support product teams and engineers build, evaluate and deploy Large Language Model (LLM) applications to production.

What Is Scorecard?

Let’s walk through the journey of an LLM Developer, from start to finish. Buckle up! 🤠

The LLM Developer's Journey

Technical System Overview

Projects

Prompt Management

Tracing

Preserving the privacy of our clients and ensuring secure processes is of top priority at Scorecard.

Privacy By Design

Automated Scoring

Best In Class Metrics

Imagine these scenarios: The performance of your LLM application drops suddenly, you lose track of which LLM calls you have made, or you do not notice potential security leaks. 

Logging

A Scorecard Testset is a collection of Testcases used to evaluate the performance of an LLM application across a variety of inputs and scenarios.

Testset Management

During the development of LLM applications, it is common practice to iteratively adjust the system to find the optimal setup that produces the best results. 

A/B Comparison

In order to develop controllable and safe LLM applications, you can integrate Scorecard with the open-source toolkits [NeMo Guardrails](https://github.com/NVIDIA/NeMo-Guardrails) and [Guardrails AI](https://github.com/guardrails-ai/guardrails). 

AI Guardrails

Run Inspection

Scorecard leverages the open source AI Proxy LiteLLM to allow you to manage multiple LLM APIs and easily swap out different AI models.

AI Proxy

Retrieval Augmented Generation (RAG) is the use of retrieval methods (e.g. via search and vector stores) to provide generative models with additional context, or *grounding*.

RAG: Retrieval Augmented Generation

CI/CD With GitHub Actions

API Reference

Retrieve all Testsets with cursor-based pagination

List Testsets

Delete

Update Testset

Create Testset

Read Testset Schema

Read Testcases

Create Testcase

Read Testcase

Delete Testcase

Update Testcase

Batch Copy Testcases

Batch Delete Testcases

Read Testrecord

Create Testrecord

Create Run

Read Run

Update Run Status

Create Score

Update Score

Read Run Metrics

Get Traces

Get Trace

Get Trace Span

Get Prompt By Id

Two types of prompts can be created - a root prompt or a child prompt (aka Prompt Version in the app).

        A root prompt can be created by providing the `name` param, and it will always be tagged as production.

        A child prompt can be created by providing the `parent_id` param. Note that the `name` param in this case will be ignored as all descendants from a root prompt would share the root's name. `is_prod` can also be provided to configure whether a child should be tagged as production.

Create Prompt

List all prompts with cursor-based pagination

List Prompts

Delete a root prompt and all of its children.

Delete Root Prompt

Update a prompt.

        `is_prod` tags the provided prompt as the production prompt within the prompt graph.

Update Prompt

Retrieve a prompt by name, defaulting to the production prompt, unless a tag to select the prompt by is specified

Introduction

Features

How To Use Scorecard

Cookbooks

Scorecard Recipes

E2E Text Extraction and Vector Search Python Demo

Anthropic Retrieval Augmented Generation (RAG) Example

Create a Testset with Custom Schema

Multi-Message Prompt using Scorecard SDK

Prompt with Custom Variables using Scorecard SDK

Heuristic Scoring Example - Exact String Match

Tracing with Scorecard - Python Demo

Tracing with Scorecard - Node Demo

Prompt Management Example

Using Second-Party Metrics from MLflow

Using Second-Party Metrics from RAGAS

Introduction

Features

How To Use Scorecard

​Scorecard Recipes

E2E Text Extraction and Vector Search Python Demo

Anthropic Retrieval Augmented Generation (RAG) Example

Create a Testset with Custom Schema

Multi-Message Prompt using Scorecard SDK

Prompt with Custom Variables using Scorecard SDK

Heuristic Scoring Example - Exact String Match

Tracing with Scorecard - Python Demo

Tracing with Scorecard - Node Demo

Prompt Management Example

Using Second-Party Metrics from MLflow

Using Second-Party Metrics from RAGAS

Scorecard Recipes