Testsets

A Testset is a collection of Testcases used to evaluate the performance of an AI agent across various inputs and scenarios. Think of it as a curated dataset specifically designed for testing AI agents. Testsets belong to a central theme like “Core Functionality”, “Edge Cases”, or “Customer Support Queries”. A Testcase is an individual test data point containing inputs, expected outputs, and metadata used for evaluation.

Screenshot of the testset details page in the UI.

Testset details page showing eight Testcases for a "Message tone rewriter" AI agent.

Create a Testset

Go to the Testsets page in your project and click the “New Testset” button to create a new, empty Testset.

Checking “Add example Testcases” will automatically generate 3 sample Testcases using AI based on your Testset’s description. This provides a starting point for your Testset.

Screenshot of the create testset modal in the UI.

Create testset modal with AI generation option

Testset schema

Each Testset has a schema, which defines which fields a Testcase has, the type of each field, and the role each field plays in evaluation.

Input fields are sent to your AI agent.
Expected fields are expected or ideal outputs, which metrics compare your agent’s output to.
Metadata fields are additional context for analysis, not used by evaluation or your agent.

You can update the schema of a Testset by clicking the “Edit Schema” button in the Testset actions menu. This allows you to add or remove fields, modify field types, and update field descriptions. Existing Testcases are not modified, but are validated against the new schema.

Testset schema editor.

With the SDK

You can also create and update Testsets with the Scorecard SDK. You define a Testset’s schema using the JSON Schema format. For example, here’s a schema for a customer support system:

{
  "type": "object",
  "title": "Customer Support Schema",
  "properties": {
    "userQuery": {
      "type": "string",
      "description": "The customer's question or request"
    },
    "context": {
      "type": "string", 
      "description": "Additional context about the customer"
    },
    "ideal": {
      "type": "string",
      "description": "The ideal response from support"
    },
    "expectedSentiment": {
      "type": "string",
      "description": "The expected predicted sentiment of the user query."
    },
    "difficulty": {
      "type": "number",
      "description": "How difficult the customer support request is to solve (1-10)"
    }
  },
  "required": ["userQuery", "ideal"]
}

Supported Data Types

string: Text content
number: Numeric values (integers or floats)
boolean: either true or false
object: Nested JSON objects
array: Lists of JSON values

You also need to define the field mapping when creating a Testset with the SDK. A field mapping categorizes schema fields by their role in evaluation. For example, here’s a field mapping for the customer support schema above:

{
  "inputs": ["userQuery", "context"],
  "expected": ["ideal", "expectedSentiment"],
  "metadata": ["difficulty"]
}

Understanding Input Fields

Input fields contain the actual data that gets sent to your AI system, workflow, or agent during testing. These should match exactly what your system expects to receive in production.

Quick tip: Your input fields should match what goes INTO your AI system. Think about:

What do users type into your UI?
What data does your API receive?
What would a user or another system send to trigger your workflow?

If you’re unsure, ask an engineer on your team: “What JSON/data do we send to our AI system?”

Common patterns for structuring inputs

Chatbot/LLM
AI Agent/Workflow
API Endpoint
Document Processing

For conversational AI systems:

{
  "user_message": "How do I reset my password?",
  "conversation_history": [...],
  "system_prompt": "You are a helpful assistant"
}

For multi-step AI workflows or agents:

{
  "task_description": "Analyze this document and extract key points",
  "input_data": "Document content or reference",
  "workflow_parameters": {
    "mode": "detailed",
    "output_format": "bullet_points"
  }
}

For AI-powered API endpoints:

{
  "query": "Find similar products",
  "filters": {"category": "electronics"},
  "max_results": 10
}

For document analysis systems:

{
  "document_content": "Full text or path to document",
  "extraction_query": "What are the payment terms?",
  "document_type": "contract"
}

Your input fields should mirror your production system’s interface. If your system expects a single prompt string, use a single string field. If it expects structured JSON with multiple parameters, reflect that structure in your schema.

Simple Testcase Schema Examples

The following examples show complete testcase schemas with both input and expected fields. The field mapping below each schema identifies which fields are inputs vs expected outputs.

Basic Chatbot
Document Q&A System
AI Agent/Workflow

{
  "type": "object",
  "properties": {
    "user_message": {
      "type": "string",
      "description": "What the user types in the chat box"
    },
    "expected_response": {
      "type": "string",
      "description": "What the bot should say back"
    }
  }
}

Field mapping:

{
  "inputs": ["user_message"],
  "expected": ["expected_response"]
}

{
  "type": "object",
  "properties": {
    "question": {
      "type": "string",
      "description": "The question about the document"
    },
    "document": {
      "type": "string",
      "description": "The document text to search through"
    },
    "expected_answer": {
      "type": "string",
      "description": "The correct answer from the document"
    }
  }
}

Field mapping:

{
  "inputs": ["question", "document"],
  "expected": ["expected_answer"]
}

{
  "type": "object",
  "properties": {
    "task_description": {
      "type": "string",
      "description": "What you want the AI agent to do"
    },
    "input_data": {
      "type": "string",
      "description": "Any data the agent needs to complete the task"
    },
    "expected_output": {
      "type": "string",
      "description": "What the agent should produce"
    }
  }
}

Field mapping:

{
  "inputs": ["task_description", "input_data"],
  "expected": ["expected_output"]
}

The key is matching your test inputs to your actual system. If your system takes a single text field, use a single text field. If it takes multiple parameters, include those as separate fields.

Create and edit Testcases

From the UI

Click the “New Testcase” button to create a new Testcase matching your Testset’s schema.

Testcase creation modal.

You can edit a particular Testcase’s fields from the Testcase table, or from the Testcase details page.

Testcase details page.

Importing from a file

The Scorecard UI supports importing Testcases in CSV, TSV, JSON, and JSONL formats. Scorecard automatically maps your file’s columns to the testset’s schema fields and validates data.

Upsert behavior: If your file includes testcases with IDs that already exist in the testset, those testcases will be updated with the new values rather than creating duplicates. This makes it easy to bulk-update existing testcases by re-uploading a modified file.

userQuery,context,ideal,category
"How do I cancel my order?","Order placed 1 hour ago","You can cancel orders within 2 hours...","cancellation"
"Where is my package?","Order shipped yesterday","Track your package using the link...","tracking"

Using the API

With our SDKs, you can create, update, and delete Testcases.

Export Testset

You can export a Testset’s Testcases to a CSV file by clicking the Export as CSV button in the Testset’s details page.

Other Testset features

Testset tags You can add custom tags to your Testsets to categorize them. For example, regression or edge-cases. Duplicate Testset You can create a copy of a Testset by clicking the “Duplicate” button in the Testset actions menu. This maintains original field mappings, so it’s useful for creating variants of your Testsets without having to recreate the schema.

Best practices

Testset strategy by use case

Hillclimbing Testsets

Purpose: Iterative improvement and developmentSize: 5-20 TestcasesContent: Your favorite prompts and edge cases that matter mostUsage: Quick feedback during development cycles

Regression Testsets

Purpose: Ensure new changes don’t break existing functionalitySize: 50-100 TestcasesContent: Representative examples of core use casesUsage: Run regularly (nightly builds, CI/CD pipelines)

Launch Evaluation Testsets

Purpose: Comprehensive testing before major releasesSize: 100+ TestcasesContent: Broad coverage of all use cases and edge casesUsage: Pre-launch validation and confidence building

Must-Pass Testsets

Purpose: Critical functionality that must never failSize: Variable (focus on precision over coverage)Content: High-precision Testcases for essential featuresUsage: Early checks in deployment pipelines

Remember that testcase data may contain sensitive information. Follow your organization’s data handling policies and avoid including PII, secrets, or confidential data in Testsets.

Introduction

Quickstarts

Core features

Advanced features

Governance, Risk, and Compliance

Create a Testset

Testset schema

With the SDK

Understanding Input Fields

Common patterns for structuring inputs

Simple Testcase Schema Examples

Create and edit Testcases

From the UI

Importing from a file

Using the API

Export Testset

Other Testset features

Best practices

Introduction

Quickstarts

Core features

Advanced features

Governance, Risk, and Compliance

​Create a Testset

​Testset schema

​With the SDK

​Understanding Input Fields

​Common patterns for structuring inputs

​Simple Testcase Schema Examples

​Create and edit Testcases

​From the UI

​Importing from a file

​Using the API

​Export Testset

​Other Testset features

​Best practices

Create a Testset

Testset schema

With the SDK

Understanding Input Fields

Common patterns for structuring inputs

Simple Testcase Schema Examples

Create and edit Testcases

From the UI

Importing from a file

Using the API

Export Testset

Other Testset features

Best practices