Skip to main content
POST
/
projects
/
{projectId}
/
testsets
JavaScript
import Scorecard from 'scorecard-ai';

const client = new Scorecard({
  apiKey: 'My API Key',
});

const testset = await client.testsets.create('314', {
  description: 'Testset for long context Q&A chatbot.',
  fieldMapping: { inputs: ['question'], expected: ['idealAnswer'], metadata: [] },
  jsonSchema: {
    type: 'object',
    properties: {
      question: { type: 'string' },
      idealAnswer: { type: 'string' },
      provenance: { type: 'string' },
      geo: { type: 'string' },
    },
  },
  name: 'Long Context Q&A',
});

console.log(testset.id);
{
  "id": "246",
  "name": "Long Context Q&A",
  "description": "Testset for long context Q&A chatbot.",
  "jsonSchema": {
    "type": "object",
    "properties": {
      "question": {
        "type": "string"
      },
      "idealAnswer": {
        "type": "string"
      },
      "provenance": {
        "type": "string"
      },
      "geo": {
        "type": "string"
      }
    }
  },
  "fieldMapping": {
    "inputs": [
      "question"
    ],
    "expected": [
      "idealAnswer"
    ],
    "metadata": [
      "provenance",
      "geo"
    ]
  }
}

Authorizations

​
Authorization
string
header
required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Path Parameters

​
projectId
string
required

The ID of the Project to create the Testset in.

Example:

"314"

Body

application/json
​
name
string
required

The name of the Testset.

​
description
string
required

The description of the Testset.

​
jsonSchema
object
required

The JSON schema for each Testcase in the Testset.

​
fieldMapping
object
required

Maps top-level keys of the Testcase schema to their roles (input/expected output). Unmapped fields are treated as metadata.

Response

Testset created successfully

A collection of Testcases that share the same schema. Each Testset defines the structure of its Testcases through a JSON schema. The fieldMapping object maps top-level keys of the Testcase schema to their roles (input/expected output). Fields not mentioned in the fieldMapping during creation or update are treated as metadata.

JSON Schema validation constraints supported:

  • Required fields - Fields listed in the schema's required array must be present in Testcases.
  • Type validation - Values must match the specified type (string, number, boolean, null, integer, object, array).
  • Enum validation - Values must be one of the options specified in the enum array.
  • Object property validation - Properties of objects must conform to their defined schemas.
  • Array item validation - Items in arrays must conform to the items schema.
  • Logical composition - Values must conform to at least one schema in the anyOf array.

Testcases that fail validation will still be stored, but will include validationErrors detailing the issues. Extra fields in the Testcase data that are not in the schema will be stored but are ignored during validation.

​
id
string
required

The ID of the Testset.

​
name
string
required

The name of the Testset.

​
description
string
required

The description of the Testset.

​
jsonSchema
object
required

The JSON schema for each Testcase in the Testset.

​
fieldMapping
object
required

Maps top-level keys of the Testcase schema to their roles (input/expected output). Unmapped fields are treated as metadata.