POST
/
projects
/
{projectId}
/
testsets
import Scorecard from 'scorecard-ai';

const client = new Scorecard({
  apiKey: process.env['SCORECARD_API_KEY'], // This is the default and can be omitted
});

const testset = await client.testsets.create('314', {
  description: 'Testset for long context Q&A chatbot.',
  fieldMapping: { inputs: ['question'], expected: ['idealAnswer'], metadata: [] },
  jsonSchema: {
    type: 'object',
    properties: {
      question: { type: 'string' },
      idealAnswer: { type: 'string' },
      provenance: { type: 'string' },
      geo: { type: 'string' },
    },
  },
  name: 'Long Context Q&A',
});

console.log(testset.id);
{
  "id": "246",
  "name": "Long Context Q&A",
  "description": "Testset for long context Q&A chatbot.",
  "jsonSchema": {
    "type": "object",
    "properties": {
      "question": {
        "type": "string"
      },
      "idealAnswer": {
        "type": "string"
      },
      "provenance": {
        "type": "string"
      },
      "geo": {
        "type": "string"
      }
    }
  },
  "fieldMapping": {
    "inputs": [
      "question"
    ],
    "expected": [
      "idealAnswer"
    ],
    "metadata": [
      "provenance",
      "geo"
    ]
  }
}

Authorizations

​
Authorization
string
header
required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Path Parameters

​
projectId
string
required

The ID of the Project to create the Testset in.

Example:

"314"

Body

application/json

Response

201
application/json

Testset created successfully

A collection of Testcases that share the same schema. Each Testset defines the structure of its Testcases through a JSON schema. The fieldMapping object maps top-level keys of the Testcase schema to their roles (input/expected output). Fields not mentioned in the fieldMapping during creation or update are treated as metadata.

JSON Schema validation constraints supported:

  • Required fields - Fields listed in the schema's required array must be present in Testcases.
  • Type validation - Values must match the specified type (string, number, boolean, null, integer, object, array).
  • Enum validation - Values must be one of the options specified in the enum array.
  • Object property validation - Properties of objects must conform to their defined schemas.
  • Array item validation - Items in arrays must conform to the items schema.
  • Logical composition - Values must conform to at least one schema in the anyOf array.

Testcases that fail validation will still be stored, but will include validationErrors detailing the issues. Extra fields in the Testcase data that are not in the schema will be stored but are ignored during validation.