> ## Documentation Index > Fetch the complete documentation index at: https://docs.scorecard.io/llms.txt > Use this file to discover all available pages before exploring further. # Create Testset > Create a new Testset for a Project. The Testset will be created in the Project specified in the path. ## OpenAPI ````yaml post /projects/{projectId}/testsets openapi: 3.1.0 info: title: Scorecard API description: REST API for Scorecard version: 1.0.0 servers: - url: https://api2.scorecard.io/api/v2 security: - ApiKeyAuth: [] paths: /projects/{projectId}/testsets: post: summary: Create Testset description: >- Create a new Testset for a Project. The Testset will be created in the Project specified in the path. operationId: createTestset parameters: - in: path name: projectId description: The ID of the Project to create the Testset in. schema: type: string example: '314' required: true requestBody: content: application/json: schema: type: object properties: name: type: string description: The name of the Testset. description: type: string description: The description of the Testset. jsonSchema: type: object description: The JSON schema for each Testcase in the Testset. additionalProperties: true fieldMapping: type: object properties: inputs: type: array items: type: string description: Fields that represent inputs to the AI system. expected: type: array items: type: string description: Fields that represent expected outputs. metadata: type: array items: type: string description: Fields that are not inputs or expected outputs. required: - inputs - expected - metadata description: >- Maps top-level keys of the Testcase schema to their roles (input/expected output). Unmapped fields are treated as metadata. required: - name - description - jsonSchema - fieldMapping examples: Create Q&A Testset: value: name: Long Context Q&A description: Testset for long context Q&A chatbot. jsonSchema: type: object properties: question: type: string idealAnswer: type: string provenance: type: string geo: type: string fieldMapping: inputs: - question expected: - idealAnswer metadata: [] summary: Create Q&A Testset description: >- Request to create a Testset for evaluating long context Q&A with fields for the question, ideal answer, and metadata like provenance and geographical context. responses: '201': description: Testset created successfully content: application/json: schema: $ref: '#/components/schemas/Testset' examples: Created Testset response: value: id: '246' name: Long Context Q&A description: Testset for long context Q&A chatbot. jsonSchema: type: object properties: question: type: string idealAnswer: type: string provenance: type: string geo: type: string fieldMapping: inputs: - question expected: - idealAnswer metadata: - provenance - geo summary: Created Testset response description: >- Response after successfully creating a Testset, showing the assigned ID and complete schema with automatically populated field mappings. '401': $ref: '#/components/responses/UnauthenticatedError' '500': $ref: '#/components/responses/ServiceError' x-codeSamples: - lang: JavaScript source: |- import Scorecard from 'scorecard-ai'; const client = new Scorecard({ apiKey: 'My API Key', }); const testset = await client.testsets.create('314', { description: 'Testset for long context Q&A chatbot.', fieldMapping: { inputs: ['question'], expected: ['idealAnswer'], metadata: [] }, jsonSchema: { type: 'object', properties: { question: { type: 'string' }, idealAnswer: { type: 'string' }, provenance: { type: 'string' }, geo: { type: 'string' }, }, }, name: 'Long Context Q&A', }); console.log(testset.id); - lang: Python source: |- from scorecard_ai import Scorecard client = Scorecard( api_key="My API Key", ) testset = client.testsets.create( project_id="314", description="Testset for long context Q&A chatbot.", field_mapping={ "inputs": ["question"], "expected": ["idealAnswer"], "metadata": [], }, json_schema={ "type": "object", "properties": { "question": { "type": "string" }, "idealAnswer": { "type": "string" }, "provenance": { "type": "string" }, "geo": { "type": "string" }, }, }, name="Long Context Q&A", ) print(testset.id) - lang: cURL source: >- curl https://api2.scorecard.io/api/v2/projects/$PROJECT_ID/testsets \ -H 'Content-Type: application/json' \ -H "Authorization: Bearer $SCORECARD_API_KEY" \ -d '{ "description": "Testset for long context Q&A chatbot.", "fieldMapping": { "expected": [ "idealAnswer" ], "inputs": [ "question" ], "metadata": [ "string" ] }, "jsonSchema": { "type": "bar", "properties": "bar" }, "name": "Long Context Q&A" }' components: schemas: Testset: type: object properties: id: type: string description: The ID of the Testset. name: type: string description: The name of the Testset. description: type: string description: The description of the Testset. jsonSchema: type: object description: The JSON schema for each Testcase in the Testset. additionalProperties: true fieldMapping: type: object properties: inputs: type: array items: type: string description: Fields that represent inputs to the AI system. expected: type: array items: type: string description: Fields that represent expected outputs. metadata: type: array items: type: string description: Fields that are not inputs or expected outputs. required: - inputs - expected - metadata description: >- Maps top-level keys of the Testcase schema to their roles (input/expected output). Unmapped fields are treated as metadata. required: - id - name - description - jsonSchema - fieldMapping description: >- A collection of Testcases that share the same schema. Each Testset defines the structure of its Testcases through a JSON schema. The `fieldMapping` object maps top-level keys of the Testcase schema to their roles (input/expected output). Fields not mentioned in the `fieldMapping` during creation or update are treated as metadata. ## JSON Schema validation constraints supported: - **Required fields** - Fields listed in the schema's `required` array must be present in Testcases. - **Type validation** - Values must match the specified type (string, number, boolean, null, integer, object, array). - **Enum validation** - Values must be one of the options specified in the `enum` array. - **Object property validation** - Properties of objects must conform to their defined schemas. - **Array item validation** - Items in arrays must conform to the `items` schema. - **Logical composition** - Values must conform to at least one schema in the `anyOf` array. Testcases that fail validation will still be stored, but will include `validationErrors` detailing the issues. Extra fields in the Testcase data that are not in the schema will be stored but are ignored during validation. ApiError: type: object properties: code: type: string message: type: string details: type: object additionalProperties: true x-stainless-any: true required: - code - message - details description: An API error. responses: UnauthenticatedError: description: Error indicating that the request is not authenticated. content: application/json: schema: $ref: '#/components/schemas/ApiError' examples: Authentication failure: value: code: UNAUTHORIZED message: Invalid or missing authentication token details: {} summary: Authentication failure description: >- Error returned when authentication credentials are invalid or missing. ServiceError: description: >- An internal service error indicating an issue with the Scorecard service. content: application/json: schema: $ref: '#/components/schemas/ApiError' examples: Internal error: value: code: INTERNAL_ERROR message: An unexpected error occurred while processing your request. details: {} summary: Internal error description: Generic error when an unexpected internal issue occurs. securitySchemes: ApiKeyAuth: type: http scheme: bearer bearerFormat: starts with ak_ ````