POST
/
projects
/
{projectId}
/
systems
import Scorecard from 'scorecard-ai';

const client = new Scorecard({
  bearerToken: process.env['SCORECARD_API_KEY'], // This is the default and can be omitted
});

async function main() {
  const system = await client.systems.create('314', {
    configSchema: {
      type: 'object',
      properties: {
        temperature: { type: 'number' },
        maxTokens: { type: 'integer' },
        model: { type: 'string', enum: ['gpt-4', 'gpt-4-turbo'] },
      },
      required: ['model'],
    },
    description: 'Production chatbot powered by GPT-4',
    inputSchema: {
      type: 'object',
      properties: {
        messages: {
          type: 'array',
          items: {
            type: 'object',
            properties: {
              role: { type: 'string', enum: ['system', 'user', 'assistant'] },
              content: { type: 'string' },
            },
            required: ['role', 'content'],
          },
        },
      },
      required: ['messages'],
    },
    name: 'GPT-4 Chatbot',
    outputSchema: { type: 'object', properties: { response: { type: 'string' } }, required: ['response'] },
  });

  console.log(system.id);
}

main();
{
  "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "name": "GPT-4 Chatbot",
  "description": "Production chatbot powered by GPT-4",
  "inputSchema": {
    "type": "object",
    "properties": {
      "messages": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "role": {
              "type": "string",
              "enum": [
                "system",
                "user",
                "assistant"
              ]
            },
            "content": {
              "type": "string"
            }
          },
          "required": [
            "role",
            "content"
          ]
        }
      }
    },
    "required": [
      "messages"
    ]
  },
  "outputSchema": {
    "type": "object",
    "properties": {
      "response": {
        "type": "string"
      }
    },
    "required": [
      "response"
    ]
  },
  "configSchema": {
    "type": "object",
    "properties": {
      "temperature": {
        "type": "number"
      },
      "maxTokens": {
        "type": "integer"
      },
      "model": {
        "type": "string",
        "enum": [
          "gpt-4",
          "gpt-4-turbo"
        ]
      }
    },
    "required": [
      "model"
    ]
  }
}

Authorizations

​
Authorization
string
header
required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Path Parameters

​
projectId
string
required

The ID of the system's Project.

Example:

"314"

Body

application/json
​
name
string
required

The name of the system.

​
description
string
required

The description of the system.

​
inputSchema
object
required

The schema of the system's inputs.

​
outputSchema
object
required

The schema of the system's outputs.

​
configSchema
object
required

The schema of the system's configuration.

Response

201
application/json
System created successfully

A System Under Test (SUT) defines the interface to a component or service you want to evaluate.

It specifies three contracts through schemas:

  • inputSchema: The structure of data the system accepts.
  • outputSchema: The structure of data the system produces.
  • configSchema: The parameters that modify system behavior.

This abstraction lets you evaluate any system as a black box, focusing on its interface rather than implementation details. It's particularly useful for systems with variable outputs or complex internal state.

Systems are templates - to run evaluations, pair them with a SystemConfig that provides specific parameter values.

​
id
string
required

The ID of the system.

​
name
string
required

The name of the system.

​
description
string
required

The description of the system.

​
inputSchema
object
required

The schema of the system's inputs.

​
outputSchema
object
required

The schema of the system's outputs.

​
configSchema
object
required

The schema of the system's configuration.