System configurations

Create system configuration

Create a new configuration for a system.

Each configuration contains specific parameter values that match the system’s configSchema - things like model parameters, thresholds, or processing options. Once created, configurations cannot be modified, ensuring stable reference points for evaluations.

When creating a configuration:

The ‘config’ object is validated against the parent system’s configSchema
Configurations with validation errors are still stored, with errors included in the response
Validation errors indicate fields that don’t match the schema but don’t prevent creation
Having validation errors may affect how some evaluation metrics are calculated

POST

systems

{systemId}

configs

import Scorecard from 'scorecard-ai';

const client = new Scorecard({
  bearerToken: process.env['SCORECARD_API_KEY'], // This is the default and can be omitted
});

async function main() {
  const systemConfig = await client.systemConfigs.create('12345678-0a8b-4f66-b6f3-2ddcfa097257', {
    config: { temperature: 0.1, maxTokens: 1024, model: 'gpt-4-turbo' },
    name: 'Production (Low Temperature)',
  });

  console.log(systemConfig.id);
}

main();

{
  "id": "5fa85f64-5717-4562-b3fc-2c963f66afa7",
  "systemId": "12345678-0a8b-4f66-b6f3-2ddcfa097257",
  "name": "Production (Low Temperature)",
  "config": {
    "temperature": 0.1,
    "maxTokens": 1024,
    "model": "gpt-4-turbo"
  }
}

Authorizations

Authorization

string

header

required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Path Parameters

systemId

string

required

The ID of the system to create the configuration for.

Example:

"12345678-0a8b-4f66-b6f3-2ddcfa097257"

Body

application/json

name

string

required

The name of the system configuration.

config

object

required

The configuration of the system.

validationErrors

object[]

Validation errors found in the configuration. If present, the configuration doesn't fully conform to its system's configSchema.

Response

201

application/json

System configuration created successfully

A SystemConfig defines the specific settings for a System Under Test.

Configurations contain parameter values that determine system behavior during evaluation. They are immutable snapshots - once created, they never change.

When running evaluations, you reference a specific configId to establish which configuration to test.

Configurations will be validated against the system's configSchema, with non-conforming values generating warnings.

string

required

The ID of the system configuration.

systemId

string

required

The ID of the system the configuration belongs to.

name

string

required

The name of the system configuration.

config

object

required

The configuration of the system.

validationErrors

object[]

Validation errors found in the configuration. If present, the configuration doesn't fully conform to its system's configSchema.

Get system configurationsRetrieve a paginated list of configurations for a specific system. System configurations provide concrete parameter values for a System Under Test, defining exactly how the system should be configured during an evaluation run.

import Scorecard from 'scorecard-ai';

const client = new Scorecard({
  bearerToken: process.env['SCORECARD_API_KEY'], // This is the default and can be omitted
});

async function main() {
  const systemConfig = await client.systemConfigs.create('12345678-0a8b-4f66-b6f3-2ddcfa097257', {
    config: { temperature: 0.1, maxTokens: 1024, model: 'gpt-4-turbo' },
    name: 'Production (Low Temperature)',
  });

  console.log(systemConfig.id);
}

main();

{
  "id": "5fa85f64-5717-4562-b3fc-2c963f66afa7",
  "systemId": "12345678-0a8b-4f66-b6f3-2ddcfa097257",
  "name": "Production (Low Temperature)",
  "config": {
    "temperature": 0.1,
    "maxTokens": 1024,
    "model": "gpt-4-turbo"
  }
}

Overview

Projects

Testsets

Testcases

Runs

Records

Systems

System configurations

Create system configuration

Authorizations

Path Parameters

Body

Response