Skip to main content
Prompts in Scorecard are configurable templates that generate inputs for language models. They support dynamic variable substitution, multi-message conversations, and integrated testing workflows.
Screenshot of viewing prompt details in the UI.Screenshot of viewing prompt details in the UI.

Prompt management page for a "docs search" LLM system.

​
Create a Prompt

Go to the Prompts page in Scorecard and click the “New Prompt +” button. Provide a name and description for your prompt, then choose your AI model and configure parameters:
  • Model: Select from available models (e.g. gpt-4.1-mini, opus-4).
  • Temperature: Control randomness (0.0 = deterministic, 1.0 = creative).
  • Max Tokens: Set maximum response length.
  • Top-P: Fine-tune token selection probability.
Then, write your prompt content using the built-in editor with Jinja template support.
Screenshot of creating a prompt in the UI.Screenshot of creating a prompt in the UI.

Prompt creation dialog with model settings.

​
Prompt Templates & Jinja Syntax

​
Template Structure

Scorecard prompts support multi-message conversations with different roles:
  • System: Instructions for the AI model’s behavior
  • User: The main prompt or question
  • Assistant: Previous responses (for conversation context)

​
Variable substitution

Scorecard supports Jinja template syntax for variable substitution. This lets you insert dynamic content from your testcases into your prompt by wrapping the Testcase’s input’s field name in {{ and }}. For example, if your Testcase has a customer_message field, your prompt might be: 
You are a helpful customer service assistant.

Customer inquiry: {{customer_message}}
Product: {{product_name}}
Customer tier: {{customer_tier}}

Please provide a helpful response addressing their concern.
Scorecard also has a special {{allInputs}} variable that contains all the testcase input fields as a formatted string.
Scorecard supports full Jinja functionality including:
  • Conditional logic: {% if customer_tier == "premium" %} User is a premium customer. {% endif %}
  • Loops: {% for item in product_list %} {{item.name}} {% endfor %}
  • Filters: {{customer_message | upper}}
  • Comments: {# This is a comment #}

​
Version Management

The prompt details page shows a version history of the prompt. You can create a new version of the prompt by modifying the existing prompt or parameters and clicking “Save Changes”. Selecting “Publish this version to production” will mark that version as the “production” version. Otherwise, it’ll simply be an experimental version. You can mark any version as production by clicking the “Publish” button, but exactly one version is marked as production at any time.
Screenshot of the prompt dropdown in the UI.Screenshot of the prompt dropdown in the UI.

Prompt version dropdown with publish button.

Before publishing to production, test a new version in the playground to be confident that it behaves as expected and won’t regress on any testcases.

​
SDK Integration

You can programmatically manage prompts and their versions using the Scorecard SDK. Since prompts are implemented as systems in Scorecard, you’ll use the systems API methods.

​
Creating or Updating Prompts

Use the systems.upsert() method to create a new prompt or update an existing one. For complete API details, see the Create (upsert) system API reference. 
import Scorecard from 'scorecard-ai';

const client = new Scorecard({
  apiKey: process.env['SCORECARD_API_KEY'],
});

// Create or update a prompt (system)
const prompt = await client.systems.upsert('your-project-id', {
  name: 'Customer Service Assistant',
  description: 'AI assistant for handling customer inquiries',
  config: {
    modelName: 'gpt-4-turbo',
    temperature: 0.7,
    maxTokens: 1024,
    topP: 0.9,
    promptTemplate: [
      {
        role: 'user',
        content: `You are a helpful assistant.

{{allInputs}}`
      }
    ]
  }
});

console.log('Prompt created with ID:', prompt.id);
The config object contains your prompt’s model parameters and template:
  • modelName: The LLM model to use (e.g., gpt-4-turbo, claude-3-opus)
  • temperature: Controls randomness (0.0-1.0)
  • maxTokens: Maximum response length
  • topP: Token selection probability
  • promptTemplate: Array of message objects defining the conversation structure. Each message has a role (system, user, or assistant) and content with your prompt text and Jinja variables.
For a system to be flagged as a prompt in Scorecard, it must include the promptTemplate configuration in its config object. Without this field, the system will not be recognized as a prompt.

​
Creating Prompt Versions

Use the systems.versions.upsert() method to create new versions of your prompt. For complete API details, see the Upsert system version API reference. 
// Create a new version of the prompt
const promptVersion = await client.systems.versions.upsert('your-system-id', {
  name: 'v2.0 - Improved tone',
  config: {
    modelName: 'gpt-4-turbo',
    temperature: 0.5,
    maxTokens: 1024,
    topP: 0.8,
    promptTemplate: [
      {
        role: 'user',
        content: `You are a helpful assistant.

{# You can replace this with the specific inputs from your testcase #}
{{allInputs}}`
      }
    ]
  }
});

console.log('New prompt version created:', promptVersion.id);
Ready to evaluate your prompts? Follow the Playground guide to test your prompts on Testsets.
⌘I