Use Projects to organize your Testsets, Runs, and Results
Projects help you organize and navigate your Scorecard assets
Sometimes your Scorecard assets such as Testsets or Runs can multiply to the point that it’s hard to find what you’re looking for. Projects help you organize these assets into buckets of related work so that it’s easier to get your work done. Projects group Testsets together, as well as the Runs and Results that are generated from them.
Now, when you load the Scorecard app the homepage is the Projects page, where you will see all of your Organization’s projects. You can click into a Project to see all of the Testsets, Runs, and Results of that Project.
Schemas define how your Testsets are structured
A new feature of Projects are Schemas. Schemas are created at the Project level, and define the Input, Label, and Output Variables of your evaluation pipeline.
Defining a Schema
You define the Schema of a Project when you create a new Project. Once created, it is expected that all testsets within a Project adhere to the same Schema. If a testset, or certain testcase does not require an Input or Label Variable, it can simply be left blank.
What’s a Variable?
Variables are the individual components of testsets and runs. Testsets have Input and Label Variables. Runs have Output Variables. Variables are one of three types: text, file_url, or json_object.
Input Variables
Input Variables are the inputs to your systems. There are two default, required Input Variables in Scorecard testsets: user_query and context. These are the standard inputs to an LLM system. context is provided in the Context window, and the user_query is added to the Prompt template. You can define additional, custom Input Variables for your Testsets which allows you to define separate inputs in separate columns. For example, you might add an image_url Input Variable that allows your Prompt template to reference both the user_query and image_url in different places.
Label Variables
Label Variables are typically used to define ideal values for model outputs, which are useful in evaluation. Every testset has one required Label Variable in the Schema called ideal. You can optionally define additional Label Variables in your testset. For example, if you find that for a certain test case your system often provides a common, incorrect response, you could add an incorrect label which you can reference in your Metric.
Output Variables
Output Variables are the various oututs you expect from your system. Typically this is just model_response which is the single required Output Variable in a Schema. However, if your system returns multiple outputs, you can write each of them to a respective Output Variable. For example your system might generate an image, and a description of it. In this case, the description would be provided as the model_response Output variable, and the location of the image might be written to a custom image_response Output Variable you define.
Creating and Editing a New Project
You can create as many Projects as you like from the Projects page. When you create a Project, you will define its name, a description, and the Schema of the Project. The Scorecard Default Schema of user_query, context, ideal, and model_response is always included in a Project Schema, however you can add additional variables if you like. You can edit a Schema along with all other Project details any time in the Project Settings tab.