Scorecard uses a hierarchical structure to organize your evaluation work: Organizations contain multiple Projects, and each Project contains your testsets, metrics, runs, and results.

​
Organizations

Organizations are the top-level entity in Scorecard that enables team collaboration and workspace isolation. All members of an organization can access all projects within that organization. API keys are scoped to a specific organization, not user or project.
Screenshot of the organization member management dialog with a few users.Screenshot of the organization member management dialog with a few users.

Organization member management dialog with a few users.

​
Projects

Each project serves as a container for:
  • Testsets: Collections of Testcases for evaluation
  • Metrics: Custom evaluation criteria and scoring methods
  • Runs: Evaluation executions and their results
  • Prompts and Systems: The AI system versions being evaluated
  • Traces: Live traces and logs of AI system calls
Screenshot of the projects overview page with two projects.Screenshot of the projects overview page with two projects.

Projects overview page for an organization.

​
Creating a New Project

To create a project, click “New Project” in the top right of the Projects page and fill in the project details.
Choose a descriptive project name for the system or use case you’re evaluating, such as “Customer Support Chatbot” or “Product Recommendation Engine.”
Screenshot of the create project modal in the UI.Screenshot of the create project modal in the UI.

Creating a new project.