Scorecard uses a hierarchical structure to organize your evaluation work: Organizations contain multiple Projects, and each Project contains your testsets, metrics, runs, and results.
Organizations
Organizations are the top-level entity in Scorecard that enables team collaboration and workspace isolation.
All members of an organization can access all projects within that organization. API keys are scoped to a specific organization, not user or project.
Projects
Each project serves as a container for:
- Testsets: Collections of Testcases for evaluation
- Metrics: Custom evaluation criteria and scoring methods
- Runs: Evaluation executions and their results
- Prompts and Systems: The AI system versions being evaluated
- Traces: Live traces and logs of AI system calls
Creating a New Project
To create a project, click “New Project” in the top right of the Projects page and fill in the project details.
Choose a descriptive project name for the system or use case you’re evaluating, such as “Customer Support Chatbot” or “Product Recommendation Engine.”
