Running Evaluations
Use ct run eval to run a policy or control protocol against one or more tasks. Start with a test-policy smoke run before spending model tokens:
uv run ct run eval --policy test --task-file simple --tags test
Running a full sabotage evaluation? Use
ct run sabotage-evalinstead — it orchestrates honest evals, attack evals, monitoring, and safety metrics in a single command.
Guides
| Guide | What it covers |
|---|---|
| Setup and configuration | Environments, credentials, models, policies, and external sandboxes |
| Task selection | Exact tasks, task files, trajectories, replay, and expansion modes |
| Examples | Honest, attack, batch, and live-development runs |
| Config files | YAML configuration, composition, and the current schema |
| EC2 fleet operation | Worker AMIs and streaming fleet events |
| Output | Eval logs, Inspect's viewer, and trajectory conversion |
| Runs | Uploading, listing, inspecting, sampling, and downloading runs |
| Troubleshooting | Docker address pools and container service startup |
ct run eval CLI reference | Every current argument and option, generated from Click |
See Also
- Sabotage Evaluations - End-to-end sabotage evaluation pipeline
- Auditing Environments - Environment security and side task quality review
- Monitoring Trajectories - Evaluating trajectory suspiciousness
- Making Main Tasks - Creating main tasks
- Making Side Tasks - Creating side tasks
- Making Strajs - Creating sabotage trajectories