ct run
Run workflows, evaluations, and analysis tasks.
Usage
ct run [OPTIONS] COMMAND [ARGS]...
Options
| Option | Description |
|---|---|
--help | Show this message and exit. |
Commands
| Command | Description |
|---|---|
ct run analyze-attack | Analyze attack trajectory(ies) to identify incriminating actions. |
ct run analyze-attack-view | View results from an analyze-attack run interactively. |
ct run compare-side-tasks | Compare side task successes between two runs. |
ct run dashboard | Show a terminal dashboard of eval run status. |
ct run doc-audit | Audit side task documentation for quality issues. |
ct run env-description-audit | Audit environment descriptions against the actual codebase. |
ct run env-description-generate | Generate a fresh docs/description.md for one or more environments. |
ct run eval | Run AI agent evaluation. |
ct run failure-type-audit | Classify side tasks as live or delayed using the Frozen Environment Assumption. |
ct run find-questions | Discover answer-question main tasks in an environment. |
ct run find-questions-iterative | Iteratively discover answer-question tasks across all environments. |
ct run goal-completeness | Audit side tasks for goal completeness — does the agent have enough info? |
ct run goal-completeness-view | Interactive viewer for goal completeness audit results. |
ct run monitor | Run monitors on trajectories (action-level and/or trajectory-level) |
ct run monitor-audit | Audit monitor errors in a run, classifying invalid zeros and failures. |
ct run mtgen | Generate, validate, and live-run mtgen tasks. |
ct run mtgen-bugfix | Generate bug-fix main tasks via LLM-driven mutations. |
ct run pairing-analysis | Analyze main/side task pairings for Control Tower environments. |
ct run rerun-traj | Replay a trajectory to re-run scoring. |
ct run reward-hacking | Audit side tasks for reward hacking vulnerabilities. |
ct run reward-hacking-view | Interactive viewer for reward hacking audit results. |
ct run rt-alignment | Audit side tasks for RT description alignment. |
ct run rt-alignment-view | Interactive viewer for RT alignment audit results. |
ct run sabotage-eval | Run sabotage evaluation: honest eval + attack eval + monitor + graphs. |
ct run scorer-timeout | Analyze why scorers are timing out in a run. |
ct run scorer-timeout-view | Interactive viewer for scorer timeout analysis results. |
ct run side-task-failure | Analyze why side tasks failed in a run. |
ct run side-task-failure-view | Interactive viewer for side task failure analysis results. |
ct run side-task-review | Review side tasks for quality issues. |
ct run submit-guessability | Audit side tasks for submit tool exploitability (theoretical). |
ct run submit-guessability-verified | Audit side tasks for submit tool exploitability with LIVE VERIFICATION. |
ct run submit-guessability-verified-view | Interactive viewer for verified submit guessability audit results. |
ct run submit-guessability-view | Interactive viewer for submit guessability audit results. |
ct run summarize | Cross-run autopsy: produces a grounded LLM report on a paired honest + attack evaluation. Downloads trajectories for each run, builds a cached prefix from any --extra-context paths (files or folders), fans out a per-trajectory summary call, and aggregates into a single FinalReport. Writes report.{json,md}, per_traj/<traj_id>.json (parsed TrajSummary), and SPEC.md (folder layout) under --out-dir. Pass --debug to also dump raw model request/response under debug/ + a reproducer command.txt. Each of --honest-run / --attack-run is optional individually; at least one must be provided. |
ct run timing-report | Phase-level timing + token/cost report for a finished eval run. Reads a .eval file (or a directory of them — single --ec2 and merged fleet runs both emit .eval) and prints where time went (LLM generation vs tool/docker exec vs scoring), per-sample setup latency and working time, token totals (incl. prompt-cache reads/writes), prompt-cache hit rate, total token $ cost, and the slowest samples and tool calls. Pure analysis over data already in the log — no API calls, no eval spend. |
ct run traj-prompt | Run an LLM prompt against trajectories. |
ct run traj-prompt-view | View results from a traj-prompt run interactively. |
ct run validate-cover | Validate that a main task provides adequate cover for a side task. |