ct run summarize
Cross-run autopsy: produces a grounded LLM report on a paired honest + attack evaluation. Downloads trajectories for each run, builds a cached prefix from any --extra-context paths (files or folders), fans out a per-trajectory summary call, and aggregates into a single FinalReport. Writes report.{json,md}, per_traj/<traj_id>.json (parsed TrajSummary), and SPEC.md (folder layout) under --out-dir. Pass --debug to also dump raw model request/response under debug/ + a reproducer command.txt. Each of --honest-run / --attack-run is optional individually; at least one must be provided.
Usage
ct run summarize [OPTIONS]
Options
| Option | Description |
|---|---|
--honest-run TEXT | Optional. Honest run URL or 32-hex run id. Pass alone to autopsy only the honest side; cross-run sections (safety @ audit budgets) skip. |
--attack-run TEXT | Optional. Attack run URL or 32-hex run id. Pass alone to autopsy only the attack side; cross-run sections (safety @ audit budgets) skip. |
--extra-context PATH | Optional. File OR folder to include in the cached prefix. Repeatable. For a folder, every *.md, *.py, and *.txt inside (depth ≤ 2) is concatenated under one cache block labeled by the folder's basename. For a file, the file content becomes its own block. Use this for any design-intent docs the LLM should see — blue/red iteration folders, proposal.md, match config.yml, issues.md, etc. |
--dimensions PATH | Optional. Petri-style scored-dimension rubric. File OR folder, repeatable — loaded the same way as --extra-context. Each file is JSON: {"name", "description", "dimensions": [...]} where each dimension has name, type (score | categorical | phrase), description, arm, and (per type) range / higher_is / values. When passed, every trajectory is additionally labeled per dimension with a message-citing justification (score dims emit {score, justification}; categorical / phrase dims emit {value, justification}), under TrajSummary.dimensions. Omit for the exact pre-rubric behaviour. A default control rubric ships at src/control_tower/ct_runs/summarize/rubrics/control_default.json. |
--out-dir PATH | Directory to write report.{json,md}, per_traj/<traj_id>.json, SPEC.md, and (with --debug) debug/ + command.txt. [required] |
--debug | When set, also write debug/<traj_id>.{input,output}.json (raw model request/response per trajectory) and command.txt (reproducer for each side). Off by default — the regular run produces per_traj JSONs + the aggregate report only. |
--limit INTEGER | If set, only summarize the first N trajectories from each run (smoke). |
--concurrency INTEGER | Concurrent model calls for the per-trajectory fan-out. [default: 8] |
--model TEXT | Model identifier with provider prefix; the first segment picks the routing path (same convention as ct run eval -p model=…): openrouter/<…> routes via inspect-ai's OpenRouter adapter (needs OPENROUTER_API_KEY); anthropic/<…> routes via the Anthropic Messages API directly (needs ANTHROPIC_API_KEY); openai/<…> / google/<…> and aliases (sonnet, haiku, gemini_flash, …) go through inspect-ai's get_model with the matching per-provider key. Default openrouter/google/gemini-3.5-flash works out of the box with OPENROUTER_API_KEY alone. Used for BOTH the per-trajectory TrajSummary call and the aggregate FinalReport call. |
--help | Show this message and exit. |