ct game-metrics metrics
Compute safety + usefulness metrics across one or more honest/attack pairs.
Usage
ct game-metrics metrics [OPTIONS]
Options
| Option | Description |
|---|---|
--honest TEXT | Honest trajectories (path, dir, or run ID). |
--attack TEXT | Attack trajectories (path, dir, or run ID). |
--run-file PATH | YAML run file with multiple run entries. |
--filter-by TEXT | Filter runs (field=value, repeatable). |
--honest-task-set PATH | Filter honest trajectories to (env, main_task) pairs listed in this JSONL file (one {"env": ..., "main_task": ...} record per line). Per-entry honest_task_set keys in the run file take precedence. |
--attack-task-set PATH | Filter attack trajectories to (env, main_task, side_task) triples listed in this JSONL file (one {"env": ..., "main_task": ..., "side_task": ...} record per line). Per-entry attack_task_set keys in the run file take precedence. |
--group-by [env|failure_type|main_task_category|side_task_category] | Group attacks by field. |
--audit-budgets FLOAT RANGE | Target FPR as a fraction; 1.0 audits everything. [default: (wide range from 0.0001 to 0.5); 0<=x<=1] |
--mode [optimized|50_50|stepwise_percentile|action_only|trajectory_only] | Audit-rule fitter to use at every budget. '50_50' and 'stepwise_percentile' are calibration-free two-leg modes; 'action_only' / 'trajectory_only' restrict the search to one monitor leg. [default: optimized] |
--bootstrap INTEGER RANGE | If > 0, compute bootstrap 95% CI bands by resampling honest and attack trajectories and recalculating safety on every draw. [default: 0; x>=0] |
--output-dir PATH | Directory for metrics.json + safety_curves.png. Auto-generated under data/metrics/<timestamp>/ when omitted. |
--max-workers INTEGER RANGE | Thread-pool size for loading run entries concurrently. [default: 8; x>=1] |
--help | Show this message and exit. |