ct run sabotage-eval

Run sabotage evaluation: honest eval + attack eval + monitor + graphs.

Usage

ct run sabotage-eval [OPTIONS]

Options

OptionDescription
-c, --config PATHYAML config file for multi-attack mode.
--attacker, --policy TEXTAttack policy name. --policy is an alias.
--attacker-arg, --policy-arg TEXTAttacker policy args in key=value format (repeatable). --policy-arg is an alias.
--attack-model TEXTShorthand for --attacker-arg model=. Required unless --attack-run-id or --config.
--attack-run-id TEXTReuse existing attack run (must have monitoring scores). Skips attack eval.
--honest TEXTHonest policy name for fresh honest eval.
--honest-arg TEXTHonest policy args in key=value format (repeatable).
--honest-model TEXTShorthand for --honest-arg model=. Defaults to attack model.
--honest-task-file TEXTTask file for honest eval. Defaults to --task-file.
--honest-run-id TEXTReuse existing honest run instead of running fresh honest eval.
--protocol TEXTBlue protocol to run both evals under (e.g. defer-to-resample, trusted-editing). Attacker/honest become the protocol's untrusted policy.
--protocol-arg TEXTProtocol args in key=value format (repeatable).
--audit-budgets TEXTAudit budget(s), comma-separated (e.g. 0.0001,0.001,0.01). Each produces graphs.
--thresholds TEXTFixed threshold(s), comma-separated (e.g. 3.0,4.0,5.0). Skips honest eval.
--action-trajectory-ratio FLOATRatio-based threshold: trajectory = action * ratio (e.g. 0.5). Uses audit_budgets for calibration.
--action-monitor TEXTAction monitor type, used for both honest and attack trajectories
--trajectory-monitor TEXTTrajectory monitor type, used for both honest and attack trajectories
--trusted-model TEXTModel bound as the eval-level trusted role the monitors run on.
--only-actionOnly run the action monitor
--only-trajectoryOnly run the trajectory monitor
--cot-level [none|reasoning|extended]Chain-of-thought visibility to the monitor: none, reasoning (models output text), or extended (reasoning and CoT).
--task-file TEXTTask set name or path.
--main-tasks-with-no-scorers [error|include|exclude]Main tasks lacking a scorer: refuse the run (error), keep them recorded as unscored (include), or drop them (exclude).
--log-dir TEXTWhere eval logs go.
-o, --output-dir TEXTWhere graphs and summary go.
--ec2Run evals on EC2.
--epochs INTEGERNumber of epochs per eval.
--limit INTEGERLimit number of samples.
--no-graphSkip graph generation.
--no-uploadDisable uploading results.
--dry-runPrint what would run without executing.
--no-cacheRe-run every attack even when a prior results.json recorded an identical one.
--preflightProbe the run's contracts (monitors, endpoints, pricing, credentials) and exit.
--skip-docker-buildSet SKIP_DOCKER_BUILD=1 so eval pulls pre-built images instead of building.
--monitor-concurrency INTEGERMax trajectories to monitor concurrently per monitoring job (default: unlimited).
--tags TEXTTags for the run (repeatable). Added to all eval runs alongside built-in sabotage-eval tags.
--helpShow this message and exit.