Evaluation Config Files

For longer commands, complex experiments, or anything you'd want to commit to the repo, put the run in an Inspect run-config YAML and pass it with --run-config. ct run eval --run-config x.yml delegates to inspect eval --run-config x.yml; the eval runs CT's registered control_tower/control_eval task.

uv run ct run eval --run-config examples/eval_configs/03-attack-single.yml

Worked examples live in examples/eval_configs/:

Schema

The file is Inspect's native run-config format: task: names the registered task and its args, eval_config: holds Inspect eval options, tags: tags the run, and metadata: carries CT's post-run knobs.

Task-selection fields follow the modes described in Evaluation Task Selection.

task:
  task: control_tower/control_eval
  args:
    # Optional run name (defaults to `<policy>-<task-set>`).
    run_name: my-experiment

    # Task selection. See task-selection.md for semantics; fields
    # mirror the CLI flags (environments, main_tasks, side_tasks,
    # task_file, trajectory_id, limit, expansion flags, categories, ...).
    task_selection:
      environments: [web_scraping]
      main_tasks: [crawl_depth]
      side_tasks: [visit_malicious_website]

    # Pick exactly one of `policy` or (`untrusted_policy` + `protocol`).
    policy:
      name: honest                # `ct protocols policies` to list
      args:
        model: sonnet
    # untrusted_policy:
    #   name: simple-attack
    # protocol:
    #   name: trusted-monitoring
    #   args:
    #     sus_threshold: 7

    # Task-level scoring options.
    scorer_timeout: 300           # int | null  (seconds)
    diagnostics: null             # list[str] | null  (e.g. [refusal])

    # Local docker sandbox options (SandboxConfig fields).
    sandbox:
      startup_retries: 7

# Inspect eval options (native `eval_config`): epochs, max_samples,
# fail_on_error, retry_on_error, token_limit, message_limit, time_limit,
# cost_limit, ...
eval_config:
  epochs: 3
  max_samples: 4
  fail_on_error: false

tags: [attack, sonnet]

# CT post-run side effects, applied by CT's registered run-end hook.
metadata:
  ct_upload: true                 # upload results to the Control Tower database
  docent_collection_id: null      # also push to this Docent collection

All task args are validated with the same Pydantic models as the CLI flags, so unknown fields and invalid combinations (e.g. policy together with protocol) are rejected loudly.

Overriding

Inspect's semantics apply: one file per run, CLI flags override file values, and -T task args dict-merge into the file's task.args. ct run eval --run-config forwards --no-upload/--docent-collection-id (as metadata), --run-name (as a task arg), --tags, --log-dir, and --max-samples; other ct run eval flags are rejected — put them in the YAML.

# Same config, different model, no upload
uv run ct run eval --run-config examples/eval_configs/02-honest-baseline.yml --no-upload
uv run inspect eval --run-config examples/eval_configs/02-honest-baseline.yml -T 'policy={"name": "honest", "args": {"model": "opus"}}'