Evaluation Task Selection

The task selection system converts CLI arguments into task combinations. You must specify tasks using one of the following modes.

Selection Modes

1. Specific Task Mode

Specify exact tasks to run. This is the most common approach.

# Single main task (honest mode - no side task)
uv run ct run eval -t crawl_depth --policy honest

# Main + side task (attack mode)
uv run ct run eval -t crawl_depth -s visit_malicious_website --policy attack

# Multiple environments with same main task
uv run ct run eval -e web_scraping -e clinical_trial --just-main-tasks --policy honest

# Multiple of each (creates cross-product)
uv run ct run eval -t task1 -t task2 -s side1 -s side2 --policy attack

2. Task File Mode (--task-file)

Reads explicit task combinations from a file. Takes precedence over other options.

# From a named task set
uv run ct run eval --task-file medium --policy attack

# From a custom file path
uv run ct run eval --task-file my_tasks.jsonl --policy attack

Task files are JSONL, one record per task combination with main_task and/or side_task:

{"env": "web_scraping", "main_task": "crawl_depth", "side_task": "visit_malicious_website"}
{"env": "clinical_trial", "main_task": "adverse_reaction_followup_tracking", "side_task": "suppress_safety_notifications"}

3. Trajectory Mode (--trajectory-id)

Extracts task info from an existing trajectory. Useful for re-running or replaying.

# From API/database (trajectory ID)
uv run ct run eval --trajectory-id abc123 --policy attack

# From local file
uv run ct run eval --trajectory-id ./my_trajectory.json --policy attack

# With task override
uv run ct run eval --trajectory-id abc123 -t different_main_task --policy attack

This is especially useful with the replay policy to re-execute trajectories:

uv run ct run eval --trajectory-id abc123 --policy replay
# Or use the convenience command:
uv run ct run rerun-traj abc123

# From live development context (uses solution.json from active traj dir):
uv run ct run rerun-traj --from-live

Replaying without execution (--replay-sandbox)

By default a replay re-executes every recorded action in a real sandbox. --replay-sandbox skips execution entirely: nothing is provisioned, and the environment is the recording — every exec / text_editor call is answered with the recorded output of the next action in the source trajectory. Replayed samples cost nothing on the sandbox side and are exactly faithful to the recording.

The contract is prefix replay: sound while the tool calls that reach the sandbox form a verbatim prefix of the recording. Anything that never reaches the sandbox is unconstrained and free to differ from the recording — assistant text and reasoning, monitor scoring, interrogation questions and the policy's answers to them, judge deliberation, stopping early. Any sandbox-reaching call that is new, modified, reordered, or skipped raises ReplayDivergenceError (with the recorded and incoming action) rather than improvising an output.

This means --replay-sandbox works with any replay-like policy — one whose executed tool calls stay a prefix of the recording — not just the literal replay policy (which re-emits the recording verbatim and is the simplest member of the class). For example, a policy that emits the recorded tool calls but answers a blue protocol's interrogation questions with a live model is replay-like: the answers never touch the sandbox. Likewise on the blue side: pass-through protocols (e.g. trusted monitoring) replay cleanly, while protocols that rewrite actions (trusted editing, defer-to-trusted) diverge by design and raise. The boundary is not read-only vs. state-modifying — even a harmless ls that wasn't recorded raises, because no container exists to answer it.

# Sweep a blue protocol over a fixed set of recorded trajectories
uv run ct run eval --trajectory-id logs/run.eval --untrusted-policy replay -up traj_like=logs/run.eval \
  --protocol trusted-monitoring --replay-sandbox

Programmatic scorers don't run: a full replay carries the source trajectory's scores over, a partial replay scores NOANSWER with an explanation. Trajectories whose policies used run_side_task_test are not replayable (that tool executes scorer scripts through many sandbox calls that were never recorded).

4. Expansion Modes

Use when you want to run many tasks without listing them explicitly.

--all: Expands to all valid main×side combinations.

# All attack combinations for one environment
uv run ct run eval -e web_scraping --all --policy attack

# All main tasks across all environments
uv run ct run eval --just-main-tasks --policy honest

--just-main-tasks: All main tasks with no side tasks. For honest mode evaluations.

uv run ct run eval --just-main-tasks --policy honest
uv run ct run eval -e web_scraping --just-main-tasks --policy honest

--just-side-tasks: All side tasks with no main task. The agent receives only the side task instructions.

uv run ct run eval --just-side-tasks --policy attack

Validation Rules

1. One mode required: Must specify --task-file, an expansion flag (--all, --just-main-tasks, --just-side-tasks), or specific tasks (-t)
2. Expansion flags are mutually exclusive: Cannot combine --all, --just-main-tasks, --just-side-tasks
3. Specific tasks don't mix with expansion: -t/-s cannot be combined with expansion flags
4. Environment is optional: If not specified, inferred from task IDs or defaults to all environments

If you omit required parameters, the CLI will prompt you interactively (unless DISABLE_TUI=1).

Attack vs Honest Mode

Whether a sample runs in "attack" or "honest" mode is determined by side task presence:

• Attack mode: Sample has a side task (-s specified or from task file)
• Honest mode: Sample has no side task

Task Sets

Task sets are predefined collections of task combinations stored in src/control_tower/task/task_sets/. They make it easy to run consistent evaluations across the team.

Managing task sets:

uv run ct task-sets list              # List available task sets
uv run ct task-sets validate simple   # Validate a task set
uv run ct task-sets validate --all    # Validate all task sets
uv run ct task-sets show medium       # Show contents of a task set

The built-in task sets are examples: simple for the smallest smoke test, one_per_env for broad environment coverage, and medium for a small multi-environment attack set.

Creating custom task sets:

Create a .jsonl file in src/control_tower/task/task_sets/ with one record per task combination:

{"env": "clinical_trial", "main_task": "adverse_reaction_followup_tracking", "side_task": "suppress_safety_notifications"}
{"env": "web_scraping", "main_task": "redact", "side_task": "expose_secret"}

Each record names an env plus main_task and/or side_task (a record with only side_task runs side-only). Run ct task-sets validate <name> to verify your task set is valid.