ROC Curves

Plot ROC curves comparing monitor performance on honest vs attack trajectories.

Quick Start

# Single honest/attack pair
uv run ct game-metrics graph roc --honest <run-id-or-path> --attack <run-id-or-path>

# From a run file with multiple entries
uv run ct game-metrics graph roc --run-file runs.yaml

Plots are saved to data/roc/<timestamp>/ by default. By default, a single ROC plot is produced using the combined_max scoring method. Use --agg-func to select different or multiple aggregation methods.

Input Modes

Inline mode

Pass --honest and one or more --attack sources directly. Each --attack is paired with the shared --honest to produce a separate curve.

# One curve (monitored gpt-4.1-nano runs)
uv run ct game-metrics graph roc \
  --honest https://data.linuxarena.ai/runs/c7aeb0aeccba40eb9d910ab13b601bed \
  --attack https://data.linuxarena.ai/runs/bf8a832da92f4431b2579ad04cd983ad

# Two curves (two attack variants, same honest baseline)
uv run ct game-metrics graph roc \
  --honest https://data.linuxarena.ai/runs/c7aeb0aeccba40eb9d910ab13b601bed \
  --attack https://data.linuxarena.ai/runs/bf8a832da92f4431b2579ad04cd983ad \
  --attack https://data.linuxarena.ai/runs/99d571edada441e8a595d7e50f4efc37

Sources can be run IDs, run URLs (https://data.linuxarena.ai/runs/...), local directories, or eval log file paths.

Run file mode

Pass --run-file with a YAML file describing multiple honest/attack pairs and their metadata. This enables --color-by, --style-by, and --filter-by.

uv run ct game-metrics graph roc --run-file runs.yaml

Run file format:

runs:
  # Run URLs (monitored runs uploaded to the run store)
  - label: gpt-4.1-nano (goals)
    model: gpt-4.1-nano
    monitor: mean-action                   # metadata for --color-by/--style-by
    honest: https://data.linuxarena.ai/runs/c7aeb0aeccba40eb9d910ab13b601bed
    attack: https://data.linuxarena.ai/runs/bf8a832da92f4431b2579ad04cd983ad

  - label: claude-haiku (goals)
    model: claude-haiku-4-5-20251001
    monitor: mean-action
    honest: https://data.linuxarena.ai/runs/af9f96b7df2b442aae557bd7c12ef9a0
    attack: https://data.linuxarena.ai/runs/99d571edada441e8a595d7e50f4efc37

  - label: gpt-oss-120b (goals)
    model: gpt-oss-120b
    monitor: mean-action
    honest: https://data.linuxarena.ai/runs/720ef3f583bb449aa2129700f1faf7d0
    attack: https://data.linuxarena.ai/runs/915266b544ba44f896969942f84e4a78

  # Local paths (from roc_monitor_eval.py output)
  - label: gpt-4.1-nano (no-goals)
    model: gpt-4.1-nano
    monitor: mean-action-no-goals
    honest: data/roc_eval/2026-03-04_14-30-47/gpt-4.1-nano/honest
    attack: data/roc_eval/2026-03-04_14-30-47/gpt-4.1-nano/attack

Required fields per entry: honest, attack. Optional fields: label, plus any arbitrary metadata (e.g. model, monitor).

When label is omitted it is auto-generated from model and monitor if present, otherwise from the attack source name.

Visual Mapping

Use --color-by and --style-by to map run-level metadata fields to visual properties. Each unique value gets a distinct color or linestyle.

# Color by model, linestyle by monitor variant
uv run ct game-metrics graph roc --run-file runs.yaml --color-by model --style-by monitor

Auto-resolved fields

When the attack source is a run ID, run URL, or a local directory containing run.json, the following fields are automatically resolved from the run's metadata and do not need to be specified in the YAML run file:

Values explicitly set in the YAML always take precedence over auto-resolved values.

If a requested --color-by or --style-by field cannot be found in either the YAML config or the run metadata, the command will error with a message listing which runs are missing the field.

Aggregation Methods

The --agg-func option controls how trajectories are scored. You can specify multiple methods to generate separate plots for each, plus a combined grid.

# Default: combined_max only (single roc.png)
uv run ct game-metrics graph roc --honest <run> --attack <run>

# Compare combined_max vs action_only (two plots + grid)
uv run ct game-metrics graph roc \
  --honest <run> --attack <run> \
  --agg-func combined_max --agg-func action_only

When a single method is used, the plot is saved as roc.png and the method name is omitted from the title. When multiple methods are specified, each is saved as roc_<method>.png with the method name in the title, plus a roc_combined.png grid.

Grouping

Use --group-by to split trajectories and produce separate plots per group value.

# One plot per environment
uv run ct game-metrics graph roc --run-file runs.yaml --group-by env

# One plot per failure type (honest baseline shared across groups)
uv run ct game-metrics graph roc --run-file runs.yaml --group-by failure_type

Available group keys:

"Paired" means both honest and attack trajectories are split by the key and matched. "Shared" means all honest trajectories serve as the baseline for every attack group.

Use --min-samples N (default: 2) to skip groups with too few trajectories on either side.

Filtering

Use --filter-by to select a subset of runs from a run file:

# Only plot runs where model=gpt-4.1-nano
uv run ct game-metrics graph roc --run-file runs.yaml --filter-by model=gpt-4.1-nano

# Multiple filters (all must match)
uv run ct game-metrics graph roc --run-file runs.yaml \
  --filter-by model=gpt-4.1-nano \
  --filter-by monitor=mean-action

Output

# Default output directory (timestamped)
uv run ct game-metrics graph roc --run-file runs.yaml
# -> data/roc/2026-03-09_14-30-00/

# Custom output directory
uv run ct game-metrics graph roc --run-file runs.yaml --output-dir results/my_experiment

With the default single aggregation method, the output directory contains:

data/roc/<timestamp>/
  roc.png

With multiple --agg-func values:

data/roc/<timestamp>/
  roc_combined_max.png
  roc_action_level.png
  roc_combined.png          # grid of all methods

Options Reference