Compare Side Tasks

Compares side task success rates between two runs.

Purpose

Analyzes differences in side task outcomes between two evaluation runs to identify:

  • Which side tasks succeeded in one run but failed in another
  • Success rate changes across runs
  • Task-specific performance differences
  • Regression or improvement patterns

Commands

ct run compare-side-tasks

Compare side task successes between two runs.

Usage:

ct run compare-side-tasks RUN_ID_1 RUN_ID_2

Arguments:

  • RUN_ID_1 - First run ID
  • RUN_ID_2 - Second run ID

Examples:

# Compare two runs
ct run compare-side-tasks abc123 def456

Output

Displays:

  • Run metadata (name, policy, tags, date) for identification
  • Side tasks that succeeded in run 1 but failed in run 2
  • Side tasks that failed in run 1 but succeeded in run 2
  • Side tasks only present in one run (when runs cover different task sets)
  • Side tasks with the same result in both runs
  • Overall success rate comparison
  • Environment-level breakdown

Use Cases

  • Comparing different policies (honest vs attack)
  • Evaluating model improvements
  • Regression testing after changes
  • Understanding policy-specific side task behavior

Technical Details

  • Downloads runs from MongoDB if not cached
  • Groups by environment and side task
  • Shows detailed diff of outcomes
  • Useful for A/B testing and policy comparison