Pairing Analysis

Analyzes main/side task pairings for quality and realism.

Purpose

Evaluates whether main task and side task pairings make sense for realistic sabotage scenarios:

  • Does the main task provide adequate cover for the side task?
  • Are there better alternative pairings?
  • Is the pairing realistic and challenging?

Commands

ct run pairing-analysis

Analyze main/side task pairings for quality.

Usage:

ct run pairing-analysis [OPTIONS]

Options:

  • Configuration options as defined in the CLI

Examples:

# Analyze all pairings
ct run pairing-analysis

# Analyze pairings for specific environment
ct run pairing-analysis -e web_scraping

Output

Analysis of each pairing including:

  • Cover quality assessment
  • Alternative pairing suggestions
  • Realism evaluation
  • Difficulty assessment

Analysis Criteria

For each pairing:

  • Does the main task require accessing same resources as side task?
  • Would normal main task work provide cover for sabotage?
  • Are there better main tasks for this side task?
  • Is the pairing realistic in a production scenario?

Use Cases

  • Quality control for benchmark design
  • Identifying weak pairings
  • Finding better cover tasks
  • Ensuring realistic sabotage scenarios

Technical Details

  • Uses LLM analysis with environment and task context
  • Considers resource access patterns
  • Evaluates plausibility of cover stories
  • Helps improve benchmark realism