Side Task Failure Analysis

Analyzes why side tasks failed in evaluation runs.

Purpose

Uses LLM analysis to determine the root cause of side task failures:

  • Agent errors (legitimate failures)
  • Side task scorer issues (bugs in scorer)
  • Environment problems
  • Ambiguous task descriptions

Commands

ct run side-task-failure

Analyze why side tasks failed in a run, eval file, trajectory file, trajectory directory, or dataset.

Usage:

ct run side-task-failure TRAJ_LIKE [OPTIONS]

Arguments:

  • TRAJ_LIKE - Run ID, eval file, trajectory file, trajectory directory, or dataset reference to analyze.

Options:

  • -l, --limit - Maximum number of failures to analyze
  • -c, --max-concurrent - Maximum concurrent API calls (default: 4)
  • --thinking-budget - Token budget for extended thinking (default: 10000)

Examples:

# Analyze all failures in a run
ct run side-task-failure abc123def456

# Limit to 10 failures
ct run side-task-failure abc123def456 --limit 10

ct run side-task-failure-view

Interactive viewer for side task failure analysis results.

Usage:

ct run side-task-failure-view [RUN_PATH]

Examples:

# Select from available runs
ct run side-task-failure-view

# View specific run
ct run side-task-failure-view data/side_task_failure_runs/20240115_120000_abc123

Output

Results saved to data/side_task_failure_runs/<timestamp>_<run_id>/:

  • results.jsonl - Analysis results for each failure
  • config.json - Run configuration
  • Categorization: agent error vs side task issue

Technical Details

  • Only analyzes trajectories where side_task_success == False
  • Uses prompts from prompts/system.md and prompts/user.md
  • Provides full trajectory context, scorer code, and environment details
  • Helps identify problematic side tasks that need fixing