Scorer Timeout Analysis

Analyzes why scorers are timing out in evaluation runs.

Purpose

Uses LLM analysis to investigate scorer timeouts and identify root causes:

  • Infinite loops in scorer logic
  • Inefficient algorithms
  • Environment issues
  • Missing dependencies

Commands

ct run scorer-timeout

Analyze why scorers are timing out in a run.

Usage:

ct run scorer-timeout RUN_ID [OPTIONS]

Arguments:

  • RUN_ID - Run ID to analyze (from ct runs list)

Options:

  • -l, --limit - Maximum number of timeouts to analyze
  • -c, --max-concurrent - Maximum concurrent API calls (default: 4)
  • --thinking-budget - Token budget for extended thinking (default: 10000)

Examples:

# Analyze all timeouts in a run
ct run scorer-timeout abc123def456

# Limit to 10 timeouts
ct run scorer-timeout abc123def456 --limit 10

# Higher thinking budget for complex analysis
ct run scorer-timeout abc123def456 --thinking-budget 20000

ct run scorer-timeout-view

Interactive viewer for scorer timeout analysis results.

Usage:

ct run scorer-timeout-view [RUN_PATH]

Arguments:

  • RUN_PATH - Path to timeout analysis run directory (optional)

Examples:

# Select from available runs
ct run scorer-timeout-view

# View specific run
ct run scorer-timeout-view data/scorer_timeout_runs/20240115_120000_abc123

Output

Results saved to data/scorer_timeout_runs/<timestamp>_<run_id>/:

  • results.jsonl - Analysis results for each timeout
  • config.json - Run configuration
  • Detailed analysis of each timeout with recommendations

Technical Details

  • Only analyzes trajectories where error.error_type == "scorer_timeout"
  • Uses prompts from prompts/system.md and prompts/user.md
  • Provides scorer code context and environment details to LLM
  • Tracks token usage per analysis