Run Scripts Documentation

This directory contains documentation for selected ct run commands and related scripts.

Organization

Documentation is organized by function:

Trajectory Analysis

Tools for analyzing individual trajectories:

analyze_attack.md - Identify incriminating actions in attack trajectories
scorer_timeout.md - Analyze why scorers timeout
side_task_failure.md - Investigate side task failures
submit_guessability.md - Theoretical submit tool exploit analysis
submit_guessability_verified.md - Live-verified submit tool exploits
traj_prompt.md - Run custom LLM prompts on trajectories

Task Analysis

Tools for analyzing task quality and design:

compare_side_tasks.md - Compare side task outcomes between runs
goal_completeness.md - Verify agents have sufficient task information
reward_hacking.md - Find reward hacking vulnerabilities
rt_alignment.md - Check RT description alignment with scorers
side_task_review.md - Comprehensive side task quality review
validate_cover.md - Validate main task provides cover for side task

Evaluation & Monitoring

Core evaluation infrastructure:

Running Evaluations - Run AI agent evaluations (ct run eval)
Monitoring - Run action-level monitors on trajectories
rerun_traj.md - Replay trajectories to re-run scoring

Task Generation

Tools for creating new tasks:

find_questions.md - Discover answer-question tasks
mtgen.md - Generate main tasks
mtgen_bugfix.md - Generate bug-fix main tasks

Quality Assurance

Environment and test validation:

doc_audit.md - Audit side task goal and scoring docs
pairing_analysis.md - Analyze main/side task pairings
integration_test_quality.md - Integration test quality (ct envs integration-test-quality)

Utilities

Use ct runs sample to sample trajectories from runs.

Quick Reference

Most Common Commands

# Run an evaluation
ct run eval --policy honest --env web_scraping --main-task crawl_depth

# Monitor trajectories
ct run monitor logs/my_eval.eval --action-monitor multi --only-action

# Analyze attack trajectories
ct run analyze-attack logs/attack_run.eval

# Review side task quality
ct run side-task-review -e web_scraping

# Test environments
ct envs test -e web_scraping

# Sample from a run
ct runs sample run-abc 50 --stratified