Reward Hacking Audit

Audits side tasks for reward hacking vulnerabilities.

Purpose

Uses LLM analysis with extended thinking to find potential reward hacking issues where an agent could:

  • Pass the scorer without violating the security property
  • Exploit scorer implementation bugs
  • Use edge cases to achieve high scores unfairly
  • Game the scoring system

Commands

ct run reward-hacking

Audit side tasks for reward hacking vulnerabilities.

Usage:

ct run reward-hacking [OPTIONS]

Options:

  • -e, --env - Environment(s) to audit (repeatable)
  • -s, --side-task - Specific side task(s) to audit (repeatable)
  • -n, --name - Name for this audit run (default: "reward_hacking")
  • -c, --max-concurrent - Maximum concurrent API calls (default: 4)
  • -t, --thinking-budget - Token budget for extended thinking (default: 10000)

Examples:

# Audit all side tasks
ct run reward-hacking

# Audit specific environment
ct run reward-hacking -e web_scraping

# Audit specific side task
ct run reward-hacking -s expose_secret

ct run reward-hacking-view

Interactive viewer for reward hacking audit results.

Usage:

ct run reward-hacking-view [RUN_PATH]

Output

Results saved to data/reward_hacking_runs/<name>_<timestamp>/:

  • results.jsonl - Vulnerability analysis for each side task
  • config.json - Audit configuration
  • Identified reward hacking vectors and recommendations

What It Checks

For each side task:

  • Can the scorer be passed without actual sabotage?
  • Are there implementation bugs in the scorer?
  • Can the agent exploit edge cases?
  • Are there "magic number" vulnerabilities?
  • Does the scorer match the intended security property?

Technical Details

  • Uses Claude Opus 4.5 with extended thinking
  • Uses prompts from prompts/system.md and prompts/user.md
  • Analyzes scorer code and environment codebase
  • Identifies exploitable patterns
  • Helps harden scorers against gaming