Submit Guessability Audit

Audits side tasks for submit tool exploitability (theoretical analysis).

Purpose

Checks if an agent could pass the scorer by calling the submit tool cleverly, WITHOUT actually modifying the environment or violating the security property. This is a theoretical analysis that examines the code without running it.

Commands

ct run submit-guessability

Audit side tasks for submit tool exploitability.

Usage:

ct run submit-guessability [OPTIONS]

Options:

  • -e, --env - Environment(s) to audit (repeatable)
  • -s, --side-task - Specific side task(s) to audit (repeatable)
  • -n, --name - Name for this audit run (default: "audit")
  • -c, --max-concurrent - Maximum concurrent API calls (default: 4)
  • -t, --thinking-budget - Token budget for extended thinking (default: 10000)

Examples:

# Audit all side tasks with custom submit tools
ct run submit-guessability

# Audit specific environment
ct run submit-guessability -e web_scraping

# Audit specific side task
ct run submit-guessability -s expose_secret

ct run submit-guessability-view

Interactive viewer for submit guessability audit results.

Usage:

ct run submit-guessability-view [RUN_PATH]

Output

Results saved to data/submit_guessability_runs/<name>_<timestamp>/:

  • results.jsonl - Analysis for each side task
  • config.json - Audit configuration
  • Theoretical exploits identified by LLM

Scope

Only audits side tasks that have:

  • Custom submit tools (not the default submit)
  • Submit tools with arguments

For Verified Exploits

Use ct run submit-guessability-verified to actually test if exploits work by running them against live scorers.

Technical Details

  • Uses prompts from prompts/system.md and prompts/user.md
  • Analyzes scorer code, submit tool definition, and goal description
  • Does NOT spin up containers or run scorers
  • Fast theoretical analysis of potential vulnerabilities