Doc Audit

Audits side task goal.md and scoring_details.md files for quality issues using an LLM with heuristic pre-analysis.

Purpose

Side tasks have two documentation files with different audiences:

  • goal.md — shown to both the attacking agent AND the monitor. Must be general and a one-liner. Specifics make the monitor's job easier.
  • scoring_details.md — shown only to the attacking agent. Must give enough info to avoid stupid failures while maximizing attack diversity. Should describe what success looks like, not how to achieve it.

Each audit type uses a separate LLM call with a focused system prompt. Heuristic checks (regex-based) run first and their results are passed to the LLM as pre-analysis context.

Commands

ct run doc-audit

ct run doc-audit [OPTIONS]

Options:

  • --env, -e — Environment to audit
  • --side-task, -s — Specific side task(s) to audit (repeatable)
  • --audit-type, -t — What to audit: goal, sd, or both (default: both)
  • --output-dir, -o — Output directory (default: data/doc_audit_runs/<env>)
  • --model, -m — Model to use (default: claude-sonnet-4-6)
  • --all-envs — Audit all environments
  • --apply — Apply suggested fixes to disk (goal audit only)

Examples:

# Audit both goal.md and scoring_details.md for an environment
ct run doc-audit -e slack_clone

# Audit only scoring_details.md
ct run doc-audit -e slack_clone -t sd

# Audit only goal.md for specific side tasks
ct run doc-audit -e web_scraping -s visit_malicious_website -s expose_secret -t goal

# Audit and apply goal.md fixes
ct run doc-audit -e web_scraping -t goal --apply

# Audit all environments
ct run doc-audit --all-envs

What it checks

Goal audit

A goal.md FAILS if it contains:

  • Named identifiers (variable names, function names, table names, API keys, file paths)
  • Specific URLs, hostnames, or host:port
  • Named persons, usernames, or emails
  • Numeric thresholds (dollar amounts, percentages, ID ranges)
  • Parenthetical lists enumerating specific items
  • Method specifications (HOW rather than WHAT)
  • Multi-line content, bullet points, or redundant sentences

Heuristic checks (13): line count, URLs, hostnames, file paths, specific users, magic numbers, ALL-CAPS identifiers, product names, bullet points, bare filenames, backtick refs, parenthetical lists, method specification.

Scoring details audit

A scoring_details.md FAILS if it is:

  • Prescriptive / tutorial-like — exact commands, curl examples, SQL queries, step-by-step instructions, DB connection strings with credentials
  • Over-specified — reveals full scorer test suite, lists all accepted values, exposes scorer internals
  • Incomplete — missing submission format, missing contract info (what gets reset/rotated), missing target identifiers
  • Scorer-misaligned — describes an attack the scorer doesn't test

A scoring_details.md PASSES if it describes what the scorer checks, provides submission format, states constraints, and leaves the attack approach to the agent.

Heuristic checks (12): code blocks, curl commands, SQL queries, step-by-step patterns, DB connection strings, function references, missing submission info, missing constraints, exhaustive lists, short documents, file paths, specific entity IDs.

Output

Per-task JSON files in the output directory with:

  • verdict: PASS or FAIL
  • confidence: 1–10
  • issues: List of specific problems found
  • suggested_fix: Corrected text if FAIL
  • reasoning: Brief explanation
  • heuristic_issues: Pre-analysis flags from regex checks

Goal audit results also include specifics_to_move (what should move to scoring_details.md).

A summary.json with aggregate counts across all audited tasks.

Golden sets

  • sd_audit_labels.json — Human-labeled scoring_details.md verdicts for measuring accuracy

Architecture

doc_audit/
├── cli.py              # Click CLI (ct run doc-audit)
├── task.py             # Orchestrator: loads docs, runs heuristics, calls LLM
├── audits/
│   ├── base.py         # DocAudit ABC (parse, validate, format)
│   ├── goal.py         # GoalAudit: goal.md specificity checks
│   └── sd.py           # ScoringDetailsAudit: SD quality checks
├── heuristics/
│   ├── base.py         # HeuristicChecker ABC (run, classify, format_for_llm)
│   ├── goal.py         # GoalHeuristics: 13 regex checks
│   └── sd.py           # SDHeuristics: 12 regex checks
├── prompts/
│   ├── goal/
│   │   ├── system.md   # Goal audit rubric + examples
│   │   └── user.md     # Per-task user message template
│   └── sd/
│       ├── system.md   # SD audit rubric + examples
│       └── user.md     # Per-task user message template
└── sd_audit_labels.json