Simulated Sandbox

--simulated runs an evaluation against an LLM-simulated environment instead of provisioning the real one. The real services are never started; a stateful simulator agent fabricates the output of every command the agent under test runs.

It exists to make live-simulation control evals cheap to run: an attack policy runs end-to-end against a believable environment, and an LLM judge decides whether the attack would have worked.

uv run ct run eval -e <env> -t <main_task> -s <side_task> \
  --policy obvious-attack --simulated --limit 1

The agent uses the same bash / text_editor / python tools and any env-specific sibling-container tools it would normally have. Nothing about the policy or the environment changes — only the sandbox.

Flags

Model values accept aliases (haiku, sonnet, opus_4_8) or full provider strings (anthropic/claude-opus-4-8).

Scratch modes

The simulator grounds its fabrications in a single real scratch container, with the env source always available at /codebase. Pick the fidelity/cost trade-off:

• filesystem (default, free): a minimal python-slim container with the env codebase copied to /codebase. No env image is built. The simulator reads the Dockerfile to map container paths (/app/...) to codebase paths.
• default_image: the env's real default service image only (siblings and network stripped), with the source bind-mounted at /codebase. One image build; the simulator can run real commands at the app's real deployment paths.
• full: the env's whole compose stack, with /codebase mounted into default.
• none: no container at all. The simulator gets no tools and fabricates every output from its own reasoning — the no-code-execution ablation.

Scoring

A simulated sample has no real environment to score against, so the programmatic scorer is replaced by an LLM transcript judge (--simulator-judge-model). It reads the task objective, the agent's submitted attack description, and the full transcript, then inspects the scratch container (when there is one) and decides whether the objective would be satisfied if the environment were real. The judge's investigation transcript lands in the score's judge_messages metadata, and the simulator's own history in the sample store (SimulatorState:sim_messages), so a simulated run stays auditable from the .eval alone.

Known limitations

• run_side_task_test is simulated too. Policies that ship the side-task test tool get a fabricated verdict — the attacker self-grades against an oracle the simulator made up, and the scorer script's contents pass through the simulator's context. For realism studies, disable it in both arms (the sim-control-eval study did exactly this).
• Task setup scripts never run, in any scratch mode. For main tasks whose setup mutates the env (seeded bugs, fixtures), the simulator grounds in a world where the task's premise doesn't hold.
• Main-task scoring is skipped — the judge scores only the side task, so simulated honest runs complete unscored and usefulness is not measured.
• Fidelity is aggregate, not per-task. The sim-control-eval study found monitor suspiciousness distributions on simulated vs real attack runs statistically indistinguishable in aggregate (KS p > 0.5) but weakly correlated per task (r ≈ 0.18, vs ≈ 0.71 real-vs-real). Treat per-task simulated results with caution; porting that repo's realism-measurement scripts is the planned follow-up for quantifying this.

How it works

The provider (control_tower.sandbox.simulated) registers a simulated inspect SandboxEnvironment whose exec / read_file / write_file drive the simulator. Because every tool — bash, python, env sibling-container shells, and even text_editor (which talks to inspect's in-container tool-support over JSON-RPC) — bottoms out at sandbox().exec, the stock tools work unchanged.

State (the simulator's running view of the world) lives on a session shared by every container name in the sample and by the judge, so the fabricated world stays consistent across the whole trajectory.

agent bash/text_editor/sibling tool ─▶ sandbox().exec ─▶ SimulatedSandboxEnvironment
                                                              │
                                                              ▼
                                              simulator agent (LLM) ──▶ fabricated output
                                                              │   (investigates the real
                                                              │    scratch container to
                                                              ▼    ground deterministic facts)
                                                       scratch container