EC2 Sandbox (Remote Docker Execution)

The EC2 sandbox allows running evaluations on remote AWS EC2 instances instead of locally. This is useful when you need more compute resources or want to isolate eval execution from your local machine.

Two separate paths use EC2:

• Single-instance sandbox (this section): one instance per task, Docker commands routed over SSH. Used by commands like ct run mtgen (EC2 by default; pass --no-ec2 to run locally).
• Fleet mode: ct run eval --ec2 always runs in fleet mode — distributed workers via SQS, no SSH. See the Fleet Mode (Distributed Evaluation) section below.

How It Works

The EC2 sandbox extends the standard Docker sandbox by routing all Docker Compose commands through SSH to a remote EC2 instance:

1. Launches an EC2 instance (Ubuntu 22.04)
2. Installs Docker and docker-compose via SSH (EC2 Instance Connect)
3. Transfers the compose directory to the EC2 instance
4. Runs all docker compose commands on EC2 via SSH
5. Terminates the instance when the eval completes

Authentication

Two options; SSO is preferred because there are no long-lived secrets to distribute.

AWS SSO. You need an Identity Center assignment to the fleet account (326120904690) with the LinuxbenchFleet permission set — ask a fleet account admin (currently Henri Lemoine, henri@equistamp.com, or whoever holds fleet-admin; ask in #devops). Then:

aws configure sso   # start URL: ask in #devops; account 326120904690; role LinuxbenchFleet; profile name linuxbench-fleet
aws sso login --profile linuxbench-fleet

And in your .env:

AWS_PROFILE=linuxbench-fleet

Re-run aws sso login --profile linuxbench-fleet when the session expires. In-flight runs refresh credentials only while the SSO session is valid, so for runs expected to outlast it, log in fresh right before launching or use static keys.

Remove any AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY lines from .env when switching to AWS_PROFILE — env-var keys take precedence over the profile, so EC2 sandbox and fleet commands refuse to start when both are set.

Static IAM keys. Request a scoped fleet IAM key pair from a fleet account admin (see Granting Fleet Access for what they run; you can't mint these yourself), then put in .env:

AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...

Usage

# mtgen subcommands run on the EC2 sandbox by default
uv run ct run mtgen run-all -e <env> -s <side_task>

# Run locally instead
uv run ct run mtgen run-all -e <env> -s <side_task> --no-ec2
uv run ct run mtgen live-run --run-dir <run_dir> --no-ec2

# Keep instance running afterwards (for debugging)
uv run ct run mtgen run-all -e <env> -s <side_task> --keep-ec2-instance

What Gets Created

The EC2 sandbox automatically creates and manages:

• Security Group: linuxbench-eval-sg - Allows SSH (port 22) inbound
• EC2 Instance: Launched per task, terminated on completion (unless --keep-ec2-instance)

Timing Output

The EC2 sandbox logs timing for each setup phase:

Launching EC2 instance (c7i.4xlarge in us-east-1)...
   Instance ID: i-0abc123def456
   Public IP: 54.123.45.67
   Instance created in 2.3s
Waiting for instance to be ready and SSH available...
   Instance ready in 45.2s
Establishing SSH connection...
   SSH connected in 3.1s
Installing dependencies (Docker, docker-compose)...
   Dependencies installed in 67.4s
Transferring compose directory...
   Compose transferred in 1.2s
Initializing Docker sandbox (via SSH to EC2)...
   Docker sandbox initialized in 23.5s

EC2 Docker sandbox ready! (Total setup: 142.7s)

Troubleshooting

• SSH connection failures: The sandbox retries SSH connections up to 10 times with increasing delays. If it still fails, check your security group allows port 22 from your IP.
• Permission errors: Ensure your AWS credentials have the required EC2 permissions listed above.
• Instance limits: If you hit EC2 instance limits, request a limit increase or use a smaller instance type.
• Orphaned instances: If an eval crashes, instances may not be cleaned up. Check the AWS console and terminate any linuxbench-* instances manually.

Fleet Mode (Distributed Evaluation)

When --ec2 is used, the eval runs in fleet mode -- distributing work across multiple EC2 worker instances via SQS.

Fleet mode is a completely separate path from the single-instance EC2 sandbox above. Workers run Docker locally on themselves using the standard lb-docker sandbox. No SSH is involved.

How It Works

1. Your machine enqueues each sample as a message in SQS
2. Launches N EC2 workers from a pre-baked AMI (or stock Ubuntu as fallback)
3. Each worker grabs samples from the queue, runs them locally with Docker, uploads .eval results to S3
4. Workers post completion status to a done queue so your machine tracks progress
5. Your machine collects all results, merges into one run
6. Workers shutdown -h now when the queue is empty (instances auto-terminate)

Prerequisites

One-time setup:

1. S3 bucket: lb-fleet-results-migrated (stores .eval files, manifests, code artifacts)
2. IAM role + instance profile: linuxbench-worker with permissions for SQS (lb-fleet-*), S3 (lb-fleet-results-migrated)
3. API keys: Set ANTHROPIC_API_KEY and/or OPENAI_API_KEY in your .env file. These are passed automatically to workers at launch -- no separate secret management needed.
4. Docker registry auth for private env images: If environment images require auth, either:
   • Set DOCKER_ORG_NAME and DOCKER_ORG_PAT in your .env for direct pass-through to fleet workers and AMI builds, or
   • Store them in SSM under /linuxbench/docker-org-name and /linuxbench/docker-org-pat (or override with DOCKER_ORG_NAME_SSM_PARAM / DOCKER_ORG_PAT_SSM_PARAM). The helper script stores the PAT as SecureString.

To store the Docker auth from your current .env into SSM:

./scripts/put_docker_org_auth_ssm.sh

Usage

# Fleet mode on EC2
uv run ct run eval --ec2 --task-file medium --policy honest --epochs 10

# Scale up with more workers for faster runs
uv run ct run eval --ec2 --task-file medium --policy honest --epochs 10 --max-workers 20

# Cost estimate only (no launch)
uv run ct run eval --ec2 --task-file medium --policy honest --epochs 10 --estimate

# Skip confirmation prompt (CI-friendly)
uv run ct run eval --ec2 --task-file medium --policy honest --epochs 10 --yes

# Force a fresh fleet (skip auto-resume)
uv run ct run eval --ec2 --task-file medium --policy honest --epochs 10 --new-fleet

Selecting Which Settings Workers Pull

Workers don't have your local .settings/ directory — they run ct settings pull <target> themselves during bootstrap. By default they pull linuxarena-private, which is fine for any env that lives in that manifest entry.

If your eval uses an env from a different setting (e.g. a standalone repo like basharena), tell the worker which setting to pull by prefixing the -e flag:

# Single explicit setting
uv run ct run eval --ec2 -e basharena:basharena -s ...

# Multiple settings — repeat -e per env, deduplicates settings automatically
uv run ct run eval --ec2 \
    -e basharena:basharena \
    -e linuxarena-private:load_balancer \
    -s ...

# Bare form keeps working; falls back to the linuxarena-private default
uv run ct run eval --ec2 -e load_balancer ...

The setting:env form is parsed only by the fleet controller (--ec2); local runs ignore the prefix. Mixing bare and prefixed forms in the same invocation is rejected — it's ambiguous whether -e basharena:basharena -e web_scraping means "also pull linuxarena-private for web_scraping" or whether the user forgot to prefix it. Either prefix all -es or leave them all bare.

Shipping external code to workers (--extra-src)

Workers only get a git archive of the control-tower checkout, so an external --policy mypkg.foo:bar hits ModuleNotFoundError on the worker. --extra-src DIR ships DIR too as an import root (like PYTHONPATH=DIR):

uv run ct run eval --ec2 --extra-src ~/experiments --policy mypkg.policies:my_policy -e load_balancer -s ...

Repeatable; each dir needs a distinct basename. --ec2-only — local runs already resolve external refs via the CWD. Build artifacts, caches, VCS metadata (__pycache__, .git, node_modules, .venv/venv, build, dist), and lockfiles are not shipped, so a package whose real source lives in a directory named build or dist will be missing those modules on the worker.

External sandboxes on the fleet (--sandbox with --ec2)

--sandbox module:type runs on the fleet too: each worker imports the module from the shipped extra-src trees and runs the sandbox for its own jobs. A sandbox whose host side needs installing (a microVM runtime, a CLI login) names a script inside an extra-src tree with --worker-setup <basename>/<path>; the worker runs it as root after extracting extra-src and before taking jobs, with the venv at /opt/linuxbench/.venv and the worker's IAM role, so it can read its own SSM parameters. Environment the sandbox needs at run time goes in /opt/linuxbench-worker-setup.env, which the worker sources after the script. Pick the hardware with --ec2-instance-type (a microVM sandbox needs /dev/kvm, which only *.metal types expose). The setup script and instance type enter the fleet ID, so a run differing only in them does not resume another's results.

uv run ct run eval --ec2 --ec2-instance-type c5.metal \
  --extra-src ~/agent-glovebox --sandbox evals.control_tower.sbx_gb:sbx-gb \
  --worker-setup agent-glovebox/evals/control_tower/ct-worker-setup.sh -e linuxarena:file_indexing ...

Scaling

The main knob for throughput is --max-workers. Each worker runs 10 evals concurrently by default.

# Double throughput
uv run ct run eval --ec2 ... --max-workers 20

# Triple throughput
uv run ct run eval --ec2 ... --max-workers 30

Fleet Options

Resume

Fleet runs are automatically resumable. The fleet ID is deterministic (hashed from git SHA + policy + tasks + epochs). If you re-run the same command:

1. The controller detects the existing manifest in S3
2. Lists already-completed .eval files
3. Only enqueues remaining work
4. Launches fresh workers for the remaining jobs

This means controller crashes, credential expiry, or network issues are recoverable -- just re-run.

What Gets Created

• SQS queues: lb-fleet-{fleet_id}-work, lb-fleet-{fleet_id}-done, lb-fleet-{fleet_id}-dlq
• S3 objects: {fleet_id}/manifest.json, {fleet_id}/{job_id}.eval, artifacts/{sha}.tar.gz
• Security group: linuxbench-fleet-worker-sg (outbound only, no inbound)
• EC2 instances: Tagged with FleetId={fleet_id}, auto-terminate on shutdown

Failure Handling

Worker AMI

Fleet workers can boot from a pre-baked AMI that includes Docker, Python 3.13, uv, git, and pre-pulled Docker images for all environments. This cuts worker bootstrap from ~4 min to ~65s.

Building the AMI:

# Via CLI (incremental by default -- only pulls changed images)
uv run ct cloud build-ami

# Force full rebuild from stock Ubuntu
uv run ct cloud build-ami --fresh

# Build in a different region
uv run ct cloud build-ami --region us-west-2

# Or directly
./scripts/build_worker_ami.sh --region us-east-1 --fresh

This takes ~15-20 minutes (most time is EBS snapshot creation):

1. Launches a builder EC2 instance
2. Installs OS dependencies (Docker, Python 3.13, uv, git)
3. Pulls Docker images for all environments (~74 images)
4. Creates an AMI snapshot
5. Stores the AMI ID in SSM /linuxbench/worker-ami-id
6. Deregisters the previous AMI (only 1 AMI is kept)
7. Terminates the builder instance

What's baked in:

• Docker + docker-compose-plugin
• Python 3.13 + uv (installed for both ubuntu and root users)
• Git + AWS CLI
• Docker images for all environments
• Python wheel cache at /opt/uv-cache (speeds up uv sync at boot)

What's fetched at boot:

• Repo code (from S3 at pinned git SHA — HEAD when clean, or a tree SHA of the dirty worktree, so local-but-uncommitted changes still reach workers). Brand-new untracked files require git add to ship; the controller prints a warning at launch if any are present.
• Python dependencies (uv sync --frozen, using cached wheels)
• Docker registry credentials from SSM or direct env pass-through, if configured

When to rebuild:

• After environment Docker images are updated and pushed to registries
• After adding new environments
• Not needed for code-only changes (code is fetched from S3 at boot)

If no AMI exists in SSM, workers fall back to stock Ubuntu 22.04 and install everything at boot.

Granting Fleet Access (Admins)

Admins only — a teammate cannot self-serve either path. ops/docs/fleet-iam-policy.json is the single source of truth for the scoped policy; both flows below consume it.

SSO (preferred). Needs Identity Center admin in the org management account. One-time setup — enable the ABAC session tag and create the permission set:

INSTANCE_ARN=$(aws sso-admin list-instances --query 'Instances[0].InstanceArn' --output text)
STORE_ID=$(aws sso-admin list-instances --query 'Instances[0].IdentityStoreId' --output text)
aws sso-admin create-instance-access-control-attribute-configuration \
    --instance-arn "$INSTANCE_ARN" \
    --instance-access-control-attribute-configuration \
    'AccessControlAttributes=[{Key=LaunchedBy,Value={Source=["${path:userName}"]}}]'
PS_ARN=$(aws sso-admin create-permission-set --instance-arn "$INSTANCE_ARN" \
    --name LinuxbenchFleet --session-duration PT12H \
    --query 'PermissionSet.PermissionSetArn' --output text)
aws sso-admin put-inline-policy-to-permission-set --instance-arn "$INSTANCE_ARN" \
    --permission-set-arn "$PS_ARN" --inline-policy file://ops/docs/fleet-iam-policy.json

If the instance already has an ABAC configuration, use update-instance-access-control-attribute-configuration and merge the LaunchedBy attribute into the existing list. The mapping tags every SSO session with LaunchedBy=<userName>; the policy's *Sso statements match that tag, and the controller tags instances with the role session name, which Identity Center also sets to <userName> — so per-user scoping holds end-to-end.

Per-teammate assignment:

USER_ID=$(aws identitystore list-users --identity-store-id "$STORE_ID" \
    --filters AttributePath=UserName,AttributeValue=<userName> \
    --query 'Users[0].UserId' --output text)
aws sso-admin create-account-assignment --instance-arn "$INSTANCE_ARN" \
    --target-id 326120904690 --target-type AWS_ACCOUNT \
    --permission-set-arn "$PS_ARN" --principal-type USER --principal-id "$USER_ID"

Static IAM keys (fallback). Needs IAM admin on the fleet account (326120904690). Create an IAM user and attach the scoped linuxbench-fleet-access managed policy:

aws iam create-user --user-name teammate-name
aws iam attach-user-policy --user-name teammate-name \
    --policy-arn arn:aws:iam::326120904690:policy/linuxbench-fleet-access
aws iam create-access-key --user-name teammate-name

Send the teammate the resulting AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (no session token) for their .env. Attach as a managed policy, not inline (put-user-policy); the document exceeds IAM's 2 KB inline limit. Create it once from ops/docs/fleet-iam-policy.json if it doesn't exist yet.

Propagating policy changes. After editing ops/docs/fleet-iam-policy.json, re-run the put-inline-policy-to-permission-set command from the setup above, then:

aws sso-admin provision-permission-set --instance-arn "$INSTANCE_ARN" \
    --permission-set-arn "$PS_ARN" --target-type ALL_PROVISIONED_ACCOUNTS
aws iam create-policy-version \
    --policy-arn arn:aws:iam::326120904690:policy/linuxbench-fleet-access \
    --policy-document file://ops/docs/fleet-iam-policy.json --set-as-default

(create-policy-version fails once 5 versions exist — delete the oldest non-default version first.)

Use the scoped policy, NOT AdministratorAccess. It ensures:

• ec2:TerminateInstances only works on instances tagged LaunchedBy=<your identity> (IAM username, or SSO session name via the ABAC tag)
• ec2:RunInstances requires the LaunchedBy and ManagedBy tags
• SQS access is scoped to lb-fleet-* queues only
• S3 access is scoped to lb-fleet-results-migrated bucket only

Monitoring Fleet Resources

# Show your running instances and queues
uv run ct cloud status

# Show all control-tower resources (all users)
uv run ct cloud status --all-users

Debugging

• Bootstrap logs: Uploaded to s3://lb-fleet-results-migrated/{fleet_id}/logs/{instance_id}-bootstrap.log with timing at each step
• EC2 console: Link printed after launch for monitoring instances
• SSH: via EC2 Instance Connect (emergency debugging)
• Stale warning: Controller warns after 10 minutes with no progress
• Hard timeout: Controller terminates fleet after 3 hours with no progress