Evaluation Troubleshooting

Docker Address Pool Exhaustion

Stock Docker's default bridge address pool yields only ~30 networks. Each concurrent sample needs 1+ bridge networks (e.g. basharena uses 2: inner_net + outer_net), so running many samples at once — locally or on a manually provisioned box — exhausts the pool and docker network create starts failing with one of:

all predefined address pools have been fully subnetted
could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

(the message depends on whether the network request specifies an explicit subnet). Sandbox startup retries into this error and every subsequent sample fails the same way until concurrency drops or the pool is enlarged.

Fix: enlarge Docker's address pool by writing to /etc/docker/daemon.json:

{
  "default-address-pools": [
    {"base": "100.64.0.0/10", "size": 24}
  ]
}

Then restart Docker:

sudo systemctl restart docker   # Linux
# or restart Docker Desktop on macOS

This carves 100.64.0.0/10 into /24 subnets (16,384 networks, 254 hosts each) — far more than any host can run containers for, so the pool can never be the bottleneck again. Address space isn't consumed until a network is actually created, so an oversized pool is free.

Do not use 172.17.0.0/12 (Docker's own default range, sometimes suggested elsewhere) or 10.0.0.0/8 as the base on EC2 or other VPC-hosted machines: 172.17.0.0/12 is Docker's own default bridge range, and 10.0.0.0/8-style ranges commonly overlap a VPC's private subnet (e.g. the default VPC uses 172.31.0.0/16), which breaks routing to VPC resources. 100.64.0.0/10 (RFC 6598 shared address space) avoids both collisions.

EC2 fleet workers (uv run ct run eval --ec2) already apply this automatically — bootstrap.sh and the worker AMI write the same daemon.json before Docker starts (see PR #884). This section is for local development machines and manually provisioned boxes running ct run eval directly, which still run stock Docker.

Container Services Hang at Startup

Services like rsyslog, mysql, or xinetd hang or time out during container startup when the host's high nofile ulimit is inherited by containers. Add ulimits to the environment's compose.yml:

services:
  default:
    ulimits:
      nofile:
        soft: 1024
        hard: 4096