EC2 Fleet Evaluations

See the generated ct run eval reference for the current EC2 and fleet options.

Worker AMI (Speed Optimization)

Fleet workers boot ~3x faster with a pre-baked AMI that includes Docker and pre-pulled Docker images:

uv run ct cloud build-ami  # ~15-20 min, one-time setup

Without AMI: ~230s bootstrap. With AMI: ~65s bootstrap. See EC2 Sandbox docs for details.

Fleet Streaming Logs

When you run --ec2, the controller prints only a compact carriage-returned progress line ([N/M] X done, …). To follow per-trajectory results live — from another shell or an LLM — tail logs/fleet/<fleet_id>/events.jsonl. Every line is one JSON object; types include:

  • fleet_start — fleet_id, expected jobs, worker count
  • worker_online — instance_id, bootstrap_s
  • result — job_id, task, model, per-scorer scores, error, duration_s, instance_id. Emitted the instant each sample finishes mid-batch via an Inspect AI hook (worker.py's FleetSampleStreamer) that posts a sample_done SQS message — not at batch-end. So with --concurrency-per-worker 10, you see 10 separate result events as each of the 10 concurrent samples lands, not one burst when the batch completes.
  • job_done — job_id, status, instance_id, duration_s, failure_category, error_summary. Emitted at batch-end from the SQS done queue (the worker's existing flow). Useful for operational debugging.
  • fleet_summary — final counts

The startup banner prints the exact path.

Recipes:

# Watch failures land:
jq -c 'select(.type=="job_done" and .status!="completed")' \
  logs/fleet/<fleet_id>/events.jsonl

# Live per-task scores as each trajectory finishes:
jq -c 'select(.type=="result") | {task, status, scores, duration_s}' \
  logs/fleet/<fleet_id>/events.jsonl

# Running pass-rate across completed trajectories:
jq -s '[.[] | select(.type=="result")] | {n: length, passed: ([.[] | select(.scores.linux_scorer.value.main_task_success=="C")] | length)}' \
  logs/fleet/<fleet_id>/events.jsonl

# Pull a failed .eval for inspection (workers also upload full .eval at batch-end):
aws s3 cp s3://<bucket>/<fleet_id>/<job_id>.eval /tmp/job.eval
uv run inspect view /tmp/job.eval