Running evals on Hawk

Submit evals to a shared Hawk cluster and watch them in the viewer, instead of running them only on your own machine.

0. Prerequisites

  • Ubuntu 24.04 (or similar), with Python 3.13+ and git.
  • uv, the Python project manager.
  • Optional: the GitHub CLI (gh), authenticated with gh auth login. ct settings pull (step 3) clones the public environment repos anonymously over https; gh is only needed for private repos (e.g. the linuxarena-private set).

1. Clone control-tower

git clone https://github.com/linuxarena/control-tower.git
cd control-tower

2. Install

uv sync --extra hawk

This installs control-tower plus the Hawk CLI into .venv. Confirm:

uv run ct --help
uv run hawk --help

3. Pull the environments

The env registry starts empty. Pull the public LinuxArena envs:

uv run ct settings pull linuxarena

This clones the 10 public envs.

4. Configure (.env)

Add these to the .env in the control-tower root (or create one if you skipped the quickstart):

export HAWK_API_URL=https://api.hawk.production.redwood-hawk.com
export CT_IMAGE_REGISTRY_PREFIX=linuxarena
export USE_OPENROUTER=1
export OPENROUTER_API_KEY=<your OpenRouter API key>

Models route through OpenRouter, so OPENROUTER_API_KEY is the only model credential you need. CT_IMAGE_REGISTRY_PREFIX=linuxarena points image pulls at the public Docker Hub org.

5. Log in to Hawk

source .env
uv run hawk login

This prints a login URL. Open it in a browser and sign in with the Hawk account email and password the organizers issued you. The browser redirects to a local callback and the login completes.

Logging in from a remote or headless machine

If control-tower runs on a box with no local browser (an SSH session to a cloud VM, for example), use the device-code flow instead of the default browser-callback flow:

uv run hawk login --device

This prints a code and a URL — open the URL on any device with a browser, enter the code, and the login completes on the headless machine once you confirm.

Confirm:

uv run hawk list eval-sets

6. Run an eval on Hawk

Here is an example command (you don't have to run this exact one). It submits the clinical_trial environment's suppress_warnings side task with the obvious-attack policy:

source .env
uv run ct run eval -e clinical_trial -s suppress_warnings \
  --untrusted-policy obvious-attack \
  --untrusted-policy-arg model=openrouter/anthropic/claude-haiku-4.5 \
  --protocol untrusted-agent \
  --main-tasks-with-no-scorers include \
  --hawk --hawk-no-wait --hawk-no-browser

The Hawk flags:

  • --hawk submits to the cluster instead of running locally.
  • --hawk-no-wait returns right after submitting instead of blocking.
  • --hawk-no-browser skips auto-opening the viewer.
  • model=... takes an OpenRouter slug (the openrouter/ prefix routes it through your key); any OpenRouter model works, e.g. openrouter/anthropic/claude-haiku-4.5.

It prints an eval-set ID and a viewer URL. Watch it:

uv run hawk status eval-set-<id>
uv run hawk logs -f eval-set-<id>

Open the viewer URL to see samples as they run.

Custom policies and protocols

Dotted-path selections from your own project work on --hawk with no extra flags: --untrusted-policy my_pkg.policies:my_attack resolves at submission, CT ships your package's source along with the submission (the serialized config is capped at ~900 KB), and the runner imports it; the same goes for dotted-path refs inside selection args, such as -bp action_monitors=my_pkg.monitors:my_monitor or an entry of a JSON monitor list. Filesystem .md monitor assets are not shipped, so on --hawk select monitors by packaged name or module:function. Your package's third-party dependencies must already exist in the runner environment (control-tower and its dependencies are installed; anything else is not).

7. Viewing your runs

Every ct run eval --hawk submission prints an eval-set ID and a viewer URL. Samples are viewable live at viewer.hawk.production.redwood-hawk.com (or your deployment's viewer host) — log in with the same account you used for hawk login.

Unless --hawk-no-wait was passed, CT blocks until the eval-set finishes; on success it automatically runs hawk download, converts the resulting .eval file(s) to CT trajectories, and uploads them (--no-upload skips both the conversion and the upload, leaving just the downloaded .eval files), while on failure it exits non-zero without downloading anything. With --hawk-no-wait, CT returns immediately after submission and prints the manual follow-up commands instead — nothing is downloaded or uploaded automatically.

To fetch the raw .eval files yourself at any point (finished or still running):

uv run hawk download eval-set-<id> -o <dir>

For headless-machine login, see Logging in from a remote or headless machine above.

8. Long runs and rate limits

Rate limits slow runs down; they don't kill them. Inspect retries rate-limited requests indefinitely with exponential backoff (3s initial, capped at 30 minutes per wait, honoring Retry-After). Backoff counts as waiting time, so it never consumes a sample's working limit — but it does consume --sample-time-limit wall clock, so avoid tight sample time limits on long runs and bound work with --token-limit / --message-limit instead.

Samples that error for other reasons (sandbox failures, non-retryable API errors, limit overruns) don't abort the run either: the runner pod finishes the remaining samples, then re-runs only the errored ones — up to 10 attempts, reusing completed samples. All of this happens cluster-side with no action from you.

Throughput is capped at 10 concurrent connections per provider by default. Raising max_connections in your policy's model config only helps if your API tier has headroom; more connections against the same rate limit just means more backoff.

For overnight runs, either leave the submitting terminal alive (in tmux or similar) so results are downloaded, converted, and uploaded automatically on completion, or pass --hawk-no-wait and run the printed hawk download command yourself later.

To check whether a run is stalled on rate limits, watch uv run hawk logs -f eval-set-<id> for retry messages; long-idle samples in the viewer with no new events are usually waiting out backoff. Errored samples appear in the downloaded .eval file with an error field and convert like any other sample, just without scores.

There is no rerun-only-failed-samples command: retries are automatic within a submission, and the remedy for samples that still end errored is a fresh submission scoped to those tasks.

Hawk controls retry, failure, and concurrency settings cluster-side. ct run eval --hawk rejects --retry-on-error, --no-fail-on-error, and --max-samples instead of silently ignoring them; remove those options or run locally when you need to control them.