Porting a Setting to Control Tower

How to bring an existing benchmark or environment into Control Tower as a native setting.

This is different from making-environments.md, which is about authoring a new environment from scratch. Porting starts from a setting that already exists somewhere else and reproduces it faithfully. If you are building something new, read that guide instead.

The one rule

Port thin. Reproduce the source faithfully, reuse Control Tower's own utilities, and keep the integration as small as it can be. A port is a translation, not a redesign: the setting should behave the way it does at the source, with the least Control Tower glue around it.

In practice that means preferring the smaller option at each step. Build the env config directly instead of wrapping it in a factory. Reach for an existing Control Tower utility before writing your own. Add configuration only where the source has it. Keep docs and tests proportionate to what the setting actually needs.

Fidelity comes before cleanup. If you (or a bot reviewer) spot a bug or bad practice in code you are copying verbatim, keep the logic unchanged so results stay reproducible — don't fix it in the port. Flag it instead (e.g. file a Linear issue), and if a clear bug is worth fixing, do it as a separate, clearly-versioned change so people can replicate either the original or the corrected behavior.

Layout

There is no required file layout. A setting can be a single setting.py that builds everything, which is usually the cleanest result. Split logic into more files only when it earns its keep, for example to stay close to the structure of the setting you are porting. Putting each piece (dataset loading, tasks, scorers, the submit tool) in its own file is not a goal in itself.

Every setting satisfies the same ControlEnvironment protocol (see env-architecture.md): an id, docs, a task_space of main/side tasks, and a sandbox spec. Think of a setting in those terms rather than sorting it into a fixed category — within the protocol, ports differ along independent axes that combine freely:

  • How tasks are defined — on disk (main_tasks/, side_tasks/) or built in code from a dataset or generator.
  • Which in-container tools the agent gets — declared by the env's base_tools set (the standard bash/text_editor/python). Unset means the default bash+text_editor; an empty frozenset() drops them, giving a custom-only agent (its custom_tools still flow) or, with no custom tools, a submit-only agent.
  • Whether and how Docker is used — a full agent sandbox, a minimal container just for scoring, or none at all (a local sandbox plus an LLM judge).

These do not co-vary: apps is dataset-backed and submit-only but still runs a Docker scoring sandbox; sae_interp_sabotage is dataset-backed, submit-only, and Docker-free; basharena_public vendors its tasks in code yet gives the agent in-container tools and per-task sandboxes.

Discovery

A packaged setting can register its get_environments() function in pyproject.toml:

[project.entry-points."control_tower.settings"]
my_setting = "ct_my_setting.setting:get_environments"

The entry-point name is the setting ID. An installed package is loaded directly. A pulled setting that declares control-tower as a dependency is isolated instead: Control Tower runs enumeration, validation, and eval execution in that project's own uv environment, so its package and third-party dependencies do not need to be installed in Control Tower's environment. Non-isolated source checkouts remain discoverable through a root setting.py or environment packages below .settings/<setting>/.

Reuse the framework

Before writing a utility, check whether Control Tower already ships it. A JSON disk cache (control_tower.utils.cache.Cache) and an LLM judge (control_tower.environments.llm_judge.llm_score) already exist, for example. If something broadly useful is missing from core, prefer adding it to Control Tower over keeping a private copy in the setting, so the next port can share it.

Keep docs thin

  • description.md (or an inline description string) is a concise description of the environment, not instructions for how to do the task. The task instructions come from the tasks.
  • A single-doc setting does not need the docs/ folder convention. Pass description inline on the env config, or keep just docs/description.md.
  • The attack surface belongs on the side task, as its attack_surface / scorer_description field, described in detail there. A thin port does not need a separate docs/attack_surface.md; in attack mode scoring depends on side_task_success (and, when the run passes --score-main-on-attack, the main-task scorer also runs on attack samples), so the side task is where that detail is useful.
  • Keep the README short. Avoid hardcoded paths, and avoid file-tree diagrams or option lists that drift out of date; point at description.md instead.

(The make_environment_config_from_path() helper reads docs/description.md and docs/attack_surface.md from disk, which is why the folder convention suits settings whose tasks live on disk. Ports that build tasks in code pass description inline instead, as apps does.)

Scorers

  • Register the main and side scorers separately (for example make_main_task_scorer() / make_side_task_scorer()); see making-scorers.md.
  • A scorer must test real behavior. Watch the dataset filter in particular: if it admits empty or mismatched test cases, scoring can run no tests and still report success, so make sure a pass means something actually ran.

Tasks and pairing

  • Build native MainTask / SideTask objects, and keep the source's prompts and task names.
  • env.task_space() is the single source of truth for valid (main, side) pairings:
    • Use ExplicitSpace when each side task is bound to one specific main (1-to-1 dataset rows, as in sae_interp_sabotage).
    • Use CrossProduct otherwise, and narrow it by declaring compatible_main_tasks / incompatible_main_tasks on the side task (see making-side-tasks.md) rather than enumerating pairs by hand.
  • A submit-only agent (no in-container tools) sets base_tools=frozenset() and defines no custom tools. A custom-only agent (its own tools but no standard bash/editor, as in shade_arena) sets base_tools=frozenset() and provides custom_tools.

Dependencies

A setting owns its own dependencies in its own pyproject.toml. Avoid adding a setting's third-party dependency to Control Tower's core pyproject.toml.

Validation

  • Be precise about what changed from the source. If a prompt or schema was adjusted, describe it as adapted rather than identical. Record these changes in the porting PR description for review, and capture material deviations in the setting's per-setting reference (e.g. its Linear entry) so they stay discoverable after merge.
  • Keep tests proportionate; a test that restates the implementation is not worth carrying (see the testing guidance in the repo guidelines).
  • Validate end to end with a real ct run eval before opening the PR, and include the honest and attack results.
  • Run the evals on both the source and the Control Tower port and confirm the results are aligned. Where the source only reports numbers, check that you can roughly reproduce them.

PR checklist

  • Bridge is thin and reuses Control Tower utilities.
  • description describes the environment, not the task.
  • Scorers test real behavior, with no vacuous success.
  • Pairing is correct (ExplicitSpace for 1-to-1, otherwise CrossProduct plus compatibility).
  • The setting owns its dependencies; Control Tower's pyproject.toml is unchanged.
  • Changes from the source are described accurately.
  • ct envs validate passes and the PR shows a real ct run eval.
  • Links the source.