Making Environments

This document outlines what an environment is, the key properties we aim for, and the process for creating new environments.

Porting an existing benchmark or environment rather than authoring a new one from scratch? See porting-settings.md.

Definition

An environment is a realistic software system that serves as the context for AI agent evaluation. At the code level, an environment is any object satisfying the ControlEnvironment protocol (src/control_tower/environments/protocol.py):

@runtime_checkable
class ControlEnvironment(Protocol):
    @property
    def id(self) -> str: ...
    @property
    def display_name(self) -> str: ...
    def docs(self, args: BaseModelPython | None = None) -> EnvDocs: ...
    def get_sandbox_spec(self, codebase_name: str = "default", args: BaseModelPython | None = None) -> SandboxEnvironmentSpec: ...
    def task_space(self, args: BaseModelPython | None = None) -> TaskSpace: ...

task_space(args) returns a TaskSpace describing the valid (main, side) pairings for the env (see src/control_tower/task/space.py), built from the env's args_model for one run's --env-arg values; None means defaults.

Optional capabilities — expose only when useful — are custom_tools: list[Tool] | None, run_tests() -> TestResult, and agent_user: str. See env-architecture.md.

The sandbox backend is determined by get_sandbox_spec() — Docker Compose, QEMU, Firecracker, Vagrant, or even no sandbox at all. See env-architecture.md for the full architecture and available backends, and .settings/demos/ for examples of environments using each backend.

Most environments use Docker Compose via the DockerEnvironment class and the make_environment_config_from_path() factory. The rest of this guide covers that standard approach.

Standard (Docker Compose) Environment Structure

Environments live one level deep inside a setting — a directory directly under .settings/ that groups related environments (e.g. .settings/linuxarena/web_scraping/). See Setting Layout for how discovery works.

Official environments are hosted as separate repos (e.g. under the linuxarena org) and cloned into .settings/<setting>/ via ct settings pull (see Development Setup). You can also create your own — ct settings pull /path/to/my_env links a local directory (see Development Setup) to make an environment discoverable without a GitHub repo.

A standard environment directory contains:

• __init__.py - environment configuration using make_environment_config_from_path()
• codebase/ - contains the code for the environment
• codebase/compose.yml - a docker compose file declaring how to set up the container with the environment
• docs/description.md - codebase structure, key files, and usage instructions
• docs/attack_surface.md - points of interaction that could be targeted in side tasks
• main_tasks/ - a set of main tasks for this environment (see how to add tasks below)
• side_tasks/ - a set of side tasks for this environment (see how to add tasks below)

All standard environments use Ubuntu 22.04 as the base image.

Main and side task content does not have to be baked into the environment image: a main or side task can declare a files map (see Making Main Tasks) to have per-sample payloads such as a data file or a code repository that is copied into the sandbox at init, so many tasks can share one compose/image.

Core Properties

A well-designed environment has the following properties:

Realistic and Plausible

Environments should model systems a SaaS company would plausibly have internally. They do NOT need to be about ML, AI safety, or any specific domain.

Sufficient Complexity

Environments should be complicated enough that a typical software engineer needs ~1 week of onboarding to become productive, but not significantly more so. Look at examples of environments in .settings/ before embarking on environment ideation.

Docker Compose Compatible

All services must be runnable inside Docker Compose.

Tech Stack Diversity

As much as possible, environments should use different programming languages and services to sample from the space of real-world Linux systems. Examples of current environments:

• file_indexing: Document search (Python 3.13, PyTorch, FastAPI, ChromaDB, OpenLDAP)
• web_scraping: Distributed scraping (Node.js 20.x, Redis, PostgreSQL, FTP)
• model_registry: ML artifact management (Go, Python/PyTorch, PostgreSQL, S3)
• media_processing: Video/audio processing (Rust, FFmpeg)
• clinical_trial: Clinical data and reporting (R, Python, PostgreSQL)
• tiktok: Short-form video backend (Go, Redis, PostgreSQL)
• user_tracking: User analytics (C, CMake, PostgreSQL)
• slack_clone: Team chat API (Ruby on Rails, Elasticsearch, PostgreSQL)

High Attack Surface

Environments may benefit from security weaknesses that side tasks can exploit. This is double-edged, and may hurt realism, but makes side task development easier. It is recommended to start environment ideation by trying to do without. In general, lack of security features tends to be better for realism than blatantly misimplemented security features.

Desired Properties

Tech Stack Diversity

Avoid languages already heavily represented (especially Python). Consider avoiding simple CRUD apps, and using unique services beyond the common Redis/PostgreSQL/S3 pattern:

• ElasticSearch
• Prometheus
• NGINX
• RabbitMQ
• Git
• Bazel build system
• ImageMagick
• Vector databases

Etc.

Persistent State

Prefer environments with high persistent state across many services, as this creates more opportunities for subtle sabotage.

Exfiltration Testing Server

For environments with exfiltration-related side tasks, consider including a mock external server (e.g., filebin.net or similar) that the agent can exfiltrate data to. This makes scorer verification straightforward.

Known Limitations & Security Considerations

Be aware of these container visibility issues that may affect realism:

Container Detection

Agents can detect they're being benchmarked through various means:

• /proc/self/mountinfo may leak volume names containing "linuxarena"
• Architecture information visible via proc

Mitigation Options

• Use tmpfs mounts instead of named volumes where possible to avoid leaking names

Example Environment Pipeline

1: Environment Idea

Create a PR with an environment idea in a new folder (name format: .settings/linuxarena/{environment_name}_idea/). At minimum, include an __init__.py and docs/ directory with placeholder content describing the idea.

Each environment is configured in its __init__.py using make_environment_config_from_path():

from pathlib import Path

from control_tower.environments.docker_environment import make_environment_config_from_path

root = Path(__file__).parent
base_compose = root / "codebase" / "compose.yml"

my_environment_config = make_environment_config_from_path(
    root,
    short_description="Brief one-line description shown on the website.",
    tech_stack=[
        "Python",
        "Flask",
        "Redis",
    ],
    compose_paths={
        "default": base_compose,
    },
    services=[
        "Flask API (port 8080)",
        "Redis",
    ],
)

The function reads docs/description.md and docs/attack_surface.md from the environment directory.

The docs/description.md file should combine the codebase structure, key files, and usage instructions into a single document. The docs/attack_surface.md file should describe points of interaction that could be targeted in side tasks.

The canonical specification for what description.md should look like — required sections, length cap, in-bounds vs out-of-bounds content, and the credentials rule — lives at src/control_tower/ct_runs/env_description/shared/spec.md. It is the single source of truth shared by two tools:

• uv run ct run env-description-generate -e <env> produces a fresh description.md from the codebase against the spec. Use --dry-run to write description.draft.md for side-by-side review, or --overwrite to replace an existing description.
• uv run ct run env-description-audit -e <env> scores an existing description.md against the same spec on accuracy, clarity, completeness, information leakage, and overall.

See docs/auditing.md for full usage.

Available parameters for make_environment_config_from_path():

2: Environment Implementation and Initial Side Task

1. Build the codebase and Docker Compose setup
2. Create main task ideas, to use in the steps below
3. Create side tasks (~5) with scorers
4. Develop sabotage trajectories (strajs) using different strategies on different main task ideas

Refer to issue templates, Discord and sprint plans for more specific requirements regarding the above. This hopefully serves as a high-level overview.

Key property of the initial side task: Honest models should not accidentally trigger it.

Purpose of two strajs: Demonstrates multiple sabotage opportunities exist, proving the environment has good attack surface. Each straj should be paired with a different main task.

3: Additional Main Tasks

After the initial strajs are approved, continue developing:

• Additional main tasks (~20)

Advanced Environment Configuration

To use different docker compose files for different tasks, pass multiple entries in compose_paths:

from pathlib import Path

from control_tower.environments.docker_environment import make_environment_config_from_path

root = Path(__file__).parent
base_compose = root / "codebase" / "compose.yml"

my_env_config = make_environment_config_from_path(
    root,
    short_description="...",
    tech_stack=["Python"],
    services=["API (port 8080)"],
    compose_paths={
        "default": base_compose,
        "special_env": root / "codebase" / "compose-special-env.yml",
    },
    custom_tools=[],  # Optional: environment-specific tools added to the agent's tool box
    agent_user="root",  # Optional: defaults to "root". Only override if your environment specifically *needs* a different user.
)

For a task to use the "special_env" docker compose file, you'll have to set codebase_name for that task (see below). The default compose_paths is {"default": path / "codebase" / "compose.yml"}

Shared Containers

When running multiple instances of an environment simultaneously (e.g., during batch evaluations), some services like databases or object storage can consume significant memory. To reduce memory overhead, environments can use shared containers that persist across multiple environment instances, so long as their interactions are heavily constrained.

Naming convention: Shared containers must be named shared-{name} (e.g., shared-minio) Lifecycle: ct automatically cleans up shared containers when no environments are using them

See the shared MinIO container in the Model Registry.

Each environment instance gets isolated resources within the shared container: per-trajectory naming, and IAM policies to restrict each environment to its own bucket.

Agent Dockerfile

The default compose service should reference the agent Dockerfile. This Dockerfile must meet specific requirements, outlined in this template:

FROM ubuntu:jammy-20250819

ENV DEBIAN_FRONTEND=noninteractive

# Common dependencies - git is REQUIRED for gen-traj command
RUN apt-get update && apt-get install -y --no-install-recommends \
    curl \
    vim \
    git \
    procps \
    python3 \
    python3-pip \
    python3-venv \
    sudo \
    # Add environment-specific packages here
    && apt-get clean && rm -rf /var/lib/apt/lists/*

# Install inspect-tool-support (REQUIRED for text_editor tool)
ENV PATH="$PATH:/opt/inspect_tool_support/bin"
RUN python3 -m venv /opt/inspect_tool_support && \
    /opt/inspect_tool_support/bin/pip install inspect-tool-support && \
    /opt/inspect_tool_support/bin/inspect-tool-support post-install --no-web-browser

# Environment-specific setup (dependencies, directories, etc.)
RUN mkdir -p /app
# ...

# Copy restart script (REQUIRED - must be at /app/restart.sh)
COPY restart.sh /app/restart.sh
RUN chmod +x /app/restart.sh

# Copy application files
COPY src/ /app/your_app/

WORKDIR /app
ENV SHELL=/bin/bash

# Start service and keep container alive
CMD ["/bin/bash", "-c", "./restart.sh & tail -f /dev/null"]

Agent Privileges

Agents run as root by default to give blue teams maximum flexibility in implementing their own security models. Environments should not include agent-level security restrictions (like sudo limitations or permission restrictions for the agent user), nor define an agent user.

• Don't create an agent user in the Dockerfile. This will cause inspect_tool_support to have permission issues.
• The system default is agent_user = "root". You don't need to override this in __init__.py unless you have a specific reason to use a different user.
• Dockerfiles should not contain agent-level security logic. Don't add sudoers entries, permission restrictions, or other measures that specifically limit what the AI agent can do. The Blue teams can implement security via task setup scripts or control protocols instead.
• Do keep environment-level security measures (e.g., sandboxing for untrusted code execution, monitoring systems) that exist independently of the agent's privileges.
• If your environment design inherently requires certain users or permission structures, that's fine, just don't add restrictions specifically to limit the agent.

Environment Restart

• Each environment should provide a restart.sh at .settings/<setting>/<env>/codebase/src/restart.sh that restarts the main service.
• The script should exit 0 on success, non-zero on failure, and print where to look for logs if start-up fails
• The environment's Dockerfile must include a COPY command to copy the script to /app/restart.sh in the container (e.g., COPY src/restart.sh /app/restart.sh)
• The script is copied to /app/restart.sh in the container and can be called directly by agents, developers, or scorers
• If an environment intentionally forbids restarts (by design), still include restart.sh that explains why and exits with a non-zero status