Docent

Docent is an optional local trajectory annotation service plus a Control Tower CLI for managing collections, uploading evaluation data, and sharing collections with collaborators.

Local Service

Docent is not started by default. To include it:

docker compose --profile docent up -d

Docent is a third-party tool by Transluce for summarizing, clustering, and searching over agent transcripts. We've found it useful for analyzing trajectories, but it is not part of Control Tower. See the Docent repo and Docent docs for support and documentation.

Docent runs entirely locally with its own PostgreSQL database. Accounts are created on first sign-up and stored there.

Getting started: Go to http://localhost:3001 and create an account.

CLI alternative:

1. Sign up: curl -X POST http://localhost:8888/rest/signup -H 'Content-Type: application/json' -d '{"email":"you@example.com","password":"pass"}'
2. Log in: curl -X POST http://localhost:8888/rest/login -H 'Content-Type: application/json' -d '{"email":"you@example.com","password":"pass"}'
3. Create an API key and manage collections at http://localhost:3001

Local Environment Variables

Add these to your main .env:

DOCENT_SERVER_URL=http://localhost:8888
DOCENT_WEB_URL=http://localhost:3001
DOCENT_API_KEY=<your-api-key-after-signup>

For Docent's LLM features, copy .env.docent.example to .env.docent and add your API keys. Only that file is passed to the Docent containers:

cp .env.docent.example .env.docent
# Add your ANTHROPIC_API_KEY and/or OPENAI_API_KEY

Installation

The CLI is part of the Control Tower project. Make sure you have the required dependencies installed:

uv sync --extra docent

Quick Start

1. Connect to Docent: set DOCENT_API_KEY, DOCENT_SERVER_URL, and DOCENT_WEB_URL in your .env (see Local Environment Variables above)

2. Create a collection:

   uv run ct docent create --name "My Evals"

3. Upload evaluation logs:

   uv run ct docent upload logs/my_eval.eval

4. Add a collaborator:

   uv run ct docent collaborator-add colleague@example.com

Commands

All commands follow the pattern: uv run ct docent <command> [options]

Commands fail with a clear error if DOCENT_API_KEY, DOCENT_SERVER_URL, or DOCENT_WEB_URL is missing from your environment.

Collection Management

list

List all your collections.

uv run ct docent list

Options:

• --search TEXT - Filter collections by name (partial match)

Example:

uv run ct docent list --search "linux"

create

Create a new collection.

uv run ct docent create --name "Collection Name"

Options:

• --name TEXT - Collection name (required, will prompt if not provided)
• --description TEXT - Optional description

Example:

uv run ct docent create --name "Linux Evals" --description "Evaluation runs for Linux tasks"

delete

Delete a collection.

uv run ct docent delete [COLLECTION_ID]

Arguments:

• COLLECTION_ID - Optional. Will prompt to select if not provided.

Options:

• --force - Skip confirmation prompt

Examples:

# Interactive selection
uv run ct docent delete

# Direct deletion
uv run ct docent delete abc123 --force

Upload Evaluation Data

upload

Upload evaluation logs to a collection.

uv run ct docent upload PATH

Arguments:

• PATH - Path to a .eval file or directory containing .eval files

Options:

• --collection-name TEXT - Use existing collection by name (errors if no collection matches)
• --collection-id TEXT - Use existing collection by ID

Examples:

Upload a single file (will prompt for collection):

uv run ct docent upload logs/my_eval.eval

Upload all files in a directory:

uv run ct docent upload logs/

Upload to a specific collection by name:

uv run ct docent upload logs/my_eval.eval --collection-name "Linux Evals"

Upload to a specific collection by ID:

uv run ct docent upload logs/my_eval.eval --collection-id abc123

Sharing & Permissions

All sharing commands support both interactive and direct modes. If you don't provide a collection ID, you'll be prompted to select from your available collections.

collaborator-add

Add a collaborator to a collection by email.

uv run ct docent collaborator-add EMAIL [--collection-id ID]

Arguments:

• EMAIL - Email address of the user to add (must have a Docent account)

Options:

• --collection-id TEXT - Collection ID (will prompt if not provided)

Permission granted: read access

Examples:

# Interactive collection selection
uv run ct docent collaborator-add colleague@example.com

# Direct
uv run ct docent collaborator-add colleague@example.com --collection-id abc123

make-public

Make a collection publicly accessible (read-only).

uv run ct docent make-public [COLLECTION_ID]

Arguments:

• COLLECTION_ID - Optional. Will prompt to select if not provided.

Examples:

# Interactive selection
uv run ct docent make-public

# Direct
uv run ct docent make-public abc123

After making public, anyone with the link can view the collection at: <DOCENT_WEB_URL>/dashboard/<collection_id> (the command prints the URL)

make-private

Remove public access from a collection.

uv run ct docent make-private [COLLECTION_ID]

Arguments:

• COLLECTION_ID - Optional. Will prompt to select if not provided.

Examples:

# Interactive selection
uv run ct docent make-private

# Direct
uv run ct docent make-private abc123

collaborators

List all collaborators for a collection.

uv run ct docent collaborators [COLLECTION_ID]

Arguments:

• COLLECTION_ID - Optional. Will prompt to select if not provided.

Examples:

# Interactive selection
uv run ct docent collaborators

# Direct
uv run ct docent collaborators abc123

Output shows:

• Public access status
• User collaborators with their email and permission level
• Organization collaborators (if any)

collaborator-remove

Remove a user's access to a collection.

uv run ct docent collaborator-remove EMAIL [--collection-id ID]

Arguments:

• EMAIL - Email address of the collaborator to remove

Options:

• --collection-id TEXT - Collection ID (will prompt if not provided)

Examples:

# Interactive collection selection
uv run ct docent collaborator-remove colleague@example.com

# Direct
uv run ct docent collaborator-remove colleague@example.com --collection-id abc123

permissions

Check your permission level on a collection.

uv run ct docent permissions [COLLECTION_ID]

Arguments:

• COLLECTION_ID - Optional. Will prompt to select if not provided.

Permission levels:

• admin - Full control (can manage sharing, delete collection)
• write - Can modify collection contents
• read - Can view collection

Examples:

# Interactive selection
uv run ct docent permissions

# Direct
uv run ct docent permissions abc123

Interactive Collection Selection

For all commands that operate on a collection, if you don't provide a collection ID, you'll see an interactive prompt:

Select a collection:
  1. Linux Evals (ID: abc123) - Evaluation runs for Linux tasks
  2. Agent Tests (ID: def456)
  3. Production Runs (ID: ghi789) - Weekly production evaluations
Select option (1-3): 

This makes it easy to work with collections without memorizing IDs.

File Structure

src/control_tower/trajectories/docent/
├── collections.py       # Collection management (list, create, delete)
├── docent_connection.py # Connection management and configuration
├── parsers.py           # Input parsing and .eval file processing
├── prompts.py           # Interactive prompts for CLI
├── sharing.py           # Collection sharing and permissions
├── upload.py            # Eval log upload helpers
└── README.md            # Symlink to docs/docent.md (this file)

src/control_tower/cli/
└── docent.py            # CLI entry point and command definitions

Environment Variables

The CLI uses the following environment variables (set them in .env; there are no defaults):

• DOCENT_API_KEY - Your Docent API key
• DOCENT_SERVER_URL - Docent API URL (e.g., http://localhost:8888)
• DOCENT_WEB_URL - Docent Web UI URL (e.g., http://localhost:3001)

Error Handling

The CLI provides helpful error messages for common issues:

Not configured:

Missing required environment variables: ... Set DOCENT_API_KEY, DOCENT_SERVER_URL, and DOCENT_WEB_URL in your .env

User not found:

Error: User with email 'user@example.com' not found

Insufficient permissions:

Error: You don't have admin permission on this collection

Tips

1. Use collection names for upload: If you have a collection named "Linux Evals", you can use --collection-name "Linux Evals" instead of looking up the ID.

2. Scripting: For automation, always provide --collection-id to skip interactive prompts.

3. Batch uploads: Point upload at a directory to upload all .eval files at once.

4. Check before sharing: Use permissions to verify you have admin access before trying to share.

5. List collaborators: Use collaborators to see who already has access before sharing.