Trajectories

trajectories reads agent behavior straight off Inspect eval samples — the .eval is the single source of truth.

A trajectory is the recorded behavior of one sample:

• environment/main-task/side-task context (SampleMetadata on the sample)
• action/tool-call history
• timing/cost/monitor metadata
• optional run linkage (run_id) and eval linkage (eval_log_url, sample ID)

This package handles:

• sample accessors (sample.py): committed actions, task success, monitor verdicts, costs, diagnostics
• the transcript walk (walk.py): committed actions, one walk shared by monitors, metrics, and thin docs
• score/error extraction (from_eval_log.py), per-call pricing and timing off inspect events (costs.py), tool-call helpers (tool_calls.py)
• viewer thin-doc assembly for uploads (mongo_docs.py)
• storage and retrieval of metadata+sample bundles (io.py, api.py, viewer_api.py)
• trajectory-focused CLI commands (cli.py)
• TUI support for interactive trajectory review/edit (in control_tower.live.tui)

CLI Overview

Top-level command group:

• ct traj ... (wired from control_tower.trajectories.cli:traj_cli)

Primary commands:

• ct traj list [--fs] [-e ENV] [-t MAIN_TASK] [-s SIDE_TASK] [--tag TAG] [--limit N]

  • Lists trajectories from the API (default) or local bundle files (--fs).

• ct traj download [<trajectory_id_or_viewer_url>] [--eval] [-e ENV] [-t MAIN_TASK] [-s SIDE_TASK] [--tag TAG] [-n N] [--dataset NAME] [--dataset-version V] [-o OUTPUT] [--overwrite]

  • Downloads one trajectory (by ID or viewer URL); bulk-downloads the latest N matching trajectories when filter options are passed instead of an ID (max 500, default 20); or downloads every trajectory in a dataset with --dataset NAME (no count limit).
  • --eval downloads the trajectory's full .eval archive instead of the bundle JSON.
  • --dataset fetches the dataset's full trajectory ID list, then downloads those trajectories by ID in batches, so the whole dataset arrives completely (no auth or Mongo URI required) instead of truncating on a single oversized stream. Add --dataset-version V to download a specific version (default: latest).
  • Accepts a trajectory ID from the configured API/MongoDB or a viewer trajectory URL (e.g. https://basharena-data.vercel.app/trajectories/<id>). See Downloading from the Viewer.

• ct traj patch <traj> ...

  • Generates a patch from a trajectory using live tooling.

• ct traj apply <eval_log_file> ...

  • Applies file edits from an eval log into an environment codebase.

• ct traj tui <trajectory_path> ...

  • Opens the trajectory TUI for interactive review/editing.

Cost accessors on a sample

Every cost is inspect's stamp on usage.total_cost. Sample totals come from sample.model_usage; per-call figures come from each ModelEvent. An Inspect response-cache read is left out of model_usage (actual spend) and marked on timing.model_calls[i].inspect_cache_hit. Provider prompt-cache discounts are already in the stamp.

• Per-call: costs.priced_model_calls(sample) prices every recorded model call from the cost inspect stamped on it; actual_cost_usd is $0 for an inspect response-cache replay.
• Agent total: sample.agent_cost(sample) is {model: usage.total_cost} from sample.model_usage, minus the calls that ran under role="monitor" (those are priced on their verdicts). A null amount is an unpriced model; the viewer keeps the known total and marks it $N+.
• Monitors: sample.monitor_cost(sample) -> Priced sums each verdict's cost_usd metadata (see Cost Tracking in function-registry.md). Ensemble-safe: walks members rather than double-counting the parent.
• Grand total: sample.total_cost(sample) -> Priced is agent + monitors, per model; sample.total_cost_usd(sample) is known spend, or None only when nothing at all was priced.

Cost tracking is automatic; a verdict with no cost_usd means nobody could price that call. A None in by_model is how the viewer tells the total is short and marks it $N+.

How trajectories fits with other CLI groups

• ct run ... runs experiments and can produce trajectories.
• ct runs ... manages run-level containers around groups of trajectories.
• ct traj ... is the direct trajectory manipulation and analysis surface.

Typical workflows

List trajectories from API:

• uv run ct traj list -n 20

Download one trajectory:

• uv run ct traj download <trajectory_id>
• uv run ct traj download https://basharena-data.vercel.app/trajectories/<id> (viewer URL)
• uv run ct traj download <trajectory_id> --eval (full .eval archive instead of bundle JSON)

Bulk-download the latest matching trajectories:

• uv run ct traj download -e web_scraping --tag baseline -n 100

Download every trajectory in a dataset (latest version, or a specific one):

• uv run ct traj download --dataset LaStraj
• uv run ct traj download --dataset LaStraj --dataset-version 2