Playbooks

Each guide on this page is self-contained. Pick by what you're trying to do:

Evaluator Authoring

Graders evaluate agent submissions and return scores. Synthetic Scientists loads a grader from a Python entrypoint string and runs it inside an isolated venv that the framework manages, so your grader's dependencies stay separate from Synthetic Scientists' own and from the agent's worktree.

For open-ended tasks where you'd rather have an LLM judge score against a rubric than write a programmatic evaluator, see Rubric Evaluators.

The fastest way to scaffold a new task, scientist new my-task, drops a task.yaml plus a packaged grader stub you can edit:

from scientist.grader import TaskGrader
from scientist.types import ScoreBundle

class Grader(TaskGrader):
    def evaluate(self) -> float | ScoreBundle:
        result = self.run_program("solution.py")
        return float(result.stdout.strip())

The grader ships as a small Python package so Synthetic Scientists can install it via uv pip install into the grader venv.

Layout

my-task/
├── task.yaml
├── seed/
│   └── solution.py
└── grader/
    ├── pyproject.toml
    └── src/my_task_grader/
        ├── __init__.py        # from .grader import Grader
        └── grader.py          # class Grader(TaskGrader): ...

pyproject.toml:

[project]
name = "my-task-grader"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = ["synthetic-scientists", "numpy"]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.build.targets.wheel]
packages = ["src/my_task_grader"]

Wire it up in task.yaml

grader:
  entrypoint: "my_task_grader.grader:Grader"
  setup:
    - "uv pip install -e ./grader"
  timeout: 300
  direction: maximize
  args:
    program_file: "solution.py"

When scientist launch (or scientist check) runs, it:

1. Creates .scientist/private/grader_venv/ via uv venv.
2. Installs the same Synthetic Scientists the host is running into that venv (editable, git, or pinned PyPI — whichever produced your install), so your package's synthetic-scientists dependency resolves to matching code.
3. Runs every shell command under grader.setup with VIRTUAL_ENV / PATH pointed at the grader venv.
4. Spawns a worker subprocess in that venv whenever an experiment needs grading and ships the result back as JSON.

The venv lives inside .scientist/private/, which is denied to agents by the worktree permission rules; your grader source stays hidden.

TaskGrader helpers

TaskGrader provides several helper methods.

run_program(filename, *args, timeout=300)

Run a file from the agent's codebase:

result = self.run_program("solution.py", "--input", "data.csv")
print(result.stdout)    # captured stdout
print(result.stderr)    # captured stderr
print(result.returncode)  # exit code

Hidden data files

Put answer keys, hidden test fixtures, expected outputs, and any data the agent must not see under grader.private in task.yaml. Synthetic Scientists copies those paths into .scientist/private/, which every agent runtime is denied read access to, and the grader reads them back via self.private_dir:

grader:
  private:
    - "taskdata"                    # copied to .scientist/private/taskdata
    - "answers/test_cases.json"     # individual files work too

from pathlib import Path

answers = Path(self.private_dir) / "answers" / "test_cases.json"
labels = Path(self.private_dir) / "taskdata" / "test_labels.npz"

score(value, explanation="")

Return a scored result with optional feedback:

return self.score(0.85, "Runtime: 1.2s (target: < 1.0s)")

fail(explanation="")

Return a failed evaluation (null score):

if result.returncode != 0:
    return self.fail(f"Program crashed: {result.stderr[:200]}")

Available context

Inside evaluate(), you have access to:

Handling errors

class Grader(TaskGrader):
    def evaluate(self) -> float | ScoreBundle:
        result = self.run_program("solution.py")

        if result.returncode != 0:
            return self.fail(f"Crashed: {result.stderr[:300]}")

        try:
            output = float(result.stdout.strip())
        except ValueError:
            return self.fail(f"Invalid output: {result.stdout[:100]}")

        if output < 0:
            return self.fail(f"Score must be non-negative, got {output}")

        return self.score(output, f"Score: {output:.4f}")

Score direction

By default, higher scores are better. For tasks where lower is better:

grader:
  direction: minimize

Evaluator timeout

grader:
  timeout: 600   # 10 minutes

If the grader exceeds this timeout, the worker subprocess is killed and the experiment is recorded as a failure with a "timed out" explanation.

Checking the evaluator

scientist check my-task

This builds the grader venv, runs grader.setup, then evaluates the seed code against your grader and prints the result. Fix any issues before running scientist launch.

Rubric Evaluators

Use a rubric evaluator when the output is an open-ended artifact and the acceptance criteria cannot be computed directly. Prefer programmatic checks for accuracy, runtime, pass rate, and other measurable quantities.

Static rubric

Define criteria before the session and apply the same rubric to every experiment.

grader:
  entrypoint: "artifact_eval.grader:Grader"
  setup: ["uv pip install -e ./grader"]
  direction: maximize
  args:
    files: ["report.md"]
    judge_runtime: claude_code
    judge_model: opus
    feedback_level: full
    rubrics:
      - name: Accuracy
        description: Every factual claim is supported by supplied evidence.
        weight: 2
      - name: Coverage
        description: Every required section is present.
        weight: 1

Return one Score per criterion and a weighted aggregate. Preserve the judge prompt, model, source files, and per-criterion verdicts.

Generated rubric

For exploratory work, an evaluator may generate criteria once from the objective before worker experiments begin. Freeze those criteria for the comparison. Rewriting the rubric during a ranking changes the measurement and invalidates direct score comparisons.

Verification

• Keep private references under grader.private.
• Name the required artifact in task.description.
• Record judge model and prompt versions.
• Use multiple independent judgments for claims sensitive to evaluator noise.
• Limit feedback when detailed verdicts expose private reference material.

Coordination and Labs

Multi-agent mode lets several agents work on the same task simultaneously, sharing discoveries through notes and skills.

agents:
  count: 4          # Spawn 4 agents
  model: sonnet     # All use the same model

Or override at launch:

scientist launch -c task.yaml agents.count=4

Mix-and-match runtimes

When you want to compare different agent runtimes (or different models within the same runtime) on the same task, use agents.assignments instead of agents.count. Each entry spawns its own group of agents with its own runtime / model / runtime_options:

agents:
  # agents.count is ignored when assignments is set.
  assignments:
    - runtime: claude_code
      model: opus
      count: 2
    - runtime: codex
      model: gpt-5.4
      count: 1
    - runtime: opencode
      model: openai/gpt-5
      count: 1

This spawns four agents (agent-1 through agent-4), each with its own runtime-native shared directory (.claude, .codex, .opencode, ...). Notes, skills, and the experiment ranking live in .scientist/public/ and are symlinked into every agent's shared directory, so all agents still see each other's work even though they're running different CLIs.

Empty fields on an assignment inherit from the top-level agents.* defaults (or from the runtime's default model when only runtime is set).

Mix-and-match runs require run.session=tmux or run.session=local; run.session=docker is single-runtime only.

Multi-lab runs

Multi-lab mode splits a multi-agent run into isolated labs. Agents on the same lab share experiments, notes, skills, cadence state, and personas. Agents on different labs cannot directly see each other's shared state from inside their worktrees, so each lab can explore a different region of the solution space without immediately converging on the same ideas.

agents:
  count: 4

labs:
  count: 2

Or override at launch:

scientist launch -c task.yaml agents.count=4 labs.count=2

labs.count must be no larger than the total agent count. With agents.assignments, the total is the sum of all assignment count values.

Synthetic Scientists partitions agents round-robin and prefixes IDs by birth lab. A four-agent, two-lab run starts as:

lab 0: 0-agent-1, 0-agent-2
lab 1: 1-agent-1, 1-agent-2

The prefix is the birth lab, not necessarily the current lab. If rotation later moves 1-agent-2 to lab 0, it keeps the ID 1-agent-2.

Lab-scoped CLI behavior

From inside an agent worktree, Synthetic Scientists reads .scientist_lab and scopes commands to that agent's current lab:

scientist overview          # current lab's agents and ranking
scientist results             # current lab's experiments
scientist knowledge --recent  # current lab's notes
scientist methods          # current lab's skills
scientist cadence       # current lab's cadence config

From outside an agent worktree, read-only commands aggregate across labs so operators can inspect the whole run. Mutating cadence commands must be run from an agent worktree so Synthetic Scientists knows which lab to update.

Rotation

Rotation is enabled by default when labs.count > 1. A rotation cycle selects strong agents from their current labs and moves up to rotation.max_per_cycle agents without worsening lab roster balance.

labs:
  count: 2
  rotation:
    every: 50                 # one cycle per 50 finalized real evals
    dest_weighting: score     # score | uniform | round_robin

When a worker rotates, its persona, per-agent cadence config, experiments, and eval logs move with it. Notes and skills stay on the source lab as lab-local knowledge. The agent's worktree symlinks and .scientist_lab breadcrumb are repointed to the destination lab.

All rotation options (rank_window, min_evals, max_per_cycle, ...) are documented in the configuration reference.

Full task.yaml example

A complete task.yaml that runs MNIST with four agents across three runtimes:

task:
  name: "MNIST"
  description: |
    Classify handwritten digits (0-9) from MNIST.

    Your program (`solution.py`) must define `run(train_path, test_path)` that
    returns a numpy int array of shape (10000,) with predicted labels.
  tips: |
    Metric is classification accuracy (higher is better, 0-1 scale).

grader:
  timeout: 300
  direction: maximize
  args:
    program_file: "solution.py"
    train_file: "data/train.npz"
    test_file: "data/test.npz"

agents:
  # Top-level fields act as defaults for any assignment that omits them.
  # `count` is IGNORED when `assignments` is set, total agents = sum of
  # assignment counts (here: 2 + 1 + 1 = 4).
  runtime: claude_code
  model: opus
  max_turns: 200

  assignments:
    # Two Claude Code agents on Opus (inherits runtime=claude_code from above)
    - model: opus
      count: 2

    # One Codex agent on gpt-5.4
    - runtime: codex
      model: gpt-5.4
      count: 1
      runtime_options:
        model_reasoning_effort: high

    # One OpenCode agent, model omitted, so it uses the runtime default
    - runtime: opencode
      count: 1

workspace:
  results_dir: "./results"
  repo_path: "./seed"

run:
  verbose: false
  ui: false
  session: tmux   # docker is not supported with assignments

Verbose mode prints the per-agent runtime / model assignment up front so you can confirm what's about to spawn:

[scientist] Agents:     4 (mix-and-match)
[scientist]   agent-1: runtime=claude_code  model=opus
[scientist]   agent-2: runtime=claude_code  model=opus
[scientist]   agent-3: runtime=codex        model=gpt-5.4
[scientist]   agent-4: runtime=opencode     model=openai/gpt-5

Work ownership

Shared experiments. All agents can see each other's eval scores and diffs. When agent-2 achieves a high score, agent-1 can inspect that experiment:

scientist results                    # See all experiments from all agents
scientist inspect <commit-hash>     # See the diff of a specific experiment

Notes. Agents write Markdown notes to share insights (scientist knowledge, stored in .scientist/public/notes/, visible to all agents).

Skills. Agents can package reusable tools as skills — directories with a SKILL.md and associated files (scientist methods).

Consolidation cadence. The default synthesize cadence (every 10 global evals) prompts agents to write notes about their findings, creating a shared knowledge base that new iterations draw from.

Reading sibling worktrees. Agents can read (but not write to) other agents' worktrees, so they can inspect successful approaches directly.

Monitoring coordinated sessions

scientist overview                 # Agent health overview
scientist results                    # Ranking across all agents
scientist results --agent agent-1    # Filter by agent
scientist dashboard                     # Web dashboard: per-agent score lines,
                                 # status, experiments, notes, logs

Runtime Adapters

Synthetic Scientists works with any coding agent that can run as a subprocess. Pick the runtime in task.yaml via agents.runtime. Claude Code is the default and needs no special config beyond an Anthropic API key.

OpenCode

OpenCode reads permissions and provider config from an opencode.json file. Place it in your task's seed/ directory so it gets copied into each agent's worktree.

{
  "$schema": "https://opencode.ai/config.json",
  "permission": {
    "external_directory": "allow",
    "question": "deny",
    "doom_loop": "allow",
    "bash": "allow",
    "edit": "allow",
    "read": "allow",
    "write": "allow",
    "webfetch": "deny",
    "websearch": "deny",
    "codesearch": "allow",
    "lsp": "allow",
    "skill": "allow"
  },
  "provider": {
    "claude": {
      "npm": "@ai-sdk/anthropic",
      "name": "claude",
      "options": {
        "baseURL": "http://localhost:4000/v1",
        "apiKey": "xxx"
      },
      "models": {
        "claude-opus-4-6": {
          "name": "claude-opus-4-6"
        }
      }
    }
  }
}

Key points:

• Set all permissions to "allow" except question, webfetch, websearch (set those to "deny") so the agent runs autonomously without interactive prompts.
• The provider section configures which model to use. When using the gateway, point baseURL at http://localhost:<gateway_port>/v1 and set apiKey to any placeholder value; the gateway handles authentication.
• Place opencode.json in your seed directory so it gets copied into each agent's worktree.

Task config:

agents:
  runtime: opencode
  model: claude/claude-opus-4-6  # must match a model defined in opencode.json

Cursor Agent

Install Cursor Agent and authenticate once:

curl -fsSL https://cursor.com/install | bash
cursor-agent login

Point your task at it:

agents:
  runtime: cursor          # alias for cursor_agent; "cursor-agent" also works
  model: auto              # or any model id supported by your Cursor plan

Synthetic Scientists spawns each agent as cursor-agent --print --output-format stream-json --force --workspace <wt> [--model] [--mode] [--resume] <prompt>. The --force flag is always passed (cursor-agent requires it for write tools in --print mode). The full task brief is written to AGENTS.md at the worktree root, and Synthetic Scientists also drops a .cursor/rules/scientist.mdc always-apply rule with short guardrails (use scientist eval, don't touch .scientist/private/, share via .cursor/notes/ and .cursor/skills/) so they survive context pressure.

Optional runtime_options:

agents:
  runtime: cursor
  runtime_options:
    command: /usr/local/bin/cursor-agent  # override binary path
    mode: plan                            # --mode plan|ask
    stream_partial_output: true           # --stream-partial-output

Note: Cursor Agent uses its own auth and does not route through the LiteLLM gateway. Login state lives in your Cursor account, not in Synthetic Scientists config.

Kiro

Install Kiro (kiro-cli) and authenticate via Kiro's setup. Then:

agents:
  runtime: kiro
  model: auto              # or any Kiro-supported model

Synthetic Scientists spawns each agent as kiro-cli chat <prompt> --no-interactive -a (the -a flag trusts all tools). Instructions are read from KIRO.md at the worktree root. Kiro does not currently expose a session id we can resume from, so restarted agents start fresh, but they still see all prior experiments and notes via the shared .kiro/ directory.

Like Cursor, Kiro uses its own auth and does not route through the LiteLLM gateway.

Runtime Profiles

A profile is a machine-local preset that bundles an agent runtime, its CLI command, a default model, runtime options, and an optional persona seed under a short name. Tasks reference a profile by name instead of repeating those details in every task.yaml.

This separates two concerns that the plain agents.runtime / agents.model fields conflate:

• Task / research topology: how many agents, which personas, which labs. This belongs in task.yaml and should stay portable across machines.
• Local machine setup: which CLIs are installed, where they live, which model defaults and runtime-specific options this user wants. This belongs in a user-level file and never has to be committed.

Profiles are an additive shorthand. They do not replace agents.runtime, agents.model, or agents.assignments; they expand into exactly those fields at config-load time, and the manager only ever consumes resolved agent specs. Existing task configs keep working unchanged.

Where profiles live

~/.config/scientist/profiles.yaml

The location honors $XDG_CONFIG_HOME (so it follows your XDG setup) and can be overridden wholesale with $SCIENTIST_PROFILES_CONFIG (mainly for testing).

default: claude-opus

agents:
  claude-opus:
    runtime: claude_code
    command: claude
    model: opus
    persona_file: ~/.config/scientist/personas/generalist.md

  codex-high:
    runtime: codex
    command: codex
    model: gpt-5.4
    runtime_options:
      model_reasoning_effort: high

Creating a profile

The fastest path is scientist configure with no subcommand; it scans PATH for every supported runtime CLI (claude, codex, cursor-agent, opencode, kiro-cli, pi) and offers an interactive numbered-selection wizard:

scientist configure
# Scanning PATH for agent runtimes (~/.config/scientist/profiles.yaml):
#
#   ✓ claude_code   claude        /opt/homebrew/bin/claude
#     codex         codex         not found
#   ✓ cursor_agent  cursor-agent  /Users/me/.local/bin/cursor-agent
#
# 2 detected runtime(s):
#
#   [1] claude_code   (claude, model sonnet)
#   [2] cursor_agent  (cursor-agent, model auto)
#
# Select runtimes to bind [1-2, comma/space-separated, 'all', or Enter to skip]: 1

For each pick the wizard asks for profile name, model, and an optional persona seed file (press Enter to skip). After the profile is created you'll be asked "Add another profile for X? [y/N]" — say yes to create a second profile for the same runtime, e.g. claude-opus and claude-sonnet both pointing at claude_code with different models.

Pass --non-interactive to just print the detection report (good for CI or piping to a file). For more control over a single profile, use scientist configure profile:

scientist configure profile --name claude-opus --runtime claude_code --model opus
scientist configure profile --name codex-high  --runtime codex \
  --option model_reasoning_effort=high

Each scientist configure profile finishes with a lightweight doctor pass (see below). The first profile you create becomes the default; pass --default to change it.

Inspecting and managing profiles

scientist profiles list                       # numbered list (default is marked)
scientist profiles show claude-opus           # one profile's resolved fields
scientist profiles verify                     # validate every profile
scientist profiles verify claude-opus         # validate one
scientist profiles remove                     # interactive numbered-selection wizard
scientist profiles remove claude-opus codex-high   # delete one or more by name

scientist profiles verify runs five checks per profile:

• the profile resolves to a valid agent spec,
• the CLI is found on PATH (or at the configured command path),
• the CLI's --version works,
• the persona file exists, if one is set, and
• (by default) a live hello-ping: spawns the runtime CLI with a one-word prompt and waits for a reply.

The live ping catches the failure modes the cheap checks miss: expired auth, broken provider credentials, network issues, model name typos. It costs one LLM round-trip per profile, so pass --no-live to skip it in CI or quick sanity checks, and --timeout SECS to adjust the per-ping wait (default 30s). Authentication is still owned by the runtime-native login flow — when the ping fails, doctor points you at it rather than asking for credentials.

Using a profile in a task

Single runtime:

agents:
  profile: claude-opus
  count: 4

Mixed team, one profile per assignment:

agents:
  assignments:
    - profile: investigator
      count: 1
    - profile: implementer
      count: 3

Precedence

When a profile is expanded, fields resolve in this order:

1. Explicit task fields win. Anything you write in task.yaml overrides the profile.
2. Profile fields win over runtime defaults.
3. Runtime defaults fill any remaining gaps (e.g. the default model for a runtime).

So this uses the claude-opus profile's runtime and options but overrides the model:

agents:
  profile: claude-opus
  model: sonnet   # wins over the profile's opus

Persona seeds

A profile may carry an initial persona_file. At expansion time it compiles into runtime_options.persona_file, which Synthetic Scientists copies into the run at agent setup. This is idempotent on continue: an agent's updated persona file is never overwritten, so a profile's persona seed only ever applies to a fresh agent. A task that sets its own runtime_options.persona_file keeps it; the profile seed is only used as a fallback.

How it fits together

agents.profile (and assignments[].profile) is resolved during config normalization, before agent specs are built. After expansion the profile key is gone; the run's stored config.yaml contains only concrete runtime / model / runtime_options fields, so resumes and the dashboard never depend on your local profiles file.

LiteLLM Gateway

Synthetic Scientists includes a built-in LiteLLM gateway that acts as a proxy between agents and model providers. This is useful when you want to:

• Route agent requests through a single proxy with unified API key management
• Use custom or self-hosted models
• Add request logging and per-agent tracking
• Use providers that require non-standard authentication

Setup

1. Create a LiteLLM config file (e.g. litellm_config.yaml) alongside your task.yaml:

model_list:
  - model_name: "claude-opus-4-6"
    litellm_params:
      model: "anthropic/claude-opus-4-6"
      api_key: "YOUR_ANTHROPIC_API_KEY"

litellm_settings:
  drop_params: true

Each entry in model_list defines a model the gateway will serve. The model_name is what agents request; litellm_params.model is the upstream provider model. See the LiteLLM docs for full configuration options (multiple providers, load balancing, fallbacks, etc.).

MiniMax routes

The generated configuration supports MiniMax-M3 and MiniMax-M2.7. Each model ID includes both global and China OpenAI-compatible routes. Use the corresponding -anthropic alias for Anthropic-compatible routes. All generated routes use MINIMAX_API_KEY:

2. Enable the gateway in your task config:

agents:
  runtime: opencode           # or claude_code, codex (cursor and kiro use their own auth)
  model: claude/claude-opus-4-6
  gateway:
    enabled: true
    port: 4000                # port the gateway listens on
    config: "./litellm_config.yaml"  # path relative to task.yaml

3. Point your agent at the gateway. For OpenCode, set baseURL in opencode.json to http://localhost:<port>/v1 (see OpenCode above). For Claude Code, the gateway URL is automatically injected.

When you run scientist launch, the gateway starts before agents are spawned, and all agent API requests are routed through it. The gateway automatically assigns each agent a unique proxy key for per-agent request tracking.

Host Plugin

Inside Claude Code, Codex, or OpenCode the agent can already run scientist via Bash. The plugin provides a complete orchestration skill—from user intent through evaluator design, coordination DAG, personas, verification, and delivery—plus focused task-authoring and run-management skills.

It is skills-first and multi-harness, with no MCP; the capability is text guidance plus a Bash call. The plugin lives in the repo under plugin/: one shared skills/ directory, per-harness manifests (.claude-plugin/, .codex-plugin/), and per-harness hooks/ configs.

Skills

The in-run eval loop (scientist eval) is intentionally not a skill: every in-run agent already reads it from the generated SCIENTIST.md, so a skill would duplicate it.

These are for people in their own harness who want to drive Synthetic Scientists. They are distinct from the repo's contributor skills, which target people editing Synthetic Scientists itself.

Install: Claude Code

/plugin marketplace add synthetic-sciences/scientist
/plugin install scientist@scientist-marketplace

Or from a local checkout: /plugin marketplace add . then the same install. On session start the hook checks scientist is on PATH and injects a short context block — an install hint if it's missing, which-skill-for-what if it's present. Validate the manifest locally with claude plugin validate ./plugin.

Install: Codex

Codex (v0.117.0+) has a git-backed plugin marketplace, mirroring Claude Code. The repo ships a Codex marketplace at .agents/plugins/marketplace.json:

codex plugin marketplace add synthetic-sciences/scientist
codex plugin add scientist@scientist-marketplace

The plugin's .codex-plugin/plugin.json wires the shared skills/ and the Codex SessionStart hook. Invoke skills with $scientist-bootstrap or let Codex match by description.

As a lighter alternative without the marketplace, Codex also discovers skills from filesystem dirs (it follows symlinks): ln -s /abs/path/to/scientist/plugin/skills/* ~/.agents/skills/. That route skips the hook — paste plugin/AGENTS.md into your AGENTS.md instead.

Install: OpenCode

OpenCode discovers skills from .opencode/skills/ (project) and ~/.config/opencode/skills/ (global), and follows symlinks. The plugin ships an installer that links the shared skills in:

plugin/install-opencode.sh            # global: ~/.config/opencode/skills/
plugin/install-opencode.sh --project  # this repo only: .opencode/skills/

OpenCode also scans .claude/skills/ and .agents/skills/ for compatibility, so if you already installed the plugin for Claude Code or symlinked skills for Codex, OpenCode picks the same skills up automatically. OpenCode has no SessionStart hook wiring here — paste plugin/AGENTS.md into your project's AGENTS.md for the same context.

Other harnesses

Cursor and Kimi follow the same shared-skills/ + per-harness-manifest layout; the skill text is harness-agnostic. Add a manifest pointing at ./skills/ as support grows; no skill content changes needed.

Live Interface

Synthetic Scientists includes a live interface for research sessions.

scientist dashboard

Opens http://127.0.0.1:8420 in your browser. Customize the port:

scientist dashboard --port 9000
scientist dashboard --no-open          # Don't auto-open browser

You can also launch it alongside scientist launch:

scientist launch -c task.yaml run.ui=true

Interface tabs

Pulse. Session health, score progression, ranking, worker status, recent evidence, and methods.

Work. The task-and-commit DAG:

• Each node is one experiment; edges follow parent_hash (git parentage), so forks created with scientist restore <hash> show up as branches in the tree.
• Nodes are colored by status and labeled with the short hash and score; the best experiment is ringed.
• Click a node to see its details and a ready-to-copy scientist promote command for turning that experiment into a normal git branch.
• When the run is stopped, click Continue from here to queue a fork from that experiment. The queued action applies on the next scientist continue.
• Click Mark as best to set the user-selected best experiment.

Backed by GET /api/work, which combines experiment and coordination nodes across labs. Control actions go through POST /api/control; writes are rejected while a manager is alive with 409 stop the run to steer — use scientist halt, queue the action, then scientist continue. The same behavior is available from the CLI with scientist continue --from <hash> -i "...".

Activity. Live worker session output.

Evidence. Browse the lab notebook, evidence graph, and reusable methods.

Backend

The dashboard uses Starlette (Python ASGI) with Server-Sent Events for live updates, and a React + Vite frontend. The backend reads directly from the .scientist/ directory, so the dashboard always reflects the latest state.

Targeting a specific session

By default, scientist dashboard selects the latest active session. Target a specific session:

scientist dashboard --task my-task --run 2025-03-15_10-30-00

Benchmarks

Official adapters live under benchmarks/ only after the corresponding result has been reproduced and documented. Each package contains a base source, evaluator, coordination seed, protocol, result manifest, and license notes.

scientist check benchmarks/<name>
scientist launch -c benchmarks/<name>/task.yaml

See Evaluation Design for the package contract.

Harbor-based suites

Benchmark adapters may use Harbor to run instances in isolated Docker containers.

Harbor starts Docker containers. Run Synthetic Scientists on the host for these suites because Docker-in-Docker is unsupported.