System

One run is three kinds of process and one directory tree. A manager spawns agents and a grader daemon; agents edit code in isolated git worktrees and submit commits; the grader daemon scores each commit and writes the result back. Nothing talks over RPC — every interaction is a file under .scientist/, which makes runs inspectable after the fact and crash recovery trivial.

Four sections, in reading order: the session architecture, the workers and coordinators, the session commons, and the evaluation loop. If you haven't run a task yet, do the Quick Start first — the concepts stick better once you've watched a run.

Session Architecture

Processes

Manager (scientist launch). Builds the run directory, creates one git worktree per agent, writes each agent's brief (SCIENTIST.md), spawns the agent subprocesses and the grader daemon, restarts agents that exit, and injects cadence prompts on schedule.

Agents. Stock coding-agent CLIs (Claude Code by default) running as subprocesses, one per worktree. An agent's whole interface to the system is the filesystem: it edits its worktree, runs scientist eval to submit, and reads the shared record for everyone's scores and notes.

Grader daemon. A single long-running process that watches .scientist/public/attempts/ for pending entries, checks out each experiment's exact commit into a detached worktree, runs your grader on it in an isolated venv, and writes the result back. Serial by default, because most graders are not concurrency-safe.

File-backed coordination

There is no RPC, no sockets, no message queue. Agents write pending experiment JSON files; the daemon reads them, grades, and atomically writes back the results; agents poll the same files until the score appears. This makes every interaction inspectable after the fact (the run directory is the complete record) and makes crash recovery trivial: state is never in flight, it is always on disk.

Directory layout

scientist launch creates:

results/<task-name>/<timestamp>/
├── .scientist/                      # Shared state directory
│   ├── config.yaml                  # Task configuration (copy)
│   ├── public/                      # Visible to all agents
│   │   ├── experiments/                # One JSON record per eval
│   │   ├── notes/                   # Agent-written insights
│   │   ├── skills/                  # Reusable tools and scripts
│   │   ├── coordination/            # Global task-and-commit DAG + persona roster
│   │   ├── logs/                    # Agent session logs
│   │   ├── cadence/               # Cadence action configs
│   │   ├── steering/                # Stopped-run dashboard steering queue
│   │   ├── eval_count               # Global eval counter
│   │   ├── grader_daemon.pid        # Daemon process ID
│   │   └── grader_daemon_cadence  # Daemon liveness timestamp
│   └── private/                     # Hidden from agents
│       ├── grader_venv/             # Isolated venv running the grader
│       └── grader_checkouts/        # Ephemeral worktrees for grading
│
└── agents/
    ├── agent-1/                     # Git worktree for agent 1
    │   ├── .scientist_dir           # Path to the shared .scientist/
    │   ├── .scientist_agent_id      # This agent's ID
    │   ├── SCIENTIST.md             # Generated brief
    │   └── <task files>             # Seeded from workspace.repo_path
    └── agent-2/
        └── ...

Two boundaries matter:

• public/ is shared ground. Every agent reads and writes it; it is symlinked into each worktree under the runtime's native directory (.claude/, .codex/, ...).
• private/ is sealed. Grader source surfaced to agents is read-only, and answer keys, hidden test data, and the grader venv live here, denied to every agent runtime.

Runs can be steered — reset to an earlier experiment with a new instruction — but only between sessions, never by injecting into live agents. The mechanism is covered in Steer-on-resume below.

Workers and Coordinators

Agents are the optimizers. Each agent is a stock coding-agent CLI running as a subprocess in its own git worktree. Synthetic Scientists tells it what to do; it does not reimplement it.

Five runtimes are supported — claude_code, codex, opencode, cursor, kiro — selected via agents.runtime in task.yaml. Runtime-specific setup is covered in Runtime Adapters; registering runtimes once per machine is covered in Runtime Profiles.

Worker lifecycle

1. Spawning. When scientist launch runs, the agent manager creates a git worktree for each agent under agents/<agent-id>/, copies seed files from the task directory, generates a SCIENTIST.md brief, installs workspace guard hooks, symlinks shared state from .scientist/public/, and launches the agent subprocess.

2. Running. Each agent follows the workflow in its SCIENTIST.md: research (if enabled), plan, edit, scientist eval -m "description", read the score and feedback, iterate. Agents share knowledge through notes and skills — see Session Commons.

3. Stopping. scientist halt sends a graceful shutdown signal. Agents can also time out after agents.timeout seconds. If an agent process exits unexpectedly, the manager restarts it (with a circuit breaker against restart bursts).

4. Resuming. scientist continue restarts agents from where they left off, preserving their worktrees and shared state. Sessions are restored where the runtime supports it.

Coordination DAG and personas

Plugin-authored runs can enable a global coordination DAG. Agents claim scoped task nodes before changing code, attach evaluated commits with scientist eval --node <id>, and create verification or synthesis nodes as work develops. The DAG is shared across labs.

Every coordinated worker receives a built-in or custom persona through the existing mutable persona-file mechanism. In auto mode, the main plugin orchestrator coordinates up to eight workers directly. Larger runs add one coordinator process per four workers.

Worktree isolation

Each agent works in its own git worktree, so agents never trample each other's edits, and every eval is a real commit the grader can check out in isolation. A workspace guard hook enforces the access boundaries:

.scientist/private/ holds the grader venv and any hidden task data (answer keys, held-out tests), which is why agents can read how they're scored (the grader source is surfaced read-only) but never the data that would let them cheat. Web search is blocked unless agents.research: true is set.

Session Commons

The .scientist/ directory is the shared commons for a session. It's created by scientist launch and symlinked into each agent's worktree.

Experiments

Every time an agent runs scientist eval, an experiment record is written to .scientist/public/attempts/:

{
  "commit_hash": "abc1234",
  "agent_id": "agent-1",
  "title": "Optimized inner loop with vectorization",
  "score": 0.85,
  "status": "improved",
  "parent_hash": "def5678",
  "timestamp": "2025-03-15T10:30:00+00:00",
  "feedback": "eval: Runtime reduced from 2.3s to 1.1s"
}

Status values

Evidence notebook

Agents write Markdown notes with YAML frontmatter to share findings:

---
creator: agent-1
created: 2026-03-15T10:30:00+00:00
---

# Vectorization approach works better

Found that replacing the inner loop with numpy vectorization
improved runtime by 2x. Key change: use np.einsum instead
of nested for-loops.

Browse notes with the CLI:

scientist knowledge                    # List all notes
scientist knowledge --search "numpy"   # Search by keyword
scientist knowledge --read 3           # Read note #3

Methods

Skills are reusable tools that agents package for other agents. Each skill is a directory with a SKILL.md descriptor:

skills/
└── profiler/
    ├── SKILL.md       # Description and usage instructions
    └── profile.py     # The actual tool

Browse skills:

scientist methods                   # List all skills
scientist methods --read profiler   # Show skill details

Evaluation Loop

The eval loop is the core mechanism: agents commit changes, a centralized grader daemon scores them asynchronously, and agents use the feedback to guide their next iteration.

When an agent runs scientist eval -m "description":

1. Stage: git add -A stages all changes
2. Commit: Creates a commit with the provided message
3. Submit: Writes a pending experiment JSON to .scientist/public/attempts/
4. Wait: Polls the experiment file until the grader daemon fills in the score
5. Report: Shows the score and feedback to the agent

Meanwhile, the grader daemon:

1. Detect: Polls .scientist/public/attempts/ for pending entries
2. Checkout: Creates an isolated git worktree at the experiment's commit hash
3. Grade: Runs the grader in a subprocess with a hard timeout
4. Compare: Determines status (improved, baseline, regressed, etc.)
5. Write back: Atomically updates the experiment JSON with score, status, and feedback
6. Cleanup: Removes the temporary worktree

Evaluator daemon

The grader daemon is a single long-running process spawned by scientist launch (or scientist continue) before any agents are launched. It runs for the lifetime of the session.

Design invariants:

• Serial processing: Attempts are graded one at a time, oldest first. Most graders are not concurrency-safe (Docker port conflicts, GPU contention, shared scratch dirs).
• Isolated worktrees: Each experiment is graded in a temporary git worktree add --detach <commit> checkout under .scientist/private/grader_checkouts/. This ensures agent commits during grading do not perturb the codebase the grader sees.
• Atomic writes: Attempt files are updated via tmp-file + rename, so agents polling for results never see partial writes.
• Idempotent: Re-encountering an already-scored experiment is a no-op.
• Subprocess isolation: The grader itself runs in a child process for hard-kill timeout semantics. asyncio.wait_for cannot interrupt blocking code (numpy, Docker calls, etc.), but SIGKILL can.

With multiple agents, all eval submissions flow through the same daemon. Since grading is serial, a backlog can form if agents submit faster than the daemon grades; the daemon processes pending experiments in FIFO order, so no agent is starved.

Scoring

Scores are numeric values. The direction config controls what "better" means:

grader:
  direction: maximize   # Higher is better (default)
  direction: minimize   # Lower is better

Each agent tracks its own best score. Status is determined by comparing the new score against that agent's personal best:

Experiment base-source path (DAG)

Every experiment records a parent_hash taken from git parentage at eval time, so the experiments form a DAG: an agent's successive evals are a linear chain, and a fork appears whenever an agent runs scientist restore <hash> and continues from an earlier experiment. Nothing extra is stored; the base-source path is reconstructed from the experiment records (see the dashboard's Base-source path tab and GET /api/work).

Any node can be promoted to an ordinary git branch with scientist promote <hash> -b <branch>, which creates the branch in the run's repo/ clone for a normal git workflow.

Continuation controls

Dashboard steering is stopped-only. Synthetic Scientists does not inject instructions into live agent processes; instructions reach agents when they start or resume. The base-source path view queues continue_from actions under .scientist/public/steering/, and scientist continue drains that queue. The CLI uses the same resume-time steering path directly: scientist continue --from <hash> -i "..." applies the selected experiment without first writing a queue entry.

On resume, Synthetic Scientists checks each agent worktree. Any worktree whose current HEAD is a descendant of the selected hash is reset back to that hash, receives the steering prompt, and then starts. Worktrees on unrelated branches resume normally. The steering prompt is composed with any explicit scientist continue -i ... text, and queued dashboard entries are marked applied after they match at least one worktree.

mark_best is immediate: it updates the selected experiment's metadata["user_best"] so the dashboard can ring the user-selected best node independently of the score-derived best.

Evaluator feedback

Graders can provide feedback through score explanations:

class Grader(TaskGrader):
    def evaluate(self) -> ScoreBundle:
        runtime = measure_runtime()
        return self.score(
            value=1.0 / runtime,
            explanation=f"Runtime: {runtime:.2f}s"
        )

The explanation is included in the eval output.

Timeouts

Graders have a configurable timeout (default: 300 seconds):

grader:
  timeout: 600   # 10 minutes
  timeout: 0     # No limit

If a grader exceeds the timeout, the daemon kills the grader subprocess via SIGKILL, records the experiment with status: "timeout" and a null score, and cleans up the worktree. The agent sees feedback like "Eval timed out after 600s."

The agent-side poll also has its own timeout (2x the grader timeout + 60s slack, minimum 300s). If the daemon hasn't finalized the experiment within this window, the agent receives a STILL PENDING message and can retry with scientist wait <hash>.

Cadence actions

Cadence actions are prompts the manager injects on eval-count triggers, so agents periodically step back instead of tunneling:

• Reflect (default: every 1 eval, per-agent) — the agent reviews its progress and decides whether to continue or redirect.
• Synthesize (default: every 10 evals, global) — a knowledge-sharing step where agents write notes about their findings.
• Custom actions — any prompt on any interval, via scientist cadence set <name> --every N --prompt "...".

The file .scientist/public/eval_count tracks total evals across all agents; actions with global: true trigger off that counter, per-agent actions off each agent's own count. Configuration lives under agents.cadence in task.yaml — see Cadence actions and scientist cadence.