Evaluation Design

Synthetic Scientists begins with an evaluation contract. The contract defines what may change, what must remain true, how progress is scored, and what evidence is required before a result is promoted.

Contract

A task contains three required parts:

task.yaml      objective, runtime topology, budgets, and evaluator settings
seed/          runnable base source
grader/        packaged evaluator with an isolated environment

Hidden labels, fixtures, and acceptance data live outside grader/ under a path declared by grader.private. Agents can read evaluator source while the private data remains inaccessible.

task:
  name: "Decode throughput"
  description: "Increase decode throughput while preserving output."

grader:
  entrypoint: "decode_eval.grader:Grader"
  setup: ["uv pip install -e ./grader"]
  direction: maximize
  private: ["verification_data"]

agents:
  count: 8

coordination:
  enabled: true
  mode: auto
  auto_personas: true
  seed: coordination.yaml

Evaluation patterns

Deterministic optimization

Use a programmatic evaluator when the result can be checked exactly:

• Combinatorial constructions.
• Schedulers and solvers.
• Compiler transformations.
• Formal proofs.
• Static cost models.

Apply validity checks before calculating the optimization score. An invalid result should receive no optimization credit.

Hidden-data evaluation

For predictive models, expose training and development data in seed/ and keep final labels under grader.private.

Record:

• Split hashes.
• Random seeds.
• Metric direction.
• Base-source score.
• Final hidden score.

Performance evaluation

Performance tasks need both correctness and measurement controls:

1. Build and correctness checks.
2. Warm-up procedure.
3. Fixed inputs and repetitions.
4. Hardware and software manifest.
5. Baseline and candidate measurements.
6. Independent confirmation.

Use paired measurements for hardware-sensitive comparisons.

Rubric evaluation

Open-ended artifacts can use a rubric evaluator when criteria are explicit and auditable. Keep the rubric static during a confirmation study, record the judge model and prompt, and limit feedback when detailed verdicts would expose private reference material.

Work graph

The optional coordination seed defines the initial task DAG before workers launch:

nodes:
  - id: measure
    title: Measure the base source
    type: task
    persona: performance-analyst

  - id: implement
    title: Implement the first measured improvement
    type: task
    persona: builder
    depends_on: [measure]

  - id: verify
    title: Verify the candidate
    type: verification
    persona: verifier
    depends_on: [implement]

Workers claim nodes with scientist work claim, attach evaluated commits with scientist eval --node, and complete the node after the assignment is finished.

Validation loop

scientist new my-task
cd my-task

# edit task.yaml, seed/, grader/, and optional coordination.yaml

scientist check .
scientist launch -c task.yaml

scientist check must return a sensible base score before additional workers are launched.

Verification

Before promotion:

• Inspect the selected diff.
• Re-run correctness checks.
• Reproduce the metric under registered conditions.
• Evaluate hidden or independent data.
• Record human interventions.
• Preserve failed experiments that affect interpretation.

Promote the verified commit:

scientist promote <hash> --branch verified-result

See Playbooks for evaluator APIs, rubric evaluators, runtime profiles, multi-lab sessions, and plugin orchestration.