Skip to main content

Evaluator Specifications

Eval specs define reusable evaluator definitions.

Benchmarks are defined separately and reference evaluators through their evaluators field.

Required Fields​

  • id
  • version
  • name
  • description

Common Fields​

  • category
  • evaluator_type
  • pydantic_class
  • output_kind
  • optional requires, default_config, and source metadata

Notes​

  • Evaluators are referenced from benchmark specs and runtime workflows.
  • Prefer explicit id:version references.