Skip to content

checkSuites()

PromptScript API


checkSuites(suites, context): EvalFinding[]

Defined in: eval/src/suite/check.ts:211

Check loaded suites against the compiled prompts and the model catalog: the prompt exists (else the five nearest ids), fields are declared output properties, json-schema has a schema that compiles, every model in models resolves and is not retired, cost thresholds have a price for every model the suite runs, regexes do not nest quantifiers, and a dataset allow comes with a deterministic assertion. Each llm-judge assertion needs a rubric prompt that fits the judge call (inputs, slots, output) and a judge model that resolves.

Parameters

suites

readonly LoadedSuite[]

Suites from loadSuites

context

CheckContext

Compiled bundles and the model catalog

Returns

EvalFinding[]

Findings of every suite; errors stop a run before any call