checkSuites()
checkSuites(
suites,context):EvalFinding[]
Defined in: eval/src/suite/check.ts:211
Check loaded suites against the compiled prompts and the model catalog:
the prompt exists (else the five nearest ids), fields are declared output
properties, json-schema has a schema that compiles, every model in
models resolves and is not retired, cost thresholds have a price for
every model the suite runs, regexes do not nest quantifiers, and a dataset
allow comes with a deterministic assertion. Each llm-judge assertion
needs a rubric prompt that fits the judge call (inputs, slots, output) and a
judge model that resolves.
Parameters
suites
readonly LoadedSuite[]
Suites from loadSuites
context
Compiled bundles and the model catalog
Returns
Findings of every suite; errors stop a run before any call