EvalSuiteV1
Defined in: core/src/types/eval.ts:233
An eval suite file, version 1.
Properties
$schema?
optional$schema?:"https://getpromptscript.dev/schema/eval-suite/v1.json"
Defined in: core/src/types/eval.ts:238
Schema URL, the version tag of the suite.
Example
"https://getpromptscript.dev/schema/eval-suite/v1.json"assertions
assertions:
EvalAssertion[]
Defined in: core/src/types/eval.ts:264
Assertions every case must pass.
Min Items
1
Max Items
20
dataset
dataset:
string|EvalDatasetRef
Defined in: core/src/types/eval.ts:245
Dataset file relative to the suite file, or a reference with review decisions
judge?
optionaljudge?:EvalJudgeConfig
Defined in: core/src/types/eval.ts:258
Judge settings shared by the llm-judge assertions
labels?
optionallabels?:string
Defined in: core/src/types/eval.ts:250
Human label file, relative to the suite file.
Example
"labels/refund-reply.jsonl"models?
optionalmodels?: (string|EvalModelEntry)[]
Defined in: core/src/types/eval.ts:256
Models to run the whole dataset on; absent means the prompt’s own model.
Min Items
1
Max Items
8
prompt
prompt:
string
Defined in: core/src/types/eval.ts:243
Prompt id under test.
Example
"acme-support/refund-reply"thresholds?
optionalthresholds?:EvalThresholds
Defined in: core/src/types/eval.ts:266
Limits the run must meet