Skip to content

EvalSuiteV1

PromptScript API


Defined in: core/src/types/eval.ts:233

An eval suite file, version 1.

Properties

$schema?

optional $schema?: "https://getpromptscript.dev/schema/eval-suite/v1.json"

Defined in: core/src/types/eval.ts:238

Schema URL, the version tag of the suite.

Example

"https://getpromptscript.dev/schema/eval-suite/v1.json"

assertions

assertions: EvalAssertion[]

Defined in: core/src/types/eval.ts:264

Assertions every case must pass.

Min Items

1

Max Items

20


dataset

dataset: string | EvalDatasetRef

Defined in: core/src/types/eval.ts:245

Dataset file relative to the suite file, or a reference with review decisions


judge?

optional judge?: EvalJudgeConfig

Defined in: core/src/types/eval.ts:258

Judge settings shared by the llm-judge assertions


labels?

optional labels?: string

Defined in: core/src/types/eval.ts:250

Human label file, relative to the suite file.

Example

"labels/refund-reply.jsonl"

models?

optional models?: (string | EvalModelEntry)[]

Defined in: core/src/types/eval.ts:256

Models to run the whole dataset on; absent means the prompt’s own model.

Min Items

1

Max Items

8


prompt

prompt: string

Defined in: core/src/types/eval.ts:243

Prompt id under test.

Example

"acme-support/refund-reply"

thresholds?

optional thresholds?: EvalThresholds

Defined in: core/src/types/eval.ts:266

Limits the run must meet