Skip to content

EvalConfig

PromptScript API


Defined in: core/src/types/config.ts:403

Eval settings (eval in promptscript.yaml): the providers eval suites call and the limits of a run.

Properties

maxCases?

optional maxCases?: number

Defined in: core/src/types/config.ts:432

Largest number of dataset rows a suite runs.

As Type

integer

Minimum

1

Default

1000

maxOutputTokens?

optional maxOutputTokens?: number

Defined in: core/src/types/config.ts:425

Output token cap of a model call when the prompt sets no maxTokens.

As Type

integer

Minimum

1

Default

1024

providers?

optional providers?: Record<string, EvalProviderConfig>

Defined in: core/src/types/config.ts:411

Providers by name. The name matches the provider of the resolved model profile, such as anthropic or openai.

Example

providers:
anthropic: { kind: anthropic, apiKeyEnvVar: ANTHROPIC_API_KEY }

reportDir?

optional reportDir?: string

Defined in: core/src/types/config.ts:437

Directory for eval reports, relative to the project root.

Example

".promptscript/eval-reports"

timeoutMs?

optional timeoutMs?: number

Defined in: core/src/types/config.ts:418

Time limit of one model call in milliseconds, response body included.

As Type

integer

Minimum

1

Default

60000