Skip to content

RunEvalOptions

PromptScript API


Defined in: eval/src/run.ts:58

Options of runEval.

Properties

allowedPatterns?

optional allowedPatterns?: readonly string[]

Defined in: eval/src/run.ts:87

validation.allowedPatterns of the project, for the dataset scan


budget?

optional budget?: object

Defined in: eval/src/run.ts:77

Spending limit in USD; required by every mode that calls a provider

maxCostUsd

maxCostUsd: number


bundles

bundles: readonly PromptBundleV1[]

Defined in: eval/src/run.ts:64

Prompt bundles the project compiles; each is verified again before use


cache

cache: object

Defined in: eval/src/run.ts:75

Cache mode and directory, relative to the project root

dir

dir: string

mode

mode: EvalCacheMode


catalog

catalog: ModelCatalog

Defined in: eval/src/run.ts:71

Model catalog with the project’s profiles


concurrency?

optional concurrency?: number

Defined in: eval/src/run.ts:81

Calls in flight at once; default 4, at most 16


maxCases?

optional maxCases?: number

Defined in: eval/src/run.ts:79

Row limit of a dataset; default 1000


maxOutputTokens?

optional maxOutputTokens?: number

Defined in: eval/src/run.ts:83

Output token cap when a prompt sets no maxTokens; default 1024


modelOverride?

optional modelOverride?: string

Defined in: eval/src/run.ts:85

Run every suite on this model only


onProgress?

optional onProgress?: (event) => void

Defined in: eval/src/run.ts:91

Receives warnings and finished cases; after it throws once it gets no more

Parameters

event

EvalProgress

Returns

void


projectRoot

projectRoot: string

Defined in: eval/src/run.ts:60

Project root directory


providers

providers: EvalProviderLookup

Defined in: eval/src/run.ts:73

Provider of a name; called only for calls the cache cannot answer


retryBaseMs?

optional retryBaseMs?: number

Defined in: eval/src/run.ts:93

First retry delay in milliseconds when a provider gives none; default 500


rubrics?

optional rubrics?: readonly PromptBundleV1[]

Defined in: eval/src/run.ts:69

Bundles judge rubrics are looked up in; defaults to bundles. A rubric is not the prompt under test, so it may come from another source than the tested prompt.


signal?

optional signal?: AbortSignal

Defined in: eval/src/run.ts:89

Stops new calls; finished cases stay in the report


suites

suites: readonly LoadedSuite[]

Defined in: eval/src/run.ts:62

Suites from loadSuites