RunEvalOptions
Defined in: eval/src/run.ts:58
Options of runEval.
Properties
allowedPatterns?
optionalallowedPatterns?: readonlystring[]
Defined in: eval/src/run.ts:87
validation.allowedPatterns of the project, for the dataset scan
budget?
optionalbudget?:object
Defined in: eval/src/run.ts:77
Spending limit in USD; required by every mode that calls a provider
maxCostUsd
maxCostUsd:
number
bundles
bundles: readonly
PromptBundleV1[]
Defined in: eval/src/run.ts:64
Prompt bundles the project compiles; each is verified again before use
cache
cache:
object
Defined in: eval/src/run.ts:75
Cache mode and directory, relative to the project root
dir
dir:
string
mode
mode:
EvalCacheMode
catalog
catalog:
ModelCatalog
Defined in: eval/src/run.ts:71
Model catalog with the project’s profiles
concurrency?
optionalconcurrency?:number
Defined in: eval/src/run.ts:81
Calls in flight at once; default 4, at most 16
maxCases?
optionalmaxCases?:number
Defined in: eval/src/run.ts:79
Row limit of a dataset; default 1000
maxOutputTokens?
optionalmaxOutputTokens?:number
Defined in: eval/src/run.ts:83
Output token cap when a prompt sets no maxTokens; default 1024
modelOverride?
optionalmodelOverride?:string
Defined in: eval/src/run.ts:85
Run every suite on this model only
onProgress?
optionalonProgress?: (event) =>void
Defined in: eval/src/run.ts:91
Receives warnings and finished cases; after it throws once it gets no more
Parameters
event
Returns
void
projectRoot
projectRoot:
string
Defined in: eval/src/run.ts:60
Project root directory
providers
providers:
EvalProviderLookup
Defined in: eval/src/run.ts:73
Provider of a name; called only for calls the cache cannot answer
retryBaseMs?
optionalretryBaseMs?:number
Defined in: eval/src/run.ts:93
First retry delay in milliseconds when a provider gives none; default 500
rubrics?
optionalrubrics?: readonlyPromptBundleV1[]
Defined in: eval/src/run.ts:69
Bundles judge rubrics are looked up in; defaults to bundles. A rubric is not the
prompt under test, so it may come from another source than the tested prompt.
signal?
optionalsignal?:AbortSignal
Defined in: eval/src/run.ts:89
Stops new calls; finished cases stay in the report
suites
suites: readonly
LoadedSuite[]
Defined in: eval/src/run.ts:62
Suites from loadSuites