Eval API
Eval suites, datasets and assertions for PromptScript prompts.
Loads suite files from .promptscript/evals/, reads and checks their
datasets, scans dataset values with the validator’s security rules and PII
detectors, and checks responses with deterministic assertions and with
llm-judge assertions that a rubric prompt scores, optionally against human labels.
Classes
Interfaces
- AssertionContext
- CheckContext
- CompareOptions
- DatasetCheckContext
- DatasetRecord
- EvalFinding
- EvalPlan
- EvalPlanModel
- EvalPlanSuite
- EvalProviderContext
- EvalProviderLookup
- EvalRequest
- EvalRun
- FormatOptions
- ImportResult
- ImportSamplesOptions
- JudgeVerdict
- LabelContext
- LoadedSuite
- LoadSuitesOptions
- LoadSuitesResult
- MockCall
- ProviderConfig
- ProviderDeps
- ProviderFactoryOptions
- ProviderUse
- ReadDatasetOptions
- ReadLabels
- RunEvalOptions
- ScanOptions
- ScrubCounts
Type Aliases
- DeterministicAssertion
- EvalCode
- EvalComparisonFormat
- EvalProgress
- EvalProvider
- EvalReportFormat
- EvalResponse
- ImportContract
- ImportRow
- MockAnswer
- MockProvider
- MockScript
- PriceSource
Variables
Functions
- checkDatasetRow
- checkSuites
- cleanText
- cleanValue
- compareReports
- createAnthropicProvider
- createMockProvider
- createOpenAICompatibleProvider
- createProviderFactory
- detectPii
- evaluateAssertions
- formatComparison
- formatPlan
- formatReport
- importSamples
- loadSuites
- missingFromCandidate
- parseSample
- planEval
- readDataset
- readLabels
- readReport
- readVerdict
- refuseProviderUrl
- reportFileName
- resolveProjectPath
- runEval
- runEvalDetailed
- scanDatasetRow
- scrubPii
- unmatchedLabels