Skip to content

EVAL_CODES

PromptScript API


const EVAL_CODES: object

Defined in: eval/src/errors.ts:8

Codes of eval errors and findings: PS61xx datasets, PS62xx suites, PS63xx providers and the run, PS64xx cache, PS65xx cost, PS66xx judge, PS67xx reports and sample imports.

Type Declaration

ALLOW_WITHOUT_ASSERTION

readonly ALLOW_WITHOUT_ASSERTION: "PS6213" = 'PS6213'

A dataset allow entry without a deterministic assertion

ASSERTION_ERROR

readonly ASSERTION_ERROR: "PS6214" = 'PS6214'

An assertion could not run on a case

BUDGET_EXCEEDED

readonly BUDGET_EXCEEDED: "PS6503" = 'PS6503'

The budget does not cover the planned or the next call

BUDGET_REQUIRED

readonly BUDGET_REQUIRED: "PS6504" = 'PS6504'

A live cache mode runs without a budget

CACHE_ENTRY_INVALID

readonly CACHE_ENTRY_INVALID: "PS6401" = 'PS6401'

A cache entry is unreadable or does not match its key; it counts as a miss

CACHE_ENTRY_TOO_LARGE

readonly CACHE_ENTRY_TOO_LARGE: "PS6405" = 'PS6405'

A cache entry is over the 1 MiB read cap, so it is not written

CACHE_IO_FAILED

readonly CACHE_IO_FAILED: "PS6404" = 'PS6404'

The cache could not be read or written, such as a full disk or a denied directory

CACHE_MISS

readonly CACHE_MISS: "PS6402" = 'PS6402'

Replay found no cache entry for a call

CACHE_TREE_INVALID

readonly CACHE_TREE_INVALID: "PS6403" = 'PS6403'

The cache tree holds something that is not a plain directory or file

CONFIG_KEY_REFUSED

readonly CONFIG_KEY_REFUSED: "PS6303" = 'PS6303'

eval.providers holds a key that is not allowed, such as apiKey

DATASET_EMPTY

readonly DATASET_EMPTY: "PS6106" = 'PS6106'

Dataset has no rows

DATASET_INVALID

readonly DATASET_INVALID: "PS6101" = 'PS6101'

Dataset file cannot be read or parsed, or a row has the wrong shape

DATASET_LIMIT

readonly DATASET_LIMIT: "PS6102" = 'PS6102'

Dataset breaks a size or row limit

IMPORT_OUT_REFUSED

readonly IMPORT_OUT_REFUSED: "PS6713" = 'PS6713'

The import output path exists or is refused

JUDGE_MODEL_MISSING

readonly JUDGE_MODEL_MISSING: "PS6209" = 'PS6209'

No judge model on the assertion, the suite or the rubric

JUDGE_SAME_MODEL

readonly JUDGE_SAME_MODEL: "PS6215" = 'PS6215'

The judge runs on a model under test

JUDGE_TARGET_TOO_LONG

readonly JUDGE_TARGET_TOO_LONG: "PS6602" = 'PS6602'

The judged text is over 64 KiB

JUDGE_UNPARSABLE

readonly JUDGE_UNPARSABLE: "PS6601" = 'PS6601'

More than half of the judge runs of a case gave no valid verdict

KEY_IN_CONFIG

readonly KEY_IN_CONFIG: "PS6307" = 'PS6307'

The value of a provider key appears in the eval config

KEY_MISSING

readonly KEY_MISSING: "PS6306" = 'PS6306'

The environment variable of a provider key is not set

LABEL_ASSERTION

readonly LABEL_ASSERTION: "PS6211" = 'PS6211'

A label names an assertion that is not an llm-judge assertion of the suite

LABEL_PROMPT_MISSING

readonly LABEL_PROMPT_MISSING: "PS6311" = 'PS6311'

A suite’s prompt is not in the release label, or has no arm of the variant

LABELS_INVALID

readonly LABELS_INVALID: "PS6603" = 'PS6603'

A label file cannot be read, is too big, or holds a bad or duplicate row

LABELS_UNMATCHED

readonly LABELS_UNMATCHED: "PS6604" = 'PS6604'

Labels name rows the run did not read

MODEL_REF

readonly MODEL_REF: "PS6206" = 'PS6206'

A model reference does not resolve, is retired or deprecated

NO_SAMPLE_MATCHED

readonly NO_SAMPLE_MATCHED: "PS6715" = 'PS6715'

No sample matched the suite

NO_SCHEMA

readonly NO_SCHEMA: "PS6205" = 'PS6205'

A json-schema assertion has no usable schema

PATH_REFUSED

readonly PATH_REFUSED: "PS6202" = 'PS6202'

A path is absolute or leaves the project

PII_CARD

readonly PII_CARD: "PS6113" = 'PS6113'

A dataset value holds a payment card number

PII_EMAIL

readonly PII_EMAIL: "PS6112" = 'PS6112'

A dataset value holds an email address

PII_IBAN

readonly PII_IBAN: "PS6114" = 'PS6114'

A dataset value holds an IBAN

PRICE_LOWERED

readonly PRICE_LOWERED: "PS6505" = 'PS6505'

A project price is lower than the built-in price of the same model

PRICING_MISSING

readonly PRICING_MISSING: "PS6501" = 'PS6501'

A cost threshold needs a price the catalog does not have

PROJECT_INVALID

readonly PROJECT_INVALID: "PS6310" = 'PS6310'

The project config cannot be loaded, or the project does not compile

PROVIDER_FAILED

readonly PROVIDER_FAILED: "PS6300" = 'PS6300'

A provider call failed

PROVIDER_REJECTED

readonly PROVIDER_REJECTED: "PS6304" = 'PS6304'

A provider refused the request with a 4xx status

PROVIDER_UNCONFIGURED

readonly PROVIDER_UNCONFIGURED: "PS6301" = 'PS6301'

A model’s provider has no entry in eval.providers and no built-in default

REGEX_RISK

readonly REGEX_RISK: "PS6210" = 'PS6210'

A regex assertion repeats a group that holds a quantifier

REPORT_INVALID

readonly REPORT_INVALID: "PS6701" = 'PS6701'

A report file cannot be read or is not an eval report of this version

REPORT_WRITE_FAILED

readonly REPORT_WRITE_FAILED: "PS6702" = 'PS6702'

A report directory or file cannot be written

RESPONSE_TOO_LARGE

readonly RESPONSE_TOO_LARGE: "PS6305" = 'PS6305'

A provider response body is larger than the cap

ROW_CONTRACT

readonly ROW_CONTRACT: "PS6104" = 'PS6104'

Row does not fit the prompt contract, or uses a key that is not allowed

ROW_ID

readonly ROW_ID: "PS6103" = 'PS6103'

Row id is malformed or used twice

ROW_NO_EXPECTED

readonly ROW_NO_EXPECTED: "PS6105" = 'PS6105'

An equals assertion has no value and the row has no expected value for it

RUBRIC_INVALID

readonly RUBRIC_INVALID: "PS6207" = 'PS6207'

A judge rubric is the prompt under test, or its inputs do not fit the judge call

RUBRIC_OUTPUT

readonly RUBRIC_OUTPUT: "PS6208" = 'PS6208'

A judge rubric output is not exactly a verdict of pass or fail and a reason

RUBRIC_SLOT_IN_SYSTEM

readonly RUBRIC_SLOT_IN_SYSTEM: "PS6212" = 'PS6212'

A judge rubric system message holds the response, the case inputs or the expected value

RUBRIC_UNFENCED

readonly RUBRIC_UNFENCED: "PS6216" = 'PS6216'

A judge rubric shows the response, case inputs or expected value outside a boundary fence

RUN_OPTION_INVALID

readonly RUN_OPTION_INVALID: "PS6309" = 'PS6309'

A run option is out of range, such as a concurrency that is not a positive integer

RUN_STOPPED

readonly RUN_STOPPED: "PS6308" = 'PS6308'

The run stopped before the case ran

SAMPLE_INVALID

readonly SAMPLE_INVALID: "PS6711" = 'PS6711'

A sample file cannot be read, or has a shape the import cannot use

SAMPLE_LIMIT

readonly SAMPLE_LIMIT: "PS6712" = 'PS6712'

A sample file has more lines than the import reads

SAMPLE_TRACKED

readonly SAMPLE_TRACKED: "PS6716" = 'PS6716'

The sample file is tracked by Git

SCAN_ERROR

readonly SCAN_ERROR: "PS6110" = 'PS6110'

A validator rule reports an error in a dataset value

SCAN_WARNING

readonly SCAN_WARNING: "PS6111" = 'PS6111'

A validator rule reports a warning in a dataset value

SCRUB_LEFT_PII

readonly SCRUB_LEFT_PII: "PS6714" = 'PS6714'

The scrub left personal data in a value

SUITE_DUPLICATE

readonly SUITE_DUPLICATE: "PS6201" = 'PS6201'

Two suite files have the same name

SUITE_INVALID

readonly SUITE_INVALID: "PS6200" = 'PS6200'

Suite file cannot be read, parsed or has the wrong structure

UNKNOWN_FIELD

readonly UNKNOWN_FIELD: "PS6204" = 'PS6204'

An assertion reads a field the output contract does not declare

UNKNOWN_PROMPT

readonly UNKNOWN_PROMPT: "PS6203" = 'PS6203'

The prompt under test is not compiled by the project

URL_REFUSED

readonly URL_REFUSED: "PS6302" = 'PS6302'

A provider URL is refused, or a provider answered with a redirect

USAGE_INVALID

readonly USAGE_INVALID: "PS6502" = 'PS6502'

A provider response has no valid token counts