EVAL_CODES
constEVAL_CODES:object
Defined in: eval/src/errors.ts:8
Codes of eval errors and findings: PS61xx datasets, PS62xx suites,
PS63xx providers and the run, PS64xx cache, PS65xx cost, PS66xx judge,
PS67xx reports and sample imports.
Type Declaration
ALLOW_WITHOUT_ASSERTION
readonlyALLOW_WITHOUT_ASSERTION:"PS6213"='PS6213'
A dataset allow entry without a deterministic assertion
ASSERTION_ERROR
readonlyASSERTION_ERROR:"PS6214"='PS6214'
An assertion could not run on a case
BUDGET_EXCEEDED
readonlyBUDGET_EXCEEDED:"PS6503"='PS6503'
The budget does not cover the planned or the next call
BUDGET_REQUIRED
readonlyBUDGET_REQUIRED:"PS6504"='PS6504'
A live cache mode runs without a budget
CACHE_ENTRY_INVALID
readonlyCACHE_ENTRY_INVALID:"PS6401"='PS6401'
A cache entry is unreadable or does not match its key; it counts as a miss
CACHE_ENTRY_TOO_LARGE
readonlyCACHE_ENTRY_TOO_LARGE:"PS6405"='PS6405'
A cache entry is over the 1 MiB read cap, so it is not written
CACHE_IO_FAILED
readonlyCACHE_IO_FAILED:"PS6404"='PS6404'
The cache could not be read or written, such as a full disk or a denied directory
CACHE_MISS
readonlyCACHE_MISS:"PS6402"='PS6402'
Replay found no cache entry for a call
CACHE_TREE_INVALID
readonlyCACHE_TREE_INVALID:"PS6403"='PS6403'
The cache tree holds something that is not a plain directory or file
CONFIG_KEY_REFUSED
readonlyCONFIG_KEY_REFUSED:"PS6303"='PS6303'
eval.providers holds a key that is not allowed, such as apiKey
DATASET_EMPTY
readonlyDATASET_EMPTY:"PS6106"='PS6106'
Dataset has no rows
DATASET_INVALID
readonlyDATASET_INVALID:"PS6101"='PS6101'
Dataset file cannot be read or parsed, or a row has the wrong shape
DATASET_LIMIT
readonlyDATASET_LIMIT:"PS6102"='PS6102'
Dataset breaks a size or row limit
IMPORT_OUT_REFUSED
readonlyIMPORT_OUT_REFUSED:"PS6713"='PS6713'
The import output path exists or is refused
JUDGE_MODEL_MISSING
readonlyJUDGE_MODEL_MISSING:"PS6209"='PS6209'
No judge model on the assertion, the suite or the rubric
JUDGE_SAME_MODEL
readonlyJUDGE_SAME_MODEL:"PS6215"='PS6215'
The judge runs on a model under test
JUDGE_TARGET_TOO_LONG
readonlyJUDGE_TARGET_TOO_LONG:"PS6602"='PS6602'
The judged text is over 64 KiB
JUDGE_UNPARSABLE
readonlyJUDGE_UNPARSABLE:"PS6601"='PS6601'
More than half of the judge runs of a case gave no valid verdict
KEY_IN_CONFIG
readonlyKEY_IN_CONFIG:"PS6307"='PS6307'
The value of a provider key appears in the eval config
KEY_MISSING
readonlyKEY_MISSING:"PS6306"='PS6306'
The environment variable of a provider key is not set
LABEL_ASSERTION
readonlyLABEL_ASSERTION:"PS6211"='PS6211'
A label names an assertion that is not an llm-judge assertion of the suite
LABEL_PROMPT_MISSING
readonlyLABEL_PROMPT_MISSING:"PS6311"='PS6311'
A suite’s prompt is not in the release label, or has no arm of the variant
LABELS_INVALID
readonlyLABELS_INVALID:"PS6603"='PS6603'
A label file cannot be read, is too big, or holds a bad or duplicate row
LABELS_UNMATCHED
readonlyLABELS_UNMATCHED:"PS6604"='PS6604'
Labels name rows the run did not read
MODEL_REF
readonlyMODEL_REF:"PS6206"='PS6206'
A model reference does not resolve, is retired or deprecated
NO_SAMPLE_MATCHED
readonlyNO_SAMPLE_MATCHED:"PS6715"='PS6715'
No sample matched the suite
NO_SCHEMA
readonlyNO_SCHEMA:"PS6205"='PS6205'
A json-schema assertion has no usable schema
PATH_REFUSED
readonlyPATH_REFUSED:"PS6202"='PS6202'
A path is absolute or leaves the project
PII_CARD
readonlyPII_CARD:"PS6113"='PS6113'
A dataset value holds a payment card number
PII_EMAIL
readonlyPII_EMAIL:"PS6112"='PS6112'
A dataset value holds an email address
PII_IBAN
readonlyPII_IBAN:"PS6114"='PS6114'
A dataset value holds an IBAN
PRICE_LOWERED
readonlyPRICE_LOWERED:"PS6505"='PS6505'
A project price is lower than the built-in price of the same model
PRICING_MISSING
readonlyPRICING_MISSING:"PS6501"='PS6501'
A cost threshold needs a price the catalog does not have
PROJECT_INVALID
readonlyPROJECT_INVALID:"PS6310"='PS6310'
The project config cannot be loaded, or the project does not compile
PROVIDER_FAILED
readonlyPROVIDER_FAILED:"PS6300"='PS6300'
A provider call failed
PROVIDER_REJECTED
readonlyPROVIDER_REJECTED:"PS6304"='PS6304'
A provider refused the request with a 4xx status
PROVIDER_UNCONFIGURED
readonlyPROVIDER_UNCONFIGURED:"PS6301"='PS6301'
A model’s provider has no entry in eval.providers and no built-in default
REGEX_RISK
readonlyREGEX_RISK:"PS6210"='PS6210'
A regex assertion repeats a group that holds a quantifier
REPORT_INVALID
readonlyREPORT_INVALID:"PS6701"='PS6701'
A report file cannot be read or is not an eval report of this version
REPORT_WRITE_FAILED
readonlyREPORT_WRITE_FAILED:"PS6702"='PS6702'
A report directory or file cannot be written
RESPONSE_TOO_LARGE
readonlyRESPONSE_TOO_LARGE:"PS6305"='PS6305'
A provider response body is larger than the cap
ROW_CONTRACT
readonlyROW_CONTRACT:"PS6104"='PS6104'
Row does not fit the prompt contract, or uses a key that is not allowed
ROW_ID
readonlyROW_ID:"PS6103"='PS6103'
Row id is malformed or used twice
ROW_NO_EXPECTED
readonlyROW_NO_EXPECTED:"PS6105"='PS6105'
An equals assertion has no value and the row has no expected value for it
RUBRIC_INVALID
readonlyRUBRIC_INVALID:"PS6207"='PS6207'
A judge rubric is the prompt under test, or its inputs do not fit the judge call
RUBRIC_OUTPUT
readonlyRUBRIC_OUTPUT:"PS6208"='PS6208'
A judge rubric output is not exactly a verdict of pass or fail and a reason
RUBRIC_SLOT_IN_SYSTEM
readonlyRUBRIC_SLOT_IN_SYSTEM:"PS6212"='PS6212'
A judge rubric system message holds the response, the case inputs or the expected value
RUBRIC_UNFENCED
readonlyRUBRIC_UNFENCED:"PS6216"='PS6216'
A judge rubric shows the response, case inputs or expected value outside a boundary fence
RUN_OPTION_INVALID
readonlyRUN_OPTION_INVALID:"PS6309"='PS6309'
A run option is out of range, such as a concurrency that is not a positive integer
RUN_STOPPED
readonlyRUN_STOPPED:"PS6308"='PS6308'
The run stopped before the case ran
SAMPLE_INVALID
readonlySAMPLE_INVALID:"PS6711"='PS6711'
A sample file cannot be read, or has a shape the import cannot use
SAMPLE_LIMIT
readonlySAMPLE_LIMIT:"PS6712"='PS6712'
A sample file has more lines than the import reads
SAMPLE_TRACKED
readonlySAMPLE_TRACKED:"PS6716"='PS6716'
The sample file is tracked by Git
SCAN_ERROR
readonlySCAN_ERROR:"PS6110"='PS6110'
A validator rule reports an error in a dataset value
SCAN_WARNING
readonlySCAN_WARNING:"PS6111"='PS6111'
A validator rule reports a warning in a dataset value
SCRUB_LEFT_PII
readonlySCRUB_LEFT_PII:"PS6714"='PS6714'
The scrub left personal data in a value
SUITE_DUPLICATE
readonlySUITE_DUPLICATE:"PS6201"='PS6201'
Two suite files have the same name
SUITE_INVALID
readonlySUITE_INVALID:"PS6200"='PS6200'
Suite file cannot be read, parsed or has the wrong structure
UNKNOWN_FIELD
readonlyUNKNOWN_FIELD:"PS6204"='PS6204'
An assertion reads a field the output contract does not declare
UNKNOWN_PROMPT
readonlyUNKNOWN_PROMPT:"PS6203"='PS6203'
The prompt under test is not compiled by the project
URL_REFUSED
readonlyURL_REFUSED:"PS6302"='PS6302'
A provider URL is refused, or a provider answered with a redirect
USAGE_INVALID
readonlyUSAGE_INVALID:"PS6502"='PS6502'
A provider response has no valid token counts