Parity Testing
This document describes the parity testing mechanism for PromptScript formatters. The goal is to ensure consistent output across all formatters and detect anomalies.
Overview
PromptScript provides two complementary testing mechanisms:
- Parity Matrix - Specification-based testing that defines expected behavior
- Golden Files - Reference-based testing that compares against known-good output
flowchart TB
subgraph "Parity Testing"
AST[Canonical AST] --> PM[Parity Matrix Tests]
AST --> GF[Golden File Tests]
PM --> |Validates| Sections[Section Coverage]
PM --> |Validates| Content[Content Patterns]
GF --> |Compares| Reference[Golden Files]
Sections --> Report[Parity Report]
Content --> Report
Reference --> Report
end
Parity Matrix
The Parity Matrix is a specification that defines:
- Which AST blocks should produce which sections
- Which formatters must implement each section
- Expected content patterns for validation
- Header variations across formatters
Location
packages/formatters/src/parity-matrix.tsStructure
interface SectionSpec { id: string; // Unique section identifier name: string; // Human-readable name description: string; // Section purpose sources: SourceBlockConfig[]; // AST blocks that provide data requiredBy: FormatterName[]; // Formatters that MUST implement optionalFor: FormatterName[]; // Formatters that MAY implement contentPatterns?: RegExp[]; // Expected content patterns headerVariations: Record<FormatterName, string | string[]>;}Example
{ id: 'restrictions', name: 'Restrictions', description: "Don'ts and forbidden practices", sources: [{ block: 'restrictions' }], requiredBy: ['github', 'cursor', 'claude', 'antigravity'], optionalFor: [], contentPatterns: [/never|don't|do not|avoid|forbidden/i], headerVariations: { github: "## Don'ts", cursor: 'Never:', claude: "## Don'ts", antigravity: "## Don'ts", },}Using the Matrix
@promptscript/formatters is not published to npm, so this example works inside the PromptScript repository only.
import { PARITY_MATRIX, getRequiredSections, validateSectionContent,} from '@promptscript/formatters';
// Get required sections for a formatterconst sections = getRequiredSections('github');
// Validate content matches expected patternsconst isValid = validateSectionContent(content, 'restrictions');Golden Files
Golden files are reference outputs that formatters should match.
Location
packages/formatters/src/__tests__/__golden__/├── github.md # Expected GitHub Copilot output├── cursor.mdc # Expected Cursor output├── claude.md # Expected Claude Code output├── antigravity.md # Expected Antigravity output└── README.md # DocumentationUpdating Golden Files
When you intentionally change formatter output:
# Update all golden filesUPDATE_GOLDEN=true pnpm nx test formatters
# Review changesgit diff packages/formatters/src/__tests__/__golden__/
# Commit updated golden files with your changesgit add packages/formatters/src/__tests__/__golden__/git commit -m "test: update golden files for new formatter output"How It Works
- Tests generate output from a canonical AST
- Output is normalized (timestamps removed, whitespace standardized)
- Normalized output is compared against golden files
- Differences indicate regression or intentional change
Running Tests
# Run all parity testspnpm nx test formatters
# Run only parity matrix testspnpm nx test formatters --testNamePattern="Parity Matrix"
# Run only golden file testspnpm nx test formatters --testNamePattern="Golden Files"
# Update golden files after intentional changesUPDATE_GOLDEN=true pnpm nx test formattersAdding a New Formatter
When adding a new formatter:
-
Add to Parity Matrix
Update
parity-matrix.tsto include your formatter inrequiredByoroptionalForarrays:{id: 'restrictions',requiredBy: ['github', 'cursor', 'claude', 'antigravity', 'newformatter'],// ...} -
Add Header Variations
Add your formatter’s header style to each section:
headerVariations: {github: "## Don'ts",newformatter: "## Never Do", // Your formatter's style} -
Generate Golden File
Add configuration in
golden-files.spec.ts:{name: 'newformatter',formatter: new NewFormatter(),goldenFile: 'newformatter.md',extension: 'md',} -
Run Tests
Terminal window pnpm nx test formattersGolden file will be created automatically on first run.
Detecting Anomalies
Missing Sections
The parity matrix tests will fail if a formatter doesn’t produce required sections:
github should produce required content types
Expected: content matching restrictions patternsReceived: no match foundContent Drift
Golden file tests will fail if output differs from reference:
github formatter output differs from golden file.Run with UPDATE_GOLDEN=true to update.
Diff:Line 15: - ## Don'ts + ## RestrictionsCross-Formatter Inconsistency
Parity tests verify all formatters produce similar content:
all markdown formatters should produce content with multiple sectionsExpected: > 3 sectionsReceived: 2 sections for claudeCI/CD Integration
Parity tests are automatically run in the CI pipeline on every push and pull request.
Golden Files Drift Check
The CI workflow includes a dedicated step that detects when golden files are out of date:
- name: Golden Files Drift Check run: | pnpm nx test formatters --testNamePattern="Golden Files" if [ -n "$(git status --porcelain packages/formatters/src/__tests__/__golden__)" ]; then echo "::error::Golden files are out of date." exit 1 fiWhat it does:
- Runs Golden Files tests
- Checks if any golden files changed during the test run
- Fails the build if files are out of date
Fixing CI Failures
If CI fails with “Golden files are out of date”:
# 1. Pull latest changesgit pull
# 2. Update golden files locallyUPDATE_GOLDEN=true pnpm nx test formatters
# 3. Review the changesgit diff packages/formatters/src/__tests__/__golden__
# 4. Commit and pushgit add packages/formatters/src/__tests__/__golden__git commit -m "test: update golden files"git pushPR Workflow
flowchart LR
subgraph "Developer"
A[Change Formatter] --> B[Run Tests]
B --> C{Golden Files<br/>Changed?}
C -->|Yes| D[UPDATE_GOLDEN=true]
D --> E[Commit Golden Files]
C -->|No| F[Commit Code]
E --> F
end
subgraph "CI Pipeline"
F --> G[Run Tests]
G --> H[Golden Files Check]
H -->|Pass| I[Build & Deploy]
H -->|Fail| J[PR Blocked]
end
When to Update Golden Files
| Scenario | Action |
|---|---|
| Intentional formatter change | Update golden files, commit with your PR |
| Unintentional output change | Fix the formatter, don’t update golden files |
| New formatter added | Golden file created automatically on first run |
| CI fails unexpectedly | Check if someone else changed formatters |
Best Practices
- Run tests before commits - Catch regressions early
- Review golden file diffs - Understand what changed
- Update matrix for new features - Keep specification current
- Add content patterns - Help detect missing functionality
- Document header variations - Track format differences
API Reference
Functions
| Function | Description |
|---|---|
getRequiredSections(formatter) | Get sections formatter MUST implement |
getOptionalSections(formatter) | Get sections formatter MAY implement |
getAllSections(formatter) | Get all supported sections |
validateSectionContent(content, sectionId) | Validate content patterns |
matchesSectionHeader(content, sectionId, formatter) | Check header presence |
analyzeFormatterOutput(formatter, content, blocks) | Generate parity report |
Types
| Type | Description |
|---|---|
FormatterName | Union of formatter names |
SectionSpec | Section specification |
ExtractionRule | Block-to-section mapping |
ParityReport | Analysis result |