Skip to content

Feature Coverage Testing

This document describes the Feature Coverage Matrix and testing mechanism for PromptScript formatters.

Overview

The Feature Coverage Matrix is a specification that tracks:

  1. Tool Capabilities - What features each classified AI target supports
  2. Formatter Implementation - Which features our formatters implement
  3. Coverage Gaps - Features supported by tools but not yet implemented
flowchart TB
  subgraph "Feature Coverage System"
    FM[Feature Matrix] --> |Defines| Features[Feature Specs]
    Features --> |Tests| FC[Feature Coverage Tests]
    FC --> |Validates| Formatters[Formatter Outputs]

    Features --> |Documents| Caps[Tool Capabilities]
    Features --> |Tracks| Gaps[Implementation Gaps]
  end

Feature Matrix

The Feature Matrix is defined in packages/formatters/src/feature-matrix.ts.

Structure

interface FeatureSpec {
id: string; // Unique identifier
name: string; // Human-readable name
description: string; // Feature description
category: FeatureCategory; // Grouping category
tools: Partial<Record<ToolName, FeatureStatus>>; // Support for classified tools
testStrategy?: string; // How to test
docsUrl?: Record<ToolName, string>; // Documentation links
}
type FeatureStatus =
| 'supported' // Tool supports, formatter implements
| 'not-supported' // Tool doesn't support
| 'planned' // Tool supports, not yet implemented
| 'partial'; // Partially implemented

The target catalog contains 50 coding agent targets and two prompt targets. The matrix currently classifies 37 of the coding agent targets. A missing entry means the target has not been classified for that feature, not that the feature is unsupported. The formatter registry and generated platform matrix remain the source of truth for target availability.

Categories

CategoryDescription
output-formatMarkdown, MDC, code blocks, diagrams
file-structureSingle/multi-file output, workflows
metadataYAML frontmatter, descriptions
targetingGlob patterns, activation types
contentCharacter limits, section splitting
advancedContext inclusion, @-mentions

Using the Matrix

Get Features for a Tool

@promptscript/formatters is not published to npm, so the code examples on this page work inside the PromptScript repository only.

import { getToolFeatures } from '@promptscript/formatters';
const features = getToolFeatures('cursor');
// Returns all features where cursor status is 'supported' or 'partial'

Check Specific Feature Support

import { toolSupportsFeature } from '@promptscript/formatters';
if (toolSupportsFeature('cursor', 'yaml-frontmatter')) {
// Generate with frontmatter
}

Get Coverage Summary

import { getFeatureCoverage } from '@promptscript/formatters';
const coverage = getFeatureCoverage('cursor');
// {
// tool: 'cursor',
// supported: 20,
// partial: 3,
// planned: 2,
// notSupported: 5,
// total: 30,
// coveragePercent: 72
// }

Generate Report

import { generateFeatureMatrixReport } from '@promptscript/formatters';
const markdown = generateFeatureMatrixReport();
console.log(markdown);
// Outputs markdown table with all features and their status per tool

Running Tests

Terminal window
# Run feature coverage tests
pnpm nx test formatters --testNamePattern="Feature Coverage"
# Run all formatter tests
pnpm nx test formatters

Test Structure

Feature coverage tests verify that formatters correctly implement the features defined in the matrix:

packages/formatters/src/__tests__/feature-coverage.spec.ts

Test Categories

  1. Matrix Integrity - Validates the matrix structure

    • Unique feature IDs
    • Required fields present
    • Valid status values
  2. Coverage Summary - Tests coverage calculation

    • Valid coverage for each tool
    • Coverage percentages are accurate
  3. Feature Implementation - Tests actual formatter output

    • markdown-output - Valid markdown produced
    • code-blocks - Fenced code blocks preserved
    • mermaid-diagrams - Mermaid diagrams preserved
    • yaml-frontmatter - Correct frontmatter generated
    • glob-patterns - Glob patterns in output
    • workflows - Workflow files generated
    • character-limit - Warning on limit exceeded

Adding a New Feature

When adding a new feature to the matrix:

  1. Define in Matrix

    feature-matrix.ts
    {
    id: 'new-feature',
    name: 'New Feature',
    description: 'Description of the feature',
    category: 'output-format',
    tools: {
    github: 'not-supported',
    cursor: 'supported',
    claude: 'not-supported',
    antigravity: 'planned',
    },
    testStrategy: 'How to verify this feature',
    }
  2. Add Test

    feature-coverage.spec.ts
    describe('new-feature', () => {
    it.each(['github', 'cursor', 'claude', 'antigravity'] as ToolName[])(
    '%s should handle new feature correctly',
    (tool) => {
    if (!toolSupportsFeature(tool, 'new-feature')) return;
    const formatter = formatters.get(tool)!;
    const result = formatter.format(createTestAST());
    // Verify feature is correctly implemented
    expect(result.content).toContain('expected content');
    }
    );
    });
  3. Implement Feature

    Add implementation to the relevant formatter(s).

  4. Update Status

    Change status from 'planned' to 'supported' once implemented.

Best Practices

  1. Keep Matrix Updated - When implementing new features, update the matrix
  2. Test All Statuses - Tests should handle all feature statuses appropriately
  3. Document Gaps - Use 'planned' status to track intended features
  4. Add Test Strategies - Help future developers understand how to verify features
  5. Include Docs URLs - Link to official tool documentation for reference

API Reference

Functions

FunctionDescription
getToolFeatures(tool)Get all supported features for a tool
getPlannedFeatures(tool)Get features planned for a tool
getFeaturesByCategory(cat)Get features by category
toolSupportsFeature(t, f)Check if tool supports feature
getFeatureCoverage(tool)Get coverage summary
getToolComparison()Get comparison matrix
generateFeatureMatrixReport()Generate markdown report

Types

TypeDescription
ToolNameUnion of tool names
FeatureStatusFeature implementation status
FeatureSpecFeature specification
FeatureCategoryFeature category
FeatureCoverageSummaryCoverage summary object