Consolidated Testing Framework for zer0-mistakes Jekyll Theme¶

🎯 Overview¶

The zer0-mistakes testing framework provides 6 comprehensive test suites for validating the Jekyll theme across installation modes, site generation, and visual rendering.

📋 Test Suite Architecture¶

Quick Reference¶

Suite	Script	Purpose	Runtime
Core	`test_core.sh`	Unit, integration, validation	~2-3 min
Deployment	`test_deployment.sh`	Docker, E2E workflows	~5-8 min
Quality	`test_quality.sh`	Security, accessibility	~4-6 min
Installer	`test_installer.sh`	Modular installer (`scripts/bin/install`): profiles, deploy plugins, agent files, AI wizard	~30-60 sec
Fork Cleanup	`test_fork_cleanup.sh`	`scripts/fork-cleanup.sh` behavior	~30 sec
Site Generation	`test_site_generation.sh`	Config matrix builds	~5-10 min
Playwright Smoke	`test_playwright.sh` (`PLAYWRIGHT_PROJECT=smoke`)	CSS, layout, behavioral DOM	~2-3 min
Playwright Snapshots	`test_playwright.sh` (`PLAYWRIGHT_PROJECT=snapshots`)	Pixel regression for the 9 theme skins	~1 min

🔧 Core Test Suite (`test_core.sh`)¶

Purpose: Fundamental functionality validation Runtime: ~2-3 minutes Focus Areas:

Unit Tests: File structure, YAML syntax, gemspec validity, version consistency
Integration Tests: Bundle install, Jekyll build, gem build process
Validation Tests: Liquid templates, Sass compilation, JavaScript syntax

# Run core tests only
./test/test_core.sh

# With verbose output
./test/test_core.sh --verbose

# Generate JSON report
./test/test_core.sh --format json

🚀 Deployment Test Suite (`test_deployment.sh`)¶

Purpose: Installation and deployment validation Runtime: ~5-8 minutes Focus Areas:

Installation Tests: Local full/minimal installation, remote installation
Docker Tests: Environment setup, volume mounting, Jekyll build in Docker
End-to-End Tests: Complete workflow, GitHub Pages readiness

# Run deployment tests
./test/test_deployment.sh

# Skip Docker tests (if Docker unavailable)
./test/test_deployment.sh --skip-docker

# Skip remote installation tests
./test/test_deployment.sh --skip-remote

# Keep test environment for debugging
./test/test_deployment.sh --no-cleanup --verbose

🏆 Quality Test Suite (`test_quality.sh`)¶

Purpose: Security, accessibility, and performance validation Runtime: ~4-6 minutes Focus Areas:

Security Tests: Vulnerability scanning, sensitive files, hardcoded secrets
Accessibility Tests: Semantic HTML, alt text, color contrast, keyboard navigation
Compatibility Tests: Ruby/Jekyll versions, cross-platform files, browser compatibility
Performance Tests: Build performance, asset optimization, page generation

# Run quality tests
./test/test_quality.sh

# With detailed output
./test/test_quality.sh --verbose

🔧 Site Generation Test Suite (`test_site_generation.sh`)¶

Purpose: Configuration matrix site building and validation Runtime: ~5-10 minutes Focus Areas:

Full Mode: Complete theme installation with all files
Minimal Mode: Essential files only
Remote Theme Mode: GitHub Pages remote_theme configuration
Gem Mode: Ruby gem-based theme installation
Build Validation: Jekyll build success, HTML generation, asset compilation

# Test all installation modes
./test/test_site_generation.sh --all

# Test specific mode
./test/test_site_generation.sh --mode full

# Keep generated sites for inspection
./test/test_site_generation.sh --mode minimal --keep

🎭 Playwright Frontend Tests (`test_playwright.sh`)¶

Spec files live in exactly two sections under test/visual/, orthogonal to the execution tiers below:

core/ — cross-cutting quality/a11y/security/responsive baseline that applies regardless of feature (accessibility, security, styling, responsive, layout-chrome, features-registry). Bare, non-negotiable expectations.
features/ — one file per feature or tightly-scoped feature cluster, matching the feature registry (_data/features.yml's tests: links) — e.g. search.spec.js, admin.spec.js, appearance.spec.js, navbar.spec.js.

A single runner script invokes the appropriate Playwright project (tier). All tiers share test/playwright.config.js.

Tier	`PLAYWRIGHT_PROJECT`	What it covers	When CI runs it
Critical	`critical`	Only tests tagged `@critical` — the user-facing essentials (navigation, search, mobile survival, theming baseline, security)	Every code-change PR — the blocking gate (`ci.yml`)
Smoke	`smoke` (default)	Every spec in `core/` and `features/` except the pixel-snapshot test	Nightly (`nightly-extended.yml`) — failures file a sticky issue instead of blocking PRs
Snapshots	`snapshots`	Homepage pixel screenshots for the 9 theme skins (`features/appearance-snapshot.spec.js`)	PRs path-filtered on styling changes (non-blocking) + nightly
Regression	`regression-chromium` / `regression-firefox` / `regression-webkit`	All specs across all browsers	Manual `workflow_dispatch` only

Parallelism: the config is workers: 1, fullyParallel: false. The critical tier is the one exception — ci.yml passes workers: '3' to the playwright-tests action (PLAYWRIGHT_WORKERS on the runner), which parallelises across spec files only; order within a file is unchanged. It is safe there because critical grep-inverts the snapshot block, so nothing in the tier drives the shared 9-skin theme state. Never set PLAYWRIGHT_WORKERS for snapshots — those specs walk all 9 skins through one server and must stay serial.

Tagging: add { tag: '@critical' } to a test() or test.describe() to put it in the PR gate. Keep the gate honest — only behaviors a visitor would notice belong there; everything else is covered nightly. A weekly agentic UI/UX audit (ui-audit.yml + test/ui-audit/sweep.mjs + .claude/agents/ui-auditor.md) additionally reviews screenshots/axe/console output of the critical routes and files findings as source:ui-audit issues.

🔍 UI/UX audit sweep (`test/ui-audit/`)¶

[!IMPORTANT] The audit's Jekyll server must be started the same way test_playwright.sh:69-74 starts it — backgrounded with &, never --detach.

--detach prints Auto-regeneration: disabled when running server detached. Jekyll's LiveReload reactor rides that watch thread, so with --detach the port in _config_dev.yml (livereload_port: 35729) is bound but never answers — while Jekyll still injects <script src=".../livereload.js"> into every page. A subresource whose connection is accepted and never answered blocks the document's load event, so every page.goto(..., waitUntil: 'load') times out.

A curl -sf readiness gate cannot catch this: it fetches the HTML and never requests a subresource. That is why the audit passed its readiness check and still captured nothing for six consecutive weeks while reporting success (#321).

Piece	What it does
`sweep.mjs`	Walks 6 routes × 3 viewports; writes `output/report.json`, `output/report.md`, `output/screens/.png`. Findings exit 0 (they are data); capturing nothing* exits 1 (that is a broken harness, not a clean audit). `report.capture` = `{attempted, captured}` is the harness's own health, kept separate from findings.
`check-audit-serve.sh`	Regression check for the serve invocation: serves as `ui-audit.yml` does, sweeps, asserts ≥1 screenshot. Run `UI_AUDIT_SERVE_DETACH=1 ./test/ui-audit/check-audit-serve.sh` to watch it go red on the old invocation.

# Full sweep against a site you're already serving
BASE_URL=http://127.0.0.1:4000 node test/ui-audit/sweep.mjs

# Narrow it (UI_AUDIT_ROUTES is for harness checks; CI never sets it)
UI_AUDIT_ROUTES=/,/docs/ BASE_URL=http://127.0.0.1:4000 node test/ui-audit/sweep.mjs

# The serve-invocation regression check (skips cleanly without Ruby/Bundler)
./test/ui-audit/check-audit-serve.sh

Prerequisites: Node.js 18+, Playwright (auto-installed by the runner)

# Critical tier — the PR gate
PLAYWRIGHT_PROJECT=critical ./test/test_playwright.sh

# …as CI runs it (3 workers, file-level parallelism)
PLAYWRIGHT_PROJECT=critical PLAYWRIGHT_WORKERS=3 ./test/test_playwright.sh

# Smoke tier (default) — starts Jekyll on port 4011 unless BASE_URL is set
./test/test_playwright.sh

# Snapshot tier — pixel regression
PLAYWRIGHT_PROJECT=snapshots ./test/test_playwright.sh

# Reuse an existing Jekyll server (e.g. docker compose on :4000)
BASE_URL=http://localhost:4000 ./test/test_playwright.sh

# Run just one section
npx playwright test --config=test/playwright.config.js --project=smoke test/visual/core/
npx playwright test --config=test/playwright.config.js --project=smoke test/visual/features/

# npm aliases
npm run test:critical
npm run test:smoke
npm run test:snapshots
npm run test:regression

Updating snapshot baselines¶

Baselines are platform-specific; CI runs on Linux. macOS/Windows contributors should regenerate baselines via the Linux Playwright Docker image:

# Starts Jekyll via docker compose, runs Playwright in a Linux container,
# writes baselines into test/visual/snapshots/
./test/update-snapshots.sh

git add test/visual/snapshots/
git commit -m "test: refresh skin homepage snapshot baselines"

Or let CI produce them (the visual-evidence autogen)¶

update-snapshots.sh also carries three hooks — PRE_TEST_SCRIPT, POST_TEST_SCRIPT (repo-relative bash scripts run inside the jammy container, before/after the Playwright pass) and SKIP_PLAYWRIGHT=1 — used by scripts/ci/visual_evidence_autogen.py, the orchestrator behind .github/workflows/visual-evidence-autogen.yml. On every same-repo PR that lane renders the branch, runs the PR's test/visual/*-evidence.mjs generators (or the generic base-vs-head test/visual/pr-evidence.mjs), verifies the baselines, composes an expected | actual | diff montage (test/visual/snapshot-diff-montage.mjs), and refreshes the baselines only when the visual-evidence-reviewer agent judges the diff to be the change the PR describes. The same flow on a Docker host:

python3 scripts/ci/visual_evidence_autogen.py all --base origin/main   # plan + render + verify
python3 scripts/ci/visual_evidence_autogen.py bless --force            # after viewing test/visual-results/autogen/snapshot-diff.png
git add -- $(python3 scripts/ci/visual_evidence_autogen.py stage)
python3 scripts/ci/visual_evidence_autogen.py teardown

🎮 Unified Test Runner (`test_runner.sh`)¶

The consolidated test runner orchestrates all test suites with advanced features:

Basic Usage¶

# Run all core test suites (excludes visual for speed)
./test/test_runner.sh

# Run ALL suites including visual tests
./test/test_runner.sh --suites full

# Run specific suites
./test/test_runner.sh --suites core
./test/test_runner.sh --suites core,deployment

# Run with advanced options
./test/test_runner.sh --suites all --verbose --format json --parallel

Advanced Options¶

# CI/CD Integration
./test/test_runner.sh --suites all --environment ci --skip-docker --skip-remote

# Parallel execution (faster)
./test/test_runner.sh --suites all --parallel

# Fail-fast mode
./test/test_runner.sh --suites all --fail-fast

# Custom timeout
./test/test_runner.sh --suites all --timeout 600

🔄 Migration from Legacy Tests¶

Before (Legacy Structure)¶

test/
├── test_unit.sh              # ❌ Replaced by test_core.sh
├── test_integration.sh       # ❌ Replaced by test_core.sh
├── test_e2e.sh              # ❌ Replaced by test_deployment.sh
├── test_installation_complete.sh # ❌ Replaced by test_deployment.sh
├── test_docker_deployment.sh # ❌ Replaced by test_deployment.sh
├── test_security.sh         # ❌ Replaced by test_quality.sh
├── test_accessibility.sh    # ❌ Replaced by test_quality.sh
├── test_compatibility.sh    # ❌ Replaced by test_quality.sh
├── test_performance.sh      # ❌ Replaced by test_quality.sh
└── ... (6+ more scripts)

After (Consolidated Structure)¶

test/
├── test_core.sh             # ✅ Unit + Integration + Validation
├── test_deployment.sh       # ✅ Installation + Docker + E2E
├── test_quality.sh          # ✅ Security + Accessibility + Compatibility + Performance
├── test_site_generation.sh  # ✅ Config Matrix + Jekyll Build + Content Validation
├── test_playwright.sh       # ✅ Playwright runner (smoke / snapshots / regression)
├── update-snapshots.sh      # ✅ Generate Linux snapshot baselines via Docker
├── test_runner.sh           # ✅ Orchestrates all suites
├── playwright.config.js     # ✅ Single Playwright config (projects = tiers)
├── lib/                     # ✅ Shared test utilities
│   ├── install_test_utils.sh
│   └── config_matrix_generator.sh
├── visual/                  # ✅ Playwright specs + snapshot baselines
│   ├── core/                # ✅ Cross-cutting quality baseline (6 files: a11y, security,
│   │                        #    styling, responsive, layout-chrome, features-registry)
│   ├── features/            # ✅ One file per feature/cluster (15 files, matches the
│   │                        #    feature registry's tests: links — search, admin, …)
│   ├── fixtures.js          # Shared helpers (SKINS, VIEWPORTS, UI_ROUTES, setSkin, …)
│   ├── *-evidence.mjs       # Visual-evidence generators (test/visual/evidence/<slug>/)
│   ├── pr-evidence.mjs      # Generic base-vs-head evidence (no bespoke spec needed)
│   ├── snapshot-diff-montage.mjs  # expected | actual | diff montage of a snapshot run
│   └── snapshots/           # Committed Linux baselines for the snapshots tier
├── visual-results/          # ⚙️ Run output (gitignored): traces, html report, jekyll.log
├── results/                 # ✅ Test results (JSON)
├── reports/                 # ✅ Aggregated reports
└── coverage/                # ✅ Coverage reports

🚀 Quick Start Guide¶

For Developers¶

# Quick validation during development
./test/test_runner.sh --suites core

# Full validation before commit
./test/test_runner.sh --suites core,deployment

# Complete quality check
./test/test_runner.sh --suites all

# Playwright smoke tier (CSS/layout/behavior; starts Jekyll on port 4011)
./test/test_runner.sh --suites playwright
# or: npm run test:smoke   (set BASE_URL=http://localhost:4000 to reuse a running site)

# Playwright pixel snapshots (skin homepage regression)
./test/test_runner.sh --suites playwright_snapshots

For CI/CD¶

# Fast feedback in PR checks
./test/test_runner.sh --suites core --environment ci

# Comprehensive testing on main branch
./test/test_runner.sh --suites all --environment ci --skip-remote

# Docker integration testing
./test/test_runner.sh --suites deployment --environment docker

For Quality Assurance¶

# Security and accessibility audit
./test/test_runner.sh --suites quality --verbose

# Performance benchmarking
./test/test_quality.sh --verbose

# Cross-platform compatibility
./test/test_quality.sh

📊 Test Reporting¶

Output Formats¶

Text: Human-readable console output (default)
JSON: Machine-readable for CI/CD integration
XML: JUnit-compatible for test reporting tools
HTML: Rich web-based reports

# Generate JSON reports for CI/CD
./test/test_runner.sh --format json

# Generate HTML reports for review
./test/test_runner.sh --format html

# All formats
./test/test_runner.sh --format json
./test/test_runner.sh --format xml
./test/test_runner.sh --format html

Report Locations¶

test/
├── results/          # Individual test results (JSON)
├── reports/          # Aggregated reports (JSON/XML/HTML)
└── coverage/         # Coverage reports (when enabled)

🔧 Configuration & Environment Variables¶

Environment Detection¶

The test framework automatically detects and adapts to different environments:

local: Developer workstation with full toolchain
ci: Continuous Integration environment (GitHub Actions)
docker: Docker-based testing environment

Skip Options¶

--skip-docker: Skip Docker-related tests (when Docker unavailable)
--skip-remote: Skip remote installation tests (for offline/private environments)

Timeout Configuration¶

Default: 300 seconds per test suite
Deployment: 600 seconds (includes Docker operations)
Custom: Use --timeout <seconds>

🎯 CI/CD Integration¶

GitHub Actions Workflows¶

New Consolidated Workflow¶

# .github/workflows/consolidated-testing.yml
- name: Run Core Tests
  run: ./test/test_runner.sh --suites core --environment ci

- name: Run Deployment Tests
  run: ./test/test_runner.sh --suites deployment --environment ci --skip-docker

- name: Run Quality Tests
  run: ./test/test_runner.sh --suites quality --environment ci

Legacy Workflow Updates¶

# Before
- run: ./test/test_runner.sh --verbose --format json

# After
- run: ./test/test_runner.sh --suites all --verbose --format json --environment ci

Test Matrix Strategy¶

Pull Requests: Core tests only (fast feedback)
Main Branch: Core + Deployment tests
Releases: All test suites (comprehensive validation)
Nightly: All tests + Docker integration

📈 Performance Improvements¶

Execution Time Comparison¶

Test Scope	Legacy Framework	Consolidated Framework	Improvement
Core Tests	~8-10 minutes	~2-3 minutes	65% faster
Deployment	~12-15 minutes	~5-8 minutes	50% faster
Quality	~10-12 minutes	~4-6 minutes	55% faster
Full Suite	~25-30 minutes	~8-12 minutes	60% faster

Benefits Achieved¶

✅ Reduced Complexity: 15+ scripts → 3 main suites
✅ Faster Execution: 60% reduction in total runtime
✅ Better Maintainability: Unified interfaces and consistent patterns
✅ Improved CI/CD: Flexible suite selection and parallel execution
✅ Enhanced Reporting: Consolidated results and better visualization

🛠️ Troubleshooting¶

Common Issues¶

"Unknown test suite" Error¶

# Error: Unknown test suite: xyz
./test/test_runner.sh --suites xyz

# Solution: Use valid suite names
./test/test_runner.sh --suites core,deployment,quality,installer,site_generation,obsidian,playwright,playwright_snapshots

Docker Tests Failing¶

# Skip Docker tests if Docker unavailable
./test/test_runner.sh --suites deployment --skip-docker

# Or run Docker-specific tests separately
./test/test_deployment.sh --verbose

Remote Installation Timeouts¶

# Skip remote tests in restricted environments
./test/test_runner.sh --suites deployment --skip-remote

Debug Mode¶

# Keep test environments for inspection
./test/test_deployment.sh --no-cleanup --verbose

# Check individual test results
ls -la test/results/
cat test/results/core_test_*.json

Performance Issues¶

# Use parallel execution for faster runs
./test/test_runner.sh --suites all --parallel

# Increase timeout for slow environments
./test/test_runner.sh --suites all --timeout 900

🧰 Configuration Matrix Generator¶

The config_matrix_generator.sh utility creates Jekyll sites for each installation mode:

Available Modes¶

Mode	Description	Use Case
`full`	Complete theme with all files	Local development
`minimal`	Essential files only	Lightweight setup
`remote_theme`	GitHub Pages remote_theme	GitHub Pages deployment
`gem`	Ruby gem-based installation	Gem distribution

Usage¶

# Generate a site for a specific mode
./test/lib/config_matrix_generator.sh --mode full --output ./my-test-site

# Generate sites for all modes
./test/lib/config_matrix_generator.sh --all --output-base ./test-sites

# List available modes
./test/lib/config_matrix_generator.sh --list

Generated Files¶

Each mode generates: - _config.yml - Mode-specific Jekyll configuration - Gemfile - Appropriate gem dependencies - index.md - Sample homepage - pages/about.md - Sample about page - pages/_docs/getting-started.md - Sample docs - pages/_posts/ - Sample blog post

🔮 Future Enhancements¶

Planned Features¶

Test Coverage Reporting: Detailed coverage metrics across all suites
Baseline Comparison: Performance regression detection
Smart Test Selection: Run only tests affected by code changes
Enhanced Parallel Execution: Fine-grained parallel test execution
Visual Test Reports: Rich HTML dashboards with trends and insights
Cross-Browser Snapshots: Wire regression-firefox / regression-webkit projects into a nightly workflow
Accessibility Automation: WCAG compliance checking via the existing axe-core specs (currently test.fixme pending PR #57)

Contributing¶

The consolidated testing framework is designed for easy extension:

Adding Tests: Add new test functions to appropriate suite files
New Test Categories: Extend existing suites or propose new ones
New Installation Modes: Add to config_matrix_generator.sh
Visual Baselines: Update with --update-baseline flag
CI/CD Integration: Update workflow files to leverage new features
Documentation: Keep this README updated with changes

🎉 Success Criteria¶

The consolidated testing framework is working correctly when:

✅ All six test suites execute successfully
✅ CI/CD workflows complete without errors
✅ Test reports are generated in expected formats
✅ Performance targets are met (< 20 minutes for full suite)
✅ No regressions in test coverage or quality
✅ All installation modes (full, minimal, remote_theme, gem) build successfully
✅ Visual tests show no unexpected regressions

Ready for production use! 🚀

📚 Additional Documentation¶

Test Framework Version: 3.0 (Extended with Installation & Visual Tests) Last Updated: January 2026 Compatibility: Jekyll 4.0+, Ruby 3.0+, Node.js 18+, Docker (optional), Playwright (for visual tests)