Setting Up Accessibility Testing with GitHub Actions CI

How we added automated WCAG 2AA accessibility testing to our landing page CI pipeline using Pa11y CI with the axe runner — catching ~750 color contrast errors and preventing regressions on every pull request.

February 2026 GitHub Actions / Pa11y CI Landing Page (15 URLs) WCAG 2.1 Level AA

~750

Contrast Errors Fixed

Pages Tested

False Positives

~2 min

CI Pipeline Time

The Challenge

The CodeFrog landing page (codefrog.app) is a PHP site with 15 pages covering features, pricing, documentation, and case studies. Despite the site using a dark theme with carefully chosen colors, there was no automated way to verify that text maintained sufficient contrast against backgrounds — especially when CSS changes were made across pages.

No automated accessibility testing existed for the landing page
Color contrast issues could be introduced by any CSS change and go unnoticed
Manual testing with browser dev tools was error-prone and easily skipped
The existing CI/CD pipeline (test.yml) only covered the Flutter app
Semi-transparent rgba() backgrounds made contrast calculations unreliable

ResearchTool Selection: Pa11y CI with axe Runner

We evaluated several accessibility testing tools before choosing Pa11y CI:

Google Lighthouse CI: Comprehensive but slow (~30s per page), includes non-a11y audits we didn’t need, and produces false positives on contrast checks
axe-core CLI: Excellent engine but requires custom scripting for multi-page testing
Pa11y CI with htmlcs runner: Fast but produces many false positives, making CI gates unreliable
Pa11y CI with axe runner: Combines Pa11y’s multi-page CI tooling with axe-core’s precision — no false positives observed in our runs, covers many WCAG 2.1 AA checks (note: automated tools cannot verify all WCAG criteria; manual testing is still recommended)

The axe runner was the deciding factor. In a CI gate, false positives are worse than missed issues — developers learn to ignore a noisy check. With axe, every failure is a real accessibility violation.

Configuration

The .pa11yci config file at the repo root:

{
  "defaults": {
    "standard": "WCAG2AA",
    "runners": ["axe"],
    "timeout": 30000,
    "wait": 1000,
    "chromeLaunchConfig": {
      "args": ["--no-sandbox", "--disable-setuid-sandbox"]
    }
  },
  "urls": [
    "http://localhost:8000/",
    "http://localhost:8000/features",
    "http://localhost:8000/pricing",
    ... 12 more URLs ...
  ]
}

InfrastructureCI Pipeline Setup

PHP Router for Built-in Server

Our landing page uses Apache .htaccess rules for extensionless URLs (e.g., /features instead of /features.php). PHP’s built-in server doesn’t support .htaccess, so we created a lightweight router script:

<?php
// router.php — Replicates .htaccess URL rewriting for PHP built-in server
$uri = parse_url($_SERVER['REQUEST_URI'], PHP_URL_PATH);

// Serve existing files directly (CSS, JS, images)
if ($uri !== '/' && file_exists(__DIR__ . $uri)) {
    return false;
}

// Try appending .php for extensionless URLs
if ($uri !== '/' && file_exists(__DIR__ . $uri . '.php')) {
    require __DIR__ . $uri . '.php';
    return true;
}

// Default to index.php
require __DIR__ . '/index.php';

GitHub Actions Workflow

The workflow starts a PHP server, waits for it to be ready, then runs Pa11y CI against all 15 URLs. The paths filter ensures it only triggers on PRs that modify landing page files — zero overhead for Flutter-only PRs.

name: Landing Page Accessibility

on:
  pull_request:
    branches: [ main ]
    paths:
      - 'landing-page/main-site/**'

jobs:
  accessibility:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: Setup PHP
      uses: shivammathur/setup-php@v2
      with:
        php-version: '8.2'
    - name: Start PHP server
      run: |
        php -S localhost:8000 -t landing-page/main-site \
          landing-page/main-site/router.php &
        # Wait for server to be ready
        for i in $(seq 1 10); do
          curl -s http://localhost:8000 > /dev/null && break
          sleep 1
        done
    - name: Install Pa11y CI
      run: npm install -g pa11y-ci
    - name: Run accessibility tests
      run: pa11y-ci

FixFixing ~750 Color Contrast Errors

The first Pa11y CI run revealed ~750 color contrast violations across 14 of 15 pages. Nearly all errors had the same root cause: semi-transparent rgba() backgrounds.

When axe-core encounters a semi-transparent background, it cannot determine the final rendered color (because it depends on what’s behind it). This means it flags the element as a potential contrast failure, even if the visual contrast is adequate.

The Pattern

The dark theme used semi-transparent overlays for visual depth:

/* Before: semi-transparent backgrounds fail axe contrast checks */
header {
  background: linear-gradient(
    to bottom,
    rgba(15, 18, 33, 0.95),
    rgba(15, 18, 33, 0.85)
  );
  backdrop-filter: blur(8px);
}

.card {
  background: linear-gradient(
    180deg,
    rgba(255, 255, 255, 0.02),
    rgba(0, 0, 0, 0.15)
  );
}

The Fix

We computed the opaque equivalent of each semi-transparent color (blending against its actual parent background) and replaced them across all 14 pages:

/* After: opaque backgrounds pass axe contrast checks */
header {
  background: #0f1221;
}

.card {
  background: var(--card); /* #171a2e */
}

The same approach was applied to note boxes, badge backgrounds, button gradients, and code block highlighting — any element where rgba() was used for a background color that text sat on top of.

TestRunning Locally

Developers can run the same accessibility tests locally before pushing:

# Start the PHP server with the router
php -S localhost:8000 -t landing-page/main-site \
  landing-page/main-site/router.php &

# Run Pa11y CI against all configured URLs
npx pa11y-ci

# Expected output:
Running Pa11y on 15 URLs:
 > http://localhost:8000/ - 0 errors
 > http://localhost:8000/features - 0 errors
 > http://localhost:8000/pricing - 0 errors
  ... (12 more pages) ...
 > http://localhost:8000/case-studies/flutter-accessibility - 0 errors

✔ 15/15 URLs passed

Results

Pa11y CI runs on every PR that touches landing-page/main-site/ files
All 15 pages pass with zero accessibility errors
The CI step completes in ~2 minutes
No false positives observed — every failure from the axe runner in our runs has been a real WCAG violation
Path-scoped triggering means zero impact on Flutter-only PRs
Developers can run the exact same tests locally with npx pa11y-ci

Lessons Learned

Use axe over htmlcs for CI gates: The htmlcs runner catches more potential issues but produces false positives. In a CI gate that blocks merging, false positives erode trust. The axe runner ensures every failure is actionable.
Semi-transparent backgrounds are a contrast trap: Designers often use rgba() overlays for visual depth on dark themes. These look fine visually but fail automated contrast checks because the tool cannot compute the final rendered color. Convert to opaque equivalents.
Path-scoped workflows keep CI fast: Using the paths: filter in GitHub Actions means the accessibility check only runs when relevant files change. This avoids adding latency to unrelated PRs.
A custom router is required for PHP’s built-in server: Unlike Apache with .htaccess, PHP’s built-in server has no built-in URL rewriting. A simple router script (15 lines) bridges this gap for CI testing.
Fix errors systematically, not one-by-one: The ~750 errors across 14 pages came from a handful of repeated CSS patterns. Identifying the root patterns and doing bulk replacements was far more efficient than fixing individual elements.

← Back to Case Studies