Phase: 5 — Advanced Specialist Topics
Every great technical SEO audit eventually runs into the same enemy: the next deploy. A developer merges a pull request,
a CMS plugin auto-updates, or a template gets refactored — and suddenly canonical tags are broken, a noindex
directive has spread to production, or your LCP hero image is lazy-loaded again. Without automated guardrails, you find
out from a traffic drop, not a failed test.
This module teaches you how to embed SEO quality gates directly into your development pipeline so that regressions are caught before they ship — not after they've affected rankings.
robots directives.CI/CD stands for Continuous Integration / Continuous Deployment (or Delivery). In practice it means:
As a technical SEO practitioner, your interest is the test stage. If you can write a test that catches "canonical tag is missing on all product pages," and that test runs on every pull request, you have effectively turned yourself into a persistent, automated member of the engineering review process.
The pipeline stages where SEO checks typically live:
Before you can automate detection, you need to know what you are protecting. An SEO regression is any unintended change that degrades crawlability, indexability, performance, or structured-data validity. Regressions cluster into five categories:
noindex added to pages that should be indexed (the most catastrophic and the most common).robots.txt updated to block Googlebot from JavaScript, CSS, or entire sections.X-Robots-Tag HTTP header incorrectly set on responses.<a href> to JavaScript click handlers (not crawlable).price missing from Offer).@type value).<title> or <meta name="description"> emptied or duplicated.
Lighthouse CI (@lhci/cli) is the official tool for running Lighthouse audits
programmatically in a CI environment. It measures Core Web Vitals proxies (TBT for INP, LCP, CLS), accessibility,
SEO, and best practices against a deployed URL and asserts thresholds.
npm install -g @lhci/cli
# or as a dev dependency
npm install --save-dev @lhci/clilighthouserc.jsPlace this file at the root of your project. Lighthouse CI reads it automatically.
// lighthouserc.js
module.exports = {
ci: {
collect: {
// URL(s) to audit — use your preview/staging URL in CI
url: [
'https://staging.example.com/',
'https://staging.example.com/products/example-product/',
'https://staging.example.com/blog/example-post/',
],
numberOfRuns: 3, // median of 3 runs for stability
settings: {
// Simulate a mid-range mobile device
preset: 'desktop', // or 'mobile'
throttlingMethod: 'simulate',
},
},
assert: {
// Fail the CI build if these thresholds are breached
assertions: {
// Core Web Vitals proxies
'largest-contentful-paint': ['error', { maxNumericValue: 2500 }],
'cumulative-layout-shift': ['error', { maxNumericValue: 0.1 }],
'total-blocking-time': ['warn', { maxNumericValue: 200 }],
// SEO category score must stay above 90
'categories:seo': ['error', { minScore: 0.9 }],
// Specific SEO audits
'document-title': ['error', { minScore: 1 }],
'meta-description': ['error', { minScore: 1 }],
'canonical': ['error', { minScore: 1 }],
'hreflang': ['warn', { minScore: 1 }],
'robots-txt': ['error', { minScore: 1 }],
'crawlable-anchors': ['error', { minScore: 1 }],
'link-text': ['warn', { minScore: 1 }],
'image-alt': ['error', { minScore: 1 }],
// Performance budgets
'uses-optimized-images': ['warn', { minScore: 1 }],
'unused-javascript': ['warn', { minScore: 0.8 }],
'render-blocking-resources':['warn', { minScore: 1 }],
},
},
upload: {
// Store results in the temporary public storage (or self-host LHCI server)
target: 'temporary-public-storage',
},
},
};# Collect, assert, and upload in one command
lhci autorun
The following workflow runs Lighthouse CI on every pull request targeting main. It first deploys to a
preview environment, then audits it. Adjust the deploy step for your platform (Vercel, Netlify, or a staging server).
# .github/workflows/lhci.yml
name: Lighthouse CI
on:
pull_request:
branches: [main]
jobs:
lighthouse:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install dependencies
run: npm ci
- name: Build project
run: npm run build
# Example: deploy to a preview URL first (Vercel CLI shown)
# - name: Deploy to preview
# run: vercel --token ${{ secrets.VERCEL_TOKEN }} --yes > preview-url.txt
- name: Run Lighthouse CI
run: npx lhci autorun
env:
LHCI_GITHUB_APP_TOKEN: ${{ secrets.LHCI_GITHUB_APP_TOKEN }}
The LHCI_GITHUB_APP_TOKEN allows Lighthouse CI to post results as a status check on the pull request, so
reviewers see pass/fail directly in the GitHub UI without reading logs.
Lighthouse audits Lighthouse-specific SEO signals, but it will not crawl your full site. You need a dedicated link checker for broader coverage. Two solid options:
broken-link-checker (Node.js)npm install -g broken-link-checker
# Check all internal links on a staging URL, report 4xx and 5xx
blc https://staging.example.com --recursive \
--exclude-external \
--filter-level 3lychee (Rust, very fast)
lychee is a high-performance link checker available as a GitHub Action. It is well-suited to large sites
because it parallelises requests efficiently.
# .github/workflows/link-check.yml
name: Link Check
on:
pull_request:
branches: [main]
schedule:
- cron: '0 6 * * 1' # Also run weekly on Monday morning
jobs:
links:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Link Checker
uses: lycheeverse/lychee-action@v1
with:
args: |
--verbose
--no-progress
--exclude-loopback
--exclude 'mailto:*'
--exclude 'tel:*'
--accept 200,206,429
'https://staging.example.com'
fail: true
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
A redirect chain longer than two hops wastes crawl budget and loses PageRank through each hop. You can assert chain
length with a simple Node script using the built-in https module:
// scripts/check-redirects.mjs
import https from 'https';
const urlsToCheck = [
'https://staging.example.com/old-page/',
'https://staging.example.com/another-old-url/',
];
const MAX_HOPS = 2;
async function followRedirects(url, hops = 0) {
return new Promise((resolve, reject) => {
const req = https.get(url, { method: 'HEAD' }, (res) => {
const { statusCode, headers } = res;
if ([301, 302, 307, 308].includes(statusCode) && headers.location) {
if (hops >= MAX_HOPS) {
reject(new Error(`Redirect chain too long (>${MAX_HOPS} hops) starting at ${url}`));
} else {
resolve(followRedirects(headers.location, hops + 1));
}
} else {
resolve({ url, statusCode, hops });
}
});
req.on('error', reject);
});
}
let failed = false;
for (const url of urlsToCheck) {
try {
const result = await followRedirects(url);
console.log(`✓ ${result.url} — ${result.hops} hop(s) → ${result.statusCode}`);
} catch (err) {
console.error(`✗ ${err.message}`);
failed = true;
}
}
if (failed) process.exit(1);# Add to your CI workflow as a step:
- name: Check redirect chains
run: node scripts/check-redirects.mjsLighthouse checks whether a canonical tag exists, but it does not verify that the canonical points to the correct URL. A self-referencing canonical on every page is the baseline expectation. You need custom assertions for that.
The following script fetches a list of URLs and checks that each page's canonical tag matches its own URL (i.e., is self-referencing, as expected for canonical pages):
// scripts/check-canonicals.mjs
import { JSDOM } from 'jsdom';
const pagesToCheck = [
'https://staging.example.com/',
'https://staging.example.com/products/example-product/',
'https://staging.example.com/blog/example-post/',
];
let failed = false;
for (const url of pagesToCheck) {
const response = await fetch(url);
const html = await response.text();
const dom = new JSDOM(html);
const document = dom.window.document;
// --- Canonical check ---
const canonicalEl = document.querySelector('link[rel="canonical"]');
if (!canonicalEl) {
console.error(`✗ [canonical] Missing canonical tag on: ${url}`);
failed = true;
} else {
const canonical = canonicalEl.getAttribute('href');
if (canonical !== url) {
console.error(`✗ [canonical] Expected "${url}", got "${canonical}"`);
failed = true;
} else {
console.log(`✓ [canonical] ${url}`);
}
}
// --- noindex check ---
const robotsMeta = document.querySelector('meta[name="robots"]');
if (robotsMeta) {
const content = robotsMeta.getAttribute('content') || '';
if (content.includes('noindex')) {
console.error(`✗ [noindex] NOINDEX found on indexable page: ${url} — content="${content}"`);
failed = true;
} else {
console.log(`✓ [robots meta] ${url} — "${content}"`);
}
}
// --- Title check ---
const title = document.querySelector('title')?.textContent?.trim();
if (!title) {
console.error(`✗ [title] Missing <title> on: ${url}`);
failed = true;
} else if (title.length > 60) {
console.warn(`⚠ [title] Title may be truncated (${title.length} chars): ${url}`);
} else {
console.log(`✓ [title] "${title}" — ${url}`);
}
// --- Meta description check ---
const desc = document.querySelector('meta[name="description"]')?.getAttribute('content')?.trim();
if (!desc) {
console.error(`✗ [meta description] Missing on: ${url}`);
failed = true;
} else {
console.log(`✓ [meta description] ${url}`);
}
}
if (failed) process.exit(1);npm install jsdom # dependency for the script above
The Google Rich Results Test is a browser tool — not scriptable in CI. For automation, use the
Schema.org validator or the unofficial schema-dts type checker together with a JSON-LD
extraction and validation script.
// scripts/check-schema.mjs
import { JSDOM } from 'jsdom';
const pagesToCheck = [
{ url: 'https://staging.example.com/products/example-product/', requiredType: 'Product' },
{ url: 'https://staging.example.com/blog/example-post/', requiredType: 'Article' },
{ url: 'https://staging.example.com/', requiredType: 'Organization' },
];
let failed = false;
for (const { url, requiredType } of pagesToCheck) {
const response = await fetch(url);
const html = await response.text();
const dom = new JSDOM(html);
const scripts = dom.window.document.querySelectorAll(
'script[type="application/ld+json"]'
);
if (scripts.length === 0) {
console.error(`✗ [schema] No JSON-LD found on: ${url}`);
failed = true;
continue;
}
let found = false;
for (const script of scripts) {
let parsed;
try {
parsed = JSON.parse(script.textContent);
} catch (e) {
console.error(`✗ [schema] Invalid JSON in JSON-LD block on: ${url} — ${e.message}`);
failed = true;
continue;
}
// Handle @graph arrays
const blocks = parsed['@graph'] ? parsed['@graph'] : [parsed];
for (const block of blocks) {
if (block['@type'] === requiredType) {
found = true;
console.log(`✓ [schema] Found @type="${requiredType}" on: ${url}`);
// Type-specific required-property checks
if (requiredType === 'Product') {
if (!block.name) { console.error(`✗ [schema] Product missing "name" on: ${url}`); failed = true; }
if (!block.offers){ console.error(`✗ [schema] Product missing "offers" on: ${url}`); failed = true; }
}
if (requiredType === 'Article') {
if (!block.headline) { console.error(`✗ [schema] Article missing "headline" on: ${url}`); failed = true; }
if (!block.datePublished){ console.error(`✗ [schema] Article missing "datePublished" on: ${url}`); failed = true; }
}
}
}
}
if (!found) {
console.error(`✗ [schema] @type="${requiredType}" not found on: ${url}`);
failed = true;
}
}
if (failed) process.exit(1);The Schema Markup Validator provides a testable endpoint. You can POST a URL to it and parse the response for errors. At the time of writing, the API is not officially versioned for public automation; use the extraction script above for robust CI integration, and reserve the online validator for manual audits and debugging.
Your sitemap must be reachable, valid XML, and not contain URLs that return 4xx or are marked noindex. A
simple check in CI:
// scripts/check-sitemap.mjs
import { JSDOM } from 'jsdom';
const SITEMAP_URL = 'https://staging.example.com/sitemap.xml';
const MAX_URLS_TO_SAMPLE = 20; // check a sample, not every URL in large sitemaps
const response = await fetch(SITEMAP_URL);
if (!response.ok) {
console.error(`✗ [sitemap] Could not fetch ${SITEMAP_URL} — status ${response.status}`);
process.exit(1);
}
const xml = await response.text();
const dom = new JSDOM(xml, { contentType: 'text/xml' });
const locs = Array.from(dom.window.document.querySelectorAll('loc'))
.map(el => el.textContent.trim());
if (locs.length === 0) {
console.error('✗ [sitemap] Sitemap contains no <loc> entries.');
process.exit(1);
}
console.log(`✓ [sitemap] ${locs.length} URL(s) found.`);
// Sample up to MAX_URLS_TO_SAMPLE URLs and HEAD-check them
const sample = locs.slice(0, MAX_URLS_TO_SAMPLE);
let failed = false;
for (const url of sample) {
const res = await fetch(url, { method: 'HEAD' });
if (res.status !== 200) {
console.error(`✗ [sitemap] ${url} returned ${res.status}`);
failed = true;
} else {
console.log(`✓ [sitemap] ${url} — 200`);
}
}
if (failed) process.exit(1);The following workflow runs on every pull request. It installs dependencies, builds, and sequentially runs all SEO checks. Any failure blocks the merge.
# .github/workflows/seo-qa.yml
name: SEO QA
on:
pull_request:
branches: [main]
jobs:
seo-checks:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Use Node.js 20
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Build
run: npm run build
# --- Deploy to preview URL first ---
# (Replace with your actual preview deployment step)
# - name: Deploy preview
# run: npx vercel --token ${{ secrets.VERCEL_TOKEN }} --yes
- name: Lighthouse CI
run: npx lhci autorun
env:
LHCI_GITHUB_APP_TOKEN: ${{ secrets.LHCI_GITHUB_APP_TOKEN }}
- name: Check canonical tags and meta
run: node scripts/check-canonicals.mjs
- name: Check structured data
run: node scripts/check-schema.mjs
- name: Check sitemap
run: node scripts/check-sitemap.mjs
- name: Check redirect chains
run: node scripts/check-redirects.mjs
- name: Link check (internal only)
uses: lycheeverse/lychee-action@v1
with:
args: '--verbose --no-progress --exclude-external https://staging.example.com'
fail: true
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}CI gates protect before deploy. Post-deploy smoke tests verify that the deployment to production succeeded without introducing unexpected changes. Keep these lightweight — they run against live production, so they should be read-only and fast.
A minimal post-deploy smoke test checks:
<loc> entry.robots.txt returns HTTP 200 and does not contain Disallow: /.noindex directive.// scripts/smoke-test.mjs
const PRODUCTION_URL = 'https://www.example.com';
const checks = [
{ url: `${PRODUCTION_URL}/`, expectStatus: 200 },
{ url: `${PRODUCTION_URL}/sitemap.xml`, expectStatus: 200 },
{ url: `${PRODUCTION_URL}/robots.txt`, expectStatus: 200 },
];
let failed = false;
for (const { url, expectStatus } of checks) {
const res = await fetch(url, { method: 'HEAD' });
if (res.status !== expectStatus) {
console.error(`✗ ${url} — expected ${expectStatus}, got ${res.status}`);
failed = true;
} else {
console.log(`✓ ${url} — ${res.status}`);
}
}
// robots.txt sanity check
const robots = await (await fetch(`${PRODUCTION_URL}/robots.txt`)).text();
if (robots.includes('Disallow: /\n') || robots.trim() === 'User-agent: *\nDisallow: /') {
console.error('✗ [robots.txt] Site appears to be blocking all crawlers!');
failed = true;
} else {
console.log('✓ [robots.txt] No blanket Disallow found.');
}
if (failed) process.exit(1);
Trigger this script as a final step in your CD workflow, after the deployment step, via a workflow_run
event or a deployment environment hook.
CI/CD gates catch regressions at deploy time. But some issues arise outside deploys: a third-party script changes its payload, a CMS editor accidentally removes a canonical, or a hosting provider's CDN starts stripping a header. Schedule your checks to run regularly even when no code is being pushed.
# Add a scheduled trigger to any workflow:
on:
push:
branches: [main]
pull_request:
branches: [main]
schedule:
# Run full SEO QA suite every Monday at 07:00 UTC
- cron: '0 7 * * 1'For production monitoring at scale, consider complementing your CI scripts with:
Automation is powerful, but it is not a substitute for SEO expertise. Know the boundary:
noindex
on thin-content pages, crawl-budget prioritisation decisions, whether a page should be indexed,
structured-data accuracy relative to visible content, hreflang strategy, and any situation requiring
business context.
A test suite that fails noisily on legitimate editorial decisions (e.g., a deliberate noindex on a
staging page that crept into the URL list) becomes a nuisance that the team starts ignoring. Keep your automated
assertions tight and deterministic. Use warnings (warn in Lighthouse CI, console warnings in scripts)
for grey-area signals that should prompt review without blocking deployment.
@lhci/cli — Lighthouse CI for performance and Lighthouse-native SEO audits.lychee / broken-link-checker — automated link checking.jsdom — DOM parsing in Node.js for canonical, meta, and schema extraction.You have completed this module when you can do the following without referring to these notes:
noindex, presence of a non-empty <title>, and valid JSON-LD with the correct
@type for each page.
robots.txt
is not blocking all crawlers.
noindex to a page in your test environment) and
show that your pipeline catches it before it could be merged.