Automated Performance Regression Testing for React: A Practical Setup Guide

A common assumption is that once a React app passes its Core Web Vitals audit, performance is a solved problem. It isn’t. Every merged pull request that adds a dependency, changes a component’s render logic, or swaps out an image is a chance for that score to quietly drift backward — and without something watching for it, nobody notices until a user complains or the next quarterly audit turns up a surprise. Performance, unlike a unit test, doesn’t fail loudly on its own. It has to be measured, on purpose, on every change.

This post works through the practical questions that come up when setting up automated performance regression testing for a React codebase: what to measure, how to wire it into CI, and how to avoid the two failure modes most teams hit first — measuring nothing that matters, or measuring so much noise that nobody trusts the results.

Isn’t a Passing Lighthouse Score in CI Enough?

Running Lighthouse once in CI and checking that the score is above some threshold is a reasonable start, but it misses the point of regression testing, which is about change over time, not a single snapshot. A score of 92 today tells you nothing about whether it was 96 last week. Without a stored baseline to compare against, a gradual decline — three points lost per week across a dozen small PRs — can go entirely unnoticed while each individual change looks harmless in isolation.

Regression testing requires three things a one-off Lighthouse run doesn’t provide on its own: a stored history of past measurements, an automated comparison against that history, and a defined threshold for what counts as a meaningful regression versus normal run-to-run variance. Lighthouse CI (the @lhci/cli package) handles the first two out of the box; the third is a decision your team has to make deliberately.

What Should Actually Be Measured?

Not every metric belongs in a CI gate. Some are too noisy to be trustworthy on shared CI runners, and others matter more at the component level than the page level. A workable split looks like this:

Page-level metrics, tracked via Lighthouse CI:

• Largest Contentful Paint (LCP)
• Total Blocking Time (TBT) — a reasonable CI-friendly proxy for Interaction to Next Paint
• Cumulative Layout Shift (CLS)
• Total JavaScript transferred

Build-level metrics, tracked via bundle analysis:

• Total bundle size, and size per route/chunk
• Size of individual heavy dependencies, tracked separately so a single library swap doesn’t get lost in an aggregate number

Component-level metrics, tracked via profiling, not CI gates:

• Render count for components under frequent parent re-renders
• Render duration for components doing expensive work (large lists, charts, complex forms)

Trying to gate a PR on component render counts in CI tends to produce more false alarms than useful signal, since render behavior can shift based on test data shape rather than code quality. That kind of measurement is better suited to a manual or scheduled profiling pass, discussed further down.

How Do You Wire Lighthouse CI Into an Actual Pipeline?

The lighthouserc.json file defines what gets tested, how many runs to average, and what the pass/fail assertions look like:

{
  "ci": {
    "collect": {
      "staticDistDir": "./build",
      "numberOfRuns": 3
    },
    "assert": {
      "assertions": {
        "categories:performance": ["error", { "minScore": 0.85 }],
        "largest-contentful-paint": ["error", { "maxNumericValue": 2500 }],
        "cumulative-layout-shift": ["error", { "maxNumericValue": 0.1 }],
        "total-blocking-time": ["warn", { "maxNumericValue": 300 }]
      }
    },
    "upload": {
      "target": "temporary-public-storage"
    }
  }
}

Running three passes per commit (numberOfRuns: 3) and comparing the median, rather than a single run, absorbs a meaningful amount of CI runner jitter. A single-run measurement on a shared GitHub Actions runner can vary by 15–20% between otherwise identical commits, purely from resource contention with other jobs — averaging across runs is not optional if the goal is a signal worth trusting.

A minimal GitHub Actions job to run this on every pull request:

name: performance-regression
on: [pull_request]

jobs:
  lighthouse:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npm run build
      - run: npx @lhci/cli autorun

Note the error versus warn distinction in the config above. LCP and CLS regressions fail the build outright; TBT is set to warn rather than block, since it tends to be the noisiest of the three on CI hardware. Which metrics get hard-blocked and which get soft-warned is a judgment call worth revisiting after a few weeks of real data.

What About Regressions Lighthouse Won’t Catch?

Lighthouse measures a page load. It has nothing to say about a component that re-renders forty times during a user interaction, or a useEffect that fires an extra network request after a refactor. Those regressions live at the component level, and catching them automatically requires a different approach — one built on bundle size and render-count assertions rather than page-load audits.

Bundle size regressions are the easier of the two to automate. A tool like size-limit can fail a build the moment a specific chunk crosses a defined threshold:

{
  "size-limit": [
    {
      "path": "build/static/js/main.*.js",
      "limit": "180 KB"
    },
    {
      "path": "build/static/js/vendor.*.js",
      "limit": "250 KB"
    }
  ]
}

This catches the specific case where a new dependency gets added without anyone noticing its transitive weight — a date-formatting library that pulls in an entire locale database, for instance. The failure message names the exact chunk that crossed the line, which makes the regression easy to trace back to the responsible PR.

Render-count regressions are harder to gate automatically without generating false positives, but they can still be tracked with a repeatable, scripted profiling pass rather than manual spot-checks. A Playwright script that drives a known interaction and reads render counts via React.Profiler gives a consistent, repeatable number to compare across commits:

import { Profiler } from 'react';

let renderCount = 0;

function onRenderCallback() {
  renderCount += 1;
}

function ProfiledDashboard() {
  return (
    <Profiler id="Dashboard" onRender={onRenderCallback}>
      <Dashboard />
    </Profiler>
  );
}

Recording renderCount after a scripted sequence of interactions (open the dashboard, apply a filter, sort a column) and comparing it against a stored baseline number turns “does this feel slower” into a specific, trackable integer. It won’t run on every PR the way a Lighthouse check does — the setup cost is higher — but as a weekly or pre-release check, it catches the class of regression that page-load audits are structurally blind to.

How Do You Set Thresholds Without Drowning in False Positives?

The single biggest reason teams abandon performance regression testing within a month of setting it up is threshold miscalibration — either the bar is so tight that unrelated CI noise fails builds constantly, or so loose that it never catches anything and the checks become background noise everyone ignores.

A few things that held up well in practice:

• Set the initial threshold from an average of the last 10–15 runs on the main branch, not a single “good” run picked by hand. A single run can be an outlier in either direction.
• Leave 10–15% headroom above the current baseline before a metric is treated as a hard failure. Tighter than that, and normal CI variance starts triggering false alarms.
• Revisit thresholds deliberately after intentional performance work, rather than letting them silently ratchet upward as a side effect of unrelated changes. If a genuine improvement drops LCP from 2.3s to 1.8s, update the budget to reflect that — don’t leave 15% of headroom sitting unused indefinitely.
• Separate error from warn the way the config example above does. Not every regression needs to block a merge; some just need a visible flag for someone to look at before the next release.

What Should Happen When a Regression Is Actually Caught?

A failed performance check is only useful if there’s a defined next step attached to it. Treating it the same way a failed unit test gets treated works well: the PR doesn’t merge until the regression is understood, and “understood” means one of three outcomes — the change is reverted, the regression is fixed, or the threshold is deliberately updated with a documented reason (a new feature that legitimately adds weight, for instance).

What doesn’t work well: treating a red performance check as advisory and merging anyway “to unblock the release.” That pattern, repeated a handful of times, is how thresholds stop meaning anything and the whole setup degrades back into the snapshot-only approach this post started by arguing against.

Is This Worth Setting Up for a Small Team?

The Lighthouse CI and bundle-size pieces are worth adopting even for a two-person team — the setup cost is a few hours, and the GitHub Actions config above is close to drop-in ready. The component-level profiling script is the piece worth deferring until there’s a specific, recurring problem (a dashboard that keeps getting slower, a form that re-renders too often) that justifies the extra maintenance. Start with the page-level and bundle-level checks; add render-count tracking only once there’s a component where “is this getting worse” is a question worth answering with data instead of a guess.

What does your current setup catch automatically, and what would have to happen before your team noticed a 20% regression in LCP on your own?