We had to ask our developers to slow down. How we scaled CI in the age of AI

One of our core values is that we support every device and every kind of user. That means we need validated support for every variation of macOS, Windows, Linux, iOS and Android, including different browsers and system modes (like the language, preferences and license a system runs with). That's why, from the very beginning of XFA, we've focused heavily on automated testing.

This scalable approach to testing makes sure that any user, on any device, has a great experience self-installing XFA, and is never blocked when verifying the security of their device, so they can log in and get to work. But with the improvements in agentic coding, we're now choking this CI. This is how we're solving it.

Three years ago, we wrote about how we set up our automated testing and deployments in The superpower of automatic deployments. This post builds on that approach: testing every change in full, in its own environment, before it's merged.

Why we needed our developers to slow down

Today we release more than 500 times a month, more than 20 times every working day. This month, we're on track for more than 150,000 minutes of CI. That's about fourteen times as many as last October, while our releases grew about seven times: every release now gets about twice as much CI. That's our test coverage growing, even though we optimized the pipeline several times in that same year.

Because developers and AI agents need quick feedback on every change, every release goes through the full pipeline. Each change gets its own full, isolated stack on its pull request, with a full build and the full test suite. After code review, the merge queue tests changes again, rebased on main, together with some longer-running tests we considered a bit too heavy to run early on every pull request, like different architectures and simulating slow devices.

After merging on main come sequential upgrade and acceptance testing on a shared test environment. That's replicated across about 30 separate repositories and 60 separate AWS accounts, and still gets a change to customers within 30 to 40 minutes, review included. If you want to know more about this approach, read The superpower of automatic deployments.

Those tests run both virtually, on cloud machines, and on a shelf of physical test devices in our office, one for every setup we support. At some point, we had to ask our developers to slow down and focus their work, or start less urgent work in the evening or at the weekend, because we were choking the series of test devices (and our cloud bill).

We're not the only company whose CI can't keep up. Anthropic saw their CI jobs grow 25 times in six months, as Claude came to write most of their code. We've optimized our own pipeline multiple times over the years, a story very similar to theirs. As they put it:

First code was the blocker, then review, and now it's CI.

Why not just buy more devices?

You might think we should just buy more devices to test on (or "runners", as we call them). Don't get us wrong: the shelf in our office keeps growing with physical test devices. They pay for themselves in about three months, and we're very happy giving every device a funny name (we went for coffee variants). But we would much rather invest in more unique variants of devices and chipsets than in more copies of the same ones. And with how fast our development speed is growing, buying, setting up and maintaining more of them just doesn't scale.

So we built our own test selector

We never want to compromise on quality. Everything has to run before a change goes live. Amazon and Google look at CI/CD in an entirely different way: Amazon tests a change before it's merged, Google mostly after. We're firmly on the pre-merge side: we want to catch problems before merge.

We'd already done the usual optimizations: every service and app in its own repository so testing stays well isolated, and very lean deploy times. We could already tell CI to ignore or isolate specific files (for example with GitHub Actions' own path filters, test suites that check the diff, or caching). At this scale and speed, it doesn't matter whether you have separate repositories or a monorepo: you need something that deterministically selects the right tests to run, to fail early and efficiently (for example, don't run the full end-to-end tests when the unit tests would already catch it).

Anthropic had the same issue and built a test impact analysis system for exactly that, but only explains it in broad strokes. We're happy to be a bit more specific about ours.

The test selector is a small service of our own (ours runs on AWS Lambda, with S3 and Athena), and it does two things. Before a CI run starts, we send it the files that changed on the pull request and in the commit. It answers which tests should run, based on which tests failed with those files changed often enough before, so whatever is likely to fail runs early. After every job, it gets the results back: which tests passed and which failed, for the files that were changed. When a test fails, it links that test to run when those files change, and it drops the link again once those files have passed enough times.

Everything still runs before a change lands. But from commit to commit, if unrelated files changed, or if the selector can tell that the unit tests would already catch a failure, it doesn't run the end-to-end tests on the pull request. Those run in the merge queue, a later stage, so we don't run them twice. The selector picks the specific jobs to run, which saves our devices from waiting, being claimed, or blocking someone else. And within a job, it picks the specific expensive checks and tests to leave for later.

What it looks like in a workflow

Every repository decides for itself which jobs may be skipped. First, a plan job at the start that asks the selector what to run:

jobs:
  plan:  # sends the files changed in this commit or pull request to the selector
    uses: xfa-tech/test-selector/.github/workflows/plan.yml@v3
    with:
      DEFAULT_PULL_REQUEST: selective             # only what those files can reach
      DEFAULT_MERGE_QUEUE: full                   # everything
      DEFAULT_MAIN: skip_if_passed_identically    # not again if the queue passed it
      DEFAULT_SCHEDULED: full                     # every night, everything

Then each job says when it may be skipped. This is an example of a job running end-to-end tests on a Windows Home laptop on our shelf:

  e2e-desktop-windows-home:
    needs: plan
    # runs only when the plan lists this job
    if: contains(fromJSON(needs.plan.outputs.run), 'e2e-desktop-windows-home')
    # the Windows Home laptop with an ARM processor, on our shelf
    runs-on: [self-hosted, windows-home, arm64, laptop]
    env:
      ON_PULL_REQUEST: selective           # when the changed files reach it
      ON_MERGE_QUEUE: full                 # always, every test
      ON_MAIN: skip_if_passed_identically  # not again if the queue passed it
      ON_SCHEDULED: full                   # every night, every test
    steps:
      - uses: actions/checkout@v7
      - uses: xfa-tech/test-selector/job@v3
        with:
          plan: ${{ needs.plan.outputs.plan }}  # what the plan decided
          run: make e2e                         # our usual test command

Inside the job, the selector leaves out the individual tests that the changed files can't reach, and reports the results back so it keeps learning.

And at the end, one job that every other job feeds into:

  required_checks_passed:
    name: Required checks passed  # the one check GitHub requires
    needs: [<every other job in the workflow>]  # each one passed or was skipped
    uses: xfa-tech/test-selector/.github/workflows/gate.yml@v3
    with: { needs: "${{ toJSON(needs) }}" }  # every job's result

The required checks job has a second benefit: we don't rely on admins anymore to add new tests as required. There's one job that is always required, and it checks that everything completed, or was left out for a reason.

Extra: testing a variation remotely, on demand

Not every test can run on a developer's (or an agent's) own machine. Some need a specific variation (as mentioned above), like a Windows Home laptop with an ARM processor or a slow device, that only exists on our shelf or in the cloud.

By adding a dedicated plan job to CI, we can now pass inputs to run specific jobs manually, so developers and agents don't have to trigger a full CI run to test a specific outcome. Agents can use their full intelligence here, knowing what their change might impact.

For example:

# only this job, in full
gh workflow run ci.yml --ref my-branch -f jobs="e2e-desktop-windows-home"
# what a pull request of this branch would run
gh workflow run ci.yml --ref my-branch -f mode=selective

That way, every test is defined once, in CI, and runs the same way whether a developer, an agent or a pull request starts it.

The first results

On pull requests, the test selector now skips more than half (52%) of the CI minutes. That optimization alone means we use almost a third (31%) less CI across all our CI usage, while releasing more than ever and getting faster feedback.

And the time spent waiting for our test devices halved. At full rollout, once our end-to-end suites pick individual tests too, we expect to save more than half of our CI minutes with the same coverage, and most of the waiting.

Using fewer minutes and fewer runners also means we can actually turn on more tests, earlier. Since they only run when they're relevant, they're cheap enough to add.

What's next

Today, still, no code goes to production without a human review and without every test running at least once, and we're not changing that.

We do keep on looking to make the feedback loop on every piece of code even faster, now focusing on scaling and automating running any tests "downstream" (the tests of services, modules and apps that use the code change), in a fully deterministic, efficient and automatically expanding way.