Cutting CI pipeline time by 40%

The problem

Our end-to-end test files, or specs, were divided between containers that ran in parallel in CI, and the division was uneven. Some containers finished early while others were still running, and a pipeline is only done when its slowest container is.

The longer the pipeline ran, the more it cost, so the imbalance meant a higher bill as well as a longer wait.

Writing it down

In maths you state a problem precisely before you try to solve it. This is the pipeline problem, stated that way.

Given

The end-to-end specs, each with its average duration from past runs, and the parallel CI containers.

Rules

Every spec runs in exactly one container. A container is done when its last spec is done.

Goal

Make the containers finish at about the same time, to get the lowest cost for the pipeline time we wanted.

Both time and cost mattered, which made it a multi-criteria optimization problem.

What I did

I extracted the run data with Python and pandas and worked out the average duration of each spec.

Then I used PuLP, a Python library for optimization, to build a mixed-integer linear programming (MILP) model. From those averages, the model decides which container runs each spec. It's like loading parcels of different sizes into vans so that every van ends up with about the same load.

The result

I deployed the model. Pipeline time fell by 40%, at no extra infrastructure cost, and the result was validated in production.

The split is only as good as the averages behind it. When specs are added or their durations change, the split has to be recomputed from fresh data. CI Shard Advisor, below, does that from new run reports each time.

What came after

Later I turned the idea into CI Shard Advisor, a public tool for Cypress suites. It reads your run reports and shows how evenly your containers finish. Then it recommends the split in which the slowest container finishes earliest, as a list of specs for each container that you can paste into GitHub Actions or Bitbucket Pipelines. It searches with branch and bound, an exact method that can prove no better split exists for the durations in your reports.

• It opens in the browser with a demo run loaded. You can drop in your own reports, and they never leave the browser. There's nothing to install and no account to create.
• It also runs as a command-line tool with a quality-gate mode, and as a local REST API.

How it's tested

All the optimization logic sits in a pure core with a hexagonal architecture, so it can be tested separately from the web app, the command-line tool and the API.

An optimizer is hard to test with fixed examples, because for most inputs nobody knows the right answer in advance. For small inputs you can find it by trying every possible split. So property-based tests generate hundreds of random small cases and check that branch and bound finds the same answer as that brute-force search, and that whenever it claims a split is optimal, it really is, even when the search is cut short. Metamorphic and invariant tests check other rules that any correct answer has to follow.

The local API also has its own Java suite in REST Assured and JUnit 5, including a check of the response against a JSON Schema.

Tech: TypeScript, React, Vite, Fastify, Node.js, pnpm, Playwright and k6.

Before you add containers

Check whether the ones you already have finish at about the same time. If they don't, fixing the split may be cheaper than adding more.