How many CI containers a test suite needs

The problem

Our end-to-end test files, or specs, ran in parallel CI containers, one container per product module. Modules are not the same size, so neither were the containers. The container of the largest module ran for more than twice as long as the shortest one, and a pipeline is only done when its slowest container is.

Every container also spent the same fixed time on setup before its first spec ran, close to half of a balanced container's run. So adding containers was not free: each new one paid the full setup again, and we paid for every minute each container ran.

Writing it down

In maths you state a problem precisely before you try to solve it. This is the pipeline problem, stated that way.

Given

The end-to-end specs, each with its average duration from real runs, the fixed setup time of a container, and the cost of a container per minute.

Rules

Every spec runs in exactly one container. A container is done when its setup and all its specs are done. The pipeline is done when its last container is.

Goals

A short pipeline and a low cost. More containers make the pipeline shorter but cost more, so the two goals pull against each other.

Two goals that pull against each other make it a multi-criteria optimization problem. There is no single best answer, only a set of best trade-offs, and the job is to find that set and then choose from it.

The data

A model is only as good as its numbers, so I started from the CI logs of real runs: every module, both the sanity and the regression suites, over several CI cycles. With Python I parsed each log and worked out the average duration of every spec, and the setup time of every container.

Failing tests needed care. A test that fails and is retried adds time that says nothing about how long the spec normally takes. So I separated them, measured how much time each CI cycle lost to failures and retries, and kept two data sets: the pipeline as it really ran, and the same pipeline without those failures.

The model

Then I used PuLP, a Python library for optimization, to build a mixed-integer linear programming (MILP) model. For each spec and each container there is a yes-or-no variable: does this container run this spec? The constraints say that every spec runs exactly once and that no container, setup included, runs longer than the pipeline time. The model makes that time as short as possible. This first model treated each container as running one test at a time, so it balanced the total test time of the containers.

It's like loading parcels of different sizes into vans so that the last van to finish its round finishes as early as possible. The CBC solver found splits that are provably optimal, or within a fraction of a second of a lower bound I derived for the pipeline time.

Time against cost

With that first model, I solved it for every number of containers from 3 to 14. Each answer is a point on the Pareto frontier: the fastest pipeline you can get for that cost. The frontier made the trade-off visible. Doubling the containers cut the pipeline time by only about a quarter and raised the cost by almost half, because every extra container pays the full setup again.

To choose a point on the frontier, I used goal programming: set a target for time and a target for cost, weight them, and find the configuration that misses them by the least. I ran it for four scenarios.

• Keep the current cost: the same number of containers we already had came out best.
• Contain the cost: fewer containers, a cheaper run and a clearly longer pipeline.
• Put time first: more containers, a faster pipeline and a higher bill.
• A slower setup than measured: the choice held, so a bad day in CI would not change it.

The answer was not to buy more containers. It was to keep the ones we had and split the specs well.

The first result

I measured the new split on real runs with one worker per container. Pipeline time fell by 42% and the cost stayed the same, within about 1% of what the model had predicted.

In production, though, each container ran 5 Playwright workers. Estimated from the same logs, the split there was only about 8% faster, and cost 3% more. The model had answered the question it was given, and the question was missing the workers.

Modelling the workers

Later I went back to the same logs and asked why so little of the gain survived. Three things ate it.

• Setup. With 5 workers, the 16 minutes of setup are more than half of every container's run, and no split changes them.
• Two suites. Each container ran sanity and then regression, and each suite has to finish before the next one starts.
• One long file. Playwright never splits a spec file across workers, and one file took about 9.5 minutes on its own. The container that runs it can't finish before its setup plus that file.

So I built a second model with the workers and the suites in it. A suite can't end before its file time divided by the workers, nor before its longest file. Then I checked each split against a simulation of how Playwright really hands out files, in file order, and improved it with a local search. With one worker, the simulation reproduces the measured CI jobs to the hundredth of a minute.

The answer changed. From 4 containers up, the pipeline sits on a floor set by the setup and that one long file, so extra containers only add cost. Compared with the 7 containers that ran in production, 4 containers would be 8% faster and 41% cheaper, and 3 would take about the same time for 52% less. These are estimates: the test times were measured with one worker, and a few real runs would have to confirm them.

A public tool

I also turned the first idea into CI Shard Advisor, a public tool for Cypress suites. It reads your run reports and shows how evenly your containers finish. Then it recommends the split in which the slowest container finishes earliest, as a list of specs for each container that you can paste into GitHub Actions or Bitbucket Pipelines. It searches with branch and bound, an exact method that can prove no better split exists for the durations in your reports.

• It opens in the browser with a demo run loaded. You can drop in your own reports, and they never leave the browser. There's nothing to install and no account to create.
• It also runs as a command-line tool with a quality-gate mode, and as a local REST API.

How it's tested

All the optimization logic sits in a pure core with a hexagonal architecture, so it can be tested separately from the web app, the command-line tool and the API.

An optimizer is hard to test with fixed examples, because for most inputs nobody knows the right answer in advance. For small inputs you can find it by trying every possible split. So property-based tests generate hundreds of random small cases and check that branch and bound finds the same answer as that brute-force search, and that whenever it claims a split is optimal, it really is, even when the search is cut short. Metamorphic and invariant tests check other rules that any correct answer has to follow.

The local API also has its own Java suite in REST Assured and JUnit 5, including a check of the response against a JSON Schema.

Tech: TypeScript, React, Vite, Fastify, Node.js, pnpm, Playwright and k6.

Before you add containers

Look at how much of each run is setup, and at your longest spec file. If the setup dominates or one file sets the floor, more containers only raise the bill, and a model that ignores your workers will promise gains you won't see.