Running our test suite in parallel on GitHub actions

A couple of years ago, Laravel introduced a great feature which allows to run PHPUnit / Pest tests in parallel. This results in a big boost in performance.

By default, it determines the concurrency level by taking a look at the number of CPU cores your machine has. So, if you're using a modern Mac that has 10 CPU cores, it will run 10 tests at the same time, greatly cutting down on the time your testsuite needs to run completely.

A default runner on GitHub doesn't have that many cores, so you can't leverage parallel testing as good as in your typical local environments.

In this blog post, I'd like to show you a way of running your tests on GitHub, by splitting them up in small chunks that can run concurrently.vWe use this technique to cut down the running time of our vast testsuite from 16 minutes to only just 4. In this blog post all examples will come from our base.

What we're going to do

Like already mentioned in the introduction, a typical test runner on GitHub hasn't got a lot of cores, meaning the default way of running tests in parallel that Laravel offers (running a test per CPU core) doesn't work well.

What you can do at GitHub however is running a lot of GitHub actions jobs in parallel. So what we are going to do is splitting our testsuite in equal chunks, and create a test job on GitHub action per chunk. These chunks can run in parallel, which will immensely decrease the total running time.

I'll tell you how to achieve this technically in the remainder of this blog post, but here's already how the end result will look like on GitHub.

In the screenshot above you can see that our test suite is split up in 12 parts which will all run concurrently. Composer / NPM will only run once to build up the dependencies and assets and they will be used in all 12 testing parts.

Splitting up the testsuite in equal parts

Let's first take a look at how we can split the testsuite in equal parts. We use Pest as a test runner, which offers a --list-tests option to output all tests.

Here's a bit of code to get all test class names from that output.

 $process = new Process([__DIR__ . '/../vendor/bin/pest', '--list-tests']);

$process->mustRun();

$index = $shardNumber - 1;

$allTestNames = Str::of($process->getOutput())
    ->explode("\n")
    ->filter(fn(string $line) => str_contains($line, ' - '))
    ->map(function (string $fullTestName) {
        $testClassName = Str::of($fullTestName)
            ->replace('- ', '')
            ->trim()
            ->between('\\\\', '::')
            ->afterLast('\\')
            ->toString();

        return $testClassName;
    })
    ->filter()
    ->unique();

In $allTestNames will be a collection containing all class names (= file names) that are inside the test suite.

To split the collection up in multiple parts, you can use the split function which accepts the number of parts you want. Here's how you would split up the tests in 12 parts, and get the first part

$testNamesOfFirstPart = $allTestNames
   ->split(12) // split the collection in 12 equal parts
   ->get(key: 0) // get the first part (the index is 0 based)

PHPUnit / Pest also offers a --filter option to only run specific tests. If you only want to run the tests in from the ArchTest class (which is displayed in the screenshot above), you could execute this.

# Will only run the tests from the ArchTest file
vendor/bin/pest --filter=ArchTest

You can use | to specify multiple patterns. Here's how you could execute the tests from multiple files

# Will only run the tests from the ArchTest file
vendor/bin/pest --filter=ArchTest|CheckSitesBeingMonitoredTest

Here's how you could use the $testNamesOfFirstPart from the previous snippet to run the first part of the tests programmatically.

$process = new Process(
    command: [
       './vendor/bin/pest',
       '--filter', 
       $testNamesOfFirstPart->join('|')],
       timeout: null // take as much time as we need
);

$process->start();

/* pipe the Pest output to the console */
foreach ($process as $data) {
    echo $data;
}

$process->wait();

// use the exit code of Pest as the exit code of the script
exit($process->getExitCode());

Running the testsuite parts in parallel at GitHub

Now that you know how you could split up a test suite in equal parts, let's take a look at how we can run all these parts in parallel on GitHub actions.

GitHub actions support a matrix parameter. Shortly said, this matrix parameter is used for testing variations of your test suite, and it will run those variations concurrently.

Here's the part of the Oh Dear GitHub workflow where the matrix is being set up. I've omitted several parts for brevity.

# .github/workflows/run-tests.yml

name: Run tests

jobs:
    run-tests:
        name: Run Tests (Part ${{ matrix.shard_number }}/${{ matrix.total_shard_count }})
        runs-on: ubuntu-latest
        strategy:
            matrix:
                total_shard_count: [12]
                shard_number: ['01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12']

        steps:
           ## 
           ## multiple set up steps omitted for brevity
           ##
            -   name: Run tests
                run: ./vendor/bin/pest

The matrix will create jobs per combination in the matrix. So it will run 1 (only one element in total_shard_count) * 12 (twelve elements in shard_number) = 12 times.

In the Run tests step, pest is executed. This will result in the whole testsuite being executed 12 times. Of course we don't want to execute the whole test suite 12 times, but only each separate 1/12 part of the testsuite.

We can achieve this by not running /vendor/bin/pest but a custom PHP script called github_parallel_test_runner that will receive the total_shard_count and the shard_number as environment variables.

# .github/workflows/run-tests.yml

name: Run tests

jobs:
    run-tests:
        name: Run Tests (Part ${{ matrix.shard_number }}/${{ matrix.total_shard_count }})
        runs-on: ubuntu-latest
        strategy:
            matrix:
                total_shard_count: [12]
                shard_number: ['01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12']

        steps:
           ## 
           ## multiple set up steps omitted for brevity
           ##
            -   name: Run tests
                run: ./bin/github_parallel_test_runner
                env:
                    TOTAL_SHARD_COUNT: ${{ matrix.total_shard_count }}
                    SHARD_NUMBER: ${{ matrix.shard_number }}

Here's the content of ./bin/github_parallel_test_runner in our code base. It will read the environment variables, execute Pest using the --list-files and --filter flags to only run a part of the tests like explained in the previous section of this post.

#!/usr/bin/env php
<?php

use Illuminate\Support\Collection;
use Illuminate\Support\Str;
use Symfony\Component\Process\Process;

require_once 'vendor/autoload.php';

$shardNumber = (int)getenv('SHARD_NUMBER');
$totalShardCount = (int)getenv('TOTAL_SHARD_COUNT');

if ($shardNumber === 0 || $totalShardCount === 0) {
    echo "SHARD_NUMBER and TOTAL_SHARD_COUNT must be set." . PHP_EOL;
    exit(1);
}

new ParallelTests($totalShardCount)->run($shardNumber);

class ParallelTests
{
    public function __construct(
        protected int $totalShardCount,
    )
    {
    }

    public function run(int $shardNumber): never
    {
        $testNames = $this->getTestNames($shardNumber);

        echo "Running {$testNames->count()} tests on node {$shardNumber} of {$this->totalShardCount}..." . PHP_EOL;

        $exitCode = $this->runPestTests($testNames);

        exit($exitCode);
    }

    /** @return Collection<string> */
    protected function getTestNames(int $shardNumber): Collection
    {
        $process = new Process([__DIR__ . '/../vendor/bin/pest', '--list-tests']);

        $process->mustRun();

        $index = $shardNumber - 1;

        $allTestNames = Str::of($process->getOutput())
            ->explode("\n")
            ->filter(fn(string $line) => str_contains($line, ' - '))
            ->map(function (string $fullTestName) {
                $testClassName = Str::of($fullTestName)
                    ->replace('- ', '')
                    ->trim()
                    ->between('\\\\', '::')
                    ->afterLast('\\')
                    ->toString();

                return $testClassName;
            })
            ->filter()
            ->unique();

        echo "Detected {$allTestNames->count()} tests:" . PHP_EOL;

        return $allTestNames
            ->split($this->totalShardCount)
            ->get($index);
    }

    protected function runPestTests(Collection $testNames): ?int
    {
        $process = new Process(
            command: ['./vendor/bin/pest', '--filter', $testNames->join('|')],
            timeout: null
        );

        $process->start();

        /* pipe the Pest output to the console */
        foreach ($process as $data) {
            echo $data;
        }

        $process->wait();

        return $process->getExitCode();
    }
}

To make this script executable you must execute this command and push the changes permissions...

chmod +x ./bin/github_parallel_test_runner

Only run composer and NPM / Yarn once

In the screenshot above, you could see how there's a "Composer and Yarn" step that is executing only once before all the test parts run. Here's that screenshot again.

A GitHub action workflow can contain multiple jobs, and you can define dependencies between them. In the snippet below you'll see the setup-dependencies job being defined (I've omitted all steps regarding to Yarn / NPM to keep things brief). We save the vendor directory as an artifact and use that saved directory in all of our test jobs. Finally, there's also a step to clean up any created artifacts.

You can see that in the needs key, you can define the steps that a job depends on.

#
# name, and concurrency setup omitted for brevity
#

jobs:
    setup-dependencies:
        name: Composer and Yarn
        runs-on: ubuntu-latest
        steps:
            -   uses: actions/checkout@v3
                with:
                    fetch-depth: 1

            -   name: Setup PHP
                uses: shivammathur/setup-php@v2
                with:
                    php-version: 8.4
                    extensions: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, soap, intl, gd, exif, iconv, imagick
                    coverage: none

            -   name: Install composer dependencies
                run: composer install --prefer-dist --no-scripts -q -o

            -   name: Upload vendor directory
                uses: actions/upload-artifact@v4
                with:
                    name: vendor-directory-${{ github.run_id }}
                    path: vendor
                    retention-days: 1

    run-tests:
        needs: [ setup-dependencies]
        name: Run Tests (Part ${{ matrix.shard_number }}/${{ matrix.total_shard_count }})
        runs-on: ubuntu-latest
        strategy:
            fail-fast: false
            matrix:
                total_shard_count: [12]
                shard_number: ['01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12']
        steps:
		        #
		        # Some setup steps omitted for brevity
		        #
        
            -   uses: actions/checkout@v3
                with:
                    fetch-depth: 1

            -   name: Download vendor directory
                uses: actions/download-artifact@v4
                with:
                    name: vendor-directory-${{ github.run_id }}
                    path: vendor

            -   name: Run tests
                run: ./bin/github-parallel-test-runner
                env:
                    TOTAL_SHARD_COUNT: ${{ matrix.total_shard_count }}
                    SHARD_NUMBER: ${{ matrix.shard_number }}

    cleanup-artifacts:
        name: Clean up artifacts
        needs: [setup-dependencies, run-tests]
        runs-on: ubuntu-latest
        if: always()
        steps:
            -   name: Delete artifacts
                uses: geekyeggo/delete-artifact@v2
                with:
                    name: |
                        vendor-directory-${{ github.run_id }}
                    failOnError: false

Cancelling stale tests

There's another neat little thing that we do in our GitHub action workflow to save some time. Whenever changes are pushed to a certain branch, we're not really interested in the results of the tests of any previous commits / pushes on that branch anymore.

Wouldn't it be nice if the test for any older commits on a branch were automatically cancelled, so the tests for the new commits / push would immediately start?

Well, with this snippet in your workflow, that's exactly what will happen.

name: Run tests

on:
    push:
        paths:
            - '**.php'
            - '.github/workflows/run-tests.yml'
            - 'phpunit.xml.dist'
            - 'composer.json'
            - 'composer.lock'

concurrency:
    group: ${{ github.workflow }}-${{ github.ref }}
    cancel-in-progress: true
    
jobs:
   #
   # omitted for brevity
   #

Our complete test workflow

To make things clear, here's our entire workflow file including the setup of all the services that our testsuite needs like MySQL, Redis, ClickHouse, Lighthouse and more.

name: Run tests

on:
    push:
        paths:
            - '**.php'
            - '.github/workflows/run-tests.yml'
            - 'phpunit.xml.dist'
            - 'composer.json'
            - 'composer.lock'

concurrency:
    group: ${{ github.workflow }}-${{ github.ref }}
    cancel-in-progress: true

jobs:
    setup-dependencies:
        name: Composer and Yarn
        runs-on: ubuntu-latest
        steps:
            -   uses: actions/checkout@v3
                with:
                    fetch-depth: 1

            -   name: Setup PHP
                uses: shivammathur/setup-php@v2
                with:
                    php-version: 8.4
                    extensions: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, soap, intl, gd, exif, iconv, imagick
                    coverage: none

            -   name: Get Composer Cache Directory
                id: composer-cache
                run: echo "dir=$(composer config cache-files-dir)" >> $GITHUB_OUTPUT

            -   name: Cache Composer dependencies
                uses: actions/cache@v3
                with:
                    path: ${{ steps.composer-cache.outputs.dir }}
                    key: ${{ runner.os }}-composer-${{ hashFiles('**/composer.lock') }}
                    restore-keys: ${{ runner.os }}-composer-

            -   name: Install composer dependencies
                run: composer install --prefer-dist --no-scripts -q -o

            -   name: Upload vendor directory
                uses: actions/upload-artifact@v4
                with:
                    name: vendor-directory-${{ github.run_id }}
                    path: vendor
                    retention-days: 1

            -   name: Cache Node Modules
                uses: actions/cache@v3
                with:
                    path: node_modules
                    key: ${{ runner.os }}-node-${{ hashFiles('**/yarn.lock') }}
                    restore-keys: ${{ runner.os }}-node-

            -   name: Cache built assets
                uses: actions/cache@v3
                with:
                    path: |
                        public/build
                        public/hot
                        public/css
                        public/js
                    key: ${{ runner.os }}-assets-${{ hashFiles('resources/**/*') }}-${{ hashFiles('**/yarn.lock') }}
                    restore-keys: ${{ runner.os }}-assets-

            -   name: Compile assets
                run: |
                    yarn install --pure-lockfile
                    yarn build
                # Only run build if node_modules or assets cache wasn't hit
                if: steps.node-cache.outputs.cache-hit != 'true' || steps.assets-cache.outputs.cache-hit != 'true'

            -   name: Upload compiled assets
                uses: actions/upload-artifact@v4
                with:
                    name: compiled-assets-${{ github.run_id }}
                    path: |
                        public/build
                        public/hot
                        public/css
                        public/js
                    retention-days: 1

    install-chrome-and-lighthouse:
        name: Install Chrome and Lighthouse
        runs-on: ubuntu-latest
        steps:
            -   uses: actions/checkout@v3
                with:
                    fetch-depth: 1

            -   name: Setup PHP
                uses: shivammathur/setup-php@v2
                with:
                    php-version: 8.4
                    extensions: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, soap, intl, gd, exif, iconv, imagick
                    coverage: none

            -   name: Setup problem matchers
                run: |
                    echo "::add-matcher::${{ runner.tool_cache }}/php.json"
                    echo "::add-matcher::${{ runner.tool_cache }}/phpunit.json"

            -   name: Install Chrome Launcher
                run: npm install chrome-launcher

            -   name: Install Lighthouse
                run: npm install lighthouse

            -   name: Cache test environment
                uses: actions/upload-artifact@v4
                with:
                    name: test-env-${{ github.run_id }}
                    path: |
                        node_modules
                    retention-days: 1

    run-tests:
        needs: [ setup-dependencies, install-chrome-and-lighthouse ]
        name: Run Tests (Part ${{ matrix.shard_number }}/${{ matrix.total_shard_count }})
        runs-on: ubuntu-latest
        strategy:
            fail-fast: false
            matrix:
                total_shard_count: [12]
                shard_number: ['01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12']

        services:
            mysql:
                image: mysql:8.0
                env:
                    MYSQL_ALLOW_EMPTY_PASSWORD: yes
                    MYSQL_DATABASE: ohdear_testing
                ports:
                    - 3306
                options: --health-cmd="mysqladmin ping" --health-interval=10s --health-timeout=5s --health-retries=3
            redis:
                image: redis
                ports:
                    - 6379:6379
                options: --entrypoint redis-server
            clickhouse:
                image: clickhouse/clickhouse-server
                options: >-
                    --health-cmd "clickhouse client -q 'SELECT 1'"
                    --health-interval 10s
                    --health-timeout 5s
                    --health-retries 5
                ports:
                    - 8123:8123
                    - 9000:9000
                    - 9009:9009
                env:
                    CLICKHOUSE_DB: ohdear
                    CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: 1

        steps:
            -   uses: actions/checkout@v3
                with:
                    fetch-depth: 1

            -   name: create db
                run: |
                    sudo /etc/init.d/mysql start
                    mysql  -u root -proot -e 'CREATE DATABASE IF NOT EXISTS ohdear_testing;'

            -   name: Setup PHP
                uses: shivammathur/setup-php@v2
                with:
                    php-version: 8.4
                    extensions: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, soap, intl, gd, exif, iconv, imagick
                    coverage: none

            -   name: Download test environment
                uses: actions/download-artifact@v4
                with:
                    name: test-env-${{ github.run_id }}
                    path: .

            -   name: Download vendor directory
                uses: actions/download-artifact@v4
                with:
                    name: vendor-directory-${{ github.run_id }}
                    path: vendor

            -   name: Download compiled assets
                uses: actions/download-artifact@v4
                with:
                    name: compiled-assets-${{ github.run_id }}
                    path: public

            -   name: Prepare Laravel Application
                run: |
                    cp .env.example .env
                    php artisan key:generate

            -   name: Set permissions for vendor binaries
                run: chmod -R +x vendor/bin/

            -   name: Run tests
                run: ./bin/github-parallel-test-runner
                env:
                    DB_PORT: ${{ job.services.mysql.ports[3306] }}
                    REDIS_PORT: ${{ job.services.redis.ports[6379] }}
                    CLICKHOUSE_HOST: localhost
                    CLICKHOUSE_PORT: 8123
                    CLICKHOUSE_DATABASE: ohdear
                    CLICKHOUSE_USERNAME: default
                    CLICKHOUSE_PASSWORD:
                    CLICKHOUSE_HTTPS: false
                    TOTAL_SHARD_COUNT: ${{ matrix.total_shard_count }}
                    SHARD_NUMBER: ${{ matrix.shard_number }}

    cleanup-artifacts:
        name: Clean up artifacts
        needs: [setup-dependencies, install-chrome-and-lighthouse, run-tests]
        runs-on: ubuntu-latest
        if: always()
        steps:
            -   name: Delete artifacts
                uses: geekyeggo/delete-artifact@v2
                with:
                    name: |
                        vendor-directory-${{ github.run_id }}
                        compiled-assets-${{ github.run_id }}
                        test-env-${{ github.run_id }}
                    failOnError: false

Pros and cons

There are pros and cons for running tests in parallel on GitHub actions. Let's start with the most important pro first: your test suite will run significantly faster. It's also very easy to increase or decrease the level of parallelism. A faster testsuite means that your entire feedback cycle is faster, which is a big win.

On the cons side, there's certainly some more complexity involved: you need a script to split tests, the workflow becomes more complex.

There's also the risk of uneven test distribution. If your tests vary significantly in execution time, you might end up with some shards finishing much earlier than others, reducing the efficiency gain.

In closing

The approach outlined in this post works exceptionally well for large test suites like we have, where we've seen significant reduction in test execution time. But even smaller projects can benefit from this technique, especially as they grow over time.