# Running our test suite in parallel on GitHub actions

A couple of years ago, Laravel introduced a great feature which allows to run PHPUnit / Pest tests in parallel. This results in a big boost in performance. 

First we had to do the groundwork: [getting our Laravel suite ready for parallel testing](/news-and-updates/making-our-laravel-test-suite-ready-for-parallel-testing).

By default, it determines the concurrency level by taking a look at the number of CPU cores your machine has. So, if you're using a modern Mac that has 10 CPU cores, it will run 10 tests at the same time, greatly cutting down on the time your testsuite needs to run completely.

A default runner on GitHub doesn't have that many cores, so you can't leverage parallel testing as good as in your typical local environments.

In this blog post, I'd like to show you a way of running your tests on GitHub, by splitting them up in small chunks that can run concurrently.vWe use this technique to cut down the running time of our vast testsuite from 16 minutes to only just 4. In this blog post all examples will come from our base.

<!--more-->

## What we're going to do

Like already mentioned in the introduction, a typical test runner on GitHub hasn't got a lot of cores, meaning the default way of running tests in parallel that Laravel offers (running a test per CPU core) doesn't work well.

What you can do at GitHub however is running a lot of GitHub actions jobs in parallel. So what we are going to do is splitting our testsuite in equal chunks, and create a test job on GitHub action per chunk. These chunks can run in parallel, which will immensely decrease the total running time.

I'll tell you how to achieve this technically in the remainder of this blog post, but here's already how the end result will look like on GitHub.

![GitHub Actions run summary with twelve parallel Run Tests jobs all passing in 6m 12s total](/media/blog/ahsGwZzeKjPu9lOG4L2ubjF8xAVOrmfDBWekhzkF.png)

In the screenshot above you can see that our test suite is split up in 12 parts which will all run concurrently. Composer / NPM will only run once to build up the dependencies and assets and they will be used in all 12 testing parts.

## Splitting up the testsuite in equal parts

Let's first take a look at how we can split the testsuite in equal parts. We use Pest as a test runner, which offers a `--list-tests` option to output all tests.

![Terminal running 'vendor/bin/pest --list-tests' showing the available Oh Dear Pest test names](/media/blog/fCyxyPAg8N2HABEnL3io7DTLTpmruuTs9gnMvxWR.png)

Here's a bit of code to get all test class names from that output.

```php
 $process = new Process([__DIR__ . '/../vendor/bin/pest', '--list-tests']);

$process->mustRun();

$index = $shardNumber - 1;

$allTestNames = Str::of($process->getOutput())
    ->explode("\n")
    ->filter(fn(string $line) => str_contains($line, ' - '))
    ->map(function (string $fullTestName) {
        $testClassName = Str::of($fullTestName)
            ->replace('- ', '')
            ->trim()
            ->between('\\\\', '::')
            ->afterLast('\\')
            ->toString();

        return $testClassName;
    })
    ->filter()
    ->unique();
```

In `$allTestNames` will be a collection containing all class names (= file names) that are inside the test suite.

To split the collection up in multiple parts, you can use the `split` function which accepts the number of parts you want. Here's how you would split up the tests in 12 parts, and get the first part

```php
$testNamesOfFirstPart = $allTestNames
   ->split(12) // split the collection in 12 equal parts
   ->get(key: 0) // get the first part (the index is 0 based)
```

PHPUnit / Pest also offers a `--filter` option to only run specific tests. If you only want to run the tests in from the `ArchTest` class (which is displayed in the screenshot above), you could execute this.

```bash
# Will only run the tests from the ArchTest file
vendor/bin/pest --filter=ArchTest
```

You can use `|` to specify multiple patterns. Here's how you could execute the tests from multiple files

```bash
# Will only run the tests from the ArchTest file
vendor/bin/pest --filter=ArchTest|CheckSitesBeingMonitoredTest
```

Here's how you could use the `$testNamesOfFirstPart` from the previous snippet to run the first part of the tests programmatically.

```php
$process = new Process(
    command: [
       './vendor/bin/pest',
       '--filter', 
       $testNamesOfFirstPart->join('|')],
       timeout: null // take as much time as we need
);

$process->start();

/* pipe the Pest output to the console */
foreach ($process as $data) {
    echo $data;
}

$process->wait();

// use the exit code of Pest as the exit code of the script
exit($process->getExitCode());
```

## Running the testsuite parts in parallel at GitHub

Now that you know how you could split up a test suite in equal parts, let's take a look at how we can run all these parts in parallel on GitHub actions.

GitHub actions support [a `matrix` parameter](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/running-variations-of-jobs-in-a-workflow). Shortly said, this matrix parameter is used for testing variations of your test suite, and it will run those variations concurrently.

Here's the part of the Oh Dear GitHub workflow where the matrix is being set up. I've omitted several parts for brevity.

```yaml
# .github/workflows/run-tests.yml

name: Run tests

jobs:
    run-tests:
        name: Run Tests (Part ${{ matrix.shard_number }}/${{ matrix.total_shard_count }})
        runs-on: ubuntu-latest
        strategy:
            matrix:
                total_shard_count: [12]
                shard_number: ['01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12']

        steps:
           ## 
           ## multiple set up steps omitted for brevity
           ##
            -   name: Run tests
                run: ./vendor/bin/pest
```

The matrix will create jobs per combination in the matrix. So it will run 1 (only one element in `total_shard_count`) * 12 (twelve elements in `shard_number`) = 12 times.

In the `Run tests` step, `pest is executed`. This will result in the whole testsuite being executed 12 times. Of course we don't want to execute the whole test suite 12 times, but only each separate 1/12 part of the testsuite.

We can achieve this by not running `/vendor/bin/pest` but a custom PHP script called `github_parallel_test_runner` that will receive the `total_shard_count` and the  `shard_number` as environment variables.

```yaml
# .github/workflows/run-tests.yml

name: Run tests

jobs:
    run-tests:
        name: Run Tests (Part ${{ matrix.shard_number }}/${{ matrix.total_shard_count }})
        runs-on: ubuntu-latest
        strategy:
            matrix:
                total_shard_count: [12]
                shard_number: ['01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12']

        steps:
           ## 
           ## multiple set up steps omitted for brevity
           ##
            -   name: Run tests
                run: ./bin/github_parallel_test_runner
                env:
                    TOTAL_SHARD_COUNT: ${{ matrix.total_shard_count }}
                    SHARD_NUMBER: ${{ matrix.shard_number }}
```

Here's the content of `./bin/github_parallel_test_runner` in our code base. It will read the environment variables, execute Pest using the `--list-files` and `--filter` flags to only run a part of the tests like explained in the previous section of this post.

```php
#!/usr/bin/env php
<?php

use Illuminate\Support\Collection;
use Illuminate\Support\Str;
use Symfony\Component\Process\Process;

require_once 'vendor/autoload.php';

$shardNumber = (int)getenv('SHARD_NUMBER');
$totalShardCount = (int)getenv('TOTAL_SHARD_COUNT');

if ($shardNumber === 0 || $totalShardCount === 0) {
    echo "SHARD_NUMBER and TOTAL_SHARD_COUNT must be set." . PHP_EOL;
    exit(1);
}

new ParallelTests($totalShardCount)->run($shardNumber);

class ParallelTests
{
    public function __construct(
        protected int $totalShardCount,
    )
    {
    }

    public function run(int $shardNumber): never
    {
        $testNames = $this->getTestNames($shardNumber);

        echo "Running {$testNames->count()} tests on node {$shardNumber} of {$this->totalShardCount}..." . PHP_EOL;

        $exitCode = $this->runPestTests($testNames);

        exit($exitCode);
    }

    /** @return Collection<string> */
    protected function getTestNames(int $shardNumber): Collection
    {
        $process = new Process([__DIR__ . '/../vendor/bin/pest', '--list-tests']);

        $process->mustRun();

        $index = $shardNumber - 1;

        $allTestNames = Str::of($process->getOutput())
            ->explode("\n")
            ->filter(fn(string $line) => str_contains($line, ' - '))
            ->map(function (string $fullTestName) {
                $testClassName = Str::of($fullTestName)
                    ->replace('- ', '')
                    ->trim()
                    ->between('\\\\', '::')
                    ->afterLast('\\')
                    ->toString();

                return $testClassName;
            })
            ->filter()
            ->unique();

        echo "Detected {$allTestNames->count()} tests:" . PHP_EOL;

        return $allTestNames
            ->split($this->totalShardCount)
            ->get($index);
    }

    protected function runPestTests(Collection $testNames): ?int
    {
        $process = new Process(
            command: ['./vendor/bin/pest', '--filter', $testNames->join('|')],
            timeout: null
        );

        $process->start();

        /* pipe the Pest output to the console */
        foreach ($process as $data) {
            echo $data;
        }

        $process->wait();

        return $process->getExitCode();
    }
}
```

To make this script executable you must execute this command and push the changes permissions...

```bash
chmod +x ./bin/github_parallel_test_runner
```

## Only run composer and NPM / Yarn once

In the screenshot above, you could see how there's a "Composer and Yarn" step that is executing only once before all the test parts run. Here's that screenshot again.

![GitHub Actions run summary with twelve parallel Run Tests jobs all passing in 6m 12s total](/media/blog/CNKYnLy3HB8ynxEI90sHqHsKU1B0YpkaH4zV2LZy.png)

A GitHub action workflow can contain multiple jobs, and you can define dependencies between them. In the snippet below you'll see the `setup-dependencies` job being defined (I've omitted all steps regarding to Yarn / NPM to keep things brief). We save the `vendor` directory as an artifact and use that saved directory in all of our test jobs. Finally, there's also a step to clean up any created artifacts.

You can see that in the `needs` key, you can define the steps that a job depends on.

```yaml

#
# name, and concurrency setup omitted for brevity
#

jobs:
    setup-dependencies:
        name: Composer and Yarn
        runs-on: ubuntu-latest
        steps:
            -   uses: actions/checkout@v3
                with:
                    fetch-depth: 1

            -   name: Setup PHP
                uses: shivammathur/setup-php@v2
                with:
                    php-version: 8.4
                    extensions: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, soap, intl, gd, exif, iconv, imagick
                    coverage: none

            -   name: Install composer dependencies
                run: composer install --prefer-dist --no-scripts -q -o

            -   name: Upload vendor directory
                uses: actions/upload-artifact@v4
                with:
                    name: vendor-directory-${{ github.run_id }}
                    path: vendor
                    retention-days: 1

    run-tests:
        needs: [ setup-dependencies]
        name: Run Tests (Part ${{ matrix.shard_number }}/${{ matrix.total_shard_count }})
        runs-on: ubuntu-latest
        strategy:
            fail-fast: false
            matrix:
                total_shard_count: [12]
                shard_number: ['01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12']
        steps:
		        #
		        # Some setup steps omitted for brevity
		        #
        
            -   uses: actions/checkout@v3
                with:
                    fetch-depth: 1

            -   name: Download vendor directory
                uses: actions/download-artifact@v4
                with:
                    name: vendor-directory-${{ github.run_id }}
                    path: vendor

            -   name: Run tests
                run: ./bin/github-parallel-test-runner
                env:
                    TOTAL_SHARD_COUNT: ${{ matrix.total_shard_count }}
                    SHARD_NUMBER: ${{ matrix.shard_number }}

    cleanup-artifacts:
        name: Clean up artifacts
        needs: [setup-dependencies, run-tests]
        runs-on: ubuntu-latest
        if: always()
        steps:
            -   name: Delete artifacts
                uses: geekyeggo/delete-artifact@v2
                with:
                    name: |
                        vendor-directory-${{ github.run_id }}
                    failOnError: false
```


## Cancelling stale tests

There's another neat little thing that we do in our GitHub action workflow to save some time. Whenever changes are pushed to a certain branch, we're not really interested in the results of the tests of any previous commits / pushes on that branch anymore.

Wouldn't it be nice if the test for any older commits on a branch were automatically cancelled, so the tests for the new commits / push would immediately start?

Well, with this snippet in your workflow, that's exactly what will happen.

```yaml
name: Run tests

on:
    push:
        paths:
            - '**.php'
            - '.github/workflows/run-tests.yml'
            - 'phpunit.xml.dist'
            - 'composer.json'
            - 'composer.lock'

concurrency:
    group: ${{ github.workflow }}-${{ github.ref }}
    cancel-in-progress: true
    
jobs:
   #
   # omitted for brevity
   #
```

## Our complete test workflow

To make things clear, here's our entire workflow file including the setup of all the services that our testsuite needs like MySQL, Redis, ClickHouse, Lighthouse and more.

This all runs inside [our GitLab CI pipeline](/news-and-updates/our-gitlab-ci-pipeline-for-laravel-applications).

```yaml
name: Run tests

on:
    push:
        paths:
            - '**.php'
            - '.github/workflows/run-tests.yml'
            - 'phpunit.xml.dist'
            - 'composer.json'
            - 'composer.lock'

concurrency:
    group: ${{ github.workflow }}-${{ github.ref }}
    cancel-in-progress: true

jobs:
    setup-dependencies:
        name: Composer and Yarn
        runs-on: ubuntu-latest
        steps:
            -   uses: actions/checkout@v3
                with:
                    fetch-depth: 1

            -   name: Setup PHP
                uses: shivammathur/setup-php@v2
                with:
                    php-version: 8.4
                    extensions: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, soap, intl, gd, exif, iconv, imagick
                    coverage: none

            -   name: Get Composer Cache Directory
                id: composer-cache
                run: echo "dir=$(composer config cache-files-dir)" >> $GITHUB_OUTPUT

            -   name: Cache Composer dependencies
                uses: actions/cache@v3
                with:
                    path: ${{ steps.composer-cache.outputs.dir }}
                    key: ${{ runner.os }}-composer-${{ hashFiles('**/composer.lock') }}
                    restore-keys: ${{ runner.os }}-composer-

            -   name: Install composer dependencies
                run: composer install --prefer-dist --no-scripts -q -o

            -   name: Upload vendor directory
                uses: actions/upload-artifact@v4
                with:
                    name: vendor-directory-${{ github.run_id }}
                    path: vendor
                    retention-days: 1

            -   name: Cache Node Modules
                uses: actions/cache@v3
                with:
                    path: node_modules
                    key: ${{ runner.os }}-node-${{ hashFiles('**/yarn.lock') }}
                    restore-keys: ${{ runner.os }}-node-

            -   name: Cache built assets
                uses: actions/cache@v3
                with:
                    path: |
                        public/build
                        public/hot
                        public/css
                        public/js
                    key: ${{ runner.os }}-assets-${{ hashFiles('resources/**/*') }}-${{ hashFiles('**/yarn.lock') }}
                    restore-keys: ${{ runner.os }}-assets-

            -   name: Compile assets
                run: |
                    yarn install --pure-lockfile
                    yarn build
                # Only run build if node_modules or assets cache wasn't hit
                if: steps.node-cache.outputs.cache-hit != 'true' || steps.assets-cache.outputs.cache-hit != 'true'

            -   name: Upload compiled assets
                uses: actions/upload-artifact@v4
                with:
                    name: compiled-assets-${{ github.run_id }}
                    path: |
                        public/build
                        public/hot
                        public/css
                        public/js
                    retention-days: 1

    install-chrome-and-lighthouse:
        name: Install Chrome and Lighthouse
        runs-on: ubuntu-latest
        steps:
            -   uses: actions/checkout@v3
                with:
                    fetch-depth: 1

            -   name: Setup PHP
                uses: shivammathur/setup-php@v2
                with:
                    php-version: 8.4
                    extensions: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, soap, intl, gd, exif, iconv, imagick
                    coverage: none

            -   name: Setup problem matchers
                run: |
                    echo "::add-matcher::${{ runner.tool_cache }}/php.json"
                    echo "::add-matcher::${{ runner.tool_cache }}/phpunit.json"

            -   name: Install Chrome Launcher
                run: npm install chrome-launcher

            -   name: Install Lighthouse
                run: npm install lighthouse

            -   name: Cache test environment
                uses: actions/upload-artifact@v4
                with:
                    name: test-env-${{ github.run_id }}
                    path: |
                        node_modules
                    retention-days: 1

    run-tests:
        needs: [ setup-dependencies, install-chrome-and-lighthouse ]
        name: Run Tests (Part ${{ matrix.shard_number }}/${{ matrix.total_shard_count }})
        runs-on: ubuntu-latest
        strategy:
            fail-fast: false
            matrix:
                total_shard_count: [12]
                shard_number: ['01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12']

        services:
            mysql:
                image: mysql:8.0
                env:
                    MYSQL_ALLOW_EMPTY_PASSWORD: yes
                    MYSQL_DATABASE: ohdear_testing
                ports:
                    - 3306
                options: --health-cmd="mysqladmin ping" --health-interval=10s --health-timeout=5s --health-retries=3
            redis:
                image: redis
                ports:
                    - 6379:6379
                options: --entrypoint redis-server
            clickhouse:
                image: clickhouse/clickhouse-server
                options: >-
                    --health-cmd "clickhouse client -q 'SELECT 1'"
                    --health-interval 10s
                    --health-timeout 5s
                    --health-retries 5
                ports:
                    - 8123:8123
                    - 9000:9000
                    - 9009:9009
                env:
                    CLICKHOUSE_DB: ohdear
                    CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: 1

        steps:
            -   uses: actions/checkout@v3
                with:
                    fetch-depth: 1

            -   name: create db
                run: |
                    sudo /etc/init.d/mysql start
                    mysql  -u root -proot -e 'CREATE DATABASE IF NOT EXISTS ohdear_testing;'

            -   name: Setup PHP
                uses: shivammathur/setup-php@v2
                with:
                    php-version: 8.4
                    extensions: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, soap, intl, gd, exif, iconv, imagick
                    coverage: none

            -   name: Download test environment
                uses: actions/download-artifact@v4
                with:
                    name: test-env-${{ github.run_id }}
                    path: .

            -   name: Download vendor directory
                uses: actions/download-artifact@v4
                with:
                    name: vendor-directory-${{ github.run_id }}
                    path: vendor

            -   name: Download compiled assets
                uses: actions/download-artifact@v4
                with:
                    name: compiled-assets-${{ github.run_id }}
                    path: public

            -   name: Prepare Laravel Application
                run: |
                    cp .env.example .env
                    php artisan key:generate

            -   name: Set permissions for vendor binaries
                run: chmod -R +x vendor/bin/

            -   name: Run tests
                run: ./bin/github-parallel-test-runner
                env:
                    DB_PORT: ${{ job.services.mysql.ports[3306] }}
                    REDIS_PORT: ${{ job.services.redis.ports[6379] }}
                    CLICKHOUSE_HOST: localhost
                    CLICKHOUSE_PORT: 8123
                    CLICKHOUSE_DATABASE: ohdear
                    CLICKHOUSE_USERNAME: default
                    CLICKHOUSE_PASSWORD:
                    CLICKHOUSE_HTTPS: false
                    TOTAL_SHARD_COUNT: ${{ matrix.total_shard_count }}
                    SHARD_NUMBER: ${{ matrix.shard_number }}

    cleanup-artifacts:
        name: Clean up artifacts
        needs: [setup-dependencies, install-chrome-and-lighthouse, run-tests]
        runs-on: ubuntu-latest
        if: always()
        steps:
            -   name: Delete artifacts
                uses: geekyeggo/delete-artifact@v2
                with:
                    name: |
                        vendor-directory-${{ github.run_id }}
                        compiled-assets-${{ github.run_id }}
                        test-env-${{ github.run_id }}
                    failOnError: false

```

## Pros and cons

There are pros and cons for running tests in parallel on GitHub actions. Let's start with the most important pro first: your test suite will run significantly faster. It's also very easy to increase or decrease the level of parallelism. A faster testsuite means that your entire feedback cycle is faster, which is a big win.

On the cons side, there's certainly some more complexity involved: you need a script to split tests, the workflow becomes more complex.

There's also the risk of uneven test distribution. If your tests vary significantly in execution time, you might end up with some shards finishing much earlier than others, reducing the efficiency gain. 

## In closing

The approach outlined in this post works exceptionally well for large test suites like we have, where we've seen significant reduction in test execution time. But even smaller projects can benefit from this technique, especially as they grow over time.

Fast CI matters to us because Oh Dear is built [for Laravel and PHP developers](/for/php-developers) shipping every day.