Playwright Sharding for GitHub Actions v4

Update sharding and merging in GitHub Actions from actions/upload-artifact@v3 and actions/download-artifact@v3 to v4.

Update artifacts from v3 to v4

UPDATE:

The example of the playwright docs is still for v3 but the main branch is updated see github.

If you use playwrights GitHub Actions example with actions/upload-artifact@v3 and actions/download-artifact@v3 you need to do more than just change @v3 to @v4.

See the breaking changes of upload-artifact@v4 and download-artifact@v4.

Both actions must have the same version v4. But the real problem is this breaking change:

Uploading to the same named Artifact multiple times

Due to how Artifacts are created in this new version, it is no longer possible to upload to the same named Artifact multiple times.

The v3 example uses the same named Artifact all-blob-reports.

In the following example the uploads are split into multiple Artifacts with different names and then merged.

GitHub Actions example

The old matrix was shard: [1/4, 2/4, 3/4, 4/4]. We need to pass 1/4 as parameter to playwright test but want to use it to name the artifacts, too. The / character is not allowed in artifacts name, therefore we split the matrix in shard: [1, 2, 3, 4] and of: [4].

1. Then we run our Playwright tests with the --shard ${{ matrix.shard }}/${{ matrix.of }} option. This will run our test command for each shard.

name: Playwright Tests
on:
  push:
    branches: [ main, master ]
  pull_request:
    branches: [ main, master ]
jobs:
  playwright-tests:
    timeout-minutes: 60
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
        of: [4]
    steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-node@v4
      with:
        node-version: 18
    - name: Install dependencies
      run: npm ci
    - name: Install Playwright browsers
      run: npx playwright install --with-deps

    - name: Run Playwright tests
      run: npx playwright test --shard ${{ matrix.shard }}/${{ matrix.of }}

    - name: Upload blob report to GitHub Actions Artifacts
      if: always()
      uses: actions/upload-artifact@v4
      with:
        name: blob-report-${{ matrix.shard }}
        path: blob-report
        retention-days: 1

1. After all shards have completed, you can run a separate job that will merge the reports and produce a combined HTML report. To ensure the execution order, we make the merge-reports job depend on our sharded playwright-tests job by adding needs: [playwright-tests].

jobs:
...
  merge-reports:
    # Merge reports after playwright-tests, even if some shards have failed
    if: always()
    needs: [playwright-tests]

    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-node@v4
      with:
        node-version: 18
    - name: Install dependencies
      run: npm ci

    - name: Download blob reports from GitHub Actions Artifacts
      uses: actions/download-artifact@v4
      with:
        path: all-blob-reports
        pattern: blob-report-*
        merge-multiple: true

    - name: Merge into HTML Report
      run: npx playwright merge-reports --reporter html ./all-blob-reports 

    - name: Upload HTML report
      uses: actions/upload-artifact@v4
      with:
        name: html-report--attempt-${{ github.run_attempt }}
        path: playwright-report
        retention-days: 14

You can now see the reports have been merged and a combined HTML report is available in the GitHub Actions Artifacts tab.

Questions, comments, ideas?

Just open an issue on the repo or get in contact via mastodon