Publish Playwright Test Reports with GitLab CI/CD Artifacts

Aron Schüler Published Dec 3, 2024

Introduction

Testing is a critical part of web development, and tools like Playwright make it easier to automate end-to-end testing for web applications. Especially with their codegen feature, I enjoy writing tests with Playwright and prefer it over tools like Cypress or Selenium.

However, debugging tests can be a hassle. If you run them in a pipeline, it can be quite hard to debug failing tests without proper reports. Sharing test results with your team to collaborate on fixing issues can also be a challenge.

Luckily, Gitlab CI/CD allows us to easily publish html output as artifacts, making it easy to share test results with your peers.

In this guide, we’ll walk you through configuring Playwright and GitLab CI/CD to generate and publish HTML test reports, making debugging and team collaboration much smoother.

Step-by-Step Guide

Configure Playwright to Generate HTML Reports

Playwright supports various reporters for its test results. To generate an HTML report, you’ll need to configure the playwright.config.ts or playwright.config.js file to use the html reporter.

Here’s how your configuration should look:

// playwright.config.js or playwright.config.ts
const config = {
  // Other configurations...
  reporter: [["html", { outputFolder: "out/report", open: "never" }]],
};
export default config;

This will generate an HTML report in the out/report directory after running the Playwright tests. The open: "never" option prevents the report from opening in the browser automatically, which would block the CI/CD pipeline and cause timeouts.

Set Up GitLab CI/CD Pipeline

Next, configure your .gitlab-ci.yml file to run the Playwright tests and provide the generated report as an artifact. Here’s a sample pipeline configuration:

Code Block for Gitlab CI

stages:
  - e2e
 
playwright:
  stage: e2e
  image: mcr.microsoft.com/playwright:v1.49.0-noble
  script:
    - npx playwright test
  except:
    - release
  after_script:
    - echo "Report will be available at https://$CI_PROJECT_ROOT_NAMESPACE.$CI_PAGES_DOMAIN/-/$CI_PROJECT_NAME/-/jobs/$CI_JOB_ID/artifacts/out/report/index.html"
  artifacts:
    paths:
      - out/report/
    when: on_failure

Explanation of Gitlab CI configuration

• image: Specifies the container image with Playwright pre-installed.
• script: Runs the Playwright tests using npx.
• except: Excludes the pipeline stage for specific branches like release.
• after_script: Outputs a clickable URL to the report in the job logs.
• artifacts: Stores the out/report directory with the playwright content as a pipeline artifact, making it available as Gitlab Pages content.

The pipeline will generate the HTML report and make it accessible as an artifact link in the GitLab job interface.

View the Report

Once the pipeline completes (successful or failed), you’ll find a link to the report in the pipeline job’s logs. The output you will look for looks something like this:

Report will be available at "https://<namespace>.<domain>/-/<project_name>/-/jobs/<job_id>/artifacts/out/report/index.html"

Clicking the link will open the Playwright HTML report, allowing you to inspect test results, trace failures, and debug issues effectively. I also like the playwright’s Visual Comparison features, as it’s great to use paired with these test reports.

Optional: Publish Report with GitLab Pages

If you’d like to make the report persistently accessible via GitLab Pages, extend the .gitlab-ci.yml configuration like this:

pages:
  stage: deploy
  script:
    - mkdir -p public
    - cp -r out/report/* public/
  artifacts:
    paths:
      - public

This step deploys the report to GitLab Pages under your project’s domain, making it publicly or privately accessible depending on your repository settings. But this is probably something that you’d rather use for feature branches or other test deployments of e. g. your SPA. If you’d like to know more about this, leave a comment down below, and I’ll happily provide a tutorial for that, too!

Conclusion

By following these steps, you can integrate Playwright’s reporting features with GitLab CI/CD, making it easy to debug failing pipelines. Especially when combined with Playwrights screenshot matching, you’ll end up with a nice way of debugging your end-to-end tests.

If you have questions or want to share your experience setting this up, feel free to drop a comment below. Happy testing!