Running Tests

Introduction

In order to allow us to make changes quickly, often and with a high level of confidence, we heavily rely on tests within the web folder of the oCIS repository.

All the steps below require you to have the oCIS repository cloned locally and the dependencies of the web folder installed. All commands on this page are run from the web directory of the oCIS repository. This can be achieved by running

$ git clone https://github.com/owncloud/ocis.git
$ cd ocis/web
$ pnpm install

Unit Tests

We have a steadily growing coverage of unit tests. You can run them locally via

$ pnpm test:unit
$ pnpm -r test:unit

You can also specify which package to run the test on, such as: pnpm --filter @ownclouders/web-pkg test:unit. Alternatively, tests can also be run by navigating to the package name and then running pnpm test:unit.

Unit Test File Structure

Our unit tests spec files follow a simple structure:

  • fixtures and mocks at the top

  • helper functions at the bottom

  • tests in between

We usually organize tests with nested describe blocks. If you would like to get feedback from the core team about the structure, scope and goals of your unit tests before actually writing some, we invite you to make a pull request with only describe blocks and nested it.todo("put your test description here") lines.

E2E Tests (Playwright)

Our end-to-end test suite is built upon the Playwright Framework, which makes it easy to write tests, debug them and have them run cross-browser with minimal overhead.

Preparation

Please make sure you have installed all dependencies and started the server(s) as described in tooling.

Prepare Web

Bundle the web frontend with the following command:

$ pnpm build

Our compose setup automatically mounts it into an oCIS backend, respectively. To have web recompiled on changes, use pnpm build:w instead.

Start Web

Start the web with the following command:

docker compose up

This will start all the services. The ENV variables specific to each services are defined in the docker-compose.yml file.

Run E2E Tests

The following command will run all available e2e tests:

$ pnpm test:e2e:playwright

To run only the tests in a specific folder, add its path, e.g. pnpm test:e2e:playwright tests/e2e/specs/smoke/.

Options

To run a particular test, simply add the spec file and line number to the test command, e.g. pnpm test:e2e:playwright tests/e2e/specs/admin-settings/users.spec.ts:14

Various options are available via ENV variables, e.g.

  • BASIC_AUTH=true use basic authorization for api requests.

  • RETRY=n to retry failures n times

  • SLOW_MO=n to slow the execution time by n milliseconds

  • TIMEOUT=n to set tests to timeout after n seconds (defaults to 180)

  • HEADLESS=true to run the browser in headless mode (by default, the browser window is opened while the tests run)

  • BROWSER=name to run tests against a specific browser. Defaults to chrome (the Google Chrome channel, which must be installed locally), available are chrome, chromium, firefox and webkit

  • ADMIN_PASSWORD to set administrator password. By default, the admin password is used in the test

The Playwright configuration defines the projects chromium, firefox and webkit. Without the --project option, the tests run once for every project. To run them only once, pass a project together with a matching BROWSER, e.g. BROWSER=chromium pnpm test:e2e:playwright --project=chromium. The tests/e2e/run-e2e.sh script used in CI does this, with BROWSER defaulting to chromium.

For debugging reasons, you may want to record a video or traces of your test run. Again, you can use the following ENV variables in your command:

  • REPORT_DIR=another/path to set a directory for your recorded files (defaults to reports/e2e, relative to the web directory)

  • REPORT_VIDEO=true to record a video of the test run

  • REPORT_HAR=true to save request information from the test run

  • REPORT_TRACING=true to record traces from the test run

To then open e.g. the tracing from the REPORT_DIR, run

$ npx playwright show-trace path/to/file.zip

Lint E2E Test Code

Run the following command to find out the lint issues early in the test codes:

$ pnpm lint

And to fix the lint problems run the following command:

$ pnpm lint --fix

If the lint problems are not fixed by --fix option, we have to manually fix the code.

Analyze the Test Report

After running tests, report artifacts are written under REPORT_DIR (defaults to reports/e2e).

  • Accessibility report: reports/e2e/a11y-report.json

  • Traces (when REPORT_TRACING=true): reports/e2e/playwright/tracing/*.zip

  • Optional videos/HAR files (when enabled via REPORT_VIDEO=true or REPORT_HAR=true) are also stored in the report directory.

To inspect a trace file:

npx playwright show-trace reports/e2e/playwright/tracing/<trace-file>.zip

If you want an HTML Playwright report for a run, execute tests with the HTML reporter enabled and then open it. Passing --reporter replaces the reporters configured in playwright.config.ts, so keep list in the list. Note that the accessibility report is only written by the configured reporters and is not created in this case:

pnpm test:e2e:playwright --reporter=list,html
npx playwright show-report

E2E Tests on oCIS with Keycloak

We can run some of the e2e tests on oCIS setup with Keycloak as an external idp. To run tests against locally, please follow the steps below:

Run oCIS with Keycloak

There’s a documentation to serve oCIS with Keycloak. Please follow each step to run oCIS with Keycloak.

Run E2E Tests

KEYCLOAK=true \
BASE_URL_OCIS=ocis.owncloud.test \
pnpm run test:e2e:playwright tests/e2e/specs/journeys

Following environment variables come in use while running e2e tests on oCIS with Keycloak:

  • BASE_URL_OCIS
    sets oCIS url (e.g.: ocis.owncloud.test)

  • KEYCLOAK_HOST
    sets Keycloak url (e.g.: keycloak.owncloud.test)

  • KEYCLOAK=true
    runs the tests with Keycloak

  • KEYCLOAK_REALM
    sets oCIS realm name used on Keycloak

E2E Tests with Predefined Users (@predefined-users)

It is possible to run e2e tests with predefined users. This is useful for running tests in a production-like environment. The following environment variables are used to run the tests with predefined users:

  • PREDEFINED_USERS: true|false

  • PREDEFINED_USERS_FILE: path to a JSON file mapping predefined users

We have to create a JSON file that contains the mapping of the users. JSON file MUST contain the following keys:

{
 "alice": {// map user},
 "brian": {// mapuser},
 "carol": {// mapuser},
}

And the user object MUST have the following properties defined:

{
  "id": "<usernmae>",
  "displayName": "<display-name>",
  "password": "<password>",
  "email": "<email>"
}

A complete example of a JSON file.

{
  "alice": {
    "id": "einstein",
    "displayName": "Albert Einstein",
    "password": "relativity",
    "email": "einstein@example.org"
  },
  "brian": {
    "id": "marie",
    "displayName": "Marie Skłodowska Curie",
    "password": "radioactivity",
    "email": "marie@example.org"
  },
  "carol": {
    "id": "moss",
    "displayName": "Maurice Moss",
    "password": "vista",
    "email": "moss@example.org"
  }
}

The test scenarios that can run with predefined users are marked with the @predefined-users tag, e.g. test.describe('rename', { tag: '@predefined-users' }, …​), and can be run with the following command:

PREDEFINED_USERS=true \
PREDEFINED_USERS_FILE='<path-to>/users.json' \
pnpm test:e2e:playwright tests/e2e/specs/file-action/rename.spec.ts --grep @predefined-users

The following tests cannot be run with predefined users:

All tests which are related to:

  • Admin Actions

  • Groups

The tests might show flakiness or fail due to the following reasons:

  • Slower network connection

  • Features enabled/disabled

  • Running latest tests against an older version of oCIS/Web

  • Large file uploads may take longer time

Test Suite Selection in CI

In CI, the e2e tests run in the e2e-tests job of the .github/workflows/acceptance-tests.yml workflow. Each entry of the job matrix passes arguments to tests/acceptance/run-e2e.py, which starts the required services and calls web/tests/e2e/run-e2e.sh with these arguments. A suite is a folder in tests/e2e/specs, e.g. admin-settings.

  • --suites runs only the given suites, comma separated, e.g. --suites search.

  • --xsuites excludes the given suites, e.g. --xsuites search,ocm.

  • --total-parts and --run-part split the remaining suites into groups that run in parallel jobs.

Suites are discovered from the folders in tests/e2e/specs, so a new suite is picked up by the part-* matrix entries automatically. Suites that need additional services, such as search, keycloak or ocm, are excluded there via --xsuites and run in their own matrix entries. A matrix entry can also pass spec file paths instead of suites.