User Interface Testing

Requirements

  • ownCloud >= 10.0. Make sure you have a running instance of ownCloud setup completely.

  • Default language set to en (in config/config.php set 'default_language' ⇒ 'en',).

  • An admin user called admin with the password admin.

  • No self-signed SSL certificates.

  • The testing app installed and enabled.

  • Testing utils (running make in your terminal from the webroot directory will install them).

  • Docker CE Installed

  • Docker Post-install done to put your developer account in the docker group so you can run Docker without sudo

  • Docker subnet enabled for any firewall that may be active such as, ufw. The example below shows how to update UFW’s firewall rules to allow the 172.17.0.0/16 Docker subnet:

    sudo ufw status
sudo ufw allow from 172.17.0.0/16

  • Docker containers pulled. It is recommended to use standalone-chrome-debug which allows seeing the browser live. The latest standalone-chrome-* containers have an issue. So make sure to pull the specific chrome container versions listed below. You will also need InBucket. Pull any or all of these Docker containers:

docker pull selenium/standalone-chrome:3.141.59-oxygen
docker pull selenium/standalone-chrome-debug:3.141.59-oxygen
docker pull selenium/standalone-firefox
docker pull selenium/standalone-firefox-debug
docker pull inbucket/inbucket

  • A vnc viewer installed (in order to view the browser action as the UI tests run). For example:

sudo apt install tigervnc-viewer

  • To run the Selenium server locally (not in Docker) see the notes at the end.

Overview

Tests are divided into suites, enabling each suite to test some logical portion of the functionality and for the total elapsed run-time of a single suite to be reasonable (up to about 30 minutes on drone). Elapsed run-time on a local developer system is very dependent on the IO as well as CPU performance. Smaller apps may have all tests in a single suite.

Each suite consists of a number of features. Each feature is described in a *.feature file. There are a number of scenarios in each feature file. Each scenario has a number of scenario steps that define the steps taken to do the test.

Set Up Test

  • Start the Selenium Docker container in a terminal:

docker run -p 4445:4444 -p 5900:5900 -v /dev/shm:/dev/shm selenium/standalone-chrome-debug

Ports on the Selenium Docker IP address are mapped to localhost so they can be accessed by the tests and the vnc viewer.

  • Start the InBucket Docker container in another terminal:

docker run -p 2500:2500 -p 9000:9000 inbucket/inbucket

Ports on the InBucket docker IP address are mapped to localhost so they can be accessed by the tests. By running these in terminal windows, it is simple to press ctrl-C to stop them when you are finished.

  • Set the following environment variables:

    • EMAIL_HOST=localhost (so that the tests know how to deliver mails)

    • TEST_SERVER_URL (The URL of your webserver)

    • TEST_SERVER_FED_URL (The alternative URL of your webserver for federation share tests - optional)

    • BROWSER (Any one of chrome, firefox, internet explorer or MicrosoftEdge. Defaults to chrome)

    • BROWSER_VERSION (version of the browser you want to use - optional)

e.g., to test an instance running on the Docker subnet with Chrome do:

export EMAIL_HOST=localhost
export TEST_SERVER_URL=http://172.17.0.1:8080/owncloud-core
export TEST_SERVER_FED_URL=http://172.17.0.1:8180/owncloud-core
export BROWSER=chrome

  • If your ownCloud install is running locally on Apache, then it should already be available on the Docker subnet at 172.17.0.1

  • To run the federation Sharing tests:

    1. Install another instance of ownCloud with its own separate database, code, data directory and URL

    2. Install and enable the testing app

    3. Make sure you have configured HTTPS with valid certificates on both servers URLs

    4. Import SSL certificates (or do not offer HTTPS).

  • Run a suite of tests:

    make test-acceptance-webui BEHAT_SUITE=webUILogin

The names of suites are found in the tests/acceptance/config/behat.yml file, and start with webUI.

  • The browser for the tests runs inside the Selenium docker container. View it by running the vnc viewer: vncviewer.

And connect to localhost. The VNC password of the docker container is secret.

Running UI Tests using IPv6

The test system must have (at least locally) functioning IPv6:

  • working loopback address ::1

  • a real routable IPv6 address (not just a link-local address)

If you have a server set up that listens on both IPv4 and IPv6 (e.g. localhost on 127.0.0.1 and ::1) then the UI tests will access the server via whichever protocol your operating system prefers. If there are tests that specifically specify IPv4 or IPv6, then those will choose a suitable local address to come from so that they access the server using the required IP version.

Because not everyone will have functional IPv6 on their test system yet, tests that specifically require IPv6 are tagged @skip @ipv6. To run those tests, follow the section below on running skipped tests and specify --tags @ipv6.

Running UI Tests for One Feature

You can run the UI tests for just a single feature by specifying the feature file:

make test-acceptance-webui BEHAT_FEATURE=tests/acceptance/features/webUITrashbin/trashbinDelete.feature

To run just a single scenario within a feature, specify the line number of the scenario:

make test-acceptance-webui BEHAT_FEATURE=tests/acceptance/features/webUITrashbin/trashbinDelete.feature<linenumber>

Running UI Tests for an App

With the app installed, run the UI tests for the app from the app root folder:

cd apps/files_texteditor
../../tests/acceptance/run.sh --suite webUITextEditor

Run UI the tests for just a single feature of the app by specifying the feature file:

cd apps/files_texteditor
../../tests/acceptance/run.sh tests/acceptance/features/webUITextEditor/editTextFiles.feature

Skipping Tests

If a UI test is known to fail because of an existing bug, then it is left in the test set but is skipped by default. Skip a test by tagging it @skip and then put another tag with text that describes the reason it is skipped. e.g.,:

@skip @trashbin-restore-problem-issue-1234
Scenario: restore a single file from the trashbin

Skipped tests are listed at the end of a default UI test run. You can locally run the skipped test(s). Run all skipped tests for a suite with:

make test-acceptance-webui BEHAT_SUITE=webUITrashbin BEHAT_FILTER_TAGS=@skip

Or run just a particular test by using its unique tag:

make test-acceptance-webui BEHAT_SUITE=webUITrashbin BEHAT_FILTER_TAGS=@trashbin-restore-problem-issue-1234

When fixing the bug, remove these skip tags in the PR along with the bug fix code.

Additional Command Options

Running all test suites in a single run is not recommended. It will take many hours on a typical development system. However, you may run all UI tests with:

make test-acceptance-webui

By default, any test scenarios that fail are automatically rerun once. This minimizes transient failures caused by browser and Selenium driver timing issues. When developing tests it can be convenient to override this behavior.

To not rerun failed test scenarios:

make test-acceptance-webui NORERUN=true BEHAT_SUITE=webUILogin

Local Selenium Setup

You may optionally run the Selenium server locally. Docker is now the recommended way, but local Selenium is also possible:

  • Selenium standalone server e.g. version 3.12.0 or newer.

  • Browser installed that you would like to test on (e.g. chrome)

  • Web driver for the browser that you want to test.

  • Place the Selenium standalone server jar file and the web driver(s) somewhere in the same folder.

  • Start the Selenium server:

java -jar selenium-server-standalone-3.12.0.jar \
  -port 4445 \
  -enablePassThrough false

  • In this configuration, the tests will continually open the browser-under-test on your local system.

  • If you run any test scenarios that need an email service (to test password reset etc.), then you need to run the InBucket Docker container.

Known Issues