User Interface Testing
ownCloud >= 10.0. Make sure you have a running instance of ownCloud setup completely.
Default language set to
'default_language' ⇒ 'en',).
An admin user called
adminwith the password
No self-signed SSL certificates.
The testing app installed and enabled.
Testing utils (running
makein your terminal from the
webrootdirectory will install them).
Docker Post-install done to put your developer account in the docker group so you can run Docker without
Docker subnet enabled for any firewall that may be active such as, ufw. The example below shows how to update UFW’s firewall rules to allow the
sudo ufw status sudo ufw allow from 172.17.0.0/16
Docker containers pulled. It is recommended to use
standalone-chrome-debugwhich allows seeing the browser live. The latest
standalone-chrome-*containers have an issue. So make sure to pull the specific chrome container versions listed below. You will also need MailHog. Pull any or all of these Docker containers:
docker pull selenium/standalone-chrome:3.141.59-oxygen docker pull selenium/standalone-chrome-debug:3.141.59-oxygen docker pull selenium/standalone-firefox docker pull selenium/standalone-firefox-debug docker pull mailhog/mailhog
vncviewer installed (in order to view the browser action as the UI tests run). For example:
sudo apt install tigervnc-viewer
To run the Selenium server locally (not in Docker) see the notes at the end.
Tests are divided into suites, enabling each suite to test some logical portion of the functionality and for the total elapsed run-time of a single suite to be reasonable (up to about 30 minutes on drone). Elapsed run-time on a local developer system is very dependent on the IO as well as CPU performance. Smaller apps may have all tests in a single suite.
Each suite consists of a number of features. Each feature is described
*.feature file. There are a number of scenarios in each feature
file. Each scenario has a number of scenario steps that define the steps
taken to do the test.
Set Up Test
Start the Selenium Docker container in a terminal:
docker run -p 4445:4444 -p 5900:5900 -v /dev/shm:/dev/shm selenium/standalone-chrome-debug
Ports on the Selenium Docker IP address are mapped to
localhost so they can be accessed by the tests and the
Start the MailHog Docker container in another terminal:
docker run -p 1025:1025 -p 8025:8025 mailhog/mailhog
Ports on the MailHog docker IP address are mapped to
localhost so they can be accessed by the tests.
By running these in terminal windows, it is simple to press
ctrl-C to stop them when you are finished.
Set the following environment variables:
TEST_SERVER_URL(The URL of your webserver)
TEST_SERVER_FED_URL(The alternative URL of your webserver for federation share tests - optional)
BROWSER(Any one of
MicrosoftEdge. Defaults to
BROWSER_VERSION(version of the browser you want to use - optional)
e.g., to test an instance running on the Docker subnet with Chrome do:
export TEST_SERVER_URL=http://172.17.0.1:8080/owncloud-core export TEST_SERVER_FED_URL=http://172.17.0.1:8180/owncloud-core export BROWSER=chrome
If your ownCloud install is running locally on Apache, then it should already be available on the Docker subnet at
To run the federation Sharing tests:
Install another instance of ownCloud with its own separate database, code, data directory and URL
Install and enable the testing app
Make sure you have configured HTTPS with valid certificates on both servers URLs
Import SSL certificates (or do not offer HTTPS).
Run a suite of tests:
make test-acceptance-webui BEHAT_SUITE=webUILogin
The names of suites are found in the
tests/acceptance/config/behat.yml file, and start with
The browser for the tests runs inside the Selenium docker container. View it by running the
And connect to
localhost. The VNC password of the docker container is
Running UI Tests using IPv6
The test system must have (at least locally) functioning IPv6:
working loopback address ::1
realroutable IPv6 address (not just a link-local address)
If you have a server set up that listens on both IPv4 and IPv6 (e.g. localhost on 127.0.0.1 and ::1) then the UI tests will access the server via whichever protocol your operating system prefers. If there are tests that specifically specify IPv4 or IPv6, then those will choose a suitable local address to come from so that they access the server using the required IP version.
Because not everyone will have functional IPv6 on their test system yet, tests that specifically require IPv6 are tagged
To run those tests, follow the section below on running skipped tests and specify
Running UI Tests for One Feature
You can run the UI tests for just a single feature by specifying the feature file:
make test-acceptance-webui BEHAT_FEATURE=tests/acceptance/features/webUITrashbin/trashbinDelete.feature
To run just a single scenario within a feature, specify the line number of the scenario:
make test-acceptance-webui BEHAT_FEATURE=tests/acceptance/features/webUITrashbin/trashbinDelete.feature<linenumber>
Running UI Tests for an App
With the app installed, run the UI tests for the app from the app root folder:
cd apps/files_texteditor ../../tests/acceptance/run.sh --suite webUITextEditor
Run UI the tests for just a single feature of the app by specifying the feature file:
cd apps/files_texteditor ../../tests/acceptance/run.sh tests/acceptance/features/webUITextEditor/editTextFiles.feature
If a UI test is known to fail because of an existing bug, then it is
left in the test set but is skipped by default. Skip a test by tagging
@skip and then put another tag with text that describes the reason
it is skipped. e.g.,:
@skip @trashbin-restore-problem-issue-1234 Scenario: restore a single file from the trashbin
Skipped tests are listed at the end of a default UI test run. You can locally run the skipped test(s). Run all skipped tests for a suite with:
make test-acceptance-webui BEHAT_SUITE=webUITrashbin BEHAT_FILTER_TAGS=@skip
Or run just a particular test by using its unique tag:
make test-acceptance-webui BEHAT_SUITE=webUITrashbin BEHAT_FILTER_TAGS=@trashbin-restore-problem-issue-1234
When fixing the bug, remove these skip tags in the PR along with the bug fix code.
Additional Command Options
Running all test suites in a single run is not recommended. It will take many hours on a typical development system. However, you may run all UI tests with:
By default, any test scenarios that fail are automatically rerun once. This minimizes transient failures caused by browser and Selenium driver timing issues. When developing tests it can be convenient to override this behavior.
To not rerun failed test scenarios:
make test-acceptance-webui NORERUN=true BEHAT_SUITE=webUILogin
Local Selenium Setup
You may optionally run the Selenium server locally. Docker is now the recommended way, but local Selenium is also possible:
Selenium standalone server e.g. version 3.12.0 or newer.
Browser installed that you would like to test on (e.g. chrome)
Web driver for the browser that you want to test.
Place the Selenium standalone server jar file and the web driver(s) somewhere in the same folder.
Start the Selenium server:
java -jar selenium-server-standalone-3.12.0.jar \ -port 4445 \ -enablePassThrough false
In this configuration, the tests will continually open the browser-under-test on your local system.
If you run any test scenarios that need MailHog (to test password reset etc.), then you need to run the MailHog Docker container. That is much simpler than trying to configure MailHog on your local system.
Tests that are known not to work in specific browsers are tagged e.g.,
@skipOnINTERNETEXPLORERand will be skipped by the script automatically
- The web driver for the current version of Firefox works differently to the old one. If you want to test FF < 56 you need to test on 47.0.2 and to use Selenium server 2.53.1 for it