Excluding Directories and Blacklisting Files

Introduction

This document describes how to manage blacklisted files and excluded directories.

Definitions of terms

Blacklisted

Files that may harm the ownCloud environment like a foreign .htaccess file. Blacklisting prevents anyone from uploading blacklisted files to the ownCloud server.

Excluded

Existing directories on your ownCloud server, including directories on external storage mounts, that are excluded from being processed by ownCloud. In effect they are invisible to ownCloud.

Both types are defined in config.php. Blacklisted files and excluded directories are not scanned by ownCloud, not viewed, not synced, and cannot be created, renamed, deleted, or accessed via direct path input from a file manager. Even when a filepath is entered manually via a file explorer, the path cannot be accessed.

For example configurations please see the config.sample.php file.

Many filesystems do not allow the coexistence of a file and folder with exactly the same name on the same directory level. Therefore no differentiation is made in processing files and folders for blacklisting or excluding, as it would just return a deny at a later stage. With the implementation made, you get an immediate error message.

Example: The storage backend has a reserved directory name ".snapshot" which is excluded by configuration. If you try to add a file or folder via the browser or sync a file or folder from the client named ".snapshot", you will get an immediate ownCloud triggered deny.

Impact on System Performance

If you have a filesystem mounted with 200,000 files and directories and 15 snapshots in rotation, you would now scan and process 200,000 elements plus 200,000 x 15 = 3,000,000 elements additionally. These additional 3,000,000 elements, 15 times more than the original quantity, would also be available for viewing and synchronisation. Because this is a big and unnecessary overhead, most times confusing to clients, further processing can be eliminated by using excluded directories.

Blacklisted Files

By default, ownCloud blacklists the file .htaccess to secure the running instance, which is important when using Apache as webserver. A foreign .htaccess file could overwrite rules defined by ownCloud. There is no explicit need to enter the file name .htaccess as parameter to the blacklisted_files array in config.php, but you can add more blacklisted file names if necessary. You can also prevent uploading, e.g., Outlook .pst files, with this mechanism to avoid massive backend space allocation.

Excluded Directories

Reasons for Excluding Directories

  1. Enterprise storage systems, or special filesystems like ZFS and BtrFS are capable of snapshots. These snapshots are directories and keep point-in-time views of the data

    1. Snapshot directories are read-only

    2. There is no common naming convention for these directories, and there most likely will never be. For example, NetApp uses .snapshot and ~snapshot, EMC e.g., .ckpt, HDS e.g., .latest and ~latest, and the ZFS filesystem uses .zfs

    3. It does not make sense for these directories to be visible to users as they are used to ease backup, restoration, and cloning

  2. Directories which are part of the mounted filesystem, but must not be user accessible/visible

  3. Manual managed but user invisible backup directories

Example:

If you have a snapshot-capable storage or filesystem where snapshots are enabled and presented to clients, each directory will contain a "special" visible directory named e.g. .snapshot. Depending on the system, you may find underneath a list of snapshots taken and in the next lower level the complete set of files and directories which were present when the snapshot was created. In most systems, this mechanism is true in all directory levels:

/.snapshot
 /nightly.0
     /home
     /dat
     /pictures
     file_1
     file_2
 /nightly.1
     /home
     /dat
     /pictures
     file_1
     file_2
 /nightly.2
     /home
     /dat
     /pictures
     file_1
     file_2
 ...
/home
/dat
/pictures
file_1
file_2
...

Example excluded_directories entries in config.php can look like this:

'excluded_directories' => [
    '.snapshot',
    '~snapshot',
    'dir1',
    'dir2',
],

Note that these are not pathnames, but directory names without any slashes. Excluding dir1 excludes:

/home/dir1
/etc/stuff/dir1

But not:

/home/.dir1
/etc/stuff/mydir1

Reasons for Blacklisting Files

The reason for blacklisting files can be explained based on the example of a .htaccess file. Such a file can lead to a unwanted behaviour of your webserver when visible to ownCloud in a directory.

Example blacklisted_files entries in config.php can look like this:

'blacklisted_files' => [
    'hosts',
    'evil_script.sh',
],