Big File Upload Configuration

Introduction

If you’re expecting to upload large files, there are some things you need to consider. Setups can be configured based on your needs.

General Considerations

  • Bear in mind that user quotas may prevent large file uploads if a user exceeds their storage limit.

  • Although the PHP setting output_buffering is already disabled in the shipped .htaccess file by ownCloud to prevent PHP memory-related errors, you may still need to manually set additional configuration options, as described below.

  • If the temporary directory used for uploading files is changed, it must be:

    • Fully accessible by the PHP/webserver user, which is usually www-data.

    • It is mounted as a Docker volume to provide the necessary space.

  • The size of your temporary directory or partition must be sufficient to accommodate multiple parallel uploads from different users. The formula for this is:

    temp_space = concurrent_uploads * chunk size

    For example, if the chunk size is 10MB (which is the default but might vary between different clients) and the average number of users uploading at the same time is 25, then you’ll need 250MB of temp space, as the formula below shows.

    10MB x 25 users = 250MB of required temp space

  • The user’s upload directory, usually in <mount-point>/files/<username>/uploads also has to be big enough. The formula is a bit different, as this directory collects all chunks of a file for that upload for that user until the upload is completed. The space needed for this directory is temporarily the same size as for the final file size. As an example, if a user wants to upload a 4GB file, temporarily 4GB of space needs to be available in the upload directory to hold the chunks. When the upload has finished, all chunks are written to the final destination and the chunks are deleted afterwards freeing that temporary space.

    The formula for the upload directory’s temp space for all users is:

    temp_space = concurrent_uploads * average_upload_size_per_user

    The location of the upload directory can be defined via the environment variable OWNCLOUD_DAV_CHUNK_BASE_DIR. You must provide a Docker volume for it if it is redefined.

    The space temporarily consumed in the upload directory will not count against the user quota. If a user has no quota left in his peronal storage and the quota excludes external mounts, uploads to a windows network drive share as example will succeed. The file temporarily created in the upload directory will not count against his personal storage.

  • If you have configured the session_lifetime setting in your configuration, or via the environment variable (see below), make sure it is not too low. This setting needs to be configured to at least the time (in seconds) that the longest upload will take. If unsure, remove this entirely from your configuration.

Configuration

If you have used PHP or .htaccess/.userini configuration before using the Dockerised ownCloud deployment, you now must switch to the environment variables that have been provided.

The following PHP keys are affected and need to be reconfigured. The values are only exemplary:

post_max_size=16G
upload_max_filesize=16G
upload_tmp_dir=/mnt/php_big_temp/
max_input_time=3600
max_execution_time=3600
memory_limit=512

Please note that output_buffering=0 is already set in the shipped .htaccess file.

The following ownCloud environment variables can be adapted when configuring big file uploads:

Environment Variable PHP or Config Key

OWNCLOUD_MAX_UPLOAD

post_max_size and
upload_max_filesize

OWNCLOUD_TEMP_DIRECTORY

upload_tmp_dir

OWNCLOUD_MAX_INPUT_TIME

max_input_time

OWNCLOUD_MAX_EXECUTION_TIME

max_execution_time

OWNCLOUD_MEMORY_LIMIT

memory_limit

OWNCLOUD_SESSION_KEEPALIVE

session_keepalive

OWNCLOUD_SESSION_LIFETIME

session_lifetime

Apache Config LimitRequestBody

The default value of the LimitRequestBody setting in Apache is 1073741824 bytes (1 GB). ownCloud has set this value to 0 (zero) for its embedded Apache configuration, which is equivalent to unlimited. This means that uploads to public folders will not be limited to any file size by Apache when chunking is not in effect.

General Upload Issues

Various environmental factors could cause a restriction of the upload size. Examples are:

  • Some services like Cloudflare are also known to cause uploading issues.

  • Upload limits enforced by proxies used by your clients.

Long-Running Uploads

For very long-running uploads those lasting longer than 1h to public folders, when chunking is not in effect, OWNCLOUD_FILELOCKING_TTL should be set to a significantly large value. If not, large file uploads will fail with a file locking error, because the Redis garbage collection will delete the initially acquired file lock after 1 hour by default.

To estimate a good value, use the following formula:

time_in_seconds = (maximum_upload_file_size / slowest_assumed_upload_connection)

For the value of "slowest assumed upload connection", take the upload speed of the user with the slowest connection and divide it by two. For example, let’s assume that the user with the slowest connection has an 8MBit/s DSL connection; which usually indicates the download speed. This type of connection would, usually, have 1MBit/s upload speed (but confirm with the ISP). Divide this value in half, to have a buffer when there is network congestion, to arrive at 512KBit/s as the final value.