Big File Upload Configuration

Introduction

If you’re expecting big file uploads, some considerations have to be taken into account. Setups based on your needs can be configured.

General Considerations

  • Make sure that a version of PHP supported by ownCloud is installed.

  • Consider that user quotas may prevent big file uploads due to a user reaching the space limitation.

  • Though ownCloud already disables the PHP setting Output Buffering in the shipped .htaccess and .user.ini to prevent PHP memory-related errors, it can be the case that you must manually set it in your php.ini or VirtualHosts when the other two configuration files can’t be used in your environment.

  • The directory used for upload_tmp_dir must be fully accessible by PHP / the webserver user, usually www-data.

  • Your temp directory or partition has to be big enough to hold multiple parallel uploads from multiple users. The formula for this is temp_space = concurrent_uploads * chunk size
    For example, if the chunk size is 10MB (which is the default but might vary between different clients) and the average number of users uploading at the same time is 25, then you’ll need 250MB of temp space, as the formula below shows.

    10MB x 25 users = 250MB required temp space

  • The user’s upload directory, usually in <datadirectory>/<username>/uploads (see Sample Config PHP Parameters for details) also has to be big enough. The formula is a bit different, as this directory collects all chunks of a file for that upload for that user until the upload is completed. The space needed for this directory is temporarily the same size as for the final file size. As an example, if a user wants to upload a 4GB file, temporarily 4GB of space needs to be available in the upload directory to hold the chunks. When the upload has finished, all chunks are written to the final destination and the chunks are deleted afterwards freeing that temporary space. The formula for the upload directory’s temp space for all users is: temp_space = concurrent_uploads * average_upload_size_per_user. The location of the upload directory can be defined via the config setting dav.chunk_base_dir.

The space temporarily consumed in the upload directory will not count against the user quota. If a user has no quota left in his peronal storage and the quota excludes external mounts, uploads to a windows network drive share as example will succeed. The file temporarily created in the upload directory will not count against his personal storage.

  • In Centos and RHEL, Apache has a few more default configurations within systemd.

    You will have to set the temp directory in two places:

    1. In php.ini, e.g., sys_temp_dir = "/scratch/tmp"

    2. In Apache systemd file e.g. sudo systemctl edit httpd and change/add:

      PrivateTmp=false

    When done, you need to reload the daemon and restart the service:

    sudo systemctl daemon-reload
sudo systemctl restart httpd

    Please do not change /usr/lib/systemd/system/httpd.service directly, only use sudo systemctl edit httpd. If not doing so, a httpd package upgrade may revert your changes.

Configuration via .htaccess/user.ini

ownCloud comes with its own owncloud/.htaccess file. When using php-fpm, PHP settings in .htaccess are not accessed. These settings must then be set in the .user.ini file. php-fpm will read settings from any .user.ini file in the same directory as the .php file that is being served via a web server.

Set the following parameters inside the corresponding file using your own desired values, as in the following examples, both files are located in the ownCloud root folder:

.user.ini
post_max_size=16G
output_buffering=0
upload_max_filesize=16G
upload_tmp_dir=/mnt/php_big_temp/
.htaccess
php_value	post_max_size=16G
php_value	output_buffering=0
php_value	upload_max_filesize=16G
php_value	upload_tmp_dir=/mnt/php_big_temp/

If you see PHP timeouts in your log files, increase the timeout values, which are in seconds, as in the example below. Use the php_value prefix like above when configuring the .htaccess file:

max_input_time=3600
max_execution_time=3600
Consider that any settings made in .htaccess or .user.ini may need to be repopulated after an upgrade of ownCloud.

Configuring via PHP Global Settings

If you don’t want to use the ownCloud .htaccess or .user.ini file, you may configure PHP globally instead. Make sure to comment out or remove any lines in .htaccess if you added any like in the section above.

If you are running ownCloud on a 32-bit system, any open_basedir directive in your php.ini file needs to be commented out.

See the Loaded Configuration File section of PHP Version and Information to find your relevant php.ini files.

Set the following parameters inside the corresponding php.ini file using your own desired file size values, as in the following example:

post_max_size=16G
output_buffering=0
upload_max_filesize=16G
upload_tmp_dir=/mnt/php_big_temp/

If you see PHP timeouts in your log files, increase the timeout values, which are in seconds, as in the example below:

max_input_time=3600
max_execution_time=3600

Configuring via a Virtual Host

You can configure php parameters also per virtual host - if you have access to the Apache configuration file. This eliminates the need to maintain custom settings in a .user.ini or .htaccess file especially on upgrades. Note the mandatory prefix php_admin_value before the php parameter.

<VirtualHost *:443>

	DocumentRoot /var/www/owncloud
	ServerName myowncloud.com

	php_admin_value	post_max_size 16G
	php_admin_value	output_buffering 0
	php_admin_value	upload_max_filesize 16G
	php_admin_value	upload_tmp_dir /mnt/php_big_temp/

	...

If you see PHP timeouts in your log files, increase the timeout values, which are in seconds, as in the example below:

php_admin_value max_input_time 3600
php_admin_value max_execution_time 3600

Configuring via ownCloud

As an alternative to the upload_tmp_dir of PHP (e.g., if you don’t have access to your php.ini) you can also configure some parameters in config.php.

  • Set a temporary location for uploaded files by using the tempdirectory setting.

  • If you have configured the session_lifetime setting in your config.php, see Sample Config PHP Parameters, make sure it is not too low. This setting needs to be configured to at least the time (in seconds) that the longest upload will take. If unsure, remove this entirely from your configuration to reset it to the default shown in the config.sample.php.

General Upload Issues

Various environmental factors could cause a restriction of the upload size. Examples are:

  • The LVE Manager of CloudLinux which sets an I/O limit.

  • Some services like Cloudflare are also known to cause uploading issues.

  • Upload limits enforced by proxies used by your clients.

  • Other web server modules like described in General Troubleshooting.

Apache Directives

Apache with mod_reqtimeout

The mod_reqtimeout Apache module could also stop large uploads from completing. If you’re using this module and uploads of large files fail, either disable it in your Apache config or increase the configured RequestReadTimeout values.

Disable mod_reqtimeout on Ubuntu

On Ubuntu, you can disable the module by running the following command:

sudo a2dismod reqtimeout
Disable mod_reqtimeout on CentOS

On CentOS, comment out the following line in /etc/httpd/conf/httpd.conf:

LoadModule reqtimeout_module modules/mod_reqtimeout.so

When you have run asdismod or updated /etc/httpd/conf/httpd.conf, restart Apache.

There are also several other configuration options in your web server config which could prevent the upload of larger files. Refer to your web server’s manual for how to configure those values correctly:

Apache with mod_fcgid

If you are using Apache 2.4 with mod_fcgid, as of February/March 2016, FcgidMaxRequestInMem still needs to be significantly increased from its default value to avoid the occurrence of segmentation faults when uploading big files. This is not a regular setting but serves as a workaround for Apache with mod_fcgid bug #51747.

Setting FcgidMaxRequestInMem significantly higher than usual may no longer be necessary, once bug #51747 is fixed.

Important Changes in Apache 2.4.54

In Apache HTTP Server 2.4.53 and earlier, the default value of the LimitRequestBody directive was 0 (unlimited). This has changed starting with Apache HTTP Server 2.4.54 where the default value is set to 1073741824 bytes (1 GB). This means, that uploads to public folders when chunking is not in effect will be limited to this file size. Change this value according to your needs in order to allow large file uploads. Please refer to the official Apache documentation LimitRequestBody for more information.

Long-Running Uploads

For very long-running uploads those lasting longer than 1h to public folders, when chunking is not in effect, filelocking.ttl should be set to a significantly large value in config.php. If not, large file uploads will fail with a file locking error, because the Redis garbage collection will delete the initially acquired file lock after 1 hour by default.

To estimate a good value, use the following formula:

time_in_seconds = (maximum_upload_file_size / slowest_assumed_upload_connection).

For the value of "slowest assumed upload connection", take the upload speed of the user with the slowest connection and divide it by two. For example, let’s assume that the user with the slowest connection has an 8MBit/s DSL connection; which usually indicates the download speed. This type of connection would, usually, have 1MBit/s upload speed (but confirm with the ISP). Divide this value in half, to have a buffer when there is network congestion, to arrive at 512KBit/s as the final value.