S3 Compatible Object Storage as Primary Storage Location

Introduction

Administrators can configure Amazon-S3-compatible object storages as the primary ownCloud storage location with the S3 Primary Object Storage app. The referencing name is files_primary_s3. Using files_primary_s3 replaces the default ownCloud owncloud/data directory. However, you need to keep the owncloud/data directory for the following reasons:

  • The ownCloud log file is saved in the data directory.

  • Legacy apps may not support using anything but the owncloud/data directory.

Even if the ownCloud log file is stored in an alternate location (by changing the location in config.php), owncloud/data may still be required for backward compatibility with some apps.

That said, Object Storage Support (objectstore) is still available, but the S3 Primary Object Storage app is the preferred and only supported way to provide S3 storage support as primary storage. ownCloud provides consulting for migrations from objectstorefiles_primary_s3.

Consider the following differentiation:

External Storage: S3

Integrate S3 object storages as external storages

S3 Primary Object Storage

Leverage object storage via S3 as primary storage

OpenStack Swift has been deprecated.

When using files_primary_s3, the Amazon S3 bucket to be used needs to be created manually first, according to the Amazon S3 developer documentation and versioning needs to be enabled for this bucket.

Implications

Read the following implications carefully BEFORE you start using files_primary_s3:

  1. Apply this configuration before the first login of any user – including the admin user; otherwise, ownCloud can no longer find the user’s files.

  2. In "object store" mode as primary storage access, ownCloud expects exclusive access to the object store container, because it only stores the binary data for each file. While in this mode, ownCloud stores the metadata in the local database for performance reasons.

  3. The current implementation is incompatible with any app that uses direct file I/O (input/output) as it circumvents the ownCloud virtual file system. An excellent example is the Encryption app, which fetches critical files in addition to any requested file, which results in significant overhead.
    Therefore encrypting the S3 primary storage via ownCloud has been disabled and can not be enabled

  4. When requiring encryption for the bucket containing the primary storage, use the bucket built-in encryption provided by the S3 API. See the configuration examples below how to enable it.

  5. When using S3 primary storage with multiple buckets, it is not recommended to use the command to transfer file ownership between users (occ files:transfer-ownership) as shares on the files can get lost. The reason for this is that file IDs are changed during such cross-storage move operations.

As ownCloud only stores the binary data in S3, while the metadata of the files is still stored on the DB, a regular database backup is highly recommended, as this is also needed for recovery in addition to the S3 backup!

Configuration

Copy the following relevant example part to your config.php file.

Any object store needs to implement \OCP\Files\ObjectStore\IObjectStore and can be passed parameters in the constructor with the arguments key, as in the following example:

<?php
$CONFIG = [
    'objectstore' => [
        'class' => 'Implementation\Of\OCP\Files\ObjectStore\IObjectStore',
        'arguments' => [
            ...
        ],
    ],
];

Amazon S3

The S3 backend mounts a bucket of the Amazon S3 object store into the virtual filesystem. The class to be used is OCA\Files_Primary_S3\S3Storage, as in the following example:

<?php
$CONFIG = [
    'objectstore' => [
        'class' => 'OCA\Files_Primary_S3\S3Storage',
        'arguments' => [
              //
              // replace with your bucket
            'bucket' => 'owncloud',
              //
              // uncomment to indicate available storage size in the objectstore in bytes (in this example 1TB),
              // without this setting, apps relying on available storage might be limited in funcionality e.g. metrics app
              //'availableStorage' => 1099511627776,
              //
              // uncomment to enable server side encryption
              //'serversideencryption' => 'AES256',
              //
              // part size in bytes, applies to uploads between ownCloud and S3. This value must be between 5MB and 5GB, inclusive.
            'part_size' => 5242880,
              // maximum number of concurrent UploadPart operations allowed during the multipart upload.
            'concurrency' => 3,
              // storage specific options
            'options' => [
                  // version and region are required
                'version' => '2006-03-01',
                  // change to your region
                'region'  => 'eu-central-1',
                'credentials' => [
                      // replace key and secret with your credentials
                    'key' => 'owncloud123456',
                    'secret' => 'secret123456',
                ],
            ],
        ],
    ],
];

Ceph S3

The S3 backend can also be used to mount the bucket of a Ceph S3 object store via the Amazon S3 API into the virtual filesystem. The class to be used is OCA\Files_Primary_S3\S3Storage:

<?php
$CONFIG = [
    'objectstore' => [
        'class' => 'OCA\Files_Primary_S3\S3Storage',
        'arguments' => [
              //
              // replace with your bucket
            'bucket' => 'owncloud',
              //
              // uncomment to indicate available storage size in the objectstore, in bytes (in this example 1TB),
              // without this setting, apps relying on available storage might be limited in funcionality e.g. metrics app
              //'availableStorage' => 1099511627776,
              //
              // uncomment to enable server side encryption
              //'serversideencryption' => 'AES256',
              //
              // part size, in bytes, to use when doing a multipart upload. this must between 5 MB and 5 GB, inclusive.
            'part_size' => 5242880,
              // maximum number of concurrent UploadPart operations allowed during the multipart upload.
            'concurrency' => 5,
              // storage specific options
            'options' => [
                  // version and region are required
                'version' => '2006-03-01',
                'region'  => '',
                  // replace key, secret and bucket with your credentials
                'credentials' => [
                      // replace key and secret with your credentials
                    'key'    => 'owncloud123456',
                    'secret' => 'secret123456',
                ],
                  // replace the ceph endpoint with your rgw url
                'endpoint' => 'http://ceph:80/',
                  // Use path style when talking to ceph
                'use_path_style_endpoint' => true,
            ],
        ],
    ],
];

Scality S3

The S3 backend can also be used to mount the bucket of a Scality S3 object store via the Amazon S3 API into the virtual filesystem. The class to be used is OCA\Files_Primary_S3\S3Storage:

<?php
$CONFIG = [
    'objectstore' => [
        'class' => 'OCA\Files_Primary_S3\S3Storage',
        'arguments' => [
              //
              // replace with your bucket
            'bucket' => 'owncloud',
              //
              // uncomment to indicate available storage size in the objectstore, in bytes (in this example 1TB),
              // without this setting, apps relying on available storage might be limited in funcionality e.g. metrics app
              //'availableStorage' => 1099511627776,
              //
              // uncomment to enable server side encryption
              //'serversideencryption' => 'AES256',
              //
              // part size, in bytes, to use when doing a multipart upload. this must between 5 MB and 5 GB, inclusive.
            'part_size' => 5242880,
              // maximum number of concurrent UploadPart operations allowed during the multipart upload.
            'concurrency' => 5,
              // storage specific options
            'options' => [
                  // version and region are required
                'version' => '2006-03-01',
                'region'  => 'us-east-1',
                'credentials' => [
                      // replace key and secret with your credentials
                    'key' => 'owncloud123456',
                    'secret' => 'secret123456',
                ],
                'use_path_style_endpoint' => true,
                'endpoint' => 'http://scality:8000/',
            ],
        ],
    ],
];

Backblaze S3

The S3 backend can also be used to mount the bucket of a Backblaze S3 object store via the Amazon S3 API into the virtual filesystem. The class to be used is OCA\Files_Primary_S3\S3Storage:

<?php
$CONFIG = [
    'objectstore' => [
        'class' => 'OCA\Files_Primary_S3\S3Storage',
        'arguments' => [
              // replace with your bucket
            'bucket' => 'owncloud',
              // uncomment to enable server side encryption
              //'serversideencryption' => 'AES256',
              //
              // part size, in bytes, to use when doing a multipart upload. this must between 5 MB and 5 GB, inclusive.
            'part_size' => 5242880,
              // maximum number of concurrent UploadPart operations allowed during the multipart upload.
            'concurrency' => 5,
              //
            'options' => [
                  // version and region are required
                'version' => 'latest',
                  // replace with your region
                'region'  => 'us-west-004',
                'credentials' => [
                      // replace key and secret with your credentials
                    'key' => 'owncloud123456',
                    'secret' => 'secret123456',
                ],
                'use_path_style_endpoint' => true,
                  // replace with your endpoint
                'endpoint' => 'https://s3.us-west-004.backblazeb2.com',
            ],
        ],
    ],
];