Backup Considerations

Introduction

To optimize your backups, this small guide aims to shed some light on the internal Infinite Scale data structure. You can also refer to it when you are just curious about how Infinite Scale stores its data.

Note, as a prerequisite for backing up Infinite Scale, the instance has to be fully shut down.

Infinite Scale Data Structure

Infinite Scale stores its data in a folder that can be configured via the environment variable OCIS_BASE_DATA_PATH. Without further configuration, services derive from that path when they store data, though individual settings for certain data types can be configured.

See the Base Data Directory description for more details and default path values.

Inside this folder, Infinite Scale will store all its data in separate subdirectories. That includes metadata, configurations, queues and stores etc. The actual bytes of files (blobs) are handled by a so called blobstore, which also stores here by default. Depending on the used blobstore, the blobs need to be backed up separately, for example if S3 is used.

Base Data Path Overview

Listing the contents of the folder will return the following:

    ~/.ocis/:tree -L 1
.
├── config
├── idm
├── idp
├── nats
├── proxy
├── search
├── storage
├── thumbnails
└── web

10 directories, 0 files

The following sections describe the content and background of the subdirectories to decide if a backup is required or recommended and its effect when it is not backed up.

config

Contains basic Infinite Scale configuration created by ocis init.
Note: The location of the configuration folder can be specified with the OCIS_CONFIG_DIR environment variable. For this document we will assume this variable is not set and the default is used. See the Configuration Directory description for more details and default path values.

    ~/.ocis/config/:tree
.
└── ocis.yaml

1 directory, 1 file

  • ocis.yaml:
    BACKUP RECOMMENDED. Holds Infinite Scale configuration data. The contents can vary depending on your environment variables. In general, most of this file can be recreated again by running ocis init. This will recreate secrets and certificates. However, if not backed up completely, some fields MUST be copied over from the old config manually to regain data access after a restore:

    Field Name Envvar Name Description If not backed up

    idp.ldap.bind_password

    OCIS_LDAP_BIND_PASSWORD

    Password for the idp

    no logins possible

    idm.service_user_passwords.idp_password

    IDM_IDPSVC_PASSWORD

    Same as above

    no logins possible

    system_user_id

    OCIS_SYSTEM_USER_ID

    The id of storage-system user

    no logins possible

    idm.service_user_passwords.reva_password

    IDM_REVASVC_PASSWORD

    The reva password

    no logins possible

    auth_basic.auth_providers.ldap.bind_password

    AUTH_BASIC_LDAP_BIND_PASSWORD

    Same as above

    no logins possible

    users.drivers.ldap.bind_password

    USERS_LDAP_BIND_PASSWORD

    Same as above

    no logins possible

    groups.drivers.ldap.bind_password

    GROUPS_LDAP_BIND_PASSWORD

    Same as above

    no logins possible

    storage_users.mount_id

    STORAGE_USERS_MOUNT_ID

    The mountid of the storage_users service

    sharing data lost

    gateway.storage_registry.storage_users_mount_id

    GATEWAY_STORAGE_USERS_MOUNT_ID

    Same as above

    sharing data lost

idm

Note: This folder will not appear if you use an external IDM. Refer to your IDM’s documentation for backup details in this case.

Contains the data for the internal Infinite Scale identity management. See the IDM service for more details.

    ~/.ocis/idm/:tree
.
├── ldap.crt
├── ldap.key
└── ocis.boltdb

1 directory, 3 files

  • ocis.boltdb:
    BACKUP REQUIRED. This is the boltdb database that stores user data. Use IDM_DATABASE_PATH to specify its path. If not backed up, Infinite Scale will have no users, therefore also all data is lost.

  • ldap.crt:
    BACKUP OPTIONAL. This is the certificate for the idm. Use IDM_LDAPS_CERT to specify its path. Will be auto-generated if not backed up.

  • ldap.key:
    BACKUP OPTIONAL. This is the certificate key for the idm. Use IDM_LDAPS_KEY to specify its path. Will be auto-generated if not backed up.

idp

Note: This folder will not appear if you use an external IDP. Refer to your IDP’s documentation for backup details in this case.

Contains the data for the internal Infinite Scale identity provider. See the IDP service for more details.

    ~/.ocis/idp/:tree
.
├── encryption.key
├── private-key.pem
└── tmp
    └── identifier-registration.yaml

2 directories, 3 files

  • encryption.key:
    BACKUP RECOMMENDED. This is the encryption secret. Use IDP_ENCRYPTION_SECRET_FILE to specify its paths. Not backing this up will force users to relogin.

  • private-key.pem:
    BACKUP RECOMMENDED. This is the encryption key. Use IDP_SIGNING_PRIVATE_KEY_FILES to specify its paths. Not backing this up will force users to relogin.

  • identifier-registration.yml:
    BACKUP OPTIONAL. It holds configuration for oidc clients (web, desktop, ios, android). Will be recreated if not backed up.

nats

Note: This folder will not appear if you use an external nats installation. In that case, data has to be secured in alignment with the external installation.

Contains nats data for streams and stores. See the NATS service for more details.

    ~/.ocis/nats/:tree -L 1
.
└── jetstream

  • jetstream:
    BACKUP RECOMMENDED. This folder contains nats data about streams and key-value stores. Use NATS_NATS_STORE_DIR to specify its path. Not backing it up can break history for multiple (non-vital) features such as history or notifications. The Infinite Scale functionality is not impacted if omitted.

proxy

Contains proxy service data. See the PROXY service for more details.

    ~/.ocis/proxy/:tree
.
├── server.crt
└── server.key

1 directory, 2 files

  • server.crt:
    BACKUP OPTIONAL. This is the certificate for the http services. Use PROXY_TRANSPORT_TLS_CERT to specify its path.

  • server.key:
    BACKUP OPTIONAL. This is the certificate key for the http services. Use PROXY_TRANSPORT_TLS_KEY to specify its path.

Contains the index created by search. See the SEARCH service for more details.

    ~/.ocis/search/:tree -L 1
.
└── bleve

2 directories, 0 files

  • bleve:
    BACKUP RECOMMENDED/OPTIONAL. This contains the search index. Can be specified via SEARCH_ENGINE_BLEVE_DATA_PATH. If not backed up, the search index needs to be recreated. This can take a long time depending on the amount of files.

storage

Contains Infinite Scale meta (and blob) data, depending on the blobstore. See the STORAGE-USERS service for more details.

    ~/.ocis/storage/:tree -L 1
.
├── metadata
├── ocm
└── users

4 directories, 0 files

  • metadata:
    BACKUP REQUIRED. Contains system data. Path can be specified via STORAGE_USERS_OCIS_ROOT. Not backing it up will remove shares from the system and will also remove custom settings.

  • ocm:
    BACKUP REQUIRED/OMITABLE. Contains ocm share data. When not using ocm sharing, this folder does not need to be backed up.

  • users:
    BACKUP REQUIRED. Contains user data. Path can be specified via STORAGE_USERS_OCIS_ROOT. Not backing it up will remove all spaces and all files. As result, you will have a configured but empty Infinite Scale instance, which is fully functional for accepting new data. Old data is lost.

thumbnails

Contains thumbnails data. See the THUMBNAILS service for more details.

    ~/.ocis/thumbnails/:tree -L 1
.
└── files

  • files:
    OPTIONAL/RECOMMENDED. This folder contains prerendered thumbnails. Can be specified via THUMBNAILS_FILESYSTEMSTORAGE_ROOT. If not backed up, thumbnails will be regenerated automatically on access which leads to some load on the thumbnails service.

web

Contains web assets such as custom logos, themes etc. See the WEB service for more details.

    ~/.ocis/web/:tree -L 1
.
└── assets

2 directories, 0 files

  • assets:
    BACKUP RECOMMENDED/OMITTABLE. This folder contains custom web assets. Can be specified via WEB_ASSET_CORE_PATH. If no custom web assets are used, there is no need for a backup. If those exist but are not backed up, they need to be reuploaded.

External Services

When using an external:

  • idp/idm like Keycloak,

  • persistant store like NATS or

  • blobstore like S3

its data needs to be backed up separately. Refer to the vendors documentation for backup and restore details.