Storage-Users Service Configuration

Introduction

The Infinite Scale Storage-Users service

Default Values

  • Storage-Users listens on port 9157 by default.

Graceful Shutdown

With Infinite Scale, you can define a graceful shutdown period for the storage-users service.

The graceful shutdown period is only applicable if the storage-users service runs as standalone service. It does not apply if the storage-users service runs as single Docker environment. To build an environment where the storage-users service runs as a standalone service, you must start two instances, one without the storage-users service and one only with the the storage-users service. Note that both instances must be able to communicate on the same network.

When hard-stopping Infinite Scale, for example with the kill <pid> command (SIGKILL), it is possible and likely that not all data from the decomposedfs (metadata) has been written to the storage which may result in an inconsistent decomposedfs. When gracefully shutting down Infinite Scale, using a command like SIGTERM, the process will no longer accept any write requests from other services and will try to write the internal open requests which can take an undefined duration based on many factors. To mitigate that situation, the following things have been implemented:

  • With the value of the environment variable STORAGE_USERS_GRACEFUL_SHUTDOWN_TIMEOUT, the storage-users service will delay its shutdown giving it time to finalize writing necessary data. This delay can be necessary if there is a lot of data to be saved and/or if storage access/throughput is slow. In such a case you would receive an error log entry informing you that not all data could be saved in time. To prevent such occurrences, you must increase the default value.

  • If a shutdown error has been logged, the command-line maintenance tool Inspect and Manipulate Node Metadata can help to fix the issue. Please contact support for details.

CLI Commands

For any command listed, use --help to get more details and possible options and arguments.

To authenticate CLI commands use:

  • OCIS_SERVICE_ACCOUNT_SECRET=<acc-secret> and

  • OCIS_SERVICE_ACCOUNT_ID=<acc-id>.

The storage-users CLI tool uses the default address to establish the connection to the gateway service. If the connection fails, check your custom gateway service GATEWAY_GRPC_ADDR configuration and set the same address in storage-users OCIS_GATEWAY_GRPC_ADDR or STORAGE_USERS_GATEWAY_GRPC_ADDR.

Manage Unfinished or Stale Uploads

When using Infinite Scale as user storage, a directory named storage/users/uploads can be found in the Infinite Scale data folder. This is an intermediate directory based on TUS which is an open protocol for resumable uploads. Each upload consists of a blob and a blob.info file. Note that the term blob is just a placeholder.

  • If an upload succeeds, the blob file will be moved to the target and the blob.info file will be deleted.

  • In case of incomplete uploads, the blob and blob.info files will continue to recieve data until either the upload succeeds in time or the upload expires based on the STORAGE_USERS_UPLOAD_EXPIRATION variable, see the table below for details.

  • In case of expired uploads, the blob and blob.info files will not be removed automatically. Thus a lot of data can pile up over time wasting storage space.

  • In the rare case of a failure, after the upload succeeded but the file was not moved to its target location, which can happen when postprocessing fails, the situation is the same as with expired uploads.

Example cases for expired uploads

  • When a user uploads a big file but the file exceeds the user-quota, the upload can’t be moved to the target after it has finished. The file stays at the upload location until it is manually cleared.

  • If the bandwith is limited and the file to transfer can’t be transferred completely before the upload expiration time is reached, the file expires and can’t be processed.

  • If the upload was technically successful, but the postprocessing step failed due to an internal error, it will not get further processed. See the procedure in the Resume Post-Processing documentation for details how to solve this.

  • If the upload and the session for that upload gets disconnected.

The following commands are available to manage unfinished uploads
ocis storage-users uploads
ocis storage-users uploads
NAME:
   ocis storage-users uploads - manage unfinished uploads

USAGE:
   ocis storage-users uploads command [command options]

COMMANDS:
   sessions            Print a list of upload sessions
   delete-stale-nodes  Delete all nodes in processing state that are not referenced by any upload session
   help, h             Shows a list of commands or help for one command

Sessions Command

The sessions command is the entry point for listing, restarting/resuming and cleaning unfinished uploads. See the Remove Stale Uploads command to manage uploads that have lost connection to a session.

There can never be a clear identification of a failed upload session due to various reasons causing them. You need to apply more critera like free space on disk, a failed service like antivirus etc. to declare an upload as failed.

If not noted otherwise, the restart option can also be replaced with the resume option. This changes behaviour slightly.

  • restart
    When restarting an upload, all steps for open items will be restarted, except if otherwise defined.

  • resume
    When resuming an upload, processing will continue unfinished items from their last completed step.

 
ocis storage-users uploads sessions
NAME:
   ocis storage-users uploads sessions - Print a list of upload sessions

USAGE:
   ocis storage-users uploads sessions [command options]

OPTIONS:
   --id value    filter sessions by upload session id (default: unset)
   --processing  filter sessions by processing status (default: unset)
   --expired     filter sessions by expired status    (default: unset)
   --has-virus   filter sessions by virus scan result (default: unset)
   --json        output as json                       (default: false)
   --restart     send restart event for all listed sessions (default: false)
   --resume      send resume event for all listed sessions (default: false)
   --clean       remove uploads                       (default: false)
   --help, -h    show help

This will always output a list of uploads that match the criteria. See the examples below.

Some additional information on returned information

  • Offset is the number of bytes the server has already received.
    If Offset == Size the server has received all bytes of the upload.

  • Processing indicates if the uploaded file is currently going through postprocessing.

  • Scan Date and Scan Result indicate the scanning status.
    If Scan Date is set and Scan Result is empty the file is not virus infected.

Command Examples

Command to list ongoing upload sessions

ocis storage-users uploads sessions --expired=false --processing=false

Sessions that are not expired:
Space Upload Id Name Offset Size Executant Owner Expires Processing Scan Date Scan Result

f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c

5e387954-7313-4223-a904-bf996da6ec0b

foo.txt

0

1234

f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c

f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c

2024-01-26T13:04:31+01:00

false

2024-04-24T11:24:14+02:00

infected: virus A

f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c

f066244d-97b2-48e7-a30d-b40fcb60cec6

bar.txt

0

4321

f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c

f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c

2024-01-26T13:18:47+01:00

false

2024-04-24T14:38:29+02:00

The sessions command can also clear and restart/resume uploads. The output is the same as if run without the --clean or --restart/--resume option. Note that it is recommended to run the command first without the --clean (--processing) option to double check which uploads would get cleaned (restarted/resumed).

Cleans all expired uploads regardless of processing and virus state.
ocis storage-users uploads sessions \
     --expired=true \
     --clean
Resume all uploads that are currently processing and are not virus infected
ocis storage-users uploads sessions \
     --processing=true \
     --has-virus=false \
     --resume

Remove Stale Uploads

Unlike the Sessions Command set above, this command deletes (or reverts) all nodes that are in processing state and not referenced by an upload session. Although this is a very rare occurrence, a command has been added to identify and remove these stale uploads, if they exist.

This command can be used to run on a specific Space, on all Spaces, and with a dry-run option to print only stale uploads as they are found.

See the Listing Space IDs to get the Space ID if you want the command to run only for a specific Space.

ocis storage-users uploads delete-stale-nodes help
NAME:
   ocis storage-users uploads delete-stale-nodes - Delete (or revert) all nodes in processing state that are not referenced by any upload session

USAGE:
   ocis storage-users uploads delete-stale-nodes [command options]

OPTIONS:
   --spaceid value  Space ID to check for processing nodes (omit to check all spaces)
   --dry-run        Only show what would be deleted without actually deleting (default: true)
   --verbose        Enable verbose logging (default: false)
   --help, -h       show help

Manage Trash-Bin Items

This command set provides commands to get an overview of trash-bin items, restore items and purge old items of personal spaces and project spaces (spaces that have been created manually). trash-bin commands require a spaceID as parameter. See Listing Space IDs for details of how to get them.

ocis storage-users trash-bin help
COMMANDS:
   purge-expired  Purge expired trash-bin items
   list           Print a list of all trash-bin items of a space.
   restore-all    Restore all trash-bin items for a space.
   restore        Restore a trash-bin item by ID.
   help, h        Shows a list of commands or help for one command

Purge Expired

Purge all expired items from the trash-bin
ocis storage-users trash-bin purge-expired

The behaviour of the purge-expired command can be configured by using the following environment variables.

  • STORAGE_USERS_PURGE_TRASH_BIN_USER_ID
    Used to obtain space trash-bin information and takes the system admin user as the default which is the OCIS_ADMIN_USER_ID but can be set individually. It should be noted, that the OCIS_ADMIN_USER_ID must be manually assigned in all other deployments. The command only considers spaces to which the assigned user has access and delete permission.

  • STORAGE_USERS_PURGE_TRASH_BIN_PERSONAL_DELETE_BEFORE
    Has a default value of 720h which equals 30 days. This means, the command will delete all files older than 30 days. The value is human-readable, for valid values see the duration type described in the Environment Variable Types. A value of 0 is equivalent to disable and prevents the deletion of personal space trash-bin files.

  • STORAGE_USERS_PURGE_TRASH_BIN_PROJECT_DELETE_BEFORE
    Has a default value of 720h which equals 30 days. This means, the command will delete all files older than 30 days. The value is human-readable, for valid values see the duration type described in the Environment Variable Types. A value of 0 is equivalent to disable and prevents the deletion of project space trash-bin files.

List and Restore Trash-Bins Items

Restoring is possible only to the original location. The personal or project spaceID is required for the items to be restored. To authenticate the CLI tool use:

OCIS_SERVICE_ACCOUNT_SECRET=<acc-secret>
OCIS_SERVICE_ACCOUNT_ID=<acc-id>

The storage-users CLI tool uses the default address to establish the connection to the gateway service. If the connection fails, check the GATEWAY_GRPC_ADDR configuration from your gateway service and set the same address to the storage-users variable STORAGE_USERS_GATEWAY_GRPC_ADDR or globally with OCIS_GATEWAY_GRPC_ADDR.

Export the gateway address if your configuration differs from the default
export STORAGE_USERS_GATEWAY_GRPC_ADDR=127.0.0.1:9142
Print a list of all trash-bin items of a space
ocis storage-users trash-bin list [command options] ['spaceID' required]

The restore option defines the behavior for an item to be restored, when the item name already exists in the target space. Supported options are: skip, replace and keep-both. The default value is skip.

When the CLI tool restores the item with the replace option, the existing item will be moved to a trash-bin. When the cli tool restores the item with the keep-both option and the designated item already exists, the name of the restored item will be changed by adding a numeric suffix in parentheses. The variable STORAGE_USERS_CLI_MAX_ATTEMPTS_RENAME_FILE defines a maximum number of attempts to rename an item.

Restore all trash-bin items for a space
ocis storage-users trash-bin restore-all [command options] ['spaceID' required]
Restore a trash-bin item by ID
ocis storage-users trash-bin restore [command options] ['spaceID' required] ['itemID' required]

Resource Optimisation

See the General Storage Considerations and Settings documentation for details about configuring the STORAGE_USERS_OCIS_MAX_CONCURRENCY environment variable.

Caching

The storage-users service caches stat, metadata and uuids of files and folders via the configured stores.

The storage-users service can use a configured store via the global OCIS_CACHE_STORE environment variable.

Note that for each global environment variable, an independent service-based one might be available additionally. For precedences see Environment Variable Notes. Check the configuration section below. Supported stores are:

Store Type Description

memory

Basic in-memory store. Will not survive a restart.
Usually the default for caches. See the store environment variable for which one is used.

nats-js-kv

Stores data using key-value-store feature of NATS JetStream.
Usually the default for stores, see the store environment variable for which one is used.

redis-sentinel

Stores data in a configured Redis Sentinel cluster.

noop

Stores nothing. Useful for testing. Not recommended in production environments.
The storage-users service can only be scaled if not using the memory store and the stores are configured identically over all instances!
If you have used one of the deprecated stores of a former version, you should reconfigure to use one of the supported ones as the deprecated stores will be removed in a later version.
Store specific notes

  • When using redis-sentinel:
    The Redis master to use is configured via e.g. OCIS_CACHE_STORE_NODES in the form of <sentinel-host>:<sentinel-port>/<redis-master> like 10.10.0.200:26379/mymaster.

  • When using nats-js-kv:

    • It is recommended to set OCIS_CACHE_STORE_NODES to the same value as OCIS_EVENTS_ENDPOINT. That way the cache uses the same nats instance as the event bus. See the Event Bus Configuration for more details.

    • Authentication can be added, if configured, via OCIS_CACHE_AUTH_USERNAME and OCIS_CACHE_AUTH_PASSWORD.

    • It is possible to set OCIS_CACHE_DISABLE_PERSISTENCE to instruct nats to not persist cache data on disc.

Event Bus Configuration

The Infinite Scale event bus can be configured by a set of environment variables.

  • In case of an orchestrated installation like with Docker or Kubernetes, the event bus must be an external service for scalability like a Redis Sentinel cluster or a key-value-store NATS JetStream. Both named stores are supported and also used in Caching and Persistence. The store used is not part of the Infinite Scale installation and must be separately provided and configured.

  • Note that from a configuration point of view, caching and persistence are independent of the event bus configuration.

Note that for each global environment variable, a service-based one might be available additionally. For precedences see Environment Variable Notes. Check the configuration section below.

Without the aim of completeness, see the list of environment variables to configure the event bus:

Envvar Description

OCIS_EVENTS_ENDPOINT

The address of the event system.

OCIS_EVENTS_CLUSTER

The clusterID of the event system. Mandatory when using NATS as event system.

OCIS_EVENTS_ENABLE_TLS

Enable TLS for the connection to the events broker.

OCIS_INSECURE

Whether to verify the server TLS certificates.

OCIS_EVENTS_AUTH_USERNAME

The username to authenticate with the events broker.

OCIS_EVENTS_AUTH_PASSWORD

The password to authenticate with the events broker.

Configuration

Environment Variables

The storage-users service is configured via the following environment variables. Read the Environment Variable Types documentation for important details. Column IV shows with which release the environment variable has been introduced.

404: Not Found

  • 8.1.0

404: Not Found

YAML Example

  • 8.1.0

404: Not Found