Storage-System Service Configuration

Introduction

The Infinite Scale Storage-System service

Default Values

  • Storage-System listens on port 9215 by default.

Caching

The storage-system service can use a configured store via the global OCIS_CACHE_STORE environment variable.

Note that for each global environment variable, a service-based one might be available additionally. For precedences see Environment Variable Notes. Check the configuration section below. Supported stores are:

Store Type Description

memory

Basic in-memory store and the default.

redis-sentinel

Stores data in a configured Redis Sentinel cluster.

nats-js-kv

Stores data using key-value-store feature of NATS JetStream.

noop

Stores nothing. Useful for testing. Not recommended in production environments.

ocmem

(deprecated) Advanced in-memory store allowing max size.

redis

(deprecated) Stores data in a configured Redis cluster.

etcd

(deprecated) Stores data in a configured etcd cluster.

nats-js

(deprecated) Stores data using the key-value-store feature of NATS JetStream.

Other store types may work but are currently not supported.

The storage-system service can only be scaled if not using the memory store and the stores are configured identically over all instances!
If you have used one of the deprecated stores of a former version, you should reconfigure to use one of the supported ones as the deprecated stores will be removed in a later version.
Store specific notes

  • When using redis-sentinel:
    The Redis master to use is configured via e.g. OCIS_CACHE_STORE_NODES in the form of <sentinel-host>:<sentinel-port>/<redis-master> like 10.10.0.200:26379/mymaster.

  • When using nats-js-kv:

    • It is recommended to set OCIS_CACHE_STORE_NODES to the same value as OCIS_EVENTS_ENDPOINT. That way the cache uses the same nats instance as the event bus.

    • Authentication can be added, if configured, via OCIS_CACHE_AUTH_USERNAME and OCIS_CACHE_AUTH_PASSWORD.

    • It is possible to set OCIS_CACHE_DISABLE_PERSISTENCE to instruct nats to not persist cache data on disc.

Deprecated Metadata Backend

Starting with Infinite Scale version 3.0.0, the default backend for metadata switched to messagepack. If the setting STORAGE_SYSTEM_OCIS_METADATA_BACKEND has not been defined manually, the backend will be migrated to messagepack automatically. Though still possible to manually configure xattrs, this setting should not be used anymore as it will be removed in a later version.

Configuration

Environment Variables

The storage-system service is configured via the following environment variables. Read the Environment Variable Types documentation for important details.

  • master

Environment variables for the storage-system service
Name Type Default Value Description

OCIS_TRACING_ENABLED
STORAGE_SYSTEM_TRACING_ENABLED

bool

false

Activates tracing.

OCIS_TRACING_TYPE
STORAGE_SYSTEM_TRACING_TYPE

string

The type of tracing. Defaults to '', which is the same as 'jaeger'. Allowed tracing types are 'jaeger' and '' as of now.

OCIS_TRACING_ENDPOINT
STORAGE_SYSTEM_TRACING_ENDPOINT

string

The endpoint of the tracing agent.

OCIS_TRACING_COLLECTOR
STORAGE_SYSTEM_TRACING_COLLECTOR

string

The HTTP endpoint for sending spans directly to a collector, i.e. http://jaeger-collector:14268/api/traces. Only used if the tracing endpoint is unset.

OCIS_LOG_LEVEL
STORAGE_SYSTEM_LOG_LEVEL

string

The log level. Valid values are: 'panic', 'fatal', 'error', 'warn', 'info', 'debug', 'trace'.

OCIS_LOG_PRETTY
STORAGE_SYSTEM_LOG_PRETTY

bool

false

Activates pretty log output.

OCIS_LOG_COLOR
STORAGE_SYSTEM_LOG_COLOR

bool

false

Activates colorized log output.

OCIS_LOG_FILE
STORAGE_SYSTEM_LOG_FILE

string

The path to the log file. Activates logging to this file if set.

STORAGE_SYSTEM_DEBUG_ADDR

string

127.0.0.1:9217

Bind address of the debug server, where metrics, health, config and debug endpoints will be exposed.

STORAGE_SYSTEM_DEBUG_TOKEN

string

Token to secure the metrics endpoint

STORAGE_SYSTEM_DEBUG_PPROF

bool

false

Enables pprof, which can be used for profiling

STORAGE_SYSTEM_DEBUG_ZPAGES

bool

false

Enables zpages, which can be used for collecting and viewing in-memory traces.

STORAGE_SYSTEM_GRPC_ADDR

string

127.0.0.1:9215

The bind address of the GRPC service.

STORAGE_SYSTEM_GRPC_PROTOCOL

string

tcp

The transport protocol of the GPRC service.

STORAGE_SYSTEM_HTTP_ADDR

string

127.0.0.1:9216

The bind address of the HTTP service.

STORAGE_SYSTEM_HTTP_PROTOCOL

string

tcp

The transport protocol of the HTTP service.

OCIS_JWT_SECRET
STORAGE_SYSTEM_JWT_SECRET

string

The secret to mint and validate jwt tokens.

OCIS_REVA_GATEWAY

string

com.owncloud.api.gateway

The CS3 gateway endpoint.

OCIS_GRPC_CLIENT_TLS_MODE

string

TLS mode for grpc connection to the go-micro based grpc services. Possible values are 'off', 'insecure' and 'on'. 'off': disables transport security for the clients. 'insecure' allows using transport security, but disables certificate verification (to be used with the autogenerated self-signed certificates). 'on' enables transport security, including server certificate verification.

OCIS_GRPC_CLIENT_TLS_CACERT

string

Path/File name for the root CA certificate (in PEM format) used to validate TLS server certificates of the go-micro based grpc services.

OCIS_SYSTEM_USER_ID

string

ID of the oCIS storage-system system user. Admins need to set the ID for the STORAGE-SYSTEM system user in this config option which is then used to reference the user. Any reasonable long string is possible, preferably this would be an UUIDv4 format.

OCIS_SYSTEM_USER_API_KEY

string

API key for the STORAGE-SYSTEM system user.

STORAGE_SYSTEM_SKIP_USER_GROUPS_IN_TOKEN

bool

false

Disables the loading of user’s group memberships from the reva access token.

OCIS_CACHE_STORE
STORAGE_SYSTEM_CACHE_STORE

string

memory

The type of the cache store. Supported values are: 'memory', 'redis-sentinel', 'nats-js-kv', 'noop'. See the text description for details.

OCIS_CACHE_STORE_NODES
STORAGE_SYSTEM_CACHE_STORE_NODES

[]string

[127.0.0.1:9233]

A list of nodes to access the configured store. This has no effect when 'memory' or 'ocmem' stores are configured. Note that the behaviour how nodes are used is dependent on the library of the configured store. See the Environment Variable Types description for more details.

OCIS_CACHE_DATABASE

string

storage-system

The database name the configured store should use.

OCIS_CACHE_TTL
STORAGE_SYSTEM_CACHE_TTL

Duration

24m0s

Default time to live for user info in the user info cache. Only applied when access tokens has no expiration. See the Environment Variable Types description for more details.

OCIS_CACHE_SIZE
STORAGE_SYSTEM_CACHE_SIZE

int

0

The maximum quantity of items in the user info cache. Only applies when store type 'ocmem' is configured. Defaults to 512 which is derived from the ocmem package though not exclicitely set as default.

OCIS_CACHE_DISABLE_PERSISTENCE
STORAGE_SYSTEM_CACHE_DISABLE_PERSISTENCE

bool

false

Disables persistence of the cache. Only applies when store type 'nats-js-kv' is configured. Defaults to false.

OCIS_CACHE_AUTH_USERNAME
STORAGE_SYSTEM_CACHE_AUTH_USERNAME

string

Username for the configured store. Only applies when store type 'nats-js-kv' is configured.

OCIS_CACHE_AUTH_PASSWORD
STORAGE_SYSTEM_CACHE_AUTH_PASSWORD

string

Password for the configured store. Only applies when store type 'nats-js-kv' is configured.

STORAGE_SYSTEM_DRIVER

string

ocis

The driver which should be used by the service.

OCIS_DECOMPOSEDFS_METADATA_BACKEND
STORAGE_SYSTEM_OCIS_METADATA_BACKEND

string

messagepack

The backend to use for storing metadata. Supported values are 'messagepack' and 'xattrs'. The setting 'messagepack' uses a dedicated file to store file metadata while 'xattrs' uses extended attributes to store file metadata. Defaults to 'messagepack'.

STORAGE_SYSTEM_OCIS_ROOT

string

/var/lib/ocis/storage/metadata

Path for the directory where the STORAGE-SYSTEM service stores it’s persistent data. If not defined, the root directory derives from $OCIS_BASE_DATA_PATH:/storage.

STORAGE_SYSTEM_OCIS_MAX_ACQUIRE_LOCK_CYCLES

int

20

When trying to lock files, ocis will try this amount of times to acquire the lock before failing. After each try it will wait for an increasing amount of time. Values of 0 or below will be ignored and the default value of 20 will be used.

STORAGE_SYSTEM_OCIS_LOCK_CYCLE_DURATION_FACTOR

int

30

When trying to lock files, ocis will multiply the cycle with this factor and use it as a millisecond timeout. Values of 0 or below will be ignored and the default value of 30 will be used.

STORAGE_SYSTEM_DATA_SERVER_URL

string

http://localhost:9216/data

URL of the data server, needs to be reachable by other services using this service.

YAML Example

  • master

# Autogenerated
# Filename: storage-system-config-example.yaml

tracing:
  enabled: false
  type: ""
  endpoint: ""
  collector: ""
log:
  level: ""
  pretty: false
  color: false
  file: ""
debug:
  addr: 127.0.0.1:9217
  token: ""
  pprof: false
  zpages: false
grpc:
  addr: 127.0.0.1:9215
  tls: null
  protocol: tcp
http:
  addr: 127.0.0.1:9216
  protocol: tcp
token_manager:
  jwt_secret: ""
reva:
  address: com.owncloud.api.gateway
  tls:
    mode: ""
    cacert: ""
system_user_id: ""
system_user_api_key: ""
skip_user_groups_in_token: false
cache:
  store: memory
  nodes:
  - 127.0.0.1:9233
  database: storage-system
  ttl: 24m0s
  size: 0
  disable_persistence: false
  auth_username: ""
  auth_password: ""
driver: ocis
drivers:
  ocis:
    metadata_backend: messagepack
    root: /var/lib/ocis/storage/metadata
    max_acquire_lock_cycles: 20
    lock_cycle_duration_factor: 30
data_server_url: http://localhost:9216/data