Storage-Users Service Configuration
Manage Unfinished Uploads
When using Infinite Scale as user storage, a directory named storage/users/uploads
can be found in the Infinite Scale data folder. This is an intermediate directory based on TUS which is an open protocol for resumable uploads. Each upload consists of a blob and a blob.info file. Note that the term blob is just a placeholder.
-
If an upload succeeds, the blob file will be moved to the target and the blob.info file will be deleted.
-
In case of incomplete uploads, the blob and blob.info files will continue to recieve data until either the upload succeeds in time or the upload expires based on the
STORAGE_USERS_UPLOAD_EXPIRATION
variable, see the table below for details. -
In case of expired uploads, the blob and blob.info files will not be removed automatically. Thus a lot of data can pile up over time wasting storage space.
-
In the rare case of a failure, after the upload succeeded but the file was not moved to its target location, which can happen when postprocessing fails, the situation is the same as with expired uploads.
- Example cases for expired uploads
-
-
When a user uploads a big file but the file exceeds the user-quota, the upload can’t be moved to the target after it has finished. The file stays at the upload location until it is manually cleared.
-
If the bandwith is limited and the file to transfer can’t be transferred completely before the upload expiration time is reached, the file expires and can’t be processed.
-
- There are two commands available to manage unfinished uploads
ocis storage-users uploads <command>
COMMANDS:
list Print a list of all incomplete uploads
clean Clean up leftovers from expired uploads
Command Examples
- Command to identify incomplete uploads
-
ocis storage-users uploads list
Incomplete uploads: - 455bd640-cd08-46e8-a5a0-9304908bd40a (file_example_PPT_1MB.ppt, Size: 1028608, Expires: 2022-08-17T12:35:34+02:00)
- Command to clear expired uploads
-
ocis storage-users uploads clean
Cleaned uploads: - 455bd640-cd08-46e8-a5a0-9304908bd40a (Filename: file_example_PPT_1MB.ppt, Size: 1028608, Expires: 2022-08-17T12:35:34+02:00)
Purge Expired Space Trash-Bin Items
This command is about purging old trash-bin items of project
spaces (spaces that have been created manually) and personal
spaces.
ocis storage-users trash-bin <command>
COMMANDS:
purge-expired Purge all expired items from the trashbin
The configuration for the purge-expired
command is done by using the following environment variables.
-
STORAGE_USERS_PURGE_TRASH_BIN_USER_ID
is used to obtain space trash-bin information and takes the system admin user as the default which is theOCIS_ADMIN_USER_ID
but can be set individually. It should be noted, that theOCIS_ADMIN_USER_ID
is only assigned automatically when using the single binary deployment and must be manually assigned in all other deployments. The command only considers spaces to which the assigned user has access and delete permission. -
STORAGE_USERS_PURGE_TRASH_BIN_PERSONAL_DELETE_BEFORE
has a default value of30 days
, which means the command will delete all files older than30 days
. The value is human-readable, valid values are24h
,60m
,60s
etc.0
is equivalent to disable and prevents the deletion ofpersonal space
trash-bin files. -
STORAGE_USERS_PURGE_TRASH_BIN_PROJECT_DELETE_BEFORE
has a default value of30 days
, which means the command will delete all files older than30 days
. The value is human-readable, valid values are24h
,60m
,60s
etc.0
is equivalent to disable and prevents the deletion ofproject space
trash-bin files.
Configuration
Environment Variables
The storage-users
service is configured via the following environment variables:
Name | Type | Default Value | Description |
---|---|---|---|
|
bool |
false |
Activates tracing. |
|
string |
|
The type of tracing. Defaults to "", which is the same as "jaeger". Allowed tracing types are "jaeger" and "" as of now. |
|
string |
|
The endpoint of the tracing agent. |
|
string |
|
The HTTP endpoint for sending spans directly to a collector, i.e. http://jaeger-collector:14268/api/traces. Only used if the tracing endpoint is unset. |
|
string |
|
The log level. Valid values are: "panic", "fatal", "error", "warn", "info", "debug", "trace". |
|
bool |
false |
Activates pretty log output. |
|
bool |
false |
Activates colorized log output. |
|
string |
|
The path to the log file. Activates logging to this file if set. |
|
string |
127.0.0.1:9159 |
Bind address of the debug server, where metrics, health, config and debug endpoints will be exposed. |
|
string |
|
Token to secure the metrics endpoint. |
|
bool |
false |
Enables pprof, which can be used for profiling. |
|
bool |
false |
Enables zpages, which can be used for collecting and viewing in-memory traces. |
|
string |
127.0.0.1:9157 |
The bind address of the GRPC service. |
|
string |
tcp |
The transport protocol of the GPRC service. |
|
string |
127.0.0.1:9158 |
The bind address of the HTTP service. |
|
string |
tcp |
The transport protocol of the HTTP service. |
|
string |
|
The secret to mint and validate jwt tokens. |
|
string |
127.0.0.1:9142 |
The CS3 gateway endpoint. |
|
string |
|
TLS mode for grpc connection to the go-micro based grpc services. Possible values are 'off', 'insecure' and 'on'. 'off': disables transport security for the clients. 'insecure' allows to use transport security, but disables certificate verification (to be used with the autogenerated self-signed certificates). 'on' enables transport security, including server ceritificate verification. |
|
string |
|
Path/File name for the root CA certificate (in PEM format) used to validate TLS server certificates of the go-micro based grpc services. |
|
bool |
false |
Disables the loading of user’s group memberships from the reva access token. |
|
string |
ocis |
The storage driver which should be used by the service. Defaults to 'ocis', Supported values are: 'ocis', 's3ng' and 'owncloudsql'. The 'ocis' driver stores all data (blob and meta data) in an POSIX compliant volume. The 's3ng' driver stores metadata in a POSIX compliant volume and uploads blobs to the s3 bucket. |
|
string |
xattrs |
The backend to use for storing metadata. Supported values are 'xattrs' and 'ini'. The setting 'xattrs' uses extended attributes to store file metadata while 'ini' uses a dedicated file to store file metadata. Defaults to 'xattrs'. |
|
string |
~/.ocis/storage/users |
The directory where the filesystem storage will store blobs and metadata. If not definied, the root directory derives from $OCIS_BASE_DATA_PATH:/storage/users. |
|
string |
{{.Id.OpaqueId}} |
Template string for the user storage layout in the user directory. |
|
string |
127.0.0.1:9191 |
Endpoint of the permissions service. The endpoints can differ for 'ocis' and 's3ng'. |
|
string |
{{.SpaceType}}/{{.User.Username | lower}} |
Template string to construct personal space aliases. |
|
string |
{{.SpaceType}}/{{.SpaceName | replace " " "-" | lower}} |
Template string to construct general space aliases. |
|
string |
/Shares |
Name of the folder jailing all shares. |
|
int |
20 |
When trying to lock files, ocis will try this amount of times to acquire the lock before failing. After each try it will wait for an increasing amount of time. Values of 0 or below will be ignored and the default value of 20 will be used. |
|
int |
30 |
When trying to lock files, ocis will multiply the cycle with this factor and use it as a millisecond timeout. Values of 0 or below will be ignored and the default value of 30 will be used. |
|
bool |
false |
Enable asynchronous file uploads. |
|
uint64 |
0 |
Set a global max quota for spaces in bytes. A value of 0 equals unlimited. If not using the global OCIS_SPACES_MAX_QUOTA, you must define the FRONTEND_MAX_QUOTA in the frontend service. |
|
string |
xattrs |
The backend to use for storing metadata. Supported values are 'xattrs' and 'ini'. The setting 'xattrs' uses extended attributes to store file metadata while 'ini' uses a dedicated file to store file metadata. Defaults to 'xattrs'. |
|
string |
~/.ocis/storage/users |
The directory where the filesystem storage will store metadata for blobs. If not definied, the root directory derives from $OCIS_BASE_DATA_PATH:/storage/users. |
|
string |
{{.Id.OpaqueId}} |
Template string for the user storage layout in the user directory. |
|
string |
127.0.0.1:9191 |
Endpoint of the permissions service. The endpoints can differ for 'ocis' and 's3ng'. |
|
string |
default |
Region of the S3 bucket. |
|
string |
|
Access key for the S3 bucket. |
|
string |
|
Secret key for the S3 bucket. |
|
string |
|
Endpoint for the S3 bucket. |
|
string |
|
Name of the S3 bucket. |
|
string |
{{.SpaceType}}/{{.User.Username | lower}} |
Template string to construct personal space aliases. |
|
string |
{{.SpaceType}}/{{.SpaceName | replace " " "-" | lower}} |
Template string to construct general space aliases. |
|
string |
/Shares |
Name of the folder jailing all shares. |
|
int |
20 |
When trying to lock files, ocis will try this amount of times to acquire the lock before failing. After each try it will wait for an increasing amount of time. Values of 0 or below will be ignored and the default value of 20 will be used. |
|
int |
30 |
When trying to lock files, ocis will multiply the cycle with this factor and use it as a millisecond timeout. Values of 0 or below will be ignored and the default value of 30 will be used. |
|
string |
~/.ocis/storage/owncloud |
The directory where the filesystem storage will store SQL migration data. If not definied, the root directory derives from $OCIS_BASE_DATA_PATH:/storage/owncloud. |
|
string |
/Shares |
Name of the folder jailing all shares. |
|
string |
{{.Username}} |
Path layout to use to navigate into a users folder in an owncloud data directory |
|
string |
~/.ocis/storage/uploadinfo |
The directory where the filesystem will store uploads temporarily. If not definied, the root directory derives from $OCIS_BASE_DATA_PATH:/storage/uploadinfo. |
|
string |
owncloud |
Username for the database. |
|
string |
owncloud |
Password for the database. |
|
string |
|
Hostname or IP of the database server. |
|
int |
3306 |
Port that the database server is listening on. |
|
string |
owncloud |
Name of the database to be used. |
|
string |
localhost:9144 |
Endpoint of the users provider. |
|
string |
http://localhost:9158/data |
URL of the data server, needs to be reachable by the data gateway provided by the frontend service or the user if directly exposed. |
|
string |
https://localhost:9200/data |
URL of the data gateway server |
|
int64 |
86400 |
the time after which the token for upload postprocessing expires |
|
string |
127.0.0.1:9233 |
The address of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture. |
|
string |
ocis-cluster |
The clusterID of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture. Mandatory when using NATS as event system. |
|
bool |
false |
Whether to verify the server TLS certificates. |
|
string |
|
The root CA certificate used to validate the server’s TLS certificate. If provided STORAGE_USERS_EVENTS_TLS_INSECURE will be seen as false. |
|
bool |
false |
Enable TLS for the connection to the events broker. The events broker is the ocis service which receives and delivers events between the services.. |
|
int |
0 |
The amount of concurrent event consumers to start. Event consumers are used for post-processing files. Multiple consumers increase parallelisation, but will also increase CPU and memory demands. The setting has no effect when the STORAGE_USERS_OCIS_ASYNC_UPLOADS is set to false. The default and minimum value is 1. |
|
string |
memory |
Store implementation for the cache. Supported values are 'memory' (default), 'redis', 'redis-sentinel', 'nats-js', and 'etcd'. See the text description for details. |
|
[]string |
[] |
A comma separated list of nodes to access the configured store. This has no effect when the 'memory' store is configured. Note that the behaviour how nodes are used is dependent on the library of the configured store. |
|
string |
users |
The database name the configured store should use. |
|
string |
|
Mount ID of this storage. |
|
bool |
false |
Exposes the data server directly to users and bypasses the data gateway. Ensure that the data server address is reachable by users. |
|
bool |
false |
Set this storage to be read-only. |
|
int64 |
86400 |
Duration in seconds after which uploads will expire. |
|
string |
|
ID of the user who collects all necessary information for deletion. |
|
Duration |
720h0m0s |
Specifies the period of time in which items that have been in the personal trash-bin for longer than this value should be deleted. A value of 0 means no automatic deletion. The value is human-readable, valid values are '24h', '60m', '60s' etc. |
|
Duration |
720h0m0s |
Specifies the period of time in which items that have been in the project trash-bin for longer than this value should be deleted. A value of 0 means no automatic deletion. The value is human-readable, valid values are '24h', '60m', '60s' etc. |
Name | Type | Default Value | Description |
---|---|---|---|
|
bool |
false |
Activates tracing. |
|
string |
|
The type of tracing. Defaults to "", which is the same as "jaeger". Allowed tracing types are "jaeger" and "" as of now. |
|
string |
|
The endpoint of the tracing agent. |
|
string |
|
The HTTP endpoint for sending spans directly to a collector, i.e. http://jaeger-collector:14268/api/traces. Only used if the tracing endpoint is unset. |
|
string |
|
The log level. Valid values are: "panic", "fatal", "error", "warn", "info", "debug", "trace". |
|
bool |
false |
Activates pretty log output. |
|
bool |
false |
Activates colorized log output. |
|
string |
|
The path to the log file. Activates logging to this file if set. |
|
string |
127.0.0.1:9159 |
Bind address of the debug server, where metrics, health, config and debug endpoints will be exposed. |
|
string |
|
Token to secure the metrics endpoint. |
|
bool |
false |
Enables pprof, which can be used for profiling. |
|
bool |
false |
Enables zpages, which can be used for collecting and viewing in-memory traces. |
|
string |
127.0.0.1:9157 |
The bind address of the GRPC service. |
|
bool |
false |
Activates TLS for the grpc based services using the server certifcate and key configured via OCIS_GRPC_TLS_CERTIFICATE and OCIS_GRPC_TLS_KEY. If OCIS_GRPC_TLS_CERTIFICATE is not set a temporary server certificate is generated - to be used with OCIS_GRPC_CLIENT_TLS_MODE=insecure. |
|
string |
|
Path/File name of the TLS server certificate (in PEM format) for the grpc services. |
|
string |
|
Path/File name for the TLS certificate key (in PEM format) for the server certificate to use for the grpc services. |
|
string |
tcp |
The transport protocol of the GPRC service. |
|
string |
127.0.0.1:9158 |
The bind address of the HTTP service. |
|
string |
tcp |
The transport protocol of the HTTP service. |
|
string |
|
The secret to mint and validate jwt tokens. |
|
string |
127.0.0.1:9142 |
The CS3 gateway endpoint. |
|
string |
|
TLS mode for grpc connection to the go-micro based grpc services. Possible values are 'off', 'insecure' and 'on'. 'off': disables transport security for the clients. 'insecure' allows to use transport security, but disables certificate verification (to be used with the autogenerated self-signed certificates). 'on' enables transport security, including server ceritificate verification. |
|
string |
|
Path/File name for the root CA certificate (in PEM format) used to validate TLS server certificates of the go-micro based grpc services. |
|
bool |
false |
Disables the loading of user’s group memberships from the reva access token. |
|
string |
ocis |
The storage driver which should be used by the service. Defaults to 'ocis', Supported values are: 'ocis', 's3ng' and 'owncloudsql'. The 'ocis' driver stores all data (blob and meta data) in an POSIX compliant volume. The 's3ng' driver stores metadata in a POSIX compliant volume and uploads blobs to the s3 bucket. |
|
string |
~/.ocis/storage/users |
The directory where the filesystem storage will store blobs and metadata. If not definied, the root directory derives from $OCIS_BASE_DATA_PATH:/storage/users. |
|
string |
{{.Id.OpaqueId}} |
Template string for the user storage layout in the user directory. |
|
string |
127.0.0.1:9191 |
Endpoint of the permissions service. The endpoints can differ for 'ocis' and 's3ng'. |
|
string |
{{.SpaceType}}/{{.User.Username | lower}} |
Template string to construct personal space aliases. |
|
string |
{{.SpaceType}}/{{.SpaceName | replace " " "-" | lower}} |
Template string to construct general space aliases. |
|
string |
/Shares |
Name of the folder jailing all shares. |
|
int |
20 |
When trying to lock files, ocis will try this amount of times to acquire the lock before failing. After each try it will wait for an increasing amount of time. Values of 0 or below will be ignored and the default value of 20 will be used. |
|
int |
30 |
When trying to lock files, ocis will multiply the cycle with this factor and use it as a millisecond timeout. Values of 0 or below will be ignored and the default value of 30 will be used. |
|
string |
~/.ocis/storage/users |
The directory where the filesystem storage will store metadata for blobs. If not definied, the root directory derives from $OCIS_BASE_DATA_PATH:/storage/users. |
|
string |
{{.Id.OpaqueId}} |
Template string for the user storage layout in the user directory. |
|
string |
127.0.0.1:9191 |
Endpoint of the permissions service. The endpoints can differ for 'ocis' and 's3ng'. |
|
string |
default |
Region of the S3 bucket. |
|
string |
|
Access key for the S3 bucket. |
|
string |
|
Secret key for the S3 bucket. |
|
string |
|
Endpoint for the S3 bucket. |
|
string |
|
Name of the S3 bucket. |
|
string |
{{.SpaceType}}/{{.User.Username | lower}} |
Template string to construct personal space aliases. |
|
string |
{{.SpaceType}}/{{.SpaceName | replace " " "-" | lower}} |
Template string to construct general space aliases. |
|
string |
/Shares |
Name of the folder jailing all shares. |
|
int |
20 |
When trying to lock files, ocis will try this amount of times to acquire the lock before failing. After each try it will wait for an increasing amount of time. Values of 0 or below will be ignored and the default value of 20 will be used. |
|
int |
30 |
When trying to lock files, ocis will multiply the cycle with this factor and use it as a millisecond timeout. Values of 0 or below will be ignored and the default value of 30 will be used. |
|
string |
~/.ocis/storage/owncloud |
The directory where the filesystem storage will store SQL migration data. If not definied, the root directory derives from $OCIS_BASE_DATA_PATH:/storage/owncloud. |
|
string |
/Shares |
Name of the folder jailing all shares. |
|
string |
{{.Username}} |
Path layout to use to navigate into a users folder in an owncloud data directory |
|
string |
~/.ocis/storage/uploadinfo |
The directory where the filesystem will store uploads temporarily. If not definied, the root directory derives from $OCIS_BASE_DATA_PATH:/storage/uploadinfo. |
|
string |
owncloud |
Username for the database. |
|
string |
owncloud |
Password for the database. |
|
string |
|
Hostname or IP of the database server. |
|
int |
3306 |
Port that the database server is listening on. |
|
string |
owncloud |
Name of the database to be used. |
|
string |
localhost:9144 |
Endpoint of the users provider. |
|
string |
http://localhost:9158/data |
URL of the data server, needs to be reachable by the data gateway provided by the frontend service or the user if directly exposed. |
|
string |
127.0.0.1:9233 |
The address of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture. |
|
string |
ocis-cluster |
The clusterID of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture. Mandatory when using NATS as event system. |
|
bool |
false |
Whether to verify the server TLS certificates. |
|
string |
|
The root CA certificate used to validate the server’s TLS certificate. If provided STORAGE_USERS_EVENTS_TLS_INSECURE will be seen as false. |
|
bool |
false |
Enable TLS for the connection to the events broker. The events broker is the ocis service which receives and delivers events between the services.. |
|
string |
memory |
Store implementation for the cache. Valid values are "memory" (default), "redis", and "etcd". |
|
[]string |
[] |
Node addresses to use for the cache store. |
|
string |
users |
Database name of the cache. |
|
string |
|
Mount ID of this storage. |
|
bool |
false |
Exposes the data server directly to users and bypasses the data gateway. Ensure that the data server address is reachable by users. |
|
bool |
false |
Set this storage to be read-only. |
|
int64 |
86400 |
Duration in seconds after which uploads will expire. |
YAML Example
Note that the filename shown below has been chosen on purpose.
See the Configuration File Naming for details when setting up your own configuration.
# Autogenerated
# Filename: storage-users-config-example.yaml
tracing:
enabled: false
type: ""
endpoint: ""
collector: ""
log:
level: ""
pretty: false
color: false
file: ""
debug:
addr: 127.0.0.1:9159
token: ""
pprof: false
zpages: false
grpc:
addr: 127.0.0.1:9157
tls: null
protocol: tcp
http:
addr: 127.0.0.1:9158
protocol: tcp
prefix: data
token_manager:
jwt_secret: ""
reva:
address: 127.0.0.1:9142
tls:
mode: ""
cacert: ""
skip_user_groups_in_token: false
driver: ocis
drivers:
ocis:
metadata_backend: xattrs
root: ~/.ocis/storage/users
user_layout: '{{.Id.OpaqueId}}'
permissions_endpoint: 127.0.0.1:9191
personalspacealias_template: '{{.SpaceType}}/{{.User.Username | lower}}'
generalspacealias_template: '{{.SpaceType}}/{{.SpaceName | replace " " "-" | lower}}'
share_folder: /Shares
max_acquire_lock_cycles: 20
lock_cycle_duration_factor: 30
async_uploads: false
max_quota: 0
s3ng:
metadata_backend: xattrs
root: ~/.ocis/storage/users
user_layout: '{{.Id.OpaqueId}}'
permissions_endpoint: 127.0.0.1:9191
region: default
access_key: ""
secret_key: ""
endpoint: ""
bucket: ""
personalspacealias_template: '{{.SpaceType}}/{{.User.Username | lower}}'
generalspacealias_template: '{{.SpaceType}}/{{.SpaceName | replace " " "-" | lower}}'
share_folder: /Shares
max_acquire_lock_cycles: 20
lock_cycle_duration_factor: 30
owncloudsql:
root: ~/.ocis/storage/owncloud
share_folder: /Shares
user_layout: '{{.Username}}'
upload_info_dir: ~/.ocis/storage/uploadinfo
db_username: owncloud
db_password: owncloud
db_host: ""
db_port: 3306
db_name: owncloud
users_provider_endpoint: localhost:9144
data_server_url: http://localhost:9158/data
data_gateway_url: https://localhost:9200/data
transfer_expires: 86400
events:
endpoint: 127.0.0.1:9233
cluster: ocis-cluster
tls_insecure: false
tls_root_ca_cert_path: ""
enable_tls: false
num_consumers: 0
cache:
store: memory
nodes: []
database: users
mount_id: ""
expose_data_server: false
readonly: false
upload_expiration: 86400
tasks:
purge_trash_bin:
user_id: ""
personal_delete_before: 720h0m0s
project_delete_before: 720h0m0s
# Autogenerated
# Filename: storage-users-config-example.yaml
tracing:
enabled: false
type: ""
endpoint: ""
collector: ""
log:
level: ""
pretty: false
color: false
file: ""
debug:
addr: 127.0.0.1:9159
token: ""
pprof: false
zpages: false
grpc:
addr: 127.0.0.1:9157
tls:
enabled: false
cert: ""
key: ""
protocol: tcp
http:
addr: 127.0.0.1:9158
protocol: tcp
prefix: data
token_manager:
jwt_secret: ""
reva:
address: 127.0.0.1:9142
tls:
mode: ""
cacert: ""
skip_user_groups_in_token: false
driver: ocis
drivers:
ocis:
root: ~/.ocis/storage/users
user_layout: '{{.Id.OpaqueId}}'
permissions_endpoint: 127.0.0.1:9191
personalspacealias_template: '{{.SpaceType}}/{{.User.Username | lower}}'
generalspacealias_template: '{{.SpaceType}}/{{.SpaceName | replace " " "-" | lower}}'
share_folder: /Shares
max_acquire_lock_cycles: 20
lock_cycle_duration_factor: 30
s3ng:
root: ~/.ocis/storage/users
user_layout: '{{.Id.OpaqueId}}'
permissions_endpoint: 127.0.0.1:9191
region: default
access_key: ""
secret_key: ""
endpoint: ""
bucket: ""
personalspacealias_template: '{{.SpaceType}}/{{.User.Username | lower}}'
generalspacealias_template: '{{.SpaceType}}/{{.SpaceName | replace " " "-" | lower}}'
share_folder: /Shares
max_acquire_lock_cycles: 20
lock_cycle_duration_factor: 30
owncloudsql:
root: ~/.ocis/storage/owncloud
share_folder: /Shares
user_layout: '{{.Username}}'
upload_info_dir: ~/.ocis/storage/uploadinfo
db_username: owncloud
db_password: owncloud
db_host: ""
db_port: 3306
db_name: owncloud
users_provider_endpoint: localhost:9144
data_server_url: http://localhost:9158/data
events:
endpoint: 127.0.0.1:9233
cluster: ocis-cluster
tls_insecure: false
tls_root_ca_cert_path: ""
enable_tls: false
cache:
store: memory
nodes: []
database: users
mount_id: ""
expose_data_server: false
readonly: false
upload_expiration: 86400