Postprocessing Service Configuration

Table of Contents

Introduction
General Prerequisites
Post-Processing Functionality
Additional Prerequisites for the postprocessing Service
Post-Processing Steps
- Virus Scanning
- Delay
Custom Post-Processing Steps
- Prerequisites
- Workflow
CLI Commands
- Health
- Resume Post-Processing
Storing
Event Bus Configuration
Configuration
- Environment Variables
- YAML Example

Introduction

The Infinite Scale postprocessing service handles the coordination of asynchronous post-processing steps.

General Prerequisites

To use the postprocessing service, an event system needs to be configured for all services. By default, Infinite Scale ships with a preconfigured nats service.

Post-Processing Functionality

The storageprovider service (storage-users) can be configured to initiate asynchronous post-processing by setting the STORAGE_USERS_OCIS_ASYNC_UPLOADS environment variable to true. If this is the case, post-processing will get initiated after uploading a file and all bytes have been received.

The postprocessing service will then coordinate configured post-processing steps like scanning the file for viruses. During post-processing, the file will be in a processing state where only a limited set of actions are available.

The processing state excludes file accessibility by users.

When all postprocessing steps have completed successfully, the file will be made accessible to users.

Additional Prerequisites for the postprocessing Service

Once post-processing has been enabled, configuring any post-processing step will require the requested services to be enabled and preconfigured. For example, to use the virusscan step, one needs to have an enabled and configured antivirus service.

Post-Processing Steps

The postprocessing service is individually configurable. This is achieved by allowing a list of post-processing steps to be performed in order of their appearance in the POSTPROCESSING_STEPS envvar. This envvar expects a comma-separated list of steps that will be executed. Currently steps known to the system are virusscan and delay. Custom steps can be added but need an existing target for processing.

Virus Scanning

To enable virus scanning as a post-processing step after uploading a file, the environment variable POSTPROCESSING_STEPS needs to contain the word virusscan at one location in the list of steps. As a result, each uploaded file gets scanned for viruses as part of the post-processing steps. Note that the antivirus service must be enabled and configured for this to work.

Delay

Though this is for development purposes only and NOT RECOMMENDED on production systems, setting the environment variable POSTPROCESSING_DELAY to a duration not equal to zero will add a delay step with the configured amount of time. Infinite Scale will continue post-processing the file after the configured delay. Use the environment variable POSTPROCESSING_STEPS and the keyword delay if you have multiple post-processing steps and want to define their order. If POSTPROCESSING_DELAY is set but the keyword delay is not contained in POSTPROCESSING_STEPS, it will be executed as the last post-processing step without being listed as the last one. In this case, a log entry will be written on service startup to notify the admin about the situation. That log entry can be avoided by adding the keyword delay to POSTPROCESSING_STEPS.

Custom Post-Processing Steps

By using the envvar POSTPROCESSING_STEPS, custom post-processing steps can be added. Any word can be used as step name but be careful not to conflict with existing keywords like virusscan and delay. In addition, if a keyword is misspelled or the corresponding service either does not exist or does not follow the necessary event communication, the postprocessing service will wait forever to get the required response to proceed and therefore does not continue with any other processing.

Prerequisites

To use custom post-processing steps, you need a custom service listening to the configured event system. For more information, see General Prerequisites.

Workflow

When defining a custom postprocessing step (eg. "customstep"), the postprocessing service will eventually send an event during postprocessing. The event will be of type StartPostprocessingStep with its field StepToStart set to "customstep". When the service defined as custom step receives this event, it can safely execute its actions. The postprocessing service will wait until it has finished its work. The event contains further information (filename, executing user, size, …) and also requires tokens and URLs to download the file in case byte inspection is necessary.

Once the service defined as custom step has finished its work, it should send an event of type PostprocessingFinished via the configured events system back to the postprocessing service. This event needs to contain a FinishedStep field set to "customstep". It also must contain the outcome of the step, which can be one of the following:

delete: Abort postprocessing, delete the file.
abort: Abort postprocessing, keep the file.
retry: There was a problem that was most likely temporary and may be solved by trying again after some backoff duration. Retry runs automatically and is defined by the backoff behavior as described below.
continue: Continue postprocessing, this is the success case.

The backoff behavior as mentioned in the retry outcome can be configured using the POSTPROCESSING_RETRY_BACKOFF_DURATION and POSTPROCESSING_MAX_RETRIES environment variables. The backoff duration is calculated using the following formula after each failure: backoff_duration = POSTPROCESSING_RETRY_BACKOFF_DURATION * 2^(number of failures - 1). This means that the time between the next round grows exponentially limited by the number of retries. Steps that still don’t succeed after the maximum number of retries will be automatically moved to the abort state.

CLI Commands

The postprocessing service provides a CLI which can be used to check its health status and to resume/restart uploads. See the following printout for a shortened list:

ocis postprocessing --help

USAGE:
   ocis postprocessing [command options]

COMMANDS:
   resume, restart  resume postprocessing for an uploadID
   health           Check health status

Health

The postprocessing service provides a health command. For details see the Maintenance Commands description.

Resume Post-Processing

If post-processing fails in one step due to an unforeseen error, current uploads will not be resumed/restarted automatically. A system administrator can instead run CLI commands to resume/restart the failed upload manually.

The postprocessing service provides a restart and resume command which share common options:

OPTIONS:
   --upload-id value, -u value  the uploadid to resume. Ignored if unset.
   --step value, -s value       resume all uploads in the given postprocessing step. Ignored if upload-id is set. (default: "finished")

If not noted otherwise, commands with the restart option can also use the resume option. This changes behaviour slightly.

restart
When restarting an upload, all steps for open items will be restarted, except if otherwise defined.
resume
When resuming an upload, processing will continue unfinished items from their last completed step.

The resume/restart command is, at a minimum, a two-step process that requires commands from the storage-users service.

For details on the storage-users command which provides many options see the Manage Unfinished Uploads documentation.

Depending if you want to restart/resume all or defined failed uploads, different commands are used.

Failed uploads by session:
- First, list ongoing upload sessions to identify possibly failed ones.
  Note that there never can be a clear identification of a failed upload session due to various reasons causing them. You need to apply more critera like free space on disk, a failed service like antivirus etc. to declare an upload as failed.
  ocis storage-users uploads sessions
- All failed uploads
  If you want to restart/resume all failed uploads, just rerun the command with the relevant flag. Note that this is the preferred command to handle failed processing steps:
  ocis storage-users uploads sessions --resume

Failed uploads by ID or step:
Use the postprocessing command to resume defined failed uploads. For postprocessing steps, the default is to resume . Note that at the moment, resume is an alias for restart to keep old functionality. restart is subject to change and will most likely be removed in a later version.

Defined by ID
If you want to resume only a specific upload, use the postprocessing resume command with the ID selected:
```
ocis postprocessing resume -u <uploadID>
```

Defined by step
Alternatively, instead of restarting one specific upload, a system admin can also resume all uploads that are currently in a specific step.

Command examples:

ocis postprocessing resume                # Resumes all uploads where postprocessing is finished, but upload is not finished.
ocis postprocessing resume -s "finished"  # Equivalent to the above.
ocis postprocessing resume -s "virusscan" # Resume all uploads currently in virusscan step.

Output example:

$ ocis postprocessing resume --step finished
┌──────────────────────────────────────┬──────────────┬──────┬──────────┬──────────┬──────┐
│              UPLOAD ID               │   FILENAME   │ SIZE │   STEP   │ FINISHED │ USER │
├──────────────────────────────────────┼──────────────┼──────┼──────────┼──────────┼──────┤
│ a5b81897-0471-4ca0-9965-259263363f35 │ New file.txt │ 4    │ finished │ false    │ paul │
└──────────────────────────────────────┴──────────────┴──────┴──────────┴──────────┴──────┘

Resumed 1 upload(s) at step "finished"

Storing

The postprocessing service needs to store some metadata about uploads to be able to orchestrate post-processing. In distributed deployments it is recommended to use a persistent store, see below for more details.

The postprocessing service can use a configured store via the global OCIS_PERSISTENT_STORE environment variable.

Note that for each global environment variable, an independent service-based one might be available additionally. For precedences see Environment Variable Notes. Check the configuration section below. Supported stores are:

Store Type Description

Store Type	Description
`memory`	Basic in-memory store. Will not survive a restart. Usually the default for caches. See the store environment variable for which one is used.
`nats-js-kv`	Stores data using key-value-store feature of NATS JetStream. Usually the default for stores, see the store environment variable for which one is used.
`redis-sentinel`	Stores data in a configured Redis Sentinel cluster.
`noop`	Stores nothing. Useful for testing. Not recommended in production environments.

memory

Basic in-memory store. Will not survive a restart.
Usually the default for caches. See the store environment variable for which one is used.

nats-js-kv

Stores data using key-value-store feature of NATS JetStream.
Usually the default for stores, see the store environment variable for which one is used.

redis-sentinel

Stores data in a configured Redis Sentinel cluster.

noop

Stores nothing. Useful for testing. Not recommended in production environments.

The postprocessing service can only be scaled if not using the memory store and the stores are configured identically over all instances!

If you have used one of the deprecated stores of a former version, you should reconfigure to use one of the supported ones as the deprecated stores will be removed in a later version.

Store specific notes

When using redis-sentinel:
The Redis master to use is configured via e.g. OCIS_PERSISTENT_STORE_NODES in the form of <sentinel-host>:<sentinel-port>/<redis-master> like 10.10.0.200:26379/mymaster.
When using nats-js-kv:
- It is recommended to set OCIS_PERSISTENT_STORE_NODES to the same value as OCIS_EVENTS_ENDPOINT. That way the cache uses the same nats instance as the event bus. See the Event Bus Configuration for more details.
- Authentication can be added, if configured, via OCIS_CACHE_AUTH_USERNAME and OCIS_CACHE_AUTH_PASSWORD.
- It is possible to set OCIS_CACHE_DISABLE_PERSISTENCE to instruct nats to not persist cache data on disc.

Event Bus Configuration

The Infinite Scale event bus can be configured by a set of environment variables.

In case of an orchestrated installation like with Docker or Kubernetes, the event bus must be an external service for scalability like a Redis Sentinel cluster or a key-value-store NATS JetStream. Both named stores are supported and also used in Caching and Persistence. The store used is not part of the Infinite Scale installation and must be separately provided and configured.
Note that from a configuration point of view, caching and persistence are independent of the event bus configuration.

Note that for each global environment variable, a service-based one might be available additionally. For precedences see Environment Variable Notes. Check the configuration section below.

Without the aim of completeness, see the list of environment variables to configure the event bus:

Envvar Description

Envvar	Description
`OCIS_EVENTS_ENDPOINT`	The address of the event system.
`OCIS_EVENTS_CLUSTER`	The clusterID of the event system. Mandatory when using NATS as event system.
`OCIS_EVENTS_ENABLE_TLS`	Enable TLS for the connection to the events broker.
`OCIS_INSECURE`	Whether to verify the server TLS certificates.
`OCIS_EVENTS_AUTH_USERNAME`	The username to authenticate with the events broker.
`OCIS_EVENTS_AUTH_PASSWORD`	The password to authenticate with the events broker.

OCIS_EVENTS_ENDPOINT

The address of the event system.

OCIS_EVENTS_CLUSTER

The clusterID of the event system. Mandatory when using NATS as event system.

OCIS_EVENTS_ENABLE_TLS

Enable TLS for the connection to the events broker.

OCIS_INSECURE

Whether to verify the server TLS certificates.

OCIS_EVENTS_AUTH_USERNAME

The username to authenticate with the events broker.

OCIS_EVENTS_AUTH_PASSWORD

The password to authenticate with the events broker.

Configuration

Environment Variables

The postprocessing service is configured via the following environment variables. Read the Environment Variable Types documentation for important details. Column IV shows with which release the environment variable has been introduced.

404: Not Found

8.2.0

404: Not Found

YAML Example

Note the file shown below must be renamed and placed in the correct folder according to the Configuration File Naming conventions to be effective.
See the Notes for Environment Variables if you want to use environment variables in the yaml file.

8.2.0

404: Not Found