Postprocessing Service Configuration

Introduction

The Infinite Scale postprocessing service handles the coordination of asynchronous post-processing steps.

General Prerequisites

To use the postprocessing service, an event system needs to be configured for all services. By default, Infinite Scale ships with a preconfigured nats service.

Post-Processing Functionality

The storageprovider service (storage-users) can be configured to initiate asynchronous post-processing by setting the STORAGE_USERS_OCIS_ASYNC_UPLOADS environment variable to true. If this is the case, post-processing will get initiated after uploading a file and all bytes have been received.

The postprocessing service will then coordinate configured post-processing steps like scanning the file for viruses. During post-processing, the file will be in a processing state where only a limited set of actions are available.

The processing state excludes file accessibility by users.

When all postprocessing steps have completed successfully, the file will be made accessible to users.

Additional Prerequisites for the postprocessing Service

Once post-processing has been enabled, configuring any post-processing step will require the requested services to be enabled and preconfigured. For example, to use the virusscan step, one needs to have an enabled and configured antivirus service.

Post-Processing Steps

The postprocessing service is individually configurable. This is achieved by allowing a list of post-processing steps to be performed in order of their appearance in the POSTPROCESSING_STEPS envvar. This envvar expects a comma-separated list of steps that will be executed. Currently steps known to the system are virusscan and delay. Custom steps can be added but need an existing target for processing.

Virus Scanning

To enable virus scanning as a post-processing step after uploading a file, the environment variable POSTPROCESSING_STEPS needs to contain the word virusscan at one location in the list of steps. As a result, each uploaded file gets scanned for viruses as part of the post-processing steps. Note that the antivirus service must be enabled and configured for this to work.

Delay

Though this is for development purposes only and NOT RECOMMENDED on production systems, setting the environment variable POSTPROCESSING_DELAY to a duration not equal to zero will add a delay step with the configured amount of time. Infinite Scale will continue post-processing the file after the configured delay. Use the enviroment variable POSTPROCESSING_STEPS and the keyword delay if you have multiple post-processing steps and want to define their order. If POSTPROCESSING_DELAY is set but the keyword delay is not contained in POSTPROCESSING_STEPS, it will be executed as the last post-processing step without being listed as the last one. In this case, a log entry will be written on service startup to notify the admin about the situation. That log entry can be avoided by adding the keyword delay to POSTPROCESSING_STEPS.

Custom Post-Processing Steps

By using the envvar POSTPROCESSING_STEPS, custom post-processing steps can be added. Any word can be used as step name but be careful not to conflict with existing keywords like virusscan and delay. In addition, if a keyword is misspelled or the corresponding service either does not exist or does not follow the necessary event communication, the postprocessing service will wait forever to get the required response to proceed and therefore does not continue with any other processing.

Prerequisites

To use custom post-processing steps, you need a custom service listening to the configured event system. For more information, see General Prerequisites.

Workflow

When setting a custom post-processing step (eg. "customstep") the postprocessing service will eventually send an event during post-processing. The event will be of the type StartPostprocessingStep with its field StepToStart set to "customstep". When the custom service receives this event, it can safely execute its actions. The post-processing service will wait until it has finished its work. The event contains further information (like filename, executing user, size, …​) and required tokens or URLs to download the file in case byte inspection is necessary.

Once the custom service has finished its work, it should send an event of the type PostprocessingFinished via the configured event system. This event needs to contain a FinishedStep field set to "customstep". It also must contain the outcome of the step, which can be one of delete (abort post-processing, delete the file), abort (abort post-processing, keep the file) and continue (continue post-processing, this is the success case).

Configuration

Environment Variables

The postprocessing service is configured via the following environment variables:

  • latest

Deprecation notes for the postprocessing service
Deprecation Info Deprecation Version Removal Version Deprecation Replacment

POSTPROCESSING_VIRUSSCAN is not longer necessary and is replaced by POSTPROCESSING_STEPS which also holds information about the order of steps

master

master

POSTPROCESSING_STEPS

Environment variables for the postprocessing service
Name Type Default Value Description

OCIS_LOG_LEVEL
POSTPROCESSING_LOG_LEVEL

string

The log level. Valid values are: "panic", "fatal", "error", "warn", "info", "debug", "trace".

OCIS_LOG_PRETTY
POSTPROCESSING_LOG_PRETTY

bool

false

Activates pretty log output.

OCIS_LOG_COLOR
POSTPROCESSING_LOG_COLOR

bool

false

Activates colorized log output.

OCIS_LOG_FILE
POSTPROCESSING_LOG_FILE

string

The path to the log file. Activates logging to this file if set.

POSTPROCESSING_EVENTS_ENDPOINT

string

127.0.0.1:9233

The address of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture.

POSTPROCESSING_EVENTS_CLUSTER

string

ocis-cluster

The clusterID of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture. Mandatory when using NATS as event system.

OCIS_INSECURE
POSTPROCESSING_EVENTS_TLS_INSECURE

bool

false

Whether the ocis server should skip the client certificate verification during the TLS handshake.

POSTPROCESSING_EVENTS_TLS_ROOT_CA_CERTIFICATE

string

The root CA certificate used to validate the server’s TLS certificate. If provided POSTPROCESSING_EVENTS_TLS_INSECURE will be seen as false.

OCIS_EVENTS_ENABLE_TLS
POSTPROCESSING_EVENTS_ENABLE_TLS

bool

false

Enable TLS for the connection to the events broker. The events broker is the ocis service which receives and delivers events between the services.

POSTPROCESSING_STEPS

[]string

[]

A comma separated list of postprocessing steps, processed in order of their appearance. Currently supported values by the system are: 'virusscan' and 'delay'. Custom steps are allowed. See the documentation for instructions.

POSTPROCESSING_VIRUSSCAN
Deprecation Note

bool

false

After uploading a file but before making it available for download, virus scanning the file can be enabled. Needs as prerequisite the antivirus service to be enabled and configured.

POSTPROCESSING_DELAY

Duration

0s

After uploading a file but before making it available for download, a delay step can be added. Intended for developing purposes only. The duration can be set as number followed by a unit identifier like s, m or h. If a duration is set but the keyword 'delay' is not explicitely added to 'POSTPROCESSING_STEPS', the delay step will be processed as last step. In such a case, a log entry will be written on service startup to remind the admin about that situation.

YAML Example

  • latest

# Autogenerated
# Filename: postprocessing-config-example.yaml

log:
  level: ""
  pretty: false
  color: false
  file: ""
postprocessing:
  events:
    endpoint: 127.0.0.1:9233
    cluster: ocis-cluster
    tls_insecure: false
    tls_root_ca_certificate: ""
    enable_tls: false
  steps: []
  virusscan: false
  delayprocessing: 0s