Caching and Persistence

Introduction

Caching can not only be a big boost to performance but can also be essential when setting up Infinite Scale via container orchestration. Persistence is recommended for some services, for example, to manage event data for later processing, or, not limited to but especially when the underlying services are scaled via container orchestration providing consistent data across instantiated services.

General Information

Main Drivers

  • Performance
    Caching does not need to be persistent and in case of outages, the cache is rebuilt and no information is lost. This is typically used for caching metadata and filesystem access (stat).

  • Persistence
    With this kind of store, saving information temporarily on a persistent media is key. Persistence stores are used, for example, for generated events returning status info to users when requested. If this data gets lost or is not consistently available over the instantiated services, users will be inconsistently informed about state changes that happened.

  • Scaling
    When services are scaled horizontally (instantiated), a common and persistent base for generated data is necessary.

Identifying Caches by Their Name

  • Performance caches
    Can be identified via environment variables ending with xxx_CACHE_STORE which is valid for both local and global ones.

  • Persistent caches
    Can be identified via environment variables ending with xxx_STORE when using local ones and OCIS_PERSISTENT_STORE(_xxx) when using global ones. Note that no other trailing name components are present when using local ones.

Store Types

  • Memory
    This type has best performance but is not suitable for all requirements as it has no persistence and is limited to the service on the running server. It can be used for deployments where:

    • persistence is not a requirement (issue: restart/reboot) and

    • scaling of services is not required (issue: instantiation).

  • External products
    These products like nats-js-kv or redis-sentinel are independent services not related to Infinite Scale. They are usually installed on separate hardware requiring their own performance, availability and scalability measures. These products are connected via a fast network. Infinite Scale connects to these store types and uses their services like a black box.

Overview of Available Types

The following table gives an overview of store types available in Infinite Scale.

Store Type Description

memory

Basic in-memory store. Will not survive a restart.
Usually the default for caches. See the store environment variable for which one is used.

nats-js-kv

Stores data using key-value-store feature of NATS JetStream.
Usually the default for stores, see the store environment variable for which one is used.

redis-sentinel

Stores data in a configured Redis Sentinel cluster.

noop

Stores nothing. Useful for testing. Not recommended in production environments.

Local vs Global Configuration

  • Performance and persistent types can be configured per service individually or via global environment variables. See the Notes for Environment Variables for details with regards to settings precedence.

    When configuring stat and metadata caches individually for each service and not via the corresponding global environment variable, all caches must have the same configuration (access to the same cache) to provide consistent results.

Identifying Services that Use Caching

See the Global Environment Variables for services where caching or persistence is used. Search for:

  • OCIS_CACHE_STORE or

  • OCIS_PERSISTENT_STORE.

Common Container Networking Pitfalls

When using a container environment like with docker compose, there are common networking pitfalls. Though these pitfalls are not restricted to caching and persistence and without completeness, here are some examples:

  • When connecting to an internal network, ports to external services should not be exposed for security reasons.

  • Check if the cache/store that is configured both in Infinite Scale and docker is reachable from Infinite Scale like:

    ping redis -> PING redis (172.19.0.2): 56 data bytes 64 bytes from 172.19.0.2: seq=0 ttl=42 time=0.107 ms