Caching and Persistence

Introduction

Caching can not only be a big boost to performance but can also be essential when setting up Infinite Scale via container orchestration. Persistence is recommended for some services, for example, to manage event data for later processing, or, not limited to but especially when the underlying services are scaled via container orchestration providing consistent data across instantiated services.

General Information

Main Drivers

  • Performance
    Caching does not need to be persistent and in case of outages, the cache is rebuilt and no information is lost. This is typically used for caching metadata and filesystem access (stat).

  • Persistence
    With this kind of store, saving information temporarily on a persistent media is key. Persistence stores are used, for example, for generated events returning status info to users when requested. If this data gets lost or is not consistently available over the instantiated services, users will be inconsistently informed about state changes that happened.

  • Scaling
    When services are scaled horizontally (instantiated), a common and persistent base for generated data is necessary.

Identifying Caches by Their Name

  • Performance caches
    Can be identified via environment variables ending with xxx_CACHE_STORE which is valid for both local and global ones.

  • Persistent caches
    Can be identified via environment variables ending with xxx_STORE when using local ones and OCIS_PERSISTENT_STORE(_xxx) when using global ones. Note that no other trailing name components are present when using local ones.

Store Types

  • Memory
    This type has best performance but is not suitable for all requirements as it has no persistence and is limited to the service on the running server. It can be used for deployments where:

    • persistence is not a requirement (issue: restart/reboot) and

    • scaling of services is not required (issue: instantiation).

  • External products
    These products like nats-js-kv or redis-sentinel are independent services not related to Infinite Scale. They are usually installed on separate hardware requiring their own performance, availability and scalability measures. These products are connected via a fast network. Infinite Scale connects to these store types and uses their services like a black box.

Overview of Available Types

The following table gives an overview of store types available in Infinite Scale.

Store Type Description

memory

Basic in-memory store. Will not survive a restart.
Usually the default for caches. See the store environment variable for which one is used.

nats-js-kv

Stores data using key-value-store feature of NATS JetStream.
Usually the default for stores, see the store environment variable for which one is used.

redis-sentinel

Stores data in a configured Redis Sentinel cluster.

noop

Stores nothing. Useful for testing. Not recommended in production environments.

Local vs Global Configuration

  • Performance and persistent types can be configured per service individually or via global environment variables. See the Notes for Environment Variables for details with regards to settings precedence.

    When configuring stat and metadata caches individually for each service and not via the corresponding global environment variable, all caches must have the same configuration (access to the same cache) to provide consistent results.

Identifying Services that Use Caching

See the Global Environment Variables for services where caching or persistence is used. Search for:

  • OCIS_CACHE_STORE or

  • OCIS_PERSISTENT_STORE.

Store Considerations

Some information generated by Infinite Scale is persisted for later use, like notifications or the activity log. This information can be important e.g. to keep track of changes. Because of the importance of this information, its data should be handled with care.

Resilience

When you connect to an external store like NATS, consider designing the store for resilience and high availability. Infinite Scale connects to the store, creates streams for various purposes (services), but is agnostic about the stores backend setup. Read the store providers documentation for more details like NATS Server Clustering or High availability with Redis Sentinel.

Store specific notes

  • When using redis-sentinel:
    The Redis master to use is configured via e.g. OCIS_PERSISTENT_STORE_NODES in the form of <sentinel-host>:<sentinel-port>/<redis-master> like 10.10.0.200:26379/mymaster.

  • When using nats-js-kv:

    • It is recommended to set OCIS_PERSISTENT_STORE_NODES to the same value as OCIS_EVENTS_ENDPOINT. That way the cache uses the same nats instance as the event bus. See the Event Bus Configuration for more details.

    • Authentication can be added, if configured, via OCIS_CACHE_AUTH_USERNAME and OCIS_CACHE_AUTH_PASSWORD.

    • It is possible to set OCIS_CACHE_DISABLE_PERSISTENCE to instruct nats to not persist cache data on disc.

Backup and Restore

When implementing a backup and restore strategy, consider adding the persistant store for the reason mentioned. See the Backup Considerations documentation for more details.

Deployments

See the oCIS with NATS deployment example for details and important notes. Note that the chart example comes as it is and is for informational purposes only.

Common Container Networking Pitfalls

When using a container environment like with docker compose, there are common networking pitfalls. Though these pitfalls are not restricted to caching and persistence and without completeness, here are some examples:

  • When connecting to an internal network, ports to external services should not be exposed for security reasons.

  • Check if the cache/store that is configured both in Infinite Scale and docker is reachable from Infinite Scale like:

    ping redis -> PING redis (172.19.0.2): 56 data bytes 64 bytes from 172.19.0.2: seq=0 ttl=42 time=0.107 ms