Container Orchestration

Introduction

This document provides guidelines for container orchestration tools suitable for Infinite Scale. These are Docker-Compose and Kubernetes. Other container orchestration tools like Docker Swarm can be derived from these.

Container orchestration tools are necessary to meet the requirements described in the Availability and Scalability guide starting at the Container section.

The pages

give a brief overview of features and characteristics of both tools.

For Kubernetes, ownCloud provides basic Helm Charts that can be used and adjusted.

ownCloud highly recommends reading the General Info as it contains valuable information about configuration rules, managing services and default paths - just to mention some of the useful topics.

Docker Compose

Similar to using docker run and handing over command-line parameters for a single container, you can use a docker-compose.yml yaml file which defines all the settings and environment variables for each container in one or more files. This is the next step when having multi-container environments.

  • Consider that when planning to run Infinite Scale via Docker Compose, the degree of freedom is not the same as when using Kubernetes with Helm as you are limited to the same server and other limitations but it usually offers more than running the pure binary when it comes to ease of configuration and maintaining your environment.

  • Use Docker Compose if you aim at an extended degree of freedom compared to the binary installation but don’t need the full capabilities of Kubernetes and Helm.

Prerequisites

Check if the package docker-compose is installed in addition to docker:

which docker-compose

If docker-compose is installed, you’ll be informed. If not, you may get no output at all or a message that it couldn’t be found. In that case you need to install docker-compose first. On most Linux distributions, you can simply use the package manager to do so.

Note that in many cases, this will install a 1.x version (python-based) only supported until June 2023. Version 2.x (go-based) is available from Docker. Follow the Install Docker Compose guide to install a 2.x version. Familiarize yourself with the differences between versions particularly with regard to syntax changes. Most of our examples are still based on Compose-V1.

When done, create a project directory like ocis-compose in your home directory to have a common location for your Infinite Scale Compose files.

Docker Compose Examples

  • See the services section for details about available environment variables and yaml files.

  • See below for example service configurations using Docker Compose, to get a first impression of how this can be achieved. Both ocis environment variables and ocis service configuration yaml files are used.

    • latest

    • 2.0.0

    Docker Compose deployment examples directory

    Using git version name: master

    Docker Compose deployment examples directory

    Using git version name: v2.0.0

    Note that github will not let you download a single directory easily. You can get the examples using the following methods requiring minimum space, replace the version accordingly:

    • Using a shallow git clone which minimizes the required clone space like:

      git clone --depth 1 https://github.com/owncloud/ocis.git -b v2.0.0

    • Using an external page providing you the folder to download as zip file like:

      https://download-directory.github.io?url=https://github.com/owncloud/ocis/tree/v2.0.0/deployments/examples

Kubernetes and Helm

The commands and examples are based on software from Kubernetes and Helm. For other software products or environments without claim to completeness like:

  • OpenShift

  • Rancher

  • K3s

  • AWS EKS

  • GCP GKE

  • Azure AKS

you may need to adapt the commands and possibly the provided yaml files. As an example, OpenShift requires, if not otherwise defined, the container user and password ID to be a very high number. Check your software or environment for details and requirements.

Note that this does not affect the supported Kubernetes versions. Any software or environment must match the version standards.

Kubernetes

Kubernetes (abbreviated as K8s) is an open-source platform for governing clusters of containerized application services. Kubernetes automates the vital aspects of container lifecycle management, including scaling, replication, monitoring, and scheduling. It offers a framework for distributed systems. Infinite Scale was designed with Kubernetes in mind. Therefore ownCloud provides Helm charts for a convenient deployment of Infinite Scale on a Kubernetes cluster.

See the Deployment Evolution description in the Availability and Scalability section for reasons to use Kubernetes.

Infinite Scale follows the Twelve-Factor App principles regarding configuration, which means almost every aspect of Infinite Scale is modifiable via environment variables.

When designing your Kubernetes cluster, you have two major approaches available which are minikube and kubeadm.

minikube

minikube lets you run a single-node Kubernetes cluster locally. It is a good way to test a deployment. It requires no extra configuration on any cloud platform as everything runs on your local machine.

kubeadm

kubeadm requires at least two nodes and builds a minimum viable, production-ready Kubernetes cluster, using best practices. It also allows the container runtime to be chosen, though it has Docker by default.

A tool to note: kubectl

kubectl is the command-line tool for Kubernetes. It allows users to run commands against a K8s cluster. It supports multiple contexts for as many clusters as you have access to. minikube also provides kubectl wrapped as minikube kubectcl.

Pods

A Pod is a Kubernetes abstraction that represents a group of one or more application containers such as Docker and some shared resources for those containers.

Helm

Helm is a Kubernetes deployment tool for automating creation, packaging, configuration and deployment of applications and services to Kubernetes clusters. Comparing Kubernetes to the operating system, Helm would be the package manager. Helm automates the maintenance of YAML manifests for Kubernetes objects. This is done by packaging information into charts — therefore Helm Charts — and advertising them to a Kubernetes cluster. The image below shows the interaction of Helm v3 with Kubernetes.

Interaction of Helm v3 with Kubernetes

Prerequisites

Installing Kubernetes

Depending on whether you want to go for a single or multi-node Kubernetes environment, follow the Kubernetes Installation Documentation to do so. This documentation will use minikube in the examples. Verify your installation by typing:

minikube version
Installing Helm

Follow the Helm Installation Documentation post installation and setup of Kubernetes. Verify your installation by typing:

helm version

Using Helm Charts with Infinite Scale

The Helm chart is still in an experimental phase and has not yet been published on a Helm chart repository. For your convenience, ownCloud provides an ocis-charts git repository.
ownCloud will publish updated data when new Helm chart releases become available. This information will be available in additional tabs in corresponding sections. Note that two Helm chart stable versions will be documented beside the development version.
When defining your own Helm charts, consider that, if you’re using config overrides in your yaml definitions, a service does not get redefined in the override again. Multiple definitions can cause chart issues that are hard to identify.

The values.yaml file provided by ownCloud uses generic configuration. You can customize this configuration with your own values, for example for different setups or sizings. This should be done by using your own values.yaml file at a different location which will overwrite or add content to the provided one. While not mandatory, the identical file name of values.yaml follows the convention of Helm. When it comes to security sensitive data like secrets, such data is usually not added in the overwrite-values file for security reasons. In such a case, you apply secrets via command from a secrets file.

Supported Infinite Scale Versions

See the following table to match the Helm chart versions with Infinite Scale releases. Note that the chart version matches the tag in the ocis-charts git repository.

Helm Chart Version Works with Infinite Scale Versions

latest

3.0.0-rc.4

0.2.0

3.0.0-alpha.1

0.1.0

2.0.0

Breaking Changes

Breaking changes are only published for Helm Chart versions that have been released. Select from the tabs possible breaking changes between Helm Chart versions. A new document containing all the details will open.

  • 0.2.0

  • 0.1.0

Breaking Changes

Breaking Changes

Get the Chart

As the Helm chart has currently not been published to a Helm repository, you need to clone ownCloud’s Helm chart git repository named ocis-charts.

  • Check the supported Kubernetes versions before you download the chart.

    • The ~ represents all patch releases for that particular version.

    • The -0 represents subversions of that particular version.

    • latest

    • 0.2.0

    • 0.1.0

    Supported Kubernetes versions

    Version

    ~1.27.0-0

    ~1.26.0-0

    ~1.25.0-0

    ~1.24.0-0

    Supported Kubernetes versions

    Version

    ~1.26.0-0

    ~1.25.0-0

    ~1.24.0-0

    Supported Kubernetes versions

    Version

    ~1.25.0

    ~1.24.0

    ~1.23.0

Start minikube

  1. Start your minikube cluster with the latest supported K8s version:

    minikube start --kubernetes-version=v1.25.0

  2. Enable the minikube ingress plugin, which acts like a reverse proxy for your cluster:

    minikube addons enable ingress

Deploy the Chart

  • Based on the Kubernetes version, you will find comments in values.yaml where content is Kubernetes version dependent. Search for comments with Kubernetes for details.

  • Deploy the chart with the deployment name ocis, use any name as desired. To do so, run the following command from the root of the cloned repository:

    helm install ocis ./charts/ocis

    • latest

    • 0.2.0

    • 0.1.0

    File Description

    values.yaml

    Helm chart with default configurations.

    Values Description

    Description of the values.yaml file.
    File Description

    values.yaml

    Helm chart with default configurations.

    Values Description

    Description of the values.yaml file.
    File Description

    values.yaml

    Helm chart with default configurations.

    Values Description

    Description of the values.yaml file.

Customize the Generic Setup

In all examples, adapt the settings according your needs.

Set Your Own Default Values

  • Create your own local values.yaml file which will overwrite parts of the provided one with the following content:

    • latest

    • 0.2.0

    • 0.1.0

    externalDomain: ocis.kube.owncloud.test
ingress:
  enabled: true
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 1024m
  tls:
    - hosts:
        - ocis.kube.owncloud.test
insecure:
  # disables ssl certificate checking for connections to the openID connect identity provider.
  # Not recommended for production setups, but we don't have valid certificates in minikube
  oidcIdpInsecure: true
  # disables ssl certificate checking for connections to the oCIS http apis.
  # Not recommended for production setups, but we don't have valid certificates in minikube
  ocisHttpApiInsecure: true
    externalDomain: ocis.kube.owncloud.test
ingress:
  enabled: true
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 1024m
  tls:
    - hosts:
        - ocis.kube.owncloud.test
insecure:
  # disables ssl certificate checking for connections to the openID connect identity provider.
  # Not recommended for production setups, but we don't have valid certificates in minikube
  oidcIdpInsecure: true
  # disables ssl certificate checking for connections to the oCIS http apis.
  # Not recommended for production setups, but we don't have valid certificates in minikube
  ocisHttpApiInsecure: true
    externalDomain: ocis.kube.owncloud.test
ingress:
  enabled: true
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 1024m
  tls:
    - hosts:
        - ocis.kube.owncloud.test
insecure:
  # disables ssl certificate checking for connections to the openID connect identity provider.
  # Not recommended for production setups, but we don't have valid certificates in minikube
  oidcIdpInsecure: true
  # disables ssl certificate checking for connections to the oCIS http apis.
  # Not recommended for production setups, but we don't have valid certificates in minikube
  ocisHttpApiInsecure: true

Enable Metrics with Prometheus

  • In order to scrape oCIS' metrics with Prometheus, you need to set up a ServiceMonitor. In order to apply the ServiceMonitor, you need to have Prometheus' CustomResourceDefinitions available, e.g. by installing the Prometheus Operator.

    • latest

    • 0.2.0

    • 0.1.0

    extraResources:
  - |
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: ocis-metrics
    spec:
      selector:
        matchLabels:
          ocis-metrics: enabled
      endpoints:
        - port: metrics-debug
          interval: 60s
          scrapeTimeout: 30s
          path: /metrics
    extraResources:
  - |
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: ocis-metrics
    spec:
      selector:
        matchLabels:
          ocis-metrics: enabled
      endpoints:
        - port: metrics-debug
          interval: 60s
          scrapeTimeout: 30s
          path: /metrics
    extraResources:
  - |
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: ocis-metrics
    spec:
      selector:
        matchLabels:
          ocis-metrics: enabled
      endpoints:
        - port: metrics-debug
          interval: 60s
          scrapeTimeout: 30s
          path: /metrics

Configure Email Notification

  • If the key features.emailNotifications.enable is set to true, the SMTP email server secret needs to be configured:

    • latest

    • 0.2.0

    • 0.1.0

    ---
apiVersion: v1
kind: Secret
metadata:
  name: notifications-smtp-secret
type: Opaque
data:
  # Username for authentication against the SMTP host.
  smtp-username: XXXXXXXXXXXXX
  # Password for authentication against the the SMTP host.
  smtp-password: XXXXXXXXXXXXX
    ---
apiVersion: v1
kind: Secret
metadata:
  name: notifications-smtp-secret
type: Opaque
data:
  # Username for authentication against the SMTP host.
  smtp-username: XXXXXXXXXXXXX
  # Password for authentication against the the SMTP host.
  smtp-password: XXXXXXXXXXXXX
    ---
apiVersion: v1
kind: Secret
metadata:
  name: notifications-smtp-secret
type: Opaque
data:
  # Username for authentication against the SMTP host.
  smtp-username: XXXXXXXXXXXXX
  # Password for authentication against the the SMTP host.
  smtp-password: XXXXXXXXXXXXX

Define Generic Secrets

Infinite Scale requires some generic secrets to work. It was decided to not create them automatically, because we didn’t find a satisfactory way for one-off generation of secrets yet. Certificates are also required which should expire and therefore need a certificate rotation from time to time, for which we didn’t document appropiate tooling yet.

For these reasons, ownCloud cannot take responsibility for these generic secrets and their lifecycle. Any information necessary to use this security-relevant data is provided by ownCloud via examples.

  • The following example shows what generic secrets need to look like and how they can be generated. The example assumes that the secretRefs are not changed . Each secret data entry holds a description of how to generate it or find the right value.

    • latest

    • 0.2.0

    • 0.1.0

    File Description

    generic-secrets.yaml

    Secrets file containing all relevant generic secrets configurations.
    File Description

    generic-secrets.yaml

    Secrets file containing all relevant generic secrets configurations.
    File Description

    generic-secrets.yaml

    Secrets file containing all relevant generic secrets configurations.
Apply Generic Secrets

Secrets can be applied by command or included in extraResources of your own values.yaml file. Adapt the data content according to your environment:

  1. To apply secrets by command, save the content as generic-secrets.yaml and use the following command with a path to the secrets file added if necessary:

    minikube kubectl -- apply -f generic-secrets.yaml

  2. To apply secrets via your own values.yaml, add the content at extraResources. Proper yaml formatting is necessary.

Define Generic Configs

Infinite Scale requires some generic configuration to work. It was decided to not create them automatically, because we didn’t find a satisfactory way for one-off generation of secrets yet.

For these reasons, ownCloud cannot take responsibility for these generic configuration. Any information necessary to use this application-relevant data is provided by ownCloud via examples.

  • The following example shows what generic configuration need to look like and how they can be generated. The example assumes that the configRefs are not changed . Each config data entry holds a description of how to generate it or find the right value.

    • latest

    • 0.2.0

    • 0.1.0

    File Description

    generic-configs.yaml

    Configuration file containing all relevant generic configurations.
    File Description

    generic-configs.yaml

    Configuration file containing all relevant generic configurations.
    File Description

    generic-configs.yaml

    Configuration file containing all relevant generic configurations.
Apply Generic Configs

Configs can be applied by command or included in extraResources of your own values.yaml file. Adapt the data content according to your environment:

  1. To apply configs by command, save the content as generic-configs.yaml and use the following command with a path to the secrets file added if necessary:

    minikube kubectl -- apply -f generic-configs.yaml

  2. To apply configs via your own values.yaml, add the content at extraResources. Proper yaml formatting is necessary.

Built-in User Management Secrets

If you’re using the built-in user management by setting features.externalUserManagement.enabled to false, you need to set these secrets. Certificates are also required which should expire and therefore need a certificate rotation from time to time, for which we didn’t document appropiate tooling yet.

As the operator of Helm Charts, you are responsible for these user management secrets and their lifecycle. Any information necessary to use this security-relevant data is provided by ownCloud via examples.

  • The following example shows what user management secrets need to look like and how they can be generated. The example assumes that the secretRefs are not changed . Each secret data entry holds a description of how to generate it or find the right value.

    • latest

    • 0.2.0

    • 0.1.0

    File Description

    builtin-user-mgmt-secrets.yaml

    Secrets file for the builtin user management.
    File Description

    builtin-user-mgmt-secrets.yaml

    Secrets file for the builtin user management.
    File Description

    builtin-user-mgmt-secrets.yaml

    Secrets file for the builtin user management.
Apply Built-in User Management Secrets

Secrets can be applied by command or included in extraResources of your own values.yaml file. Adapt the data content according to your environment:

  1. To apply secrets by command, save the content as builtin-user-mgmt-secrets.yaml and use the following command (adding the path to the secrets file if necessary):

    minikube kubectl -- apply -f builtin-user-mgmt-secrets.yaml

  2. To apply secrets via your own values.yaml, add the content at extraResources. Proper yaml formatting is necessary.

External User Management Secrets

If you’re using external user management by setting features.externalUserManagement.enabled to true, you need to set these secrets. Certificates are also required which should expire and therefore need a certificate rotation from time to time, for which we didn’t document appropiate tooling yet.

If you’re using Helm Charts, you are responsible for these user management secrets and their lifecycle. Any information necessary to use this security-relevant data is provided by ownCloud via examples.

  • The following example shows what external user management secrets need to look like and how they can be generated. The example assumes that the secretRefs are not changed . Each secret data entry holds a description of how to generate it or find the right value.

    • latest

    • 0.2.0

    • 0.1.0

    File Description

    external-user-mgmt-secrets.yaml

    Secrets file for the external user management.
    File Description

    external-user-mgmt-secrets.yaml

    Secrets file for the external user management.
    File Description

    external-user-mgmt-secrets.yaml

    Secrets file for the external user management.
Apply External User Management Secrets

Secrets can be applied by command or included in extraResources of your own values.yaml file. Adapt the data content according to your environment:

  1. To apply secrets by command, save the content as external-user-mgmt-secrets.yaml and use the following command (adding a path to the secrets file if necessary):

    minikube kubectl -- apply -f external-user-mgmt-secrets.yaml

  2. To apply secrets via your own values.yaml, add the content at extraResources. Proper yaml formatting is necessary.

NGINX Ingress Example

This is an example with NGINX ingress and certificates issued by cert-manager. To make this work, you need to have NGINX ingress and cert-manager installed in your cluster.

  • Defining NGINX ingress and cert-manager.

    • latest

    • 0.2.0

    • 0.1.0

    externalDomain: ocis.owncloud.test

ingress:
  enabled: true
  ingressClassName: nginx
  annotations:
    cert-manager.io/issuer: 'ocis-certificate-issuer'
  tls:
    - hosts:
        - ocis.kube.owncloud.test
      secretName: ocis-tls-certificate

extraResources:
  - |
    apiVersion: cert-manager.io/v1
    kind: Issuer
    metadata:
      name: ocis-certificate-issuer
    spec:
      acme:
        server: https://acme-v02.api.letsencrypt.org/directory
        email: test@example.test
        privateKeySecretRef:
          name: ocis-certificate-issuer
        solvers:
        - http01:
            ingress:
              class: nginx
    externalDomain: ocis.owncloud.test

ingress:
  enabled: true
  ingressClassName: nginx
  annotations:
    cert-manager.io/issuer: 'ocis-certificate-issuer'
  tls:
    - hosts:
        - ocis.kube.owncloud.test
      secretName: ocis-tls-certificate

extraResources:
  - |
    apiVersion: cert-manager.io/v1
    kind: Issuer
    metadata:
      name: ocis-certificate-issuer
    spec:
      acme:
        server: https://acme-v02.api.letsencrypt.org/directory
        email: test@example.test
        privateKeySecretRef:
          name: ocis-certificate-issuer
        solvers:
        - http01:
            ingress:
                class: nginx
    externalDomain: ocis.owncloud.test

ingress:
  enabled: true
  ingressClassName: nginx
  annotations:
    cert-manager.io/issuer: 'ocis-certificate-issuer'
  tls:
    - hosts:
        - ocis.kube.owncloud.test
      secretName: ocis-tls-certificate

extraResources:
  - |
    apiVersion: cert-manager.io/v1
    kind: Issuer
    metadata:
      name: ocis-certificate-issuer
    spec:
      acme:
        server: https://acme-v02.api.letsencrypt.org/directory
        email: test@example.test
        privateKeySecretRef:
          name: ocis-certificate-issuer
        solvers:
        - http01:
            ingress:
              class: nginx

Apply Chart Changes

  • Apply all changes defined in your own values.yaml file:

    helm upgrade --install --reset-values \
    ocis ./charts/ocis --values values.yaml

  • Ensure that all the pods are running:

    minikube kubectl -- get pods

Define Chart Access

Linux

  1. Get and check the minikube IP address:

    minikube ip

  2. If not already done, add the minikube IP address to your /etc/hosts file or use the Ingress DNS add-on. An example line for the /etc/hosts file could look like this:

    192.168.49.2 ocis.kube.owncloud.test
macOS
The following procedure exposes the macOS system via the necessary configured tunnel to the local network. This setup is therefore intended for development purposes only!

  1. On macOS, minikube requires a tunnel to be started to get access to the Ingress of the cluster:

    minikube tunnel --bind-address=192.168.179.3

    The IP address to be set, which should normally be the one of the en0 interface, can be fetched with the ifconfig command. Note that the interface must be reachable from within Kubernetes and 127.0.0.1 is not an option in this case. This is because the OpenID process tries to look up the host name of the cluster via the /etc/hosts file which is described in the next step. Within a pod 127.0.0.1 will not resolve to the host serving Kubernetes.

  2. Add the IP address provided by the above command to your /etc/hosts file. An example could look like this:

    192.168.179.3 ocis.kube.owncloud.test

Access Infinite Scale in Your Browser

After you have customized your setup, use the following URL to access Infinite Scale with your browser:

https://ocis.kube.owncloud.test

Uninstalling the Chart

To uninstall/delete the ocis deployment, use the following command:

helm delete ocis

This command removes all the Kubernetes components associated with the chart and deletes the deployment.

Upgrading an Existing Release to a New Major Version

A major chart version change (like v1.2.3 → v2.0.0) indicates that there is an incompatible breaking change which needs manual actions.