Container Orchestration

Introduction

This document provides guidelines for container orchestration tools suitable for Infinite Scale. These are Docker-Compose and Kubernetes. Other container orchestration tools like Docker Swarm can be derived from these.

Container orchestration tools are necessary to meet the requirements described in the Availability and Scalability guide starting at the Container section.

The pages

give a brief overview of features and characteristics of both tools.

For Kubernetes, ownCloud provides basic Helm Charts that can be used and adjusted.

ownCloud highly recommends reading the General Info as it contains valuable information about configuration rules, managing services and default paths - just to mention some of the useful topics.

Docker Compose

Similar to using docker run and handing over command-line parameters for a single container, you can use a docker-compose.yml yaml file which defines all the settings and environment variables for each container in one or more files. This is the next step when having multi-container environments.

  • Consider that when planning to run Infinite Scale via Docker Compose, the degree of freedom is not the same as when using Kubernetes with Helm as you are limited to the same server and other limitations but it usually offers more than running the pure binary when it comes to ease of configuration and maintaining your environment.

  • Use Docker Compose if you aim at an extended degree of freedom compared to the binary installation but don’t need the full capabilities of Kubernetes and Helm.

Prerequisites

Check if the package docker-compose is installed in addition to docker:

which docker-compose

If docker-compose is installed, you’ll be informed. If not, you may get no output at all or a message that it couldn’t be found. In that case you need to install docker-compose first. On most Linux distributions, you can simply use the package manager to do so.

Note that in many cases, this will install a 1.x version (python-based) only supported until June 2023. Version 2.x (go-based) is available from Docker. Follow the Install Docker Compose guide to install a 2.x version. Familiarize yourself with the differences between versions particularly with regard to syntax changes. Most of our examples are still based on Compose-V1. One major difference is, that compose is now a plugin of docker, even if there is still a standalone version. If the check above fails, you might also want to test:

docker compose version

When done, create a project directory like ocis-compose in your home directory to have a common location for your Infinite Scale Compose files.

Docker Compose Examples

  • See the services section for details about available environment variables and yaml files.

  • See below for example service configurations using Docker Compose, to get a first impression of how this can be achieved. Both ocis environment variables and ocis service configuration yaml files are used.

    • 5.0.9

    Docker Compose deployment examples directory

    Using git version name: v5.0.9

    Note that github will not let you download a single directory easily. You can get the examples using the following methods requiring minimum space. For stable releases, check or update the version accordingly like when there is a new patch release available:

    • Using a shallow git clone which minimizes the required clone space like:

      git clone --depth 1 https://github.com/owncloud/ocis.git -b 5.0.9

    • Using an external page providing you the folder to download as zip file like:

      https://download-directory.github.io?url=https://github.com/owncloud/ocis/tree/v5.0.9/deployments/examples

Kubernetes and Helm

The commands and examples are based on software from Kubernetes and Helm. For other software products or environments without claim to completeness like:

  • OpenShift

  • Rancher

  • K3s

  • AWS EKS

  • GCP GKE

  • Azure AKS

you may need to adapt the commands and possibly the provided yaml files. As an example, OpenShift requires, if not otherwise defined, the container user and password ID to be a very high number. Check your software or environment for details and requirements.

Note that this does not affect the supported Kubernetes versions. Any software or environment must match the version standards.

Kubernetes

Kubernetes (abbreviated as K8s) is an open-source platform for governing clusters of containerized application services. Kubernetes automates the vital aspects of container lifecycle management, including scaling, replication, monitoring, and scheduling. It offers a framework for distributed systems. Infinite Scale was designed with Kubernetes in mind. Therefore ownCloud provides Helm charts for a convenient deployment of Infinite Scale on a Kubernetes cluster.

See the Deployment Evolution description in the Availability and Scalability section for reasons to use Kubernetes.

Infinite Scale follows the Twelve-Factor App principles regarding configuration, which means almost every aspect of Infinite Scale is modifiable via environment variables.

When designing your Kubernetes cluster, you have two major approaches available which are minikube and kubeadm.

minikube

minikube lets you run a single-node Kubernetes cluster locally. It is a good way to test a deployment. It requires no extra configuration on any cloud platform as everything runs on your local machine.

kubeadm

kubeadm requires at least two nodes and builds a minimum viable, production-ready Kubernetes cluster, using best practices. It also allows the container runtime to be chosen, though it has Docker by default.

A tool to note: kubectl

kubectl is the command-line tool for Kubernetes. It allows users to run commands against a K8s cluster. It supports multiple contexts for as many clusters as you have access to. minikube also provides kubectl wrapped as minikube kubectcl.

Pods

A Pod is a Kubernetes abstraction that represents a group of one or more application containers such as Docker and some shared resources for those containers.

Helm

Helm is a Kubernetes deployment tool for automating creation, packaging, configuration and deployment of applications and services to Kubernetes clusters. Comparing Kubernetes to the operating system, Helm would be the package manager. Helm automates the maintenance of YAML manifests for Kubernetes objects. This is done by packaging information into charts — therefore Helm Charts — and advertising them to a Kubernetes cluster. The image below shows the interaction of Helm v3 with Kubernetes.

Interaction of Helm v3 with Kubernetes

Prerequisites

Installing Kubernetes

Depending on whether you want to go for a single or multi-node Kubernetes environment, follow the Kubernetes Installation Documentation to do so. This documentation will use minikube in the examples. Verify your installation by typing:

minikube version
Installing Helm

Follow the Helm Installation Documentation post installation and setup of Kubernetes. Verify your installation by typing:

helm version

Using Our Helm Charts with Infinite Scale

The Helm chart is still in an experimental phase and has not yet been published on a Helm chart repository. For your convenience, ownCloud provides an ocis-charts git repository.
ownCloud will publish updated data when new Helm chart releases become available. This information will be available in additional tabs in corresponding sections. Note that two Helm chart stable versions will be documented beside the development version.
When defining your own Helm charts, consider that, if you’re using config overrides in your yaml definitions, a service does not get redefined in the override again. Multiple definitions can cause chart issues that are hard to identify.

The values.yaml file provided by ownCloud uses generic configuration. You can customize this configuration with your own values, for example for different setups or sizings. This should be done by using your own values.yaml file at a different location which will overwrite or add content to the provided one. While not mandatory, the identical file name of values.yaml follows the convention of Helm. When it comes to security sensitive data like secrets, such data is usually not added in the overwrite-values file for security reasons. In such a case, you apply secrets via command from a secrets file.

Supported Infinite Scale Versions

See the following table to match the Helm chart versions with Infinite Scale releases. Note that the chart version matches the tag in the ocis-charts git repository.

Helm Chart Version Works with Infinite Scale Versions

latest

4.0.1

0.5.0

4.0.1

0.4.0

3.0.0

  • If a chart has been superseded by another for the same Infinite Scale release, only the latest one is listed.

  • Note that Helm Chart Version 0.2.0 was a necessary intermediate for Infinite Scale 3.0.0-alpha.1 only and is therefore not listed with a working Infinite Scale version.

Breaking Changes

Select possible breaking changes of a Helm chart version from the tabs. A new document with all details will be opened, directly referring to the selected version, although the document contains information about all published versions.

  • 0.5.0

  • 0.4.0

Breaking Changes

Breaking Changes

Supported Kubernetes Versions

  • latest

  • 0.5.0

  • 0.4.0

We only list non EOL versions as of the chart release date from here: https://kubernetes.io/releases/

Note that some EOL versions still might be API compatible.

Version Covered by tests Tested by developers

1.28

yes

yes

1.27

yes

yes

1.26

yes

no

1.25

yes

no

We only list non EOL versions as of the chart release date from here: https://kubernetes.io/releases/

Note that some EOL versions still might be API compatible.

Version Covered by tests Tested by developers

1.27

yes

yes

1.26

yes

yes

1.25

yes

no

1.24

yes

no

  • Check the supported Kubernetes versions before you download the chart.

    • The ~ represents all patch releases for that particular version.

    • The -0 represents subversions of that particular version.

    Version

    ~1.27.0-0

    ~1.26.0-0

    ~1.25.0-0

    ~1.24.0-0

Get the Chart

As the Helm chart has currently not been published to a Helm repository, you need to clone ownCloud’s Helm chart git repository named ocis-charts.

Start minikube

  1. Start your minikube cluster with the latest supported K8s version:

    minikube start --kubernetes-version=v1.28.1

  2. Enable the minikube ingress plugin, which acts like a reverse proxy for your cluster:

    minikube addons enable ingress

  3. Linux only: enable the ingress-dns plugin and configure it:

    minikube addons enable ingress-dns

  4. Configure the in-cluster DNS server to resolve local DNS names inside the cluster`:

    Note that this step is not optional but mandatory for an Infinite Scale installation. For details see Step 4, (optional) Configure in-cluster DNS server to resolve local DNS names inside cluster.

  5. macOS only: run minikube tunnel to be expose 80,443 ports:

    minikube tunnel

  6. Configure hosts:

    1. On Linux you need to add additional configurations to use ingress by adding the domain names to /etc/hosts. Those entries need to point to the Minikube interface IP which you can get by running minikube ip.

      192.168.49.2 ocis.kube.owncloud.test

    2. On macOS you need to add additional configurations to use ingress by adding the domain names to /etc/hosts. Since you are using minikube tunnel, those entries need to point to 127.0.0.1 because it’s listening on the localhost interface.

      127.0.0.1 ocis.kube.owncloud.test

Deploy the Chart

  • When installing Infinite Scale in Minikube on MacOS, you need to set the hostAliases option:

    hostAliases:
  - ip: "192.168.49.2" # <- needs to be the IP of `minikube ip`
    hostnames:
      - "ocis.kube.owncloud.test"

  • Based on the Kubernetes version, you will find comments in values.yaml where content depends on the Kubernetes version. Search for comments with Kubernetes for details.

  • Deploy the chart with the deployment name ocis, use any name as desired. To do so, run the following command from the root of the cloned repository:

    helm install ocis ./charts/ocis

    • latest

    • 0.5.0

    • 0.4.0

    File Description

    values.yaml

    Helm chart with default configurations.

    Values Description

    Description of the values.yaml file.
    File Description

    values.yaml

    Helm chart with default configurations.

    Values Description

    Description of the values.yaml file.
    File Description

    values.yaml

    Helm chart with default configurations.

    Values Description

    Description of the values.yaml file.

Customize the Generic Setup

In all examples, adapt the settings according your needs.

Set Your Own Default Values

  • Create your own local values.yaml file which will overwrite parts of the provided one with the following content:

    • latest

    • 0.5.0

    • 0.4.0

    externalDomain: ocis.kube.owncloud.test
ingress:
  enabled: true
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 1024m
  tls:
    - hosts:
        - ocis.kube.owncloud.test
insecure:
  # disables ssl certificate checking for connections to the openID connect identity provider.
  # Not recommended for production setups, but we don't have valid certificates in minikube
  oidcIdpInsecure: true
  # disables ssl certificate checking for connections to the oCIS http apis.
  # Not recommended for production setups, but we don't have valid certificates in minikube
  ocisHttpApiInsecure: true
    externalDomain: ocis.kube.owncloud.test
ingress:
  enabled: true
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 1024m
  tls:
    - hosts:
        - ocis.kube.owncloud.test
insecure:
  # disables ssl certificate checking for connections to the openID connect identity provider.
  # Not recommended for production setups, but we don't have valid certificates in minikube
  oidcIdpInsecure: true
  # disables ssl certificate checking for connections to the oCIS http apis.
  # Not recommended for production setups, but we don't have valid certificates in minikube
  ocisHttpApiInsecure: true
    externalDomain: ocis.kube.owncloud.test
ingress:
  enabled: true
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 1024m
  tls:
    - hosts:
        - ocis.kube.owncloud.test
insecure:
  # disables ssl certificate checking for connections to the openID connect identity provider.
  # Not recommended for production setups, but we don't have valid certificates in minikube
  oidcIdpInsecure: true
  # disables ssl certificate checking for connections to the oCIS http apis.
  # Not recommended for production setups, but we don't have valid certificates in minikube
  ocisHttpApiInsecure: true

Enable Metrics with Prometheus

  • In order to scrape oCIS' metrics with Prometheus, you need to set up a ServiceMonitor. In order to apply the ServiceMonitor, you need to have Prometheus' CustomResourceDefinitions available, e.g. by installing the Prometheus Operator.

    • latest

    • 0.5.0

    • 0.4.0

    extraResources:
  - |
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: ocis-metrics
    spec:
      selector:
        matchLabels:
          ocis-metrics: enabled
      endpoints:
        - port: metrics-debug
          interval: 60s
          scrapeTimeout: 30s
          path: /metrics
    extraResources:
  - |
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: ocis-metrics
    spec:
      selector:
        matchLabels:
          ocis-metrics: enabled
      endpoints:
        - port: metrics-debug
          interval: 60s
          scrapeTimeout: 30s
          path: /metrics
    extraResources:
  - |
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: ocis-metrics
    spec:
      selector:
        matchLabels:
          ocis-metrics: enabled
      endpoints:
        - port: metrics-debug
          interval: 60s
          scrapeTimeout: 30s
          path: /metrics

Configure Email Notification

  • If the key features.emailNotifications.enable is set to true, the SMTP email server Secret referenced in secretRefs.notificationsSmtpSecretRef needs to be configured:

    • latest

    • 0.5.0

    • 0.4.0

    ---
apiVersion: v1
kind: Secret
metadata:
  name: notifications-smtp-secret # needs to be set to `secretRefs.notificationsSmtpSecretRef`
type: Opaque
data:
  # Username for authentication against the SMTP host.
  smtp-username: XXXXXXXXXXXXX
  # Password for authentication against the the SMTP host.
  smtp-password: XXXXXXXXXXXXX
    ---
apiVersion: v1
kind: Secret
metadata:
  name: notifications-smtp-secret # needs to be set to `secretRefs.notificationsSmtpSecretRef`
type: Opaque
data:
  # Username for authentication against the SMTP host.
  smtp-username: XXXXXXXXXXXXX
  # Password for authentication against the the SMTP host.
  smtp-password: XXXXXXXXXXXXX
    ---
apiVersion: v1
kind: Secret
metadata:
  name: notifications-smtp-secret # needs to be set to `secretRefs.notificationsSmtpSecretRef`
type: Opaque
data:
  # Username for authentication against the SMTP host.
  smtp-username: XXXXXXXXXXXXX
  # Password for authentication against the the SMTP host.
  smtp-password: XXXXXXXXXXXXX

Configure S3ng Storage

  • If the key services.storageusers.storageBackend.driver is set to s3ng, the S3 access key ID / secret Secret referenced in secretRefs.s3CredentialsSecretRef needs to be configured:

    • latest

    • 0.5.0

    • 0.4.0

    ---
apiVersion: v1
kind: Secret
metadata:
  name: notifications-smtp-secret # default of `secretRefs.s3CredentialsSecretRef`
type: Opaque
data:
  # S3 access key.
  accessKey: XXXXXXXXXXXXX
  # S3 secret key.
  secretKey: XXXXXXXXXXXXX
    ---
apiVersion: v1
kind: Secret
metadata:
  name: notifications-smtp-secret # default of `secretRefs.s3CredentialsSecretRef`
type: Opaque
data:
  # S3 access key.
  accessKey: XXXXXXXXXXXXX
  # S3 secret key.
  secretKey: XXXXXXXXXXXXX
    ---
apiVersion: v1
kind: Secret
metadata:
  name: notifications-smtp-secret # default of `secretRefs.s3CredentialsSecretRef`
type: Opaque
data:
  # S3 access key.
  accessKey: XXXXXXXXXXXXX
  # S3 secret key.
  secretKey: XXXXXXXXXXXXX

Configure Userlog Global Notifications Secret

  • Configure Global Notifications Secrets referenced in secretRefs.globalNotificationsSecretRef if required:

    • latest

    ---
apiVersion: v1
kind: Secret
metadata:
  namespace: ocis
  name: userlog-notifications-secret # default of `secretRefs.globalNotificationsSecretRef`
type: Opaque
data:
  # token
  notifications-secret: XXXXXXXXXXXXX

Define Mandatory Secrets and ConfigMaps

Infinite Scale requires some mandatory Secrets and ConfigMaps to work. They are created one-off if you don’t explicitly provide them. If you’re using the builtin user management, which is not recommended, among the auto generated Secrets, there are also some certificates which expire and need to be renewed manually.

These Secrets and ConfigMaps need to be part of your backup since you need to provide them manually during a disaster recovery procedure.
Mandatory Secrets

  • If you want to manage Secrets on your own, you can look at the following example which shows what mandatory Secrets look like and how they can be generated. The example assumes that the secretRefs are not changed. Each Secret data entry holds a description of how to generate it or find the right value.

    • latest

    • 0.5.0

    • 0.4.0

    File Description

    generic-secrets.yaml

    List of all mandatory Secrets.
    File Description

    generic-secrets.yaml

    List of all mandatory Secrets.
    File Description

    generic-secrets.yaml

    List of all mandatory Secrets.
Apply Mandatory Secrets

Secrets can be applied by command or included in extraResources of your own values.yaml file. Adapt the data content according to your environment:

  1. To apply secrets by command, save the content as mandatory-secrets.yaml and use the following command with a path to the secrets file added if necessary:

    kubectl apply -f mandatory-secrets.yaml

  2. To apply secrets via your own values.yaml, add the content at extraResources. Proper yaml formatting is necessary.

Mandatory ConfigMaps

  • If you want to manage ConfigMaps on your own, you can look at the following example which shows how mandatory ConfigMaps look like and how they can be generated. The example assumes that the configRefs are not changed. Each ConfigMaps data entry holds a description of how to generate it or find the right value.

  • The following example shows what generic configuration need to look like and how they can be generated. The example assumes that the configRefs are not changed . Each config data entry holds a description of how to generate it or find the right value.

    • latest

    • 0.5.0

    • 0.4.0

    File Description

    generic-configs.yaml

    List of all mandatory ConfigMaps.
    File Description

    generic-configs.yaml

    List of all mandatory ConfigMaps.
    File Description

    generic-configs.yaml

    List of all mandatory ConfigMaps.
Apply Mandatory ConfigMaps

ConfigMaps can be applied by command or included in extraResources of your own values.yaml file. Adapt the data content according to your environment:

  1. To apply configs by command, save the content as generic-configs.yaml and use the following command with a path to the secrets file added if necessary:

    kubectl apply -f generic-configs.yaml

  2. To apply configs via your own values.yaml, add the content at extraResources. Proper yaml formatting is necessary.

Built-in User Management Secrets

If you’re using the built-in user management by setting features.externalUserManagement.enabled to false, which is the default, you’ll need additional Secrets. These are also autogenerated for you if you don’t provide them manually.

These Secrets are certificates that expire after 365 days and therefore need a certificate rotation from time to time. Rotation can be achieved by deleting the Secrets ldap-ca and ldap-cert and restarting all Infinite Scale deployments like with kubectl rollout restart deploy.

  • The following example shows what the Secrets for the built-in user management need to look like and how they can be generated. The example assumes that the secretRefs are not changed . Each Secret data entry holds a description of how to generate it or find the right value.

    • latest

    • 0.5.0

    • 0.4.0

    File Description

    builtin-user-mgmt-secrets.yaml

    Secrets file for the builtin user management.
    File Description

    builtin-user-mgmt-secrets.yaml

    Secrets file for the builtin user management.
    File Description

    builtin-user-mgmt-secrets.yaml

    Secrets file for the builtin user management.
Apply Built-in User Management Secrets

Secrets can be applied by command or included in extraResources of your own values.yaml file. Adapt the data content according to your environment:

  1. To apply secrets by command, save the content as builtin-user-mgmt-secrets.yaml and use the following command (adding the path to the secrets file if necessary):

    kubectl apply -f builtin-user-mgmt-secrets.yaml

  2. To apply secrets via your own values.yaml, add the content at extraResources. Proper yaml formatting is necessary.

External User Management Secrets

If you’re using external user management by setting features.externalUserManagement.enabled to true, you need to set these Secrets. Certificates are also required which should expire and therefore need a certificate rotation from time to time, for which we didn’t document appropiate tooling yet.

If you’re using Helm Charts, you are responsible for these user management secrets and their lifecycle. Any information necessary to use this security-relevant data is provided by ownCloud via examples.

  • The following example shows what external user management secrets need to look like and how they can be generated. The example assumes that the secretRefs are not changed . Each secret data entry holds a description of how to generate it or find the right value.

    • latest

    • 0.5.0

    • 0.4.0

    File Description

    external-user-mgmt-secrets.yaml

    Secrets file for the external user management.
    File Description

    external-user-mgmt-secrets.yaml

    Secrets file for the external user management.
    File Description

    external-user-mgmt-secrets.yaml

    Secrets file for the external user management.
Apply External User Management Secrets

Secrets can be applied by command or included in extraResources of your own values.yaml file. Adapt the data content according to your environment:

  1. To apply secrets by command, save the content as external-user-mgmt-secrets.yaml and use the following command (adding a path to the secrets file if necessary):

    kubectl apply -f external-user-mgmt-secrets.yaml

  2. To apply secrets via your own values.yaml, add the content at extraResources. Proper yaml formatting is necessary.

NGINX Ingress Example

This is an example with NGINX ingress and certificates issued by cert-manager. To make this work, you need to have NGINX ingress and cert-manager installed in your cluster.

  • Defining NGINX ingress and cert-manager.

    • latest

    • 0.5.0

    • 0.4.0

    externalDomain: ocis.owncloud.test

ingress:
  enabled: true
  ingressClassName: nginx
  annotations:
    cert-manager.io/issuer: 'ocis-certificate-issuer'
  tls:
    - hosts:
        - ocis.kube.owncloud.test
      secretName: ocis-tls-certificate

extraResources:
  - |
    apiVersion: cert-manager.io/v1
    kind: Issuer
    metadata:
      name: ocis-certificate-issuer
    spec:
      acme:
        server: https://acme-v02.api.letsencrypt.org/directory
        email: test@example.test
        privateKeySecretRef:
          name: ocis-certificate-issuer
        solvers:
        - http01:
            ingress:
              class: nginx
    externalDomain: ocis.owncloud.test

ingress:
  enabled: true
  ingressClassName: nginx
  annotations:
    cert-manager.io/issuer: 'ocis-certificate-issuer'
  tls:
    - hosts:
        - ocis.kube.owncloud.test
      secretName: ocis-tls-certificate

extraResources:
  - |
    apiVersion: cert-manager.io/v1
    kind: Issuer
    metadata:
      name: ocis-certificate-issuer
    spec:
      acme:
        server: https://acme-v02.api.letsencrypt.org/directory
        email: test@example.test
        privateKeySecretRef:
          name: ocis-certificate-issuer
        solvers:
        - http01:
            ingress:
                class: nginx
    externalDomain: ocis.owncloud.test

ingress:
  enabled: true
  ingressClassName: nginx
  annotations:
    cert-manager.io/issuer: 'ocis-certificate-issuer'
  tls:
    - hosts:
        - ocis.kube.owncloud.test
      secretName: ocis-tls-certificate

extraResources:
  - |
    apiVersion: cert-manager.io/v1
    kind: Issuer
    metadata:
      name: ocis-certificate-issuer
    spec:
      acme:
        server: https://acme-v02.api.letsencrypt.org/directory
        email: test@example.test
        privateKeySecretRef:
          name: ocis-certificate-issuer
        solvers:
        - http01:
            ingress:
              class: nginx

Apply Chart Changes

  • Apply all changes defined in your own values.yaml file:

    helm upgrade --install --reset-values \
    ocis ./charts/ocis --values values.yaml

  • Ensure that all the pods are running:

    kubectl get pods

Access Infinite Scale in Your Browser

After you have customized your setup, use the following URL to access Infinite Scale with your browser:

https://ocis.kube.owncloud.test

Uninstalling the Chart

To uninstall/delete the ocis deployment, use the following command:

helm delete ocis

This command removes all the Kubernetes components associated with the chart and deletes the deployment.