Container Orchestration

Introduction

In this section, we provide guidelines for container orchestration tools suitable for Infinite Scale like Docker-Compose and Kubernetes. Other container orchestration tools like Docker Swarm can be derived from these. Container orchestration tools are necessary to meet the requirements described in the Availability and Scalability guide starting at the Container section.

The pages

give a brief overview of features and characteristics of both tools.

For Kubernetes, ownCloud provides basic Helm Charts that can be used and adjusted.

Docker Compose

Similarly, when using docker run and handing over command-line parameters for a single container, you can define a docker-compose.yml yaml file which defines all the settings and environment variables for each container in one file. This is the next step when having multi-container environments.

Prerequisites

Check if the package docker-compose is installed in addition to docker:

which docker-compose

If docker-compose is installed, you’ll be informed. If not, you may get no output at all or a message that it couldn’t be found. In that case you need to install docker-compose first. On most Linux distributions, you can simply use the package manager to do so. Note that in many cases, this will install a 1.x version (python-based) while a 2.x version (go-based) is preferred. To check which version would be installed with the package manager, simply type:

sudo apt-cache policy docker-compose

If the output shows a version like 2.x, you can use the package manager to install docker-compose, like with the following command on Debian-based distributions:

sudo apt install docker-compose

If the output shows a version like 1.25.0-1, follow the Install Docker Compose guide to install a 2.x version.

When done, create a project directory, like ocis-compose in your home directory to have a common location for your Infinite Scale compose files.

Docker Compose Example

See the ocis individual services example which configures services individually using docker-compose, to get a first impression how this can be done. Both, ocis environment variables and ocis service configuration yaml files are used. See the services section for details of available environment variables and yaml files.

Kubernetes and Helm

Kubernetes

Kubernetes (abbreviated as K8s) is an open-source platform for governing clusters of containerized application services. Kubernetes automates the vital aspects of container lifecycle management, including scaling, replication, monitoring, and scheduling. It offers a framework for distributed systems. Infinite Scale was designed with Kubernetes in mind. Therefore ownCloud provides Helm charts for a convenient deployment of Infinite Scale on a Kubernetes cluster.

See the Deployment Evolution description in the Availability and Scalability section for reasons to use Kubernetes.

Infinite Scale follows the Twelve-Factor App principles regarding configuration, which means almost every aspect of Infinite Scale is modifiable via environment variables.

When designing your Kubernetes cluster, you have two major approaches available which are minikube and kubeadm.

minikube

minikube lets you run a single-node Kubernetes cluster locally. It is a good way to test a deployment. It requires no extra configuration on any cloud platform as everything runs on your local machine.

kubeadm

kubeadm requires at least two nodes and builds a minimum viable, production-ready Kubernetes cluster, using best practices. It also allows the container runtime to be chosen, though it has Docker by default.

A tool to note: kubectl

kubectl is the command-line tool for Kubernetes. It allows users to run commands against a K8s cluster. It supports multiple contexts for as many clusters as you have access to. minikube also provides kubectl wrapped as minikube kubectcl.

Pods

A Pod is a Kubernetes abstraction that represents a group of one or more application containers such as Docker and some shared resources for those containers.

Helm

Helm is a Kubernetes deployment tool for automating creation, packaging, configuration and deployment of applications and services to Kubernetes clusters. Comparing Kubernetes to the operating system, Helm would be the package manager. Helm automates the maintenance of YAML manifests for Kubernetes objects. This is done by packaging information into charts — therefore Helm Charts — and advertising them to a Kubernetes cluster. The image below shows the interaction of Helm v3 with Kubernetes.

Interaction of Helm v3 with Kubernetes

Prerequisites

Installing Kubernetes

Depending on whether you want to go for a single or multi-node Kubernetes enviroment, follow the Kubernetes Installation Documentation to do so. This documentation will use minikube in the examples.

Installing Helm

Follow the Helm Installing Documentation post installation and setup of Kubernetes. Verify your installation is correct by typing:

minikube status
minikube
type: Control Plane
host: Stopped
kubelet: Stopped
apiserver: Stopped
kubeconfig: Stopped

Using Helm Charts with Infinite Scale

The Helm chart is still in an experimental phase and has not yet been published on a Helm chart repository. For your convenience, ownCloud provides an ocis-charts git repository.
ownCloud will publish updated data when new Helm chart releases become available. This information will be available in additional tabs in corresponding sections. Note that two Helm chart stable versions will be documented beside the development version.

The values.yaml file provided by ownCloud uses generic configuration. You can customize this configuration with your own values, for example for different setups or sizings. This should be done by using your own values.yaml file at a different location which will overwrite or add content to the provided one. While not mandatory, the identical file name of values.yaml follows the convention of Helm. When it comes to security sensitive data like secrets, such data is usually not added in the overwrite-values file for security reasons. In such a case, you apply secrets via command from a secrets file.

Supported Infinite Scale Versions

See the following table to match the Helm chart versions with Infinite Scale releases. Note that the chart version matches the tag in the ocis-charts git repository.

Helm Chart Version Works with Ininite Scale Versions

0.0.0

2.0.0-beta.8

Get the Chart

As the Helm chart has currently not been published to a Helm repository, you need to clone ownCloud’s Helm chart git repository named ocis-charts.

  • Check the supported Kubernetes versions before you download the chart.

    The ~ represents all patch releases for that particular version:

    • 0.0.0

    Supported Kubernetes versions

    Version

    ~1.20.0

    ~1.21.0

    ~1.22.0

    ~1.23.0

    ~1.24.0

Deploy the Chart

  • Deploy the chart with the deployment name ocis, use any name as desired. To do so, run the following command from the root of the cloned repository:

    helm install ocis ./charts/ocis
    • 0.0.0

    File Descripton

    values.yaml

    Helm chart with default configurations.

    Values Description

    Description of the values.yaml file.

Start minikube

  1. Start your minikube cluster:

    minikube start
  2. Enable the minikube ingress plugin, which acts like a reverse proxy for your cluster:

    minikube addons enable ingress

Customize the Generic Setup

In all examples, adapt the settings according your needs.

Set Your Own Default Values

  • Create your own local values.yaml file which will overwrite parts of the provided one with the following content:

    • 0.0.0

    externalDomain: ocis.kube.owncloud.test
    ingress:
      enabled: true
      annotations:
        nginx.ingress.kubernetes.io/proxy-body-size: 1024m
      tls:
      - hosts:
        - ocis.kube.owncloud.test
    insecure:
      # disables ssl certificate checking for connections to the openID connect identity provider. Not recommended for production setups, but we don't have valid certificates in minikube
      oidcIdpInsecure: true
      # disables ssl certificate checking for connections to the oCIS http apis. Not recommended for production setups, but we don't have valid certificates in minikube
      ocisHttpApiInsecure: true

Configure Email Notification

  • If the key features.emailNotifications.enable is set to true, the SMTP email server secret needs to be configured:

    • 0.0.0

    ---
    apiVersion: v1
    kind: Secret
    metadata:
      name: notifications-smtp-secret
    type: Opaque
    data:
      # Username for authentication against the SMTP host.
      smtp-username: XXXXXXXXXXXXX
      # Password for authentication against the the SMTP host.
      smtp-password: XXXXXXXXXXXXX

Define Generic Secrets

Infinite Scale requires some generic secrets to work. It was decided not to create them automatically, because Helm does not support one-off generation of secrets out of the box. Certificates are also required which should expire and therefore need a certificate rotation from time to time, which is also not supported by Helm.

For these reasons, ownCloud cannot take responsibility for these generic secrets and their lifecycle. Any information necessary to use this security-relevant data is provided by ownCloud via examples.

  • The following example shows what generic secrets need to look like and how they can be generated. The example assumes that the secretRefs are not changed . Each secret data entry holds a description on how to generate it or find the right value.

    • 0.0.0

    File Descripton

    generic-secrets.yaml

    Secrets file containing all relevant generic secrets configurations.

Apply Generic Secrets

Secrets can be applied by command or included in extraResources of your own values.yaml file. Adapt the data content according to your environment:

  1. To apply secrets by command, save the content as generic-secrets.yaml and use the following command with a path to the secrets file added if necessary:

    minikube kubectl apply -f generic-secrets.yaml
  2. To apply secrets via your own values.yaml, add the content at extraResources. Proper yaml formatting is necessary.

Built-in User Management Secrets

If you’re using the built-in user management by setting features.externalUserManagement.enabled to false, you need to set these secrets. Certificates are also required which should have an expiration date and therefore need a certificate rotation from time to time, which is also not supported by Helm.

As the operator of Helm Charts, you are responsible for these user management secrets and their lifecycle. Any information necessary to use this security-relevant data is provided by ownCloud via examples.

  • The following example shows what user management secrets need to look like and how they can be generated. The example assumes that the secretRefs are not changed . Each secret data entry holds a description on how to generate it or find the right value.

    • 0.0.0

    File Descripton

    builtin-user-mgmt-secrets.yaml

    Secrets file for the builtin user management.

Apply Built-in User Management Secrets

Secrets can be applied by command or included in extraResources of your own values.yaml file. Adapt the data content according to your environment:

  1. To apply secrets by command, save the content as builtin-user-mgmt-secrets.yaml and use the following command (adding the path to the secrets file if necessary):

    minikube kubectl apply -f builtin-user-mgmt-secrets.yaml
  2. To apply secrets via your own values.yaml, add the content at extraResources. Proper yaml formatting is necessary.

External User Management Secrets

If you’re using external user management by setting features.externalUserManagement.enabled to true, you need to set these secrets. Certificates are also required which should expire and therefore need a certificate rotation from time to time, which is also not supported by Helm.

If you’re using Helm Charts, you are responsible for these user management secrets and their lifecycle. Any information necessary to use this security-relevant data is provided by ownCloud via examples.

  • The following example shows what external user management secrets need to look like and how they can be generated. The example assumes that the secretRefs are not changed . Each secret data entry holds a description on how to generate it or find the right value.

    • 0.0.0

    File Descripton

    external-user-mgmt-secrets.yaml

    Secrets file for the external user management.

Apply External User Management Secrets

Secrets can be applied by command or included in extraResources of your own values.yaml file. Adapt the data content according to your environment:

  1. To apply secrets by command, save the content as external-user-mgmt-secrets.yaml and use the following command (adding a path to the secrets file if necessary):

    minikube kubectl apply -f external-user-mgmt-secrets.yaml
  2. To apply secrets via your own values.yaml, add the content at extraResources. Proper yaml formatting is necessary.

NGINX Ingress Example

This is an example with NGINX ingress and certificate issued by cert-manager. To make this work, you need to have NGINX ingress and cert-manager installed in your cluster.

  • Defining NGINX ingress and cert-manager.

    • 0.0.0

    externalDomain: ocis.owncloud.test
    
      ingress:
        enabled: true
        ingressClassName: nginx
        annotations:
          cert-manager.io/issuer: "ocis-certificate-issuer"
        tls:
          - hosts:
            - ocis.owncloud.test
            secretName: ocis-tls-certificate
    
      extraResources:
        - |
          apiVersion: cert-manager.io/v1
          kind: Issuer
          metadata:
            name: ocis-certificate-issuer
            namespace: ocis-namespace
          spec:
            acme:
              server: https://acme-v02.api.letsencrypt.org/directory
              email: test@example.test
              privateKeySecretRef:
                name: ocis-certificate-issuer
              solvers:
              - http01:
                  ingress:
                    class: nginx

Apply Chart Changes

  • Install all changes defined in your own values.yaml file:

    helm install ocis ./charts/ocis --values values.yaml

Define Chart Access

  1. Get and check the minikube IP address:

    minikube ip
  2. If not already done, add the minikube IP address to your /etc/hosts file or use the Ingress DNS add-on. An example line for the /etc/hosts file could look like this:

    192.168.49.2 ocis.kube.owncloud.test

Access Infinite Scale in Your Browser

After you have customized your setup, use the following URL to access Infinite Scale with your browser:

https://ocis.kube.owncloud.test

Uninstalling the Chart

To uninstall/delete the ocis deployment, use the following command:

helm delete ocis

This command removes all the Kubernetes components associated with the chart and deletes the deployment.

Upgrading an Existing Release to a New Major Version

A major chart version change (like v1.2.3 → v2.0.0) indicates that there is an incompatible breaking change which needs manual actions.