Container Orchestration
- Introduction
- Docker Compose
- Kubernetes and Helm
- Using Helm Charts with Infinite Scale
Introduction
This document provides guidelines for container orchestration tools suitable for Infinite Scale. These are Docker-Compose and Kubernetes. Other container orchestration tools like Docker Swarm can be derived from these.
Container orchestration tools are necessary to meet the requirements described in the Availability and Scalability guide starting at the Container section.
The pages
give a brief overview of features and characteristics of both tools.
For Kubernetes, ownCloud provides basic Helm Charts that can be used and adjusted.
ownCloud highly recommends reading the General Info as it contains valuable information about configuration rules, managing services and default paths - just to mention some of the useful topics. |
Docker Compose
Similar to using docker run and handing over command-line parameters for a single container, you can use a docker-compose.yml
yaml file which defines all the settings and environment variables for each container in one or more files. This is the next step when having multi-container environments.
-
Consider that when planning to run Infinite Scale via Docker Compose, the degree of freedom is not the same as when using Kubernetes with Helm as you are limited to the same server and other limitations but it usually offers more than running the pure binary when it comes to ease of configuration and maintaining your environment.
-
Use Docker Compose if you aim at an extended degree of freedom compared to the binary installation but don’t need the full capabilities of Kubernetes and Helm.
Prerequisites
Check if the package docker-compose
is installed in addition to docker:
which docker-compose
If docker-compose
is installed, you’ll be informed. If not, you may get no output at all or a message that it couldn’t be found. In that case you need to install docker-compose
first. On most Linux distributions, you can simply use the package manager to do so.
Note that in many cases, this will install a 1.x version (python-based) only supported until June 2023. Version 2.x (go-based) is available from Docker. Follow the Install Docker Compose guide to install a 2.x version. Familiarize yourself with the differences between versions particularly with regard to syntax changes. Most of our examples are still based on Compose-V1.
When done, create a project directory like ocis-compose
in your home directory to have a common location for your Infinite Scale Compose files.
Docker Compose Examples
-
See the services section for details about available environment variables and yaml files.
-
See below for example service configurations using Docker Compose, to get a first impression of how this can be achieved. Both ocis environment variables and ocis service configuration yaml files are used.
Using git version name:
master
Using git version name:
v2.0.0
Note that github will not let you download a single directory easily. You can get the examples using the following methods requiring minimum space, replace the version accordingly:
-
Using a shallow git clone which minimizes the required clone space like:
git clone --depth 1 https://github.com/owncloud/ocis.git -b v2.0.0
-
Using an external page providing you the folder to download as zip file like:
https://download-directory.github.io?url=https://github.com/owncloud/ocis/tree/v2.0.0/deployments/examples
-
Kubernetes and Helm
The commands and examples are based on software from Kubernetes and Helm. For other software products or environments without claim to completeness like:
-
OpenShift
-
Rancher
-
K3s
-
AWS EKS
-
GCP GKE
-
Azure AKS
you may need to adapt the commands and possibly the provided yaml files. As an example, OpenShift requires, if not otherwise defined, the container user and password ID to be a very high number. Check your software or environment for details and requirements.
Note that this does not affect the supported Kubernetes versions. Any software or environment must match the version standards.
Kubernetes
Kubernetes (abbreviated as K8s) is an open-source platform for governing clusters of containerized application services. Kubernetes automates the vital aspects of container lifecycle management, including scaling, replication, monitoring, and scheduling. It offers a framework for distributed systems. Infinite Scale was designed with Kubernetes in mind. Therefore ownCloud provides Helm charts for a convenient deployment of Infinite Scale on a Kubernetes cluster.
See the Deployment Evolution description in the Availability and Scalability section for reasons to use Kubernetes.
Infinite Scale follows the Twelve-Factor App principles regarding configuration, which means almost every aspect of Infinite Scale is modifiable via environment variables.
When designing your Kubernetes cluster, you have two major approaches available which are minikube
and kubeadm
.
- minikube
-
minikube lets you run a single-node Kubernetes cluster locally. It is a good way to test a deployment. It requires no extra configuration on any cloud platform as everything runs on your local machine.
- kubeadm
-
kubeadm requires at least two nodes and builds a minimum viable, production-ready Kubernetes cluster, using best practices. It also allows the container runtime to be chosen, though it has Docker by default.
- A tool to note: kubectl
-
kubectl is the command-line tool for Kubernetes. It allows users to run commands against a K8s cluster. It supports multiple contexts for as many clusters as you have access to. minikube also provides kubectl wrapped as
minikube kubectcl
. - Pods
-
A Pod is a Kubernetes abstraction that represents a group of one or more application containers such as Docker and some shared resources for those containers.
Helm
Helm is a Kubernetes deployment tool for automating creation, packaging, configuration and deployment of applications and services to Kubernetes clusters. Comparing Kubernetes to the operating system, Helm would be the package manager. Helm automates the maintenance of YAML manifests for Kubernetes objects. This is done by packaging information into charts — therefore Helm Charts — and advertising them to a Kubernetes cluster. The image below shows the interaction of Helm v3 with Kubernetes.
Prerequisites
- Installing Kubernetes
-
Depending on whether you want to go for a single or multi-node Kubernetes environment, follow the Kubernetes Installation Documentation to do so. This documentation will use minikube in the examples. Verify your installation by typing:
minikube version
- Installing Helm
-
Follow the Helm Installation Documentation post installation and setup of Kubernetes. Verify your installation by typing:
helm version
Using Helm Charts with Infinite Scale
The Helm chart is still in an experimental phase and has not yet been published on a Helm chart repository. For your convenience, ownCloud provides an ocis-charts git repository. |
ownCloud will publish updated data when new Helm chart releases become available. This information will be available in additional tabs in corresponding sections. Note that two Helm chart stable versions will be documented beside the development version. |
When defining your own Helm charts, consider that, if you’re using config overrides in your yaml definitions, a service does not get redefined in the override again. Multiple definitions can cause chart issues that are hard to identify. |
The values.yaml
file provided by ownCloud uses generic configuration. You can customize this configuration with your own values, for example for different setups or sizings. This should be done by using your own values.yaml
file at a different location which will overwrite or add content to the provided one. While not mandatory, the identical file name of values.yaml
follows the convention of Helm. When it comes to security sensitive data like secrets, such data is usually not added in the overwrite-values file for security reasons. In such a case, you apply secrets via command from a secrets file.
Supported Infinite Scale Versions
See the following table to match the Helm chart versions with Infinite Scale releases. Note that the chart version matches the tag in the ocis-charts git repository.
Helm Chart Version | Works with Infinite Scale Versions |
---|---|
latest |
3.0.0-rc.4 |
0.2.0 |
3.0.0-alpha.1 |
0.1.0 |
2.0.0 |
Breaking Changes
Breaking changes are only published for Helm Chart versions that have been released. Select from the tabs possible breaking changes between Helm Chart versions. A new document containing all the details will open.
Get the Chart
As the Helm chart has currently not been published to a Helm repository, you need to clone ownCloud’s Helm chart git repository named ocis-charts.
-
Check the supported Kubernetes versions before you download the chart.
-
The
~
represents all patch releases for that particular version. -
The
-0
represents subversions of that particular version.
-
Start minikube
-
Start your minikube cluster with the latest supported K8s version:
minikube start --kubernetes-version=v1.25.0
-
Enable the minikube ingress plugin, which acts like a reverse proxy for your cluster:
minikube addons enable ingress
Deploy the Chart
-
Based on the Kubernetes version, you will find comments in
values.yaml
where content is Kubernetes version dependent. Search for comments withKubernetes
for details. -
Deploy the chart with the deployment name
ocis
, use any name as desired. To do so, run the following command from the root of the cloned repository:helm install ocis ./charts/ocis
File Description Helm chart with default configurations.
Description of the values.yaml file.
File Description Helm chart with default configurations.
Description of the values.yaml file.
File Description Helm chart with default configurations.
Description of the values.yaml file.
Customize the Generic Setup
In all examples, adapt the settings according your needs.
Set Your Own Default Values
-
Create your own local
values.yaml
file which will overwrite parts of the provided one with the following content:externalDomain: ocis.kube.owncloud.test ingress: enabled: true annotations: nginx.ingress.kubernetes.io/proxy-body-size: 1024m tls: - hosts: - ocis.kube.owncloud.test insecure: # disables ssl certificate checking for connections to the openID connect identity provider. # Not recommended for production setups, but we don't have valid certificates in minikube oidcIdpInsecure: true # disables ssl certificate checking for connections to the oCIS http apis. # Not recommended for production setups, but we don't have valid certificates in minikube ocisHttpApiInsecure: true
externalDomain: ocis.kube.owncloud.test ingress: enabled: true annotations: nginx.ingress.kubernetes.io/proxy-body-size: 1024m tls: - hosts: - ocis.kube.owncloud.test insecure: # disables ssl certificate checking for connections to the openID connect identity provider. # Not recommended for production setups, but we don't have valid certificates in minikube oidcIdpInsecure: true # disables ssl certificate checking for connections to the oCIS http apis. # Not recommended for production setups, but we don't have valid certificates in minikube ocisHttpApiInsecure: true
externalDomain: ocis.kube.owncloud.test ingress: enabled: true annotations: nginx.ingress.kubernetes.io/proxy-body-size: 1024m tls: - hosts: - ocis.kube.owncloud.test insecure: # disables ssl certificate checking for connections to the openID connect identity provider. # Not recommended for production setups, but we don't have valid certificates in minikube oidcIdpInsecure: true # disables ssl certificate checking for connections to the oCIS http apis. # Not recommended for production setups, but we don't have valid certificates in minikube ocisHttpApiInsecure: true
Enable Metrics with Prometheus
-
In order to scrape oCIS' metrics with Prometheus, you need to set up a
ServiceMonitor
. In order to apply the ServiceMonitor, you need to have Prometheus' CustomResourceDefinitions available, e.g. by installing the Prometheus Operator.extraResources: - | apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: ocis-metrics spec: selector: matchLabels: ocis-metrics: enabled endpoints: - port: metrics-debug interval: 60s scrapeTimeout: 30s path: /metrics
extraResources: - | apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: ocis-metrics spec: selector: matchLabels: ocis-metrics: enabled endpoints: - port: metrics-debug interval: 60s scrapeTimeout: 30s path: /metrics
extraResources: - | apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: ocis-metrics spec: selector: matchLabels: ocis-metrics: enabled endpoints: - port: metrics-debug interval: 60s scrapeTimeout: 30s path: /metrics
Configure Email Notification
-
If the key
features.emailNotifications.enable
is set totrue
, the SMTP email server secret needs to be configured:--- apiVersion: v1 kind: Secret metadata: name: notifications-smtp-secret type: Opaque data: # Username for authentication against the SMTP host. smtp-username: XXXXXXXXXXXXX # Password for authentication against the the SMTP host. smtp-password: XXXXXXXXXXXXX
--- apiVersion: v1 kind: Secret metadata: name: notifications-smtp-secret type: Opaque data: # Username for authentication against the SMTP host. smtp-username: XXXXXXXXXXXXX # Password for authentication against the the SMTP host. smtp-password: XXXXXXXXXXXXX
--- apiVersion: v1 kind: Secret metadata: name: notifications-smtp-secret type: Opaque data: # Username for authentication against the SMTP host. smtp-username: XXXXXXXXXXXXX # Password for authentication against the the SMTP host. smtp-password: XXXXXXXXXXXXX
Define Generic Secrets
Infinite Scale requires some generic secrets to work. It was decided to not create them automatically, because we didn’t find a satisfactory way for one-off generation of secrets yet. Certificates are also required which should expire and therefore need a certificate rotation from time to time, for which we didn’t document appropiate tooling yet.
For these reasons, ownCloud cannot take responsibility for these generic secrets and their lifecycle. Any information necessary to use this security-relevant data is provided by ownCloud via examples.
-
The following example shows what generic secrets need to look like and how they can be generated. The example assumes that the
secretRefs
are not changed . Each secret data entry holds a description of how to generate it or find the right value.File Description Secrets file containing all relevant generic secrets configurations.
File Description Secrets file containing all relevant generic secrets configurations.
File Description Secrets file containing all relevant generic secrets configurations.
Apply Generic Secrets
Secrets can be applied by command or included in extraResources
of your own values.yaml
file. Adapt the data content according to your environment:
-
To apply secrets by command, save the content as
generic-secrets.yaml
and use the following command with a path to the secrets file added if necessary:minikube kubectl -- apply -f generic-secrets.yaml
-
To apply secrets via your own
values.yaml
, add the content atextraResources
. Proper yaml formatting is necessary.
Define Generic Configs
Infinite Scale requires some generic configuration to work. It was decided to not create them automatically, because we didn’t find a satisfactory way for one-off generation of secrets yet.
For these reasons, ownCloud cannot take responsibility for these generic configuration. Any information necessary to use this application-relevant data is provided by ownCloud via examples.
-
The following example shows what generic configuration need to look like and how they can be generated. The example assumes that the
configRefs
are not changed . Each config data entry holds a description of how to generate it or find the right value.File Description Configuration file containing all relevant generic configurations.
File Description Configuration file containing all relevant generic configurations.
File Description Configuration file containing all relevant generic configurations.
Apply Generic Configs
Configs can be applied by command or included in extraResources
of your own values.yaml
file. Adapt the data content according to your environment:
-
To apply configs by command, save the content as
generic-configs.yaml
and use the following command with a path to the secrets file added if necessary:minikube kubectl -- apply -f generic-configs.yaml
-
To apply configs via your own
values.yaml
, add the content atextraResources
. Proper yaml formatting is necessary.
Built-in User Management Secrets
If you’re using the built-in user management by setting features.externalUserManagement.enabled
to false
, you need to set these secrets. Certificates are also required which should expire and therefore need a certificate rotation from time to time, for which we didn’t document appropiate tooling yet.
As the operator of Helm Charts, you are responsible for these user management secrets and their lifecycle. Any information necessary to use this security-relevant data is provided by ownCloud via examples.
-
The following example shows what user management secrets need to look like and how they can be generated. The example assumes that the
secretRefs
are not changed . Each secret data entry holds a description of how to generate it or find the right value.File Description Secrets file for the builtin user management.
File Description Secrets file for the builtin user management.
File Description Secrets file for the builtin user management.
Apply Built-in User Management Secrets
Secrets can be applied by command or included in extraResources
of your own values.yaml
file. Adapt the data content according to your environment:
-
To apply secrets by command, save the content as
builtin-user-mgmt-secrets.yaml
and use the following command (adding the path to the secrets file if necessary):minikube kubectl -- apply -f builtin-user-mgmt-secrets.yaml
-
To apply secrets via your own
values.yaml
, add the content atextraResources
. Proper yaml formatting is necessary.
External User Management Secrets
If you’re using external user management by setting features.externalUserManagement.enabled
to true
, you need to set these secrets. Certificates are also required which should expire and therefore need a certificate rotation from time to time, for which we didn’t document appropiate tooling yet.
If you’re using Helm Charts, you are responsible for these user management secrets and their lifecycle. Any information necessary to use this security-relevant data is provided by ownCloud via examples.
-
The following example shows what external user management secrets need to look like and how they can be generated. The example assumes that the
secretRefs
are not changed . Each secret data entry holds a description of how to generate it or find the right value.File Description Secrets file for the external user management.
File Description Secrets file for the external user management.
File Description Secrets file for the external user management.
Apply External User Management Secrets
Secrets can be applied by command or included in extraResources
of your own values.yaml
file. Adapt the data content according to your environment:
-
To apply secrets by command, save the content as
external-user-mgmt-secrets.yaml
and use the following command (adding a path to the secrets file if necessary):minikube kubectl -- apply -f external-user-mgmt-secrets.yaml
-
To apply secrets via your own
values.yaml
, add the content atextraResources
. Proper yaml formatting is necessary.
NGINX Ingress Example
This is an example with NGINX ingress and certificates issued by cert-manager. To make this work, you need to have NGINX ingress and cert-manager installed in your cluster.
-
Defining NGINX ingress and cert-manager.
externalDomain: ocis.owncloud.test ingress: enabled: true ingressClassName: nginx annotations: cert-manager.io/issuer: 'ocis-certificate-issuer' tls: - hosts: - ocis.kube.owncloud.test secretName: ocis-tls-certificate extraResources: - | apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: ocis-certificate-issuer spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: test@example.test privateKeySecretRef: name: ocis-certificate-issuer solvers: - http01: ingress: class: nginx
externalDomain: ocis.owncloud.test ingress: enabled: true ingressClassName: nginx annotations: cert-manager.io/issuer: 'ocis-certificate-issuer' tls: - hosts: - ocis.kube.owncloud.test secretName: ocis-tls-certificate extraResources: - | apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: ocis-certificate-issuer spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: test@example.test privateKeySecretRef: name: ocis-certificate-issuer solvers: - http01: ingress: class: nginx
externalDomain: ocis.owncloud.test ingress: enabled: true ingressClassName: nginx annotations: cert-manager.io/issuer: 'ocis-certificate-issuer' tls: - hosts: - ocis.kube.owncloud.test secretName: ocis-tls-certificate extraResources: - | apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: ocis-certificate-issuer spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: test@example.test privateKeySecretRef: name: ocis-certificate-issuer solvers: - http01: ingress: class: nginx
Apply Chart Changes
-
Apply all changes defined in your own
values.yaml
file:helm upgrade --install --reset-values \ ocis ./charts/ocis --values values.yaml
-
Ensure that all the pods are running:
minikube kubectl -- get pods
Define Chart Access
Linux
-
Get and check the minikube IP address:
minikube ip
-
If not already done, add the minikube IP address to your
/etc/hosts
file or use the Ingress DNS add-on. An example line for the/etc/hosts
file could look like this:192.168.49.2 ocis.kube.owncloud.test
macOS
The following procedure exposes the macOS system via the necessary configured tunnel to the local network. This setup is therefore intended for development purposes only! |
-
On macOS, minikube requires a tunnel to be started to get access to the Ingress of the cluster:
minikube tunnel --bind-address=192.168.179.3
The IP address to be set, which should normally be the one of the
en0
interface, can be fetched with theifconfig
command. Note that the interface must be reachable from within Kubernetes and127.0.0.1
is not an option in this case. This is because the OpenID process tries to look up the host name of the cluster via the/etc/hosts
file which is described in the next step. Within a pod127.0.0.1
will not resolve to the host serving Kubernetes. -
Add the IP address provided by the above command to your
/etc/hosts
file. An example could look like this:192.168.179.3 ocis.kube.owncloud.test
Access Infinite Scale in Your Browser
After you have customized your setup, use the following URL to access Infinite Scale with your browser:
https://ocis.kube.owncloud.test