Container Orchestration
- Introduction
- Docker Compose
- Kubernetes and Helm
- Using Our Helm Charts with Infinite Scale
- Supported Infinite Scale Versions
- Breaking Changes
- Supported Kubernetes Versions
- Get the Chart
- Start minikube
- Deploy the Chart
- Customize the Generic Setup
- Set Your Own Default Values
- Enable Metrics with Prometheus
- Configure Email Notification
- Configure S3ng Storage
- Configure Userlog Global Notifications Secret
- Define Mandatory Secrets and ConfigMaps
- Built-in User Management Secrets
- External User Management Secrets
- NGINX Ingress Example
- Apply Chart Changes
- Access Infinite Scale in Your Browser
- Uninstalling the Chart
Introduction
This document provides guidelines for container orchestration tools suitable for Infinite Scale. These are Docker-Compose and Kubernetes. Other container orchestration tools like Docker Swarm can be derived from these.
Container orchestration tools are necessary to meet the requirements described in the Availability and Scalability guide starting at the Container section.
The pages
give a brief overview of features and characteristics of both tools.
For Kubernetes, ownCloud provides basic Helm Charts that can be used and adjusted.
ownCloud highly recommends reading the General Info as it contains valuable information about configuration rules, managing services and default paths - just to mention some of the useful topics. |
Docker Compose
Similar to using docker run and handing over command-line parameters for a single container, you can use a docker-compose.yml
yaml file which defines all the settings and environment variables for each container in one or more files. This is the next step when having multi-container environments.
-
Consider that when planning to run Infinite Scale via Docker Compose, the degree of freedom is not the same as when using Kubernetes with Helm as you are limited to the same server and other limitations but it usually offers more than running the pure binary when it comes to ease of configuration and maintaining your environment.
-
Use Docker Compose if you aim at an extended degree of freedom compared to the binary installation but don’t need the full capabilities of Kubernetes and Helm.
Prerequisites
Check if the package docker-compose
is installed in addition to docker:
which docker-compose
If docker-compose
is installed, you’ll be informed. If not, you may get no output at all or a message that it couldn’t be found. In that case you need to install docker-compose
first. On most Linux distributions, you can simply use the package manager to do so.
Note that in many cases, this will install a 1.x version (python-based) only supported until June 2023. Version 2.x (go-based) is available from Docker. Follow the Install Docker Compose guide to install a 2.x version. Familiarize yourself with the differences between versions particularly with regard to syntax changes. Most of our examples are still based on Compose-V1. One major difference is, that compose is now a plugin of docker, even if there is still a standalone version. If the check above fails, you might also want to test:
docker compose version
When done, create a project directory like ocis-compose
in your home directory to have a common location for your Infinite Scale Compose files.
Docker Compose Examples
-
See the services section for details about available environment variables and yaml files.
-
See below for example service configurations using Docker Compose, to get a first impression of how this can be achieved. Both ocis environment variables and ocis service configuration yaml files are used.
Using git version name:
v5.0.8
Note that github will not let you download a single directory easily. You can get the examples using the following methods requiring minimum space. For stable releases, check or update the version accordingly like when there is a new patch release available:
-
Using a shallow git clone which minimizes the required clone space like:
git clone --depth 1 https://github.com/owncloud/ocis.git -b 5.0.8
-
Using an external page providing you the folder to download as zip file like:
https://download-directory.github.io?url=https://github.com/owncloud/ocis/tree/v5.0.8/deployments/examples
-
Kubernetes and Helm
The commands and examples are based on software from Kubernetes and Helm. For other software products or environments without claim to completeness like:
-
OpenShift
-
Rancher
-
K3s
-
AWS EKS
-
GCP GKE
-
Azure AKS
you may need to adapt the commands and possibly the provided yaml files. As an example, OpenShift requires, if not otherwise defined, the container user and password ID to be a very high number. Check your software or environment for details and requirements.
Note that this does not affect the supported Kubernetes versions. Any software or environment must match the version standards.
Kubernetes
Kubernetes (abbreviated as K8s) is an open-source platform for governing clusters of containerized application services. Kubernetes automates the vital aspects of container lifecycle management, including scaling, replication, monitoring, and scheduling. It offers a framework for distributed systems. Infinite Scale was designed with Kubernetes in mind. Therefore ownCloud provides Helm charts for a convenient deployment of Infinite Scale on a Kubernetes cluster.
See the Deployment Evolution description in the Availability and Scalability section for reasons to use Kubernetes.
Infinite Scale follows the Twelve-Factor App principles regarding configuration, which means almost every aspect of Infinite Scale is modifiable via environment variables.
When designing your Kubernetes cluster, you have two major approaches available which are minikube
and kubeadm
.
- minikube
-
minikube lets you run a single-node Kubernetes cluster locally. It is a good way to test a deployment. It requires no extra configuration on any cloud platform as everything runs on your local machine.
- kubeadm
-
kubeadm requires at least two nodes and builds a minimum viable, production-ready Kubernetes cluster, using best practices. It also allows the container runtime to be chosen, though it has Docker by default.
- A tool to note: kubectl
-
kubectl is the command-line tool for Kubernetes. It allows users to run commands against a K8s cluster. It supports multiple contexts for as many clusters as you have access to. minikube also provides kubectl wrapped as
minikube kubectcl
. - Pods
-
A Pod is a Kubernetes abstraction that represents a group of one or more application containers such as Docker and some shared resources for those containers.
Helm
Helm is a Kubernetes deployment tool for automating creation, packaging, configuration and deployment of applications and services to Kubernetes clusters. Comparing Kubernetes to the operating system, Helm would be the package manager. Helm automates the maintenance of YAML manifests for Kubernetes objects. This is done by packaging information into charts — therefore Helm Charts — and advertising them to a Kubernetes cluster. The image below shows the interaction of Helm v3 with Kubernetes.
Prerequisites
- Installing Kubernetes
-
Depending on whether you want to go for a single or multi-node Kubernetes environment, follow the Kubernetes Installation Documentation to do so. This documentation will use minikube in the examples. Verify your installation by typing:
minikube version
- Installing Helm
-
Follow the Helm Installation Documentation post installation and setup of Kubernetes. Verify your installation by typing:
helm version
Using Our Helm Charts with Infinite Scale
The Helm chart is still in an experimental phase and has not yet been published on a Helm chart repository. For your convenience, ownCloud provides an ocis-charts git repository. |
ownCloud will publish updated data when new Helm chart releases become available. This information will be available in additional tabs in corresponding sections. Note that two Helm chart stable versions will be documented beside the development version. |
When defining your own Helm charts, consider that, if you’re using config overrides in your yaml definitions, a service does not get redefined in the override again. Multiple definitions can cause chart issues that are hard to identify. |
The values.yaml
file provided by ownCloud uses generic configuration. You can customize this configuration with your own values, for example for different setups or sizings. This should be done by using your own values.yaml
file at a different location which will overwrite or add content to the provided one. While not mandatory, the identical file name of values.yaml
follows the convention of Helm. When it comes to security sensitive data like secrets, such data is usually not added in the overwrite-values file for security reasons. In such a case, you apply secrets via command from a secrets file.
Supported Infinite Scale Versions
See the following table to match the Helm chart versions with Infinite Scale releases. Note that the chart version matches the tag in the ocis-charts git repository.
Helm Chart Version | Works with Infinite Scale Versions |
---|---|
latest |
4.0.1 |
0.5.0 |
4.0.1 |
0.4.0 |
3.0.0 |
-
If a chart has been superseded by another for the same Infinite Scale release, only the latest one is listed.
-
Note that Helm Chart Version
0.2.0
was a necessary intermediate for Infinite Scale3.0.0-alpha.1
only and is therefore not listed with a working Infinite Scale version.
Breaking Changes
Select possible breaking changes of a Helm chart version from the tabs. A new document with all details will be opened, directly referring to the selected version, although the document contains information about all published versions.
Supported Kubernetes Versions
We only list non EOL versions as of the chart release date from here: https://kubernetes.io/releases/
Note that some EOL versions still might be API compatible.
Version | Covered by tests | Tested by developers |
---|---|---|
1.28 |
yes |
yes |
1.27 |
yes |
yes |
1.26 |
yes |
no |
1.25 |
yes |
no |
We only list non EOL versions as of the chart release date from here: https://kubernetes.io/releases/
Note that some EOL versions still might be API compatible.
Version | Covered by tests | Tested by developers |
---|---|---|
1.27 |
yes |
yes |
1.26 |
yes |
yes |
1.25 |
yes |
no |
1.24 |
yes |
no |
-
Check the supported Kubernetes versions before you download the chart.
-
The
~
represents all patch releases for that particular version. -
The
-0
represents subversions of that particular version.
Version ~1.27.0-0
~1.26.0-0
~1.25.0-0
~1.24.0-0
-
Get the Chart
As the Helm chart has currently not been published to a Helm repository, you need to clone ownCloud’s Helm chart git repository named ocis-charts.
Start minikube
-
Start your minikube cluster with the latest supported K8s version:
minikube start --kubernetes-version=v1.28.1
-
Enable the
minikube ingress
plugin, which acts like a reverse proxy for your cluster:minikube addons enable ingress
-
Linux only: enable the
ingress-dns
plugin and configure it:minikube addons enable ingress-dns
-
Configure the
in-cluster DNS server
to resolve local DNS names inside the cluster`:Note that this step is not optional but mandatory for an Infinite Scale installation. For details see Step 4, (optional) Configure in-cluster DNS server to resolve local DNS names inside cluster.
-
macOS only: run
minikube tunnel
to be expose80,443
ports:minikube tunnel
-
Configure hosts:
-
On Linux you need to add additional configurations to use ingress by adding the domain names to
/etc/hosts
. Those entries need to point to the Minikube interface IP which you can get by runningminikube ip
.192.168.49.2 ocis.kube.owncloud.test
-
On macOS you need to add additional configurations to use ingress by adding the domain names to
/etc/hosts
. Since you are usingminikube tunnel
, those entries need to point to127.0.0.1
because it’s listening on the localhost interface.127.0.0.1 ocis.kube.owncloud.test
-
Deploy the Chart
-
When installing Infinite Scale in Minikube on MacOS, you need to set the
hostAliases
option:hostAliases: - ip: "192.168.49.2" # <- needs to be the IP of `minikube ip` hostnames: - "ocis.kube.owncloud.test"
-
Based on the Kubernetes version, you will find comments in
values.yaml
where content depends on the Kubernetes version. Search for comments withKubernetes
for details. -
Deploy the chart with the deployment name
ocis
, use any name as desired. To do so, run the following command from the root of the cloned repository:helm install ocis ./charts/ocis
File Description Helm chart with default configurations.
Description of the values.yaml file.
File Description Helm chart with default configurations.
Description of the values.yaml file.
File Description Helm chart with default configurations.
Description of the values.yaml file.
Customize the Generic Setup
In all examples, adapt the settings according your needs.
Set Your Own Default Values
-
Create your own local
values.yaml
file which will overwrite parts of the provided one with the following content:externalDomain: ocis.kube.owncloud.test ingress: enabled: true annotations: nginx.ingress.kubernetes.io/proxy-body-size: 1024m tls: - hosts: - ocis.kube.owncloud.test insecure: # disables ssl certificate checking for connections to the openID connect identity provider. # Not recommended for production setups, but we don't have valid certificates in minikube oidcIdpInsecure: true # disables ssl certificate checking for connections to the oCIS http apis. # Not recommended for production setups, but we don't have valid certificates in minikube ocisHttpApiInsecure: true
externalDomain: ocis.kube.owncloud.test ingress: enabled: true annotations: nginx.ingress.kubernetes.io/proxy-body-size: 1024m tls: - hosts: - ocis.kube.owncloud.test insecure: # disables ssl certificate checking for connections to the openID connect identity provider. # Not recommended for production setups, but we don't have valid certificates in minikube oidcIdpInsecure: true # disables ssl certificate checking for connections to the oCIS http apis. # Not recommended for production setups, but we don't have valid certificates in minikube ocisHttpApiInsecure: true
externalDomain: ocis.kube.owncloud.test ingress: enabled: true annotations: nginx.ingress.kubernetes.io/proxy-body-size: 1024m tls: - hosts: - ocis.kube.owncloud.test insecure: # disables ssl certificate checking for connections to the openID connect identity provider. # Not recommended for production setups, but we don't have valid certificates in minikube oidcIdpInsecure: true # disables ssl certificate checking for connections to the oCIS http apis. # Not recommended for production setups, but we don't have valid certificates in minikube ocisHttpApiInsecure: true
Enable Metrics with Prometheus
-
In order to scrape oCIS' metrics with Prometheus, you need to set up a
ServiceMonitor
. In order to apply the ServiceMonitor, you need to have Prometheus' CustomResourceDefinitions available, e.g. by installing the Prometheus Operator.extraResources: - | apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: ocis-metrics spec: selector: matchLabels: ocis-metrics: enabled endpoints: - port: metrics-debug interval: 60s scrapeTimeout: 30s path: /metrics
extraResources: - | apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: ocis-metrics spec: selector: matchLabels: ocis-metrics: enabled endpoints: - port: metrics-debug interval: 60s scrapeTimeout: 30s path: /metrics
extraResources: - | apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: ocis-metrics spec: selector: matchLabels: ocis-metrics: enabled endpoints: - port: metrics-debug interval: 60s scrapeTimeout: 30s path: /metrics
Configure Email Notification
-
If the key
features.emailNotifications.enable
is set totrue
, the SMTP email server Secret referenced insecretRefs.notificationsSmtpSecretRef
needs to be configured:--- apiVersion: v1 kind: Secret metadata: name: notifications-smtp-secret # needs to be set to `secretRefs.notificationsSmtpSecretRef` type: Opaque data: # Username for authentication against the SMTP host. smtp-username: XXXXXXXXXXXXX # Password for authentication against the the SMTP host. smtp-password: XXXXXXXXXXXXX
--- apiVersion: v1 kind: Secret metadata: name: notifications-smtp-secret # needs to be set to `secretRefs.notificationsSmtpSecretRef` type: Opaque data: # Username for authentication against the SMTP host. smtp-username: XXXXXXXXXXXXX # Password for authentication against the the SMTP host. smtp-password: XXXXXXXXXXXXX
--- apiVersion: v1 kind: Secret metadata: name: notifications-smtp-secret # needs to be set to `secretRefs.notificationsSmtpSecretRef` type: Opaque data: # Username for authentication against the SMTP host. smtp-username: XXXXXXXXXXXXX # Password for authentication against the the SMTP host. smtp-password: XXXXXXXXXXXXX
Configure S3ng Storage
-
If the key
services.storageusers.storageBackend.driver
is set tos3ng
, the S3 access key ID / secret Secret referenced insecretRefs.s3CredentialsSecretRef
needs to be configured:--- apiVersion: v1 kind: Secret metadata: name: notifications-smtp-secret # default of `secretRefs.s3CredentialsSecretRef` type: Opaque data: # S3 access key. accessKey: XXXXXXXXXXXXX # S3 secret key. secretKey: XXXXXXXXXXXXX
--- apiVersion: v1 kind: Secret metadata: name: notifications-smtp-secret # default of `secretRefs.s3CredentialsSecretRef` type: Opaque data: # S3 access key. accessKey: XXXXXXXXXXXXX # S3 secret key. secretKey: XXXXXXXXXXXXX
--- apiVersion: v1 kind: Secret metadata: name: notifications-smtp-secret # default of `secretRefs.s3CredentialsSecretRef` type: Opaque data: # S3 access key. accessKey: XXXXXXXXXXXXX # S3 secret key. secretKey: XXXXXXXXXXXXX
Configure Userlog Global Notifications Secret
-
Configure Global Notifications Secrets referenced in
secretRefs.globalNotificationsSecretRef
if required:
Define Mandatory Secrets and ConfigMaps
Infinite Scale requires some mandatory Secrets and ConfigMaps to work. They are created one-off if you don’t explicitly provide them. If you’re using the builtin user management, which is not recommended, among the auto generated Secrets, there are also some certificates which expire and need to be renewed manually.
These Secrets and ConfigMaps need to be part of your backup since you need to provide them manually during a disaster recovery procedure. |
Mandatory Secrets
-
If you want to manage Secrets on your own, you can look at the following example which shows what mandatory Secrets look like and how they can be generated. The example assumes that the
secretRefs
are not changed. Each Secret data entry holds a description of how to generate it or find the right value.File Description List of all mandatory Secrets.
File Description List of all mandatory Secrets.
File Description List of all mandatory Secrets.
Apply Mandatory Secrets
Secrets can be applied by command or included in extraResources
of your own values.yaml
file. Adapt the data content according to your environment:
-
To apply secrets by command, save the content as
mandatory-secrets.yaml
and use the following command with a path to the secrets file added if necessary:kubectl apply -f mandatory-secrets.yaml
-
To apply secrets via your own
values.yaml
, add the content atextraResources
. Proper yaml formatting is necessary.
Mandatory ConfigMaps
-
If you want to manage ConfigMaps on your own, you can look at the following example which shows how mandatory ConfigMaps look like and how they can be generated. The example assumes that the
configRefs
are not changed. Each ConfigMaps data entry holds a description of how to generate it or find the right value.
-
The following example shows what generic configuration need to look like and how they can be generated. The example assumes that the
configRefs
are not changed . Each config data entry holds a description of how to generate it or find the right value.File Description List of all mandatory ConfigMaps.
File Description List of all mandatory ConfigMaps.
File Description List of all mandatory ConfigMaps.
Apply Mandatory ConfigMaps
ConfigMaps can be applied by command or included in extraResources
of your own values.yaml
file. Adapt the data content according to your environment:
-
To apply configs by command, save the content as
generic-configs.yaml
and use the following command with a path to the secrets file added if necessary:kubectl apply -f generic-configs.yaml
-
To apply configs via your own
values.yaml
, add the content atextraResources
. Proper yaml formatting is necessary.
Built-in User Management Secrets
If you’re using the built-in user management by setting features.externalUserManagement.enabled
to false
, which is the default, you’ll need additional Secrets. These are also autogenerated for you if you don’t provide them manually.
These Secrets are certificates that expire after 365 days and therefore need a certificate rotation from time to time. Rotation can be achieved by deleting the Secrets ldap-ca and ldap-cert and restarting all Infinite Scale deployments like with kubectl rollout restart deploy .
|
-
The following example shows what the Secrets for the built-in user management need to look like and how they can be generated. The example assumes that the
secretRefs
are not changed . Each Secret data entry holds a description of how to generate it or find the right value.File Description Secrets file for the builtin user management.
File Description Secrets file for the builtin user management.
File Description Secrets file for the builtin user management.
Apply Built-in User Management Secrets
Secrets can be applied by command or included in extraResources
of your own values.yaml
file. Adapt the data content according to your environment:
-
To apply secrets by command, save the content as
builtin-user-mgmt-secrets.yaml
and use the following command (adding the path to the secrets file if necessary):kubectl apply -f builtin-user-mgmt-secrets.yaml
-
To apply secrets via your own
values.yaml
, add the content atextraResources
. Proper yaml formatting is necessary.
External User Management Secrets
If you’re using external user management by setting features.externalUserManagement.enabled
to true
, you need to set these Secrets. Certificates are also required which should expire and therefore need a certificate rotation from time to time, for which we didn’t document appropiate tooling yet.
If you’re using Helm Charts, you are responsible for these user management secrets and their lifecycle. Any information necessary to use this security-relevant data is provided by ownCloud via examples.
-
The following example shows what external user management secrets need to look like and how they can be generated. The example assumes that the
secretRefs
are not changed . Each secret data entry holds a description of how to generate it or find the right value.File Description Secrets file for the external user management.
File Description Secrets file for the external user management.
File Description Secrets file for the external user management.
Apply External User Management Secrets
Secrets can be applied by command or included in extraResources
of your own values.yaml
file. Adapt the data content according to your environment:
-
To apply secrets by command, save the content as
external-user-mgmt-secrets.yaml
and use the following command (adding a path to the secrets file if necessary):kubectl apply -f external-user-mgmt-secrets.yaml
-
To apply secrets via your own
values.yaml
, add the content atextraResources
. Proper yaml formatting is necessary.
NGINX Ingress Example
This is an example with NGINX ingress and certificates issued by cert-manager. To make this work, you need to have NGINX ingress and cert-manager installed in your cluster.
-
Defining NGINX ingress and cert-manager.
externalDomain: ocis.owncloud.test ingress: enabled: true ingressClassName: nginx annotations: cert-manager.io/issuer: 'ocis-certificate-issuer' tls: - hosts: - ocis.kube.owncloud.test secretName: ocis-tls-certificate extraResources: - | apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: ocis-certificate-issuer spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: test@example.test privateKeySecretRef: name: ocis-certificate-issuer solvers: - http01: ingress: class: nginx
externalDomain: ocis.owncloud.test ingress: enabled: true ingressClassName: nginx annotations: cert-manager.io/issuer: 'ocis-certificate-issuer' tls: - hosts: - ocis.kube.owncloud.test secretName: ocis-tls-certificate extraResources: - | apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: ocis-certificate-issuer spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: test@example.test privateKeySecretRef: name: ocis-certificate-issuer solvers: - http01: ingress: class: nginx
externalDomain: ocis.owncloud.test ingress: enabled: true ingressClassName: nginx annotations: cert-manager.io/issuer: 'ocis-certificate-issuer' tls: - hosts: - ocis.kube.owncloud.test secretName: ocis-tls-certificate extraResources: - | apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: ocis-certificate-issuer spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: test@example.test privateKeySecretRef: name: ocis-certificate-issuer solvers: - http01: ingress: class: nginx