Kubernetes Deployment

OpenMetadata supports the Installation and Running of Application on kubernetes through Helm Charts.

openmetadata-helm-charts houses Kubernetes Helm charts for deploying OpenMetadata and its dependencies (Elasticsearch, MySQL and Airflow) on a Kubernetes cluster.

  • A Kubernetes cluster on any cloud
  • kubectl to manage Kubernetes resources
  • Helm to deploy resources based on Helm charts from the OpenMetadata repository

OpenMetadata ONLY supports Helm 3.

This guide assumes your helm chart release names are openmetadata and openmetadata-dependencies and the kubernetes namespace used is default.

If you are using minikube to start a local kubernetes instance on Mac OS with M1 chipset, use the following command to start the cluster required for OpenMetadata Helm Charts to install locally (with docker desktop running as container runtime engine).

minikube start --cpus=4 --memory=8192 --cni=bridge --driver=docker

Assuming Kubernetes setup is done and your Kubernetes context points to a correct Kubernetes cluster, first we create kubernetes secrets that contains MySQL and Airflow passwords as secrets.

The above commands sets the passwords as an example. Change to any password of your choice.

Next, install the OpenMetadata dependencies by adding the OpenMetadata Helm repository with the following command.

Run the command helm repo list to ensure the OpenMetadata repository was added.

Assuming kubectl context points to the correct kubernetes cluster, first create the kubernetes secrets that contain airflow mysql password as secrets.

Deploy the dependencies by running the following command:

The above command uses configurations defined here. You can modify any configuration and deploy by passing your own values.yaml

Run kubectl get pods to check whether all the pods for the dependencies are running. You should get a result similar to below.

Please note that the pods names above as openmetadata-dependencies-* are part of airflow deployments.

Helm Chart for OpenMetadata Dependencies uses the following helm charts:

If you want to customise helm values for the dependencies as per your cluster, you can follow the above links and update your custom helm values.yaml.

Airflow uses DAGs and Logs for persistence volumes with the Kubernetes Executor.

Make sure your kubernetes cluster storage provisioner has persistent volumes capability of ReadWriteMany Access Mode. Modify the Helm Values for airflow as per your requirement here.

This is not required if you are deploying on kubernetes cluster created by minkube or Docker Desktop. Check the storage provider compatibility here.

Next, deploy OpenMetadata by running the following command:

Again, this uses the values defined here. Use the --values flag to point to your own YAML configuration if needed.

The values in values.yaml are preset to match with dependencies deployed using openmetadata-dependencies with release name "openmetadata-dependencies". If you deployed helm chart using different release name, make sure to update values.yaml accordingly before installing.

Run kubectl get pods --selector=app.kubernetes.io/name=openmetadata to check the status of pods running. You should get a result similar to the output below:

To expose the OpenMetadata UI on a local Kubernetes instance, run this command:

Please follow our Enable Security Guide to configure security for your OpenMetadata installation.

Run the below command to view status of openmetadata helm chart deployed:

For more information, visit helm command line reference here.

Run the below command to list openmetadata kubernetes pods deployed in a namespace:

For example, list pods deployed by helm release name ometa in the namespace ometa-dev:

Next, view the logs of pod by running the below command,

For more information, visit the kubectl logs command line reference documentation here.

If your openmetadata pods are not in ready state at any point in time and the openmetadata pod logs speaks about the below issue -

This is due to the default JVM Heap Space configuration (1 GiB) being not enough for your workloads. In order to resolve this issue, head over to your custom openmetadata helm values and append the below environment variable

The flag Xmx specifies the maximum memory allocation pool for a Java virtual machine (JVM), while Xms specifies the initial memory allocation pool.

Upgrade the helm charts with the above changes using the following command helm upgrade --install openmetadata open-metadata/openmetadata --values <values.yml> --namespace <namespaceName>. Update this command your values.yml filename and namespaceName where you have deployed OpenMetadata in Kubernetes.

Use the below command to uninstall OpenMetadata Helm Charts completely.

MySql and ElasticSearch OpenMetadata Dependencies as deployed as StatefulSets and have persistent volumes (pv) and persistent volume claims (pvc). These will need to be manually cleaned after helm uninstall. You can use kubectl delete CLI command for the same.

If you are running OpenMetadata in AWS, it is recommended to use Amazon RDS and Amazon OpenSearch Service.

We support

  • Amazon RDS (MySQL) engine version upto 8.0.29
  • Amazon OpenSearch (ElasticSearch) engine version upto 7.10 or Amazon OpenSearch engine version upto 1.3
  • Amazon RDS (PostgreSQL) engine version upto 14.2-R1

For Production Systems, we recommend Amazon RDS to be in Multiple Availibility Zones. For Amazon OpenSearch (or ElasticSearch) Service, we recommend Multiple Availibility Zones with minimum 3 Master Nodes.

Once you have the RDS and OpenSearch Services Setup, you can update the environment variables below for OpenMetadata kubernetes deployments to connect with Database and ElasticSearch.

Make sure to create RDS and OpenSearch credentials as Kubernetes Secrets mentioned here.

One possible use case where you would need to use a custom image for the ingestion is because you have developed your own custom connectors. You can find a complete working example of this here. After you have your code ready, the steps would be the following:

For example:

where openmetadata/ingestion:x.y.z needs to point to the same version of the OpenMetadata server, for example openmetadata/ingestion:0.13.2. This image needs to be built and published to the container registry of your choice.

The ingestion containers (which is the one shipping Airflow) gets installed in the openmetadata-dependencies helm chart. In this step, we use our own custom values YAML file to point to the image we just created on the previous step. You can create a file named values.deps.yaml with the following contents:

Upgrade/Install your openmetadata-dependencies helm charts with the below single command:

Finally, follow the usual installation for the openmetadata helm charts.