# Deployment | OpenMetadata Installation & Setup Guide Source: https://docs.open-metadata.org/v2.0.x/deployment Deploy, configure, and manage OpenMetadata across environments with security and scaling guidance.
# From 0 to OpenMetadata in 5 minutes We support different kinds of deployment: ## Quick Start Choose how you want to try OpenMetadata
Try Sandbox
Try Sandbox
No setup · Hosted · Instant access

Explore OpenMetadata instantly with a hosted sandbox that mimics a real production setup no installation required.

Try Sandbox
Docker
Run locally with Docker
Local setup · Full control · Docker-based

Run OpenMetadata on your machine using Docker to get a hands-on experience with a real deployment.

  • Prepare your environment with Docker and Docker Compose
  • Launch OpenMetadata locally using the provided Docker Compose setup
  • Operate and troubleshoot the deployment via UI and Docker commands
Learn More Right arrow
## Production Deploy OpenMetadata securely and reliably for team and organization use. Deploy OpenMetadata on AWS using Amazon EKS and AWS-managed services for production workloads. Run OpenMetadata on Azure Kubernetes with Azure-native database and search services. Deploy OpenMetadata on Google Cloud using GKE with managed infrastructure and services. Deploy OpenMetadata on a self-managed, on-premises Kubernetes cluster. Deploy OpenMetadata on Kubernetes using Helm charts for a scalable, production-ready setup. Install and run OpenMetadata directly on physical or self-managed servers without Kubernetes. ## Production Configuration & Hardening
Production Ready
Production Ready Requirements

Understand hardware sizing, performance, and infrastructure requirements for production deployments.

Learn More Right arrow
Enable Security
Enable Security

Secure your OpenMetadata deployment with authentication, authorization, and secrets management.

Learn More Right arrow
## Upgrade Before upgrading, review compatibility requirements and back up your metadata.
Upgrade OM
Upgrade OpenMetadata

Safely upgrade your OpenMetadata deployment to the latest supported version with pre-checks and post-upgrade validation.

Learn More Right arrow
Understand supported upgrade paths, breaking changes, and version compatibility. Back up your metadata before upgrading and restore it if needed.
# How to enable Azure Auth Source: https://docs.open-metadata.org/v2.0.x/deployment/azure-auth # AZURE resources on Postgres/MySQL Auth [Azure Reference Doc](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-extensions#how-to-use-postgresql-extensions) ## Requirements 1. Azure Postgres or MySQL Cluster with auth enabled 2. User on DB Cluster with authentication enabled ## How to enable Azure Auth on postgresql Set the environment variables ```Commandline theme={null} DB_PARAMS="azure=true&allowPublicKeyRetrieval=true&sslmode=require&serverTimezone=UTC" DB_USER_PASSWORD=none ``` Either through helm (if deployed in kubernetes) or as env vars. The `DB_USER_PASSWORD` is still required and cannot be empty. Set it to a random/dummy string. # Azure - Enable Passwordless Database Backend Connection Source: https://docs.open-metadata.org/v2.0.x/deployment/azure-passwordless-auth # Azure - Enable Passwordless Database Backend Connection By Default, OpenMetadata supports basic authentication when connecting to MySQL/PostgreSQL as Database backend. With Azure, you can enhance the security for configuring Database configurations other the basic authentication mechanism. This guide will help you setup the application to use passwordless approach for Azure PaaS Databases (preferrably [Azure Database for PostgreSQL - Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/service-overview) and [Azure Database for MySQL - Flexible Server](https://learn.microsoft.com/en-us/azure/mysql/flexible-server/overview)). ## Prerequisites This guide requires the following prerequisites - * Azure Database Flexible Server enabled with Microsoft Entra authentication * [Azure Managed Identities](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview) * Azure Kubernetes Service (Enabled with Workload Identity) or Azure VM * OpenMetadata Application Version `1.5.9` and higher If you are looking to enable Passwordless Database Backend Configuration on Existing OpenMetadata Application hosted using Azure Cloud, you need to create perform the following prerequisites - * Create Managed Identity from Azure Portal * Create a SQL User for Managed Identity in Azure Databases * PostgreSQL Reference link [here](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/how-to-manage-azure-ad-users#create-a-userrole-using-microsoft-entra-principal-name) * MySQL Reference link [here](https://learn.microsoft.com/en-us/azure/mysql/flexible-server/how-to-azure-ad#create-microsoft-entra-users-in-azure-database-for-mysql) * Assign Existing OpenMetadata Database Tables Ownership to Managed Identities created in above step ## Enabling Passwordless connections with OpenMetadata Configure your Helm Values for Kubernetes Deployment like below - ```yaml theme={null} # For PostgreSQL commonLabels: azure.workload.identity/use: "true" serviceAccount: create: true annotations: azure.workload.identity/client-id: name: "openmetadata-sa" automountServiceAccountToken: true openmetadata: config: database: host: driverClass: org.postgresql.Driver dbParams: "azure=true&allowPublicKeyRetrieval=true&serverTimezone=UTC&sslmode=require&authenticationPluginClassName=com.azure.identity.extensions.jdbc.postgresql.AzurePostgresqlAuthenticationPlugin" dbScheme: postgresql port: 5432 auth: username: password: secretRef: database-secrets secretKey: openmetadata-database-password databaseName: # For MySQL commonLabels: azure.workload.identity/use: "true" serviceAccount: create: true annotations: azure.workload.identity/client-id: name: "openmetadata-sa" automountServiceAccountToken: true openmetadata: config: database: host: driverClass: com.mysql.cj.jdbc.Driver dbParams: "azure=true&allowPublicKeyRetrieval=trueserverTimezone=UTC&sslMode=REQUIRED&defaultAuthenticationPlugin=com.azure.identity.extensions.jdbc.mysql.AzureMysqlAuthenticationPlugin" dbScheme: mysql port: 3306 auth: username: password: secretRef: database-secrets secretKey: openmetadata-database-password databaseName: ``` In the above code snippet, the Database Credentials (Auth Password Kubernetes Secret) is still required and cannot be empty. Set it to dummy / random value. Install / Upgrade your Helm Release with the following command - ```bash theme={null} helm repo update open-metadata helm upgrade --install openmetadata open-metadata/openmetadata --values ``` For further reference, checkout the official documentation available in the below links - * [MySQL](https://learn.microsoft.com/en-us/azure/developer/java/spring-framework/migrate-mysql-to-passwordless-connection?tabs=sign-in-azure-cli%2Cjava%2Capp-service) * [PostgreSQL](https://learn.microsoft.com/en-us/azure/developer/java/spring-framework/migrate-postgresql-to-passwordless-connection?tabs=sign-in-azure-cli%2Cjava%2Capp-service%2Cassign-role-service-connector) # Bare Metal Deployment | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/bare-metal Deploy the platform on bare-metal servers to maintain full control over infrastructure, authentication, and network configuration. # Deploy on Bare Metal Requirements This guide assumes you have access to a command-line environment or shell such as bash, zsh, etc. or Linux or Mac OS X or PowerShell on Microsoft Windows. This guide also assumes that your command-line environment has access to the tar utility. Please review additional requirements listed in the subsections below. ## Java (version 21.0.0) OpenMetadata is built using Java, DropWizard, and Jetty. Type the following command to verify that you have a supported version of the Java runtime installed. ```commandline theme={null} java --version ``` To install Java or upgrade to Java 21, see the instructions for your operating system at [How do I install Java?](https://java.com/en/download/help/download_options.html#mac). ## MySQL (version 8.0.42 or higher) To install MySQL see the instructions for your operating system (OS) at [Installing and Upgrading MySQL](https://dev.mysql.com/doc/mysql-installation-excerpt/8.0/en/installing.html) or visit one of the following OS-specific guides. * [Installing MySQL on Linux](https://dev.mysql.com/doc/mysql-installation-excerpt/8.0/en/linux-installation.html) * [Installing MySQL on Windows](https://dev.mysql.com/doc/mysql-installation-excerpt/8.0/en/windows-installation.html) * [Installing MySQL on MacOS](https://dev.mysql.com/doc/mysql-installation-excerpt/8.0/en/macos-installation.html) Make sure to configure required databases and users for OpenMetadata. You can refer a sample script [here](https://github.com/open-metadata/OpenMetadata/blob/main/docker/mysql/mysql-script.sql). ## Postgres (version 15 or higher) To install Postgres see the instructions for your operating system (OS) at [Postgres Download](https://www.postgresql.org/download/) Make sure to configure required databases and users for OpenMetadata. You can refer a sample script [here](https://github.com/open-metadata/OpenMetadata/blob/main/docker/postgresql/postgres-script.sql). ## Elasticsearch / OpenSearch OpenMetadata supports ElasticSearch version 9.x (minimum 9.0.0, recommended 9.3.0) and OpenSearch version 3.x (minimum 3.0.0, recommended 3.3.0). The 9.x Elasticsearch client is not compatible with 8.x or older servers. To install or upgrade Elasticsearch to a supported version please see the instructions for your operating system at [Installing ElasticSearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html). Please follow the instructions here to [install ElasticSearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html). If you are using AWS OpenSearch Service, OpenMetadata supports AWS OpenSearch Service engine version 3.x (minimum 3.0.0, recommended 3.3.0). Note that AWS OpenSearch Service currently supports up to OpenSearch 3.3. For more information on AWS OpenSearch Service, please visit the official docs [here](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/what-is.html). ## Airflow or other workflow schedulers OpenMetadata performs metadata ingestion using the Ingestion Framework. Learn more about how to deploy and manage the ingestion workflows [here](/v2.0.x/deployment/ingestion). OpenMetadata versions have specific Airflow compatibility requirements to ensure seamless metadata ingestion. OpenMetadata 1.5 supports Airflow 2.9, 1.6.4 supports Airflow 2.9.3, and 1.6.5 supports Airflow 2.10.5. Ensure that your Airflow version aligns with your OpenMetadata deployment to maintain stability and functionality. ## Minimum Sizing Requirements * Our minimum specs recommendation for the OpenMetadata Deployment (one replica) is 2 vCPUs and 4 Gigs with 20 Gigs of volume size if using persistent volumes for logs. * For Elasticsearch, 2 vCPUs and 2 Gigs RAM (per instance) with 30 Gigs of Storage volume attached. * For the database, 2 vCPUs and 2 Gigs RAM (per instance) with 30 Gigs of Storage Volume Attached (dynamic expansion up to 100 Gigs). These settings apply as well when using managed instances, such as RDS or AWS OpenSearch. ## Procedure ## 1. Download the distribution Visit the [releases page](https://github.com/open-metadata/OpenMetadata/releases/latest) and download the latest binary release. Release binaries follow the naming convention of `openmetadata-x.y.z.tar.gz`. Where `x`, `y`, and `z` represent the major, minor, and patch release numbers. ## 2. Untar the release download Once the tar file has downloaded, run the following command, updated if necessary for the version of OpenMetadata that you downloaded. ```commandline theme={null} tar -zxvf openmetadata-*.tar.gz ``` ## 3. Navigate to the directory created ```commandline theme={null} cd openmetadata-* ``` Review and update the `openmetadata.yaml` configurations to match your environment. Specifically, consider aspects such as the connection to the MySQL database or ElasticSearch. You can find more information about these configurations [here](/v2.0.x/deployment/configuration). ## 4. Prepare the OpenMetadata Database and Indexes The command below will generate all the necessary tables and indexes in ElasticSearch. Note that if there's any data in that database, this command will drop it! ```commandline theme={null} ./bootstrap/openmetadata-ops.sh drop-create ``` ## 5. Start OpenMetadata ```commandline theme={null} ./bin/openmetadata.sh start ``` We recommend configuring `serviced` to monitor the OpenMetadata command to restart in case of any failures. ## Run OpenMetadata with a load balancer You may put one or more OpenMetadata instances behind a load balancer for reverse proxying. To do this you will need to add one or more entries to the configuration file for your reverse proxy. ### Apache mod\_proxy To use the Apache mod\_proxy module as a reverse proxy for load balancing, update the VirtualHost tag in your Apache config file to resemble the following. ```xml theme={null} BalancerMember http://127.0.0.1:8585 BalancerMember http://127.0.0.2:8686 ProxyPreserveHost On ProxyPass / balancer://mycluster/ ProxyPassReverse / balancer://mycluster/ ``` ### Nginx To use OpenMetadata behind an Nginx reverse proxy, add an entry resembling the following the http context of your Nginx configuration file for each OpenMetadata instance. ```commandline theme={null} server { access_log /var/log/nginx/stage-reverse-access.log; error_log /var/log/nginx/stage-reverse-error.log; server_name stage.open-metadata.org; location / { proxy_pass http://127.0.0.1:8585; } } ``` ## Run OpenMetadata with AWS Services or your hosted DB/ElasticSearch If you are running OpenMetadata in AWS, it is recommended to use [Amazon RDS](https://docs.aws.amazon.com/rds/index.html) and [Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/?id=docs_gateway). We support * Amazon RDS (MySQL) engine version 8 or higher * Amazon OpenSearch (ElasticSearch) engine version 9.x (minimum 9.0.0, recommended 9.3.0) or Amazon OpenSearch engine version 3.x (minimum 3.0.0, recommended 3.3.0) * Amazon RDS (PostgreSQL) engine version 15 or higher For Production Systems, we recommend Amazon RDS to be in Multiple Availability Zones. For Amazon OpenSearch (or ElasticSearch) Service, we recommend Multiple Availability Zones with minimum 3 Master Nodes. Once you have the RDS and OpenSearch Services Setup, you can update the environment variables below for OpenMetadata bare metal systems to connect with Database and ElasticSearch. Below are the environment variables for OpenMetadata Server ### Configure MySQL connection ``` # MySQL Environment Variables DB_DRIVER_CLASS='com.mysql.cj.jdbc.Driver' DB_SCHEME='mysql' DB_PARAMS='allowPublicKeyRetrieval=true&useSSL=true&serverTimezone=UTC' DB_USER='' DB_USER_PASSWORD='' DB_HOST='' DB_PORT='' OM_DATABASE='' ``` ### Configure Postgres Connection ``` # Postgres Environment Variables DB_DRIVER_CLASS='org.postgresql.Driver' DB_SCHEME='postgresql' DB_PARAMS='allowPublicKeyRetrieval=true&useSSL=true&serverTimezone=UTC' DB_USER='' DB_USER_PASSWORD='' DB_HOST='' DB_PORT='' OM_DATABASE='' ``` ### Configure ElasticSearch Connection ``` ELASTICSEARCH_SOCKET_TIMEOUT_SECS='60' ELASTICSEARCH_USER='' ELASTICSEARCH_CONNECTION_TIMEOUT_SECS='5' ELASTICSEARCH_PORT='443' ELASTICSEARCH_SCHEME='https' ELASTICSEARCH_BATCH_SIZE='10' ELASTICSEARCH_HOST='vpc-..es.amazonaws.com' ELASTICSEARCH_PASSWORD='' ELASTICSEARCH_CLUSTER_ALIAS='' ``` ### Configure OpenSearch ``` # ElasticSearch Configurations SEARCH_TYPE="opensearch" ELASTICSEARCH_HOST="" ELASTICSEARCH_PORT="" ELASTICSEARCH_SCHEME="" ELASTICSEARCH_USER="" ELASTICSEARCH_PASSWORD="" ELASTICSEARCH_CLUSTER_ALIAS="" ``` If you want to separate indexes for production and non-production environments, you can set the `clusterAlias` in the configuration file. ### Configure Ingestion ``` PIPELINE_SERVICE_CLIENT_ENDPOINT="" PIPELINE_SERVICE_CLIENT_HEALTH_CHECK_INTERVAL="300" SERVER_HOST_API_URL="/api" PIPELINE_SERVICE_CLIENT_VERIFY_SSL="no-ssl" PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH="" PIPELINE_SERVICE_CLIENT_CLASS_NAME="org.openmetadata.service.clients.pipeline.airflow.AirflowRESTClient" PIPELINE_SERVICE_IP_INFO_ENABLED="false" PIPELINE_SERVICE_CLIENT_HOST_IP="" PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER="noop" AIRFLOW_USERNAME="" AIRFLOW_PASSWORD="" AIRFLOW_TIMEOUT="10" AIRFLOW_TRUST_STORE_PATH="" AIRFLOW_TRUST_STORE_PASSWORD="" ``` When setting up environment file if your custom password includes any special characters then make sure to follow the steps [here](https://github.com/open-metadata/OpenMetadata/issues/12110#issuecomment-1611341650). ## Troubleshooting ### Java Memory Heap Issue If your openmetadata application logs speaks about the below issue - ``` Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "AsyncAppender-Worker-async-file-appender" Exception in thread "pool-5-thread-1" java.lang.OutOfMemoryError: Java heap space Exception in thread "AsyncAppender-Worker-async-file-appender" java.lang.OutOfMemoryError: Java heap space Exception in thread "dw-46" java.lang.OutOfMemoryError: Java heap space Exception in thread "AsyncAppender-Worker-async-console-appender" java.lang.OutOfMemoryError: Java heap space ``` This is due to the default JVM Heap Space configuration (1 GiB) being not enough for your workloads. In order to resolve this issue, head over to your openmetadata environment variables list and append the below environment variable ``` # environment variable file (either .bash_profile or .bashrc or add in conf/openmetadata-env.sh in release binaries) export OPENMETADATA_HEAP_OPTS="-Xmx2G -Xms2G" ``` The flag `Xmx` specifies the maximum memory allocation pool for a Java virtual machine (JVM), while `Xms` specifies the initial memory allocation pool. Restart the OpenMetadata Application using `./bin/openmetadata.sh start` which will start the service using a linux process. ## Enable Security Please follow our [Enable Security Guide](/v2.0.x/deployment/bare-metal/security) to configure security for your OpenMetadata installation. # Enable Security | OpenMetadata Deployment Security Guide Source: https://docs.open-metadata.org/v2.0.x/deployment/bare-metal/security Secure bare-metal deployments with guidance on encryption, authentication, and secret management for non-cloud environments. # Bare Metal Security Follow the steps for setting up the SSO, and then check the specific `Bare Metal` section of your chosen SSO. By default, Basic Authentication will be enabled as authentication mechanism. Configure Basic Authentication to access the UI and APIs Configure Ldap Authentication to access the UI and APIs Configure Auth0 SSO to access the UI and APIs Configure Azure SSO to access the UI and APIs Configure a Custom OIDC SSO to access the UI and APIs Configure Google SSO to access the UI and APIs Configure Okta SSO to access the UI and APIs Configure Amazon Cognito SSO to access the UI and APIs Configure OneLogin SSO to access the UI and APIs Configure Keycloak SSO to access the UI and APIs # Configuring OpenMetadata to Run Under a Subpath Source: https://docs.open-metadata.org/v2.0.x/deployment/bare-metal/subpath ## Subpath in OpenMetadata To configure **OpenMetadata** to operate under a subpath (for example `/openmetadata`), useful when deploying behind a reverse proxy or load balancer, you need to adjust specific settings in the `openmetadata.yaml` configuration file. **`BASE_PATH` must not have a trailing slash**, but `basePath` in `openmetadata.yaml` needs one. OpenMetadata builds static asset URLs by appending directly to `basePath` with no separator, so if it's missing the trailing slash, requests resolve to `/openmetadataassets/...` instead of `/openmetadata/assets/...` and every static asset 404s. ## Configuration Steps ### 1. Set the Base Path Define the `basePath` parameter to configure the application's root context, and ensure that the `publicKeyUrl` is updated accordingly to reflect the new base path. This sets the root context for the application. Note the trailing slash on `basePath`: ```yaml theme={null} basePath: ${BASE_PATH:-/openmetadata}/ ``` This configuration sets the base path to /openmetadata by default. You can override it by setting the BASE\_PATH environment variable: set `BASE_PATH` itself **without** a trailing slash (e.g., `BASE_PATH=/openmetadata`), since the `/` above is appended for you. ### 2. Configure Web Paths Configure the web application and API endpoint paths to align with the specified base path. These settings live under the top-level `server:` block: ```yaml theme={null} server: applicationContextPath: ${BASE_PATH:-/openmetadata} rootPath: ${BASE_PATH:-/openmetadata}api/* ``` * `applicationContextPath`: Defines the context path for the web application. * `rootPath`: Specifies the root path for API endpoints. [GitHub](https://github.com/open-metadata/OpenMetadata/discussions/17954) ### 3. Set Asset Paths Ensure that asset paths are correctly prefixed with the base path. ```yaml theme={null} assets: resourcePath: /openmetadata/assets/ uriPath: ${BASE_PATH:-/openmetadata} ``` * `resourcePath`: Path to static resources. * `uriPath`: URI path prefix for assets. Subpath ## Example Configuration Here's how the relevant section of your `openmetadata.yaml` might look: ```yaml theme={null} basePath: ${BASE_PATH:-/openmetadata}/ publicKeyUrl: ${BASE_PATH:-/}api/v1/system/config/jwks server: applicationContextPath: ${BASE_PATH:-/openmetadata} rootPath: ${BASE_PATH:-/openmetadata}api/* assets: resourcePath: /openmetadata/assets/ uriPath: ${BASE_PATH:-/openmetadata} ``` ## Deployment Considerations * **Reverse Proxy Configuration**: Ensure that your reverse proxy (e.g., NGINX, Apache) is configured to forward requests to the OpenMetadata application with the correct subpath. * **Environment Variables**: You can override the default base path by setting the BASE\_PATH environment variable in your deployment environment. Ensure that related parameters such as basePath, applicationContextPath, rootPath, and publicKeyUrl are updated to reflect this change. * **Static Assets**: Verify that static assets are accessible under the new subpath to prevent broken links or missing resources. # Server Configuration Reference | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/configuration Configure your platform deployment using environmental settings, secret values, service parameters, and performance tuning options. # Server Configuration Reference This document describes OpenMetadata Server Configuration ```yaml theme={null} swagger: resourcePackage: org.openmetadata.service.resources server: rootPath: '/api/*' applicationConnectors: - type: http port: 8585 adminConnectors: - type: http port: 8586 # Logging settings. # https://logback.qos.ch/manual/layouts.html#conversionWord logging: level: INFO loggers: org.openmetadata.service.common: DEBUG io.swagger: ERROR appenders: - type: file threshold: TRACE logFormat: "%level [%d{HH:mm:ss.SSS}] [%t] %logger{5} - %msg %n" currentLogFilename: ./logs/openmetadata.log archivedLogFilenamePattern: ./logs/openmetadata-%d{yyyy-MM-dd}-%i.log.gz archivedFileCount: 7 timeZone: UTC maxFileSize: 50MB database: # the name of the JDBC driver, mysql in our case driverClass: com.mysql.cj.jdbc.Driver # the username and password user: openmetadata_user password: openmetadata_password # the JDBC URL; the database is called openmetadata_db url: jdbc:mysql://localhost/openmetadata_db?useSSL=false&serverTimezone=UTC elasticsearch: host: localhost port: 9200 eventHandlerConfiguration: eventHandlerClassNames: - "org.openmetadata.service.events.AuditEventHandler" health: delayedShutdownHandlerEnabled: true shutdownWaitPeriod: 1s healthCheckUrlPaths: ["/api/v1/health-check"] healthChecks: - name: UserDatabaseCheck critical: true schedule: checkInterval: 2500ms downtimeInterval: 10s failureAttempts: 2 successAttempts: 1 ``` ## Server Port ```yaml theme={null} server: rootPath: '/api/*' applicationConnectors: - type: http port: 8585 adminConnectors: - type: http port: 8586 ``` By default, the OpenMetadata server runs on port 8585. It uses Jetty Server. The above config can be changed to make it run on a different port. Once you have updated the port details in config restart the server. ## Database OpenMetadata supports MySQL or Postgres as the database. The database configurations and connection strings must be as specified below. ### MySQL Configuration We recommend you create a MySQL user with a strong password and update this section accordingly. | Parameter | Description | Default Value | | --------------- | --------------------------- | ------------------------------------------------------------------------ | | **driverClass** | The name of the JDBC driver | `com.mysql.cj.jdbc.Driver` | | **user** | MySQL database username | `openmetadata_user` | | **password** | MySQL database password | `openmetadata_password` | | **url** | JDBC URL connection string | `jdbc:mysql://localhost/openmetadata_db?useSSL=false&serverTimezone=UTC` | ```yaml theme={null} database: driverClass: com.mysql.cj.jdbc.Driver user: openmetadata_user password: openmetadata_password url: jdbc:mysql://localhost/openmetadata_db?useSSL=false&serverTimezone=UTC ``` ### PostgreSQL Configuration OpenMetadata uses stored generated columns (supported since Postgres 12). We recommend running Postgres 15 or higher. Create a Postgres user with a strong password and update this section accordingly. | Parameter | Description | Default Value | | --------------- | ---------------------------- | ---------------------------------------------------------------------------------- | | **driverClass** | The name of the JDBC driver | `org.postgresql.Driver` | | **user** | PostgreSQL database username | `openmetadata_user` | | **password** | PostgreSQL database password | `openmetadata_password` | | **url** | JDBC URL connection string | `jdbc:postgresql://localhost:5432/openmetadata_db?useSSL=false&serverTimezone=UTC` | ```yaml theme={null} database: driverClass: org.postgresql.Driver user: openmetadata_user password: openmetadata_password url: jdbc:postgresql://localhost:5432/openmetadata_db?useSSL=false&serverTimezone=UTC ``` ## ElasticSearch ```yaml theme={null} elasticsearch: host: localhost port: 9200 ``` ElasticSearch is one of the pre-requisites to run OpenMetadata. Default configuration expects a single instance of ElasticSearch running on the local machine. Please make sure you update it with your production elastic search. ## Event Handlers ```yaml theme={null} eventHandlerConfiguration: eventHandlerClassNames: - "org.openmetadata.service.events.AuditEventHandler" ``` EventHandler configuration is optional. It will update the AuditLog in MySQL DB and also ElasticSearch indexes whenever any entity is updated either through UI or API interactions. We recommend you leave it there as it enhances the user experience. ## Healthcheck ```yaml theme={null} health: delayedShutdownHandlerEnabled: true shutdownWaitPeriod: 1s healthCheckUrlPaths: ["/api/v1/health-check"] healthChecks: - name: UserDatabaseCheck critical: true schedule: checkInterval: 2500ms downtimeInterval: 10s failureAttempts: 2 successAttempts: 1 ``` Healthcheck API provides an API endpoint to check the OpenMetadata server health. We recommend in production settings to use this API to monitor the health of your OpenMetadata instance. Please tune the above configuration according to your production needs. ## Security Please follow our [Enable Security Guide](/v2.0.x/deployment/security) to configure security for your OpenMetadata installation. # Database Connection Pooling Source: https://docs.open-metadata.org/v2.0.x/deployment/database-connection-pooling Optimize your OpenMetadata deployment with database connection pooling. Learn configuration best practices, performance tuning, and setup guides. # Database Connection Pool Dropwizard JDBI provides connection pooling by default. Enabling and properly configuring connection pooling ensures that each database query does not open a new connection but instead utilizes a pool of reusable connections. This enhances application performance, reduces latency, and efficiently manages database resources. Database connection pooling is a technique used to maintain a cache of database connections that can be reused for future requests. This approach minimizes the overhead associated with establishing a new connection for each request, leading to improved performance and resource utilization. In the context of Dropwizard JDBI, enabling and configuring connection pooling ensures that your application can handle multiple database operations efficiently, especially under high load conditions. * [Why Use a Database Connection Pool?](#why-use-a-database-connection-pool) * [Configuration Parameters](#configuration-parameters) * [Best Practices](#best-practices-for-database-connection-pooling) ## Why Use a Database Connection Pool? * **Performance Improvement**: Reusing existing connections reduces the time required to establish new connections, leading to faster query execution. * **Resource Optimization**: Limits the number of open connections to the database, preventing resource exhaustion. * **Scalability**: Efficiently handles increasing numbers of database requests by managing connections effectively. * **Stability**: Reduces the risk of connection timeouts and failures by maintaining a pool of healthy connections. ## Configuration Parameters The following configuration parameters control the behavior of the database connection pool in Dropwizard JDBI: | Parameter Name | Description | Environment Variable | Default Value | | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | ------------- | | **maxSize** | Specifies the maximum number of connections that can be active in the pool at any given time. Determines the upper limit of concurrent database connections. Should be set based on expected workload and database server capabilities. | `DB_CONNECTION_POOL_MAX_SIZE` | `50` | | **minSize** | Defines the minimum number of idle connections that the pool tries to maintain. Ensures connections are always ready to serve incoming requests and helps reduce latency for initial requests after periods of inactivity. | `DB_CONNECTION_POOL_MIN_SIZE` | `10` | | **initialSize** | Sets the number of connections created when the pool is initialized. Determines how many connections are available immediately after startup. Typically aligned with `minSize` for consistency. | `DB_CONNECTION_POOL_INITIAL_SIZE` | `10` | | **checkConnectionWhileIdle** | Indicates whether idle connections should be validated periodically to ensure they are still alive. Helps detect and remove stale or broken connections from the pool, maintaining pool health over time. | `DB_CONNECTION_CHECK_CONNECTION_WHILE_IDLE` | `true` | | **checkConnectionOnBorrow** | Determines whether a connection should be validated before being handed over to a client. Ensures clients receive only valid and live connections, preventing runtime errors from broken connections. | `DB_CONNECTION_CHECK_CONNECTION_ON_BORROW` | `true` | | **evictionInterval** | Specifies the interval at which idle connections are checked and evicted if necessary. Controls how frequently the pool checks for idle connections to remove. Works with `minIdleTime` to maintain optimal pool size. | `DB_CONNECTION_EVICTION_INTERVAL` | `5 minutes` | | **minIdleTime** | Defines the minimum amount of time a connection can remain idle before it is eligible for eviction. Helps balance resource usage by removing unnecessary idle connections. | `DB_CONNECTION_MIN_IDLE_TIME` | `1 minute` | ```yaml theme={null} maxSize: ${DB_CONNECTION_POOL_MAX_SIZE:-50} minSize: ${DB_CONNECTION_POOL_MIN_SIZE:-10} initialSize: ${DB_CONNECTION_POOL_INITIAL_SIZE:-10} checkConnectionWhileIdle: ${DB_CONNECTION_CHECK_CONNECTION_WHILE_IDLE:-true} checkConnectionOnBorrow: ${DB_CONNECTION_CHECK_CONNECTION_ON_BORROW:-true} evictionInterval: ${DB_CONNECTION_EVICTION_INTERVAL:-5 minutes} minIdleTime: ${DB_CONNECTION_MIN_IDLE_TIME:-1 minute} ``` ## Best Practices for Database Connection Pooling To ensure that your database connection pooling is optimized for performance, reliability, and resource management, consider the following best practices: ## 1. Monitor and Adjust * **Regular Monitoring**: Continuously monitor your application’s performance and database load to understand how your connection pool is performing. * **Adjust Pool Sizes**: Based on monitoring data, adjust the pool size parameters (`maxSize`, `minSize`, `initialSize`) to match your workload needs. This helps in avoiding resource bottlenecks or wastage. ## 2. Understand Workload Patterns * **Peak vs. Off-Peak**: Identify peak and off-peak usage times in your application. Configure your pool to handle peak loads effectively while conserving resources during off-peak times. * **Dynamic Scaling**: Consider implementing dynamic scaling of the connection pool size if your environment supports it. This allows the pool to grow and shrink in response to actual demand. ## 3. Ensure Database Capacity * **Max Connections**: Verify that your database server can support the maximum number of connections specified by `maxSize`. Exceeding the database’s capacity can lead to connection failures and degraded performance. * **Avoid Over-Configuration**: Setting `maxSize` too high can overwhelm your database, while setting it too low can result in connection shortages under high load. Balance is key. ## 4. Use Validation Checks * **Enable `checkConnectionWhileIdle`**: This ensures that idle connections are periodically validated, preventing broken connections from remaining in the pool. It improves the reliability of the connection pool. * **Enable `checkConnectionOnBorrow`**: Validate connections before handing them over to the application. This reduces the risk of runtime errors due to stale or broken connections. ## 5. Configure Timeouts Appropriately * **Eviction Interval**: Set `evictionInterval` to an appropriate value that balances the frequency of idle connection checks with the overhead of performing these checks. * **Idle Time**: Adjust `minIdleTime` based on typical usage patterns. This ensures that connections are retained for as long as they are likely to be needed, without wasting resources on keeping them idle for too long. ## 6. Consider Connection Pool Size Based on Application Behavior * **Transactional Applications**: For applications with short, frequent transactions, a larger pool size might be necessary to handle the high concurrency. * **Long-Lived Connections**: If your application tends to hold connections open for extended periods, ensure that your pool is large enough to accommodate other incoming requests without running out of connections. ## 7. Review and Test Configuration Changes * **Staging Environment Testing**: Before deploying changes to production, test any configuration changes in a staging environment that closely mirrors production. This helps to catch potential issues early. * **Review Logs and Metrics**: After making changes, review your application logs and database metrics to ensure that the new configuration is performing as expected. ## 8. Use Connection Pooling Libraries Effectively * **Leverage Built-In Features**: Use features provided by your connection pooling library, such as connection leak detection, to ensure optimal usage of your pool. * **Stay Updated**: Keep your connection pooling library up to date with the latest versions to benefit from performance improvements and security fixes. By following these best practices, you can ensure that your database connection pool is configured for optimal performance, reliability, and resource efficiency, resulting in a more stable and responsive application. # Docker Deployment | OpenMetadata Container Setup Source: https://docs.open-metadata.org/v2.0.x/deployment/docker Deploy the platform using Docker containers to simplify setup, scaling, and local testing without needing external dependencies. # Docker Deployment This guide will help you set up the OpenMetadata Application using Docker Deployment. Before starting with the deployment make sure you follow all the below Prerequisites. ## Docker Deployment Architecture Docker Deployment Architecture ## Prerequisites ### Configure OpenMetadata to use External Database and Search Engine For Production Deployment using Docker, we recommend bringing your own Databases and ElasticSearch Engine and not rely on quickstart packages. ### Configure External Orchestrator Service (Ingestion Service) OpenMetadata requires connectors to be scheduled to periodically fetch the metadata, or you can use the OpenMetadata APIs to push the metadata as well 1. OpenMetadata Ingestion Framework is flexible to run on any orchestrator. However, we built an ability to deploy and manage connectors as pipelines from the UI. This requires the Airflow container we ship. 2. If your team prefers to run on any other orchestrator such as prefect, dagster or even GitHub workflows. Please refer to our recent webinar on [How Ingestion Framework works](https://www.youtube.com/watch?v=i7DhG_gZMmE\&list=PLa1l-WDhLreslIS_96s_DT_KdcDyU_Itv\&index=10) ### Docker (version 20.10.0 or higher) [Docker](https://docs.docker.com/get-started/overview/) is an open-source platform for developing, shipping, and running applications. It enables you to separate your applications from your infrastructure, so you can deliver software quickly using OS-level virtualization. It helps deliver software in packages called Containers. To check what version of Docker you have, please use the following command. ```commandline theme={null} docker --version ``` If you need to install Docker, please visit [Get Docker](https://docs.docker.com/get-docker/). ### Docker Compose (version v2.2.3 or greater) The Docker compose package enables you to define and run multi-container Docker applications. The compose command integrates compose functions into the Docker platform, making them available from the Docker command-line interface ( CLI). The Python packages you will install in the procedure below use compose to deploy OpenMetadata. * **MacOS X**: Docker on MacOS X ships with compose already available in the Docker CLI. * **Linux**: To install compose on Linux systems, please visit the Docker CLI command documentation and follow the instructions. To verify that the docker compose command is installed and accessible on your system, run the following command. ```commandline theme={null} docker compose version ``` Upon running this command you should see output similar to the following. ```commandline theme={null} Docker Compose version v2.2.3 ``` #### Install Docker Compose Version 2 on Linux Follow the [Docker Compose installation instructions](https://docs.docker.com/compose/install/linux/) to install Docker Compose version 2. 1. Run the following command to download the current stable release of Docker Compose ``` DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker} mkdir -p $DOCKER_CONFIG/cli-plugins curl -SL https://github.com/docker/compose/releases/download/v2.2.3/docker-compose-linux-x86_64 -o $DOCKER_CONFIG/cli-plugins/docker-compose ``` This command installs Compose V2 for the active user under \$HOME directory. To install Docker Compose for all users on your system, replace` ~/.docker/cli-plugins` with `/usr/local/lib/docker/cli-plugins`. 2. Apply executable permissions to the binary ``` chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose ``` 3. Test your installation ``` docker compose version > Docker Compose version v2.2.3 ``` ## Steps for Deploying OpenMetadata using Docker ### 1. Create a directory for OpenMetadata Create a new directory for OpenMetadata and navigate into that directory. ```commandline theme={null} mkdir openmetadata-docker && cd openmetadata-docker ``` ### 2. Download Docker Compose Files from GitHub Releases Download the Docker Compose files from the [Latest GitHub Releases](https://github.com/open-metadata/OpenMetadata/releases/latest). The Docker compose file name will be `docker-compose-openmetadata.yml`. This docker compose file contains only the docker compose services for OpenMetadata Server. Bring up the dependencies as mentioned in the [prerequisites](#configure-openmetadata-to-use-external-database-and-search-engine) section. You can also run the below command to fetch the docker compose file directly from the terminal - ```bash theme={null} wget https://github.com/open-metadata/OpenMetadata/releases/download/2.0.1-release/docker-compose-openmetadata.yml ``` ### 3. Update Environment Variables required for OpenMetadata Dependencies In the previous [step](#2-download-docker-compose-files-from-github-releases), we download the `docker-compose` file. Identify and update the environment variables in the file to prepare openmetadata configurations. For MySQL Configurations, update the below environment variables - ```bash theme={null} ... # Database configuration for MySQL DB_DRIVER_CLASS="com.mysql.cj.jdbc.Driver" DB_SCHEME="mysql" DB_PARAMS="allowPublicKeyRetrieval=true&useSSL=true&serverTimezone=UTC" DB_USER="" DB_USER_PASSWORD="" DB_HOST="" DB_PORT="" OM_DATABASE="" ``` For ElasticSearch Configurations, update the below environment variables - ```bash theme={null} # ElasticSearch Configurations SEARCH_TYPE="elasticsearch" ELASTICSEARCH_HOST="" ELASTICSEARCH_PORT="" ELASTICSEARCH_SCHEME="" ELASTICSEARCH_USER="" ELASTICSEARCH_PASSWORD="" ELASTICSEARCH_CLUSTER_ALIAS="" ``` For OpenSearch Configurations, update the below environment variables - ```bash theme={null} # ElasticSearch Configurations SEARCH_TYPE="opensearch" ELASTICSEARCH_HOST="" ELASTICSEARCH_PORT="" ELASTICSEARCH_SCHEME="" ELASTICSEARCH_USER="" ELASTICSEARCH_PASSWORD="" ELASTICSEARCH_CLUSTER_ALIAS="" ``` If you want to separate indexes for production and non-production environments, you can set the `clusterAlias` in the configuration file. For Ingestion Configurations, update the below environment variables - ```bash theme={null} PIPELINE_SERVICE_CLIENT_ENDPOINT="" PIPELINE_SERVICE_CLIENT_HEALTH_CHECK_INTERVAL="300" SERVER_HOST_API_URL="/api" PIPELINE_SERVICE_CLIENT_VERIFY_SSL="no-ssl" PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH="" PIPELINE_SERVICE_CLIENT_CLASS_NAME="org.openmetadata.service.clients.pipeline.airflow.AirflowRESTClient" PIPELINE_SERVICE_IP_INFO_ENABLED="false" PIPELINE_SERVICE_CLIENT_HOST_IP="" PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER="noop" AIRFLOW_USERNAME="" AIRFLOW_PASSWORD="" AIRFLOW_TIMEOUT="10" AIRFLOW_TRUST_STORE_PATH="" AIRFLOW_TRUST_STORE_PASSWORD="" ``` When setting up environment file if your custom password includes any special characters then make sure to follow the steps [here](https://github.com/open-metadata/OpenMetadata/issues/12110#issuecomment-1611341650). ### 4. Start the Docker Compose Services Run the below command to deploy the OpenMetadata - ```bash theme={null} docker compose --env-file ./env-mysql up --detach ``` You can validate that all containers are up by running with command `docker ps`. ```commandline theme={null} ❯ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 470cc8149826 openmetadata/server:2.0.1 "./openmetadata-star…" 45 seconds ago Up 43 seconds 3306/tcp, 9200/tcp, 9300/tcp, 0.0.0.0:8585-8586->8585-8586/tcp openmetadata_server ``` In a few seconds, you should be able to access the OpenMetadata UI at `http://localhost:8585`. ## Next Steps For port mapping, load balancer setup, AWS services configuration, Docker volumes, and troubleshooting, see [Docker Advanced Configuration & Troubleshooting](/v2.0.x/deployment/docker/advanced). # Docker Advanced Configuration & Troubleshooting Source: https://docs.open-metadata.org/v2.0.x/deployment/docker/advanced Configure port mapping, load balancing, AWS services, volumes, and troubleshoot common Docker deployment issues for OpenMetadata. # Docker Advanced Configuration & Troubleshooting ## Port Mapping / Port Forwarding We are shipping the OpenMetadata server and UI at container port and host port `8585`. You can change the host port number according to your requirement. As an example, You could update the ports to serve OpenMetadata Server and UI at port `80` To achieve this - * You just have to update the ports mapping of the openmetadata-server in the `docker-compose.yml` file under `openmetadata-server` docker service section. ```yaml theme={null} --- ports: - "80:8585" ``` * Once the port is updated if there are any containers running remove them first using `docker compose down` command and then recreate the containers once again by below command ```commandline theme={null} docker compose up --detach ``` ## Run OpenMetadata with a load balancer You may put one or more OpenMetadata instances behind a load balancer for reverse proxying. To do this you will need to add one or more entries to the configuration file for your reverse proxy. ### Nginx To use OpenMetadata behind Nginx reverse proxy, add an entry resembling the following in the http context of your Nginx configuration file for each OpenMetadata instance. ``` server { access_log /var/log/nginx/stage-reverse-access.log; error_log /var/log/nginx/stage-reverse-error.log; server_name stage.open-metadata.org; location / { proxy_pass http://127.0.0.1:8585; } } ``` ## Run OpenMetadata with AWS Services If you are running OpenMetadata in AWS, it is recommended to use [Amazon RDS](https://docs.aws.amazon.com/rds/index.html) and [Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/?id=docs_gateway). We support * Amazon RDS (MySQL) engine version 8 or higher * Amazon OpenSearch (ElasticSearch) engine version 9.x (minimum 9.0.0, recommended 9.3.0) or Amazon OpenSearch engine version 3.x (minimum 3.0.0, recommended 3.3.0) * Amazon RDS (PostgreSQL) engine version 15 or higher Note:- When using AWS Services the SearchType Configuration for elastic search should be `opensearch`, for both cases ElasticSearch and OpenSearch, as you can see in the ElasticSearch configuration example. For Production Systems, we recommend Amazon RDS to be in Multiple Availability Zones. For Amazon OpenSearch (or ElasticSearch) Service, we recommend Multiple Availability Zones with minimum 3 Master Nodes. Once you have the RDS and OpenSearch Services Setup, you can update the environment variables below for OpenMetadata Docker Compose backed systems to connect with Database and ElasticSearch. ``` # MySQL Environment Variables DB_DRIVER_CLASS='com.mysql.cj.jdbc.Driver' DB_SCHEME='mysql' DB_PARAMS='allowPublicKeyRetrieval=true&useSSL=true&serverTimezone=UTC' DB_USER_PASSWORD='' DB_HOST='' DB_USER='' OM_DATABASE='' DB_PORT='' # ElasticSearch Environment Variables SEARCH_TYPE='opensearch' ELASTICSEARCH_SOCKET_TIMEOUT_SECS='60' ELASTICSEARCH_USER='' ELASTICSEARCH_CONNECTION_TIMEOUT_SECS='5' ELASTICSEARCH_PORT='443' ELASTICSEARCH_SCHEME='https' ELASTICSEARCH_BATCH_SIZE='10' ELASTICSEARCH_HOST='' ELASTICSEARCH_PASSWORD='' ELASTICSEARCH_CLUSTER_ALIAS='' ``` Replace the environment variables values with the RDS and OpenSearch Service ones and then provide this environment variable file as part of docker compose command. ```bash theme={null} docker compose --env-file ./env-mysql up --detach ``` ## Advanced ### Add Docker Volumes for OpenMetadata Server Compose Service There are many scenarios where you would want to provide additional files to the OpenMetadata Server and serve while running the application. In such scenarios, it is recommended to provision docker volumes for OpenMetadata Application. If you are not familiar with Docker Volumes with Docker Compose Services, Please refer to [official documentation](https://docs.docker.com/storage/volumes/#use-a-volume-with-docker-compose) for more information. For example, we would like to provide custom JWT Configuration Keys to be served to OpenMetadata Application. This requires the OpenMetadata Containers to have docker volumes sharing the private and public keys. Let's assume you have the keys available in `jwtkeys` directory in the same directory where your `docker-compose` file is available in the host machine. In scenarios where you need to provide a custom `openmetadata.yaml` configuration file to the OpenMetadata application, you can do so by mounting the file as a volume in the Docker container. This is especially useful for configurations that cannot be controlled through environment variables. We add the volumes section to mount the keys or `openmetadata.yaml` onto the docker containers create with docker compose as follows - ```yaml theme={null} services: openmetadata-server: ... volumes: - ./jwtkeys:/etc/openmetadata/jwtkeys - ./openmetadata.yaml:/opt/openmetadata/conf/openmetadata.yaml ... ``` The above example uses [bind mounts](https://docs.docker.com/storage/bind-mounts/#use-a-bind-mount-with-compose) to share files and directories between host machine and openmetadata container. Next, in your environment file, update the jwt configurations to use the right path from inside the container. ```bash theme={null} ... # JWT Configuration RSA_PUBLIC_KEY_FILE_PATH="/etc/openmetadata/jwtkeys/public_key.der" RSA_PRIVATE_KEY_FILE_PATH="/etc/openmetadata/jwtkeys/private_key.der" ... ``` Ensure that the default environment variables are set appropriately to complement the settings in your `openmetadata.yaml`. Once the changes are updated, if there are any containers running remove them first using `docker compose down` command and then recreate the containers once again by below command ```commandline theme={null} docker compose up --detach ``` ## Troubleshooting ### Java Memory Heap Issue If your openmetadata Docker Compose logs speaks about the below issue - ``` Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "AsyncAppender-Worker-async-file-appender" Exception in thread "pool-5-thread-1" java.lang.OutOfMemoryError: Java heap space Exception in thread "AsyncAppender-Worker-async-file-appender" java.lang.OutOfMemoryError: Java heap space Exception in thread "dw-46" java.lang.OutOfMemoryError: Java heap space Exception in thread "AsyncAppender-Worker-async-console-appender" java.lang.OutOfMemoryError: Java heap space ``` This is due to the default JVM Heap Space configuration (1 GiB) being not enough for your workloads. In order to resolve this issue, head over to your custom openmetadata environment variable file and append the below environment variable ``` #environment variable file OPENMETADATA_HEAP_OPTS="-Xmx2G -Xms2G" ``` The flag `Xmx` specifies the maximum memory allocation pool for a Java virtual machine (JVM), while `Xms` specifies the initial memory allocation pool. Restart the OpenMetadata Docker Compose Application using `docker compose --env-file -f docker-compose.yml up --detach` which will recreate the containers with new environment variable values you have provided. ### PostgreSQL Issue permission denied to create extension "pgcrypto" If you are facing the below issue with PostgreSQL as Database Backend for OpenMetadata Application, ``` Message: ERROR: permission denied to create extension "pgcrypto" Hint: Must be superuser to create this extension. ``` It seems the Database User does not have sufficient privileges. In order to resolve the above issue, grant usage permissions to the PSQL User. ```sql theme={null} GRANT USAGE ON SCHEMA schema_name TO ; GRANT CREATE ON EXTENSION pgcrypto TO ; ``` In the above command, replace `` with the sql user used by OpenMetadata Application to connect to PostgreSQL Database. In the above command, replace `` with the sql user used by OpenMetadata Application to connect to PostgreSQL Database. ## Security Please follow our [Enable Security Guide](/v2.0.x/deployment/docker/security) to configure security for your OpenMetadata installation. ## Next Steps 1. Refer the [How-to Guides](/v2.0.x/how-to-guides) for an overview of all the features in OpenMetadata. 2. Visit the [Connectors](/v2.0.x/connectors) documentation to see what services you can integrate with OpenMetadata. 3. Visit the [API](https://docs.open-metadata.org/api-reference) documentation and explore the rich set of OpenMetadata APIs. # Enable Security (Docker) | OpenMetadata Docker Security Source: https://docs.open-metadata.org/v2.0.x/deployment/docker/security Secure Docker-based deployments using best practices in network access, token handling, and identity provider integration. # Docker Security Follow the steps for setting up the SSO, and then check the specific `Docker` section of your chosen SSO. By default Basic Authentication will be enabled as authentication mechanism. Configure Basic Authentication to access the UI and APIs Configure Ldap Authentication to access the UI and APIs Configure Auth0 SSO to access the UI and APIs Configure Azure SSO to access the UI and APIs Configure a Custom OIDC SSO to access the UI and APIs Configure Google SSO to access the UI and APIs Configure Okta SSO to access the UI and APIs # Configuring OpenMetadata to Run Under a Subpath Source: https://docs.open-metadata.org/v2.0.x/deployment/docker/subpath ## Subpath in OpenMetadata To configure **OpenMetadata** to operate under a subpath (for example `/openmetadata`), useful when deploying behind a reverse proxy or load balancer, you need to adjust specific settings in the `openmetadata.yaml` configuration file. **`BASE_PATH` must not have a trailing slash**, but `basePath` in `openmetadata.yaml` needs one. OpenMetadata builds static asset URLs by appending directly to `basePath` with no separator, so if it's missing the trailing slash, requests resolve to `/openmetadataassets/...` instead of `/openmetadata/assets/...` and every static asset 404s. ## Configuration Steps ### 1. Set the Base Path Define the `basePath` parameter to configure the application's root context, and ensure that the `publicKeyUrl` is updated accordingly to reflect the new base path. This sets the root context for the application. Note the trailing slash on `basePath`: ```yaml theme={null} basePath: ${BASE_PATH:-/openmetadata}/ ``` This configuration sets the base path to /openmetadata by default. You can override it by setting the BASE\_PATH environment variable: set `BASE_PATH` itself **without** a trailing slash (e.g., `BASE_PATH=/openmetadata`), since the `/` above is appended for you. ### 2. Configure Web Paths Configure the web application and API endpoint paths to align with the specified base path. These settings live under the top-level `server:` block: ```yaml theme={null} server: applicationContextPath: ${BASE_PATH:-/openmetadata} rootPath: ${BASE_PATH:-/openmetadata}api/* ``` * `applicationContextPath`: Defines the context path for the web application. * `rootPath`: Specifies the root path for API endpoints. [GitHub](https://github.com/open-metadata/OpenMetadata/discussions/17954) ### 3. Set Asset Paths Ensure that asset paths are correctly prefixed with the base path. ```yaml theme={null} assets: resourcePath: /openmetadata/assets/ uriPath: ${BASE_PATH:-/openmetadata} ``` * `resourcePath`: Path to static resources. * `uriPath`: URI path prefix for assets. Subpath ## Example Configuration Here's how the relevant section of your `openmetadata.yaml` might look: ```yaml theme={null} basePath: ${BASE_PATH:-/openmetadata}/ publicKeyUrl: ${BASE_PATH:-/}api/v1/system/config/jwks server: applicationContextPath: ${BASE_PATH:-/openmetadata} rootPath: ${BASE_PATH:-/openmetadata}api/* assets: resourcePath: /openmetadata/assets/ uriPath: ${BASE_PATH:-/openmetadata} ``` ## Deployment Considerations * **Reverse Proxy Configuration**: Ensure that your reverse proxy (e.g., NGINX, Apache) is configured to forward requests to the OpenMetadata application with the correct subpath. * **Environment Variables**: You can override the default base path by setting the BASE\_PATH environment variable in your deployment environment. Ensure that related parameters such as basePath, applicationContextPath, rootPath, and publicKeyUrl are updated to reflect this change. * **Static Assets**: Verify that static assets are accessible under the new subpath to prevent broken links or missing resources. # How to enable AWS RDS IAM Auth | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/iam-auth Learn how to securely connect OpenMetadata to AWS RDS using IAM authentication with correct environment variables and configuration best practices. # Aws resources on RDS IAM Auth [AWS Reference Doc](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.html) ## Requirements 1. AWS RDS Cluster with IAM auth enabled 2. User on DB Cluster with IAM enabled 3. IAM policy with permission on RDS connect 4. Role with IAM policy attached 5. IAM role attached to an EC2 instance on which openmetadata is deployed or ServiceAccount/Kube2Iam role attached to pod. ## How to enable ADS RDS IAM Auth on postgresql Set the environment variables ```Commandline theme={null} DB_USER_PASSWORD: "dummy" DB_PARAMS: "awsRegion=eu-west-1&allowPublicKeyRetrieval=true&sslmode=require&serverTimezone=UTC" ``` Either through helm (if deployed in kubernetes) or as env vars. The `DB_USER_PASSWORD` is still required and cannot be empty. Set it to a random/dummy string. When using IAM authentication for AWS RDS, you must still provide a dummy value for the `DB_PASSWORD` environment variable. OpenMetadata automatically handles the IAM credentials internally. Ensure the following parameters are set for successful connection: * `DB_PARAMS=awsRegion=us-east-1&allowPublicKeyRetrieval=true&serverTimezone=UTC` * `DB_USE_SSL=true` These settings ensure proper token generation and secure communication with the RDS instance. # Ingestion Framework Deployment | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/ingestion Configure ingestion from external, cloud-native, or hybrid environments using deployment guides. # Ingestion Framework Deployment The Ingestion Framework is the module that takes care of bringing metadata in to OpenMetadata. It is used for any type of workflow that is supported in the platform: Metadata, Lineage, Usage, Profiler, Data Quality,... ## Manage & Schedule the Ingestion Framework In this guide, we will present the different alternatives to run and manage your ingestion workflows. There are mainly 2 ways of running the ingestion: 1. Internally, by managing the workflows from OpenMetadata. 2. Externally, by using any other tool capable of running Python code. Note that the end result is going to be the same. The only difference is that running the workflows internally, OpenMetadata will dynamically generate the processes that will perform the metadata extraction. If configuring the ingestion externally, you will be managing this processes directly on your platform of choice. ## Option 1 - From OpenMetadata If you want to learn how to configure your setup to run them from OpenMetadata, follow these guides: Deploy, configure and manage the ingestion workflows using Apache Airflow as the orchestrator. Run ingestion pipelines using native Kubernetes Jobs and CronJobs - no Airflow required. **New in 1.12**: The Kubernetes Native Orchestrator allows you to run ingestion pipelines directly as Kubernetes Jobs, eliminating the need for Apache Airflow. This is ideal for organizations that want to reduce infrastructure complexity while leveraging their existing Kubernetes cluster. ## Option 2 - Externally Any tool capable of running Python code can be used to configure the metadata extraction from your sources. In this section, we are going to give you some background on how the Ingestion Framework works, how to configure the metadata extraction, and some examples on how to host the ingestion in different platforms. Manage the Ingestion Framework from anywhere! # Run the Ingestion Framework Externally Source: https://docs.open-metadata.org/v2.0.x/deployment/ingestion/external Configure external ingestion endpoints to trigger metadata pipelines from orchestrators or custom scripts. # Ingestion Framework External Deployment Any tool capable of running Python code can be used to configure the metadata extraction from your sources. ## 1. How does the Ingestion Framework work? The Ingestion Framework contains all the logic about how to connect to the sources, extract their metadata and send it to the OpenMetadata server. We have built it from scratch with the main idea of making it an independent component that can be run from - **literally** - anywhere. In order to install it, you just need to get it from [PyPI](https://pypi.org/project/openmetadata-ingestion/). ```shell theme={null} pip install openmetadata-ingestion ``` We will show further examples later, but a piece of code is the best showcase for its simplicity. In order to run a full ingestion process, you just need to execute a single function. For example, if we wanted to run the metadata ingestion from within a simple Python script: ```python theme={null} from metadata.workflow.metadata import MetadataWorkflow # Specify your YAML configuration CONFIG = """ source: ... workflowConfig: openMetadataServerConfig: hostPort: 'http://localhost:8585/api' authProvider: openmetadata securityConfig: jwtToken: ... """ def run(): workflow_config = yaml.safe_load(CONFIG) workflow = MetadataWorkflow.create(workflow_config) workflow.execute() workflow.raise_from_status() workflow.print_status() workflow.stop() if __name__ == "__main__": run() ``` Where this function runs is completely up to you, and you can adapt it to what makes the most sense within your organization and engineering context. Below you'll see some examples of different orchestrators you can leverage to execute the ingestion process. ## 2. Ingestion Configuration In the example above, the `Workflow` class got created from a YAML configuration. Any Workflow that you execute (ingestion, profiler, lineage,...) will have its own YAML representation. You can think about this configuration as the recipe you want to execute: where is your source, which pieces do you extract, how are they processed and where are they sent. An example YAML config for extracting MySQL metadata looks like this: ```yaml theme={null} source: type: mysql serviceName: mysql serviceConnection: config: type: Mysql username: openmetadata_user authType: password: openmetadata_password hostPort: localhost:3306 databaseSchema: openmetadata_db sourceConfig: config: type: DatabaseMetadata sink: type: metadata-rest config: {} workflowConfig: openMetadataServerConfig: hostPort: 'http://localhost:8585/api' authProvider: openmetadata securityConfig: jwtToken: ... ``` You will find examples of all the workflow's YAML files at each Connector [page](/v2.0.x/connectors). We will now show you examples on how to configure and run every workflow externally by using Snowflake as an example. But first, let's digest some information that will be common everywhere, the `workflowConfig`. ### Workflow Config Here you will define information such as where are you hosting the OpenMetadata server, and the JWT token to authenticate. Review this section carefully to ensure you are properly managing service credentials and other security configurations. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](/v2.0.x/deployment/security/enable-jwt-tokens). You can refer to the JWT Troubleshooting section [link](/v2.0.x/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](/v2.0.x/deployment/secrets-manager). If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Secrets Manager Configuration** If you have configured any [Secrets Manager](/v2.0.x/deployment/secrets-manager), you need to let the Ingestion Framework know how to retrieve the credentials securely. Follow the [docs](/v2.0.x/deployment/secrets-manager) to configure the secret retrieval based on your environment. **SSL Configuration** If you have added SSL to the [OpenMetadata server](/v2.0.x/deployment/security/enable-ssl), then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](/v2.0.x/deployment/security/enable-ssl/ssl-troubleshooting). ```yaml theme={null} workflowConfig: loggerLevel: INFO # DEBUG, INFO, WARNING or ERROR openMetadataServerConfig: hostPort: "http://localhost:8585/api" authProvider: openmetadata securityConfig: jwtToken: "{bot_jwt_token}" ## Store the service Connection information # storeServiceConnection: true or false ## Secrets Manager Configuration # secretsManagerProvider: aws, azure or noop # secretsManagerLoader: airflow or env ## If SSL, fill the following # verifySSL: validate # or ignore # sslConfig: # caCertificate: /local/path/to/certificate ``` #### JWT Token with Secrets Manager If you are using the [Secrets Manager](/v2.0.x/deployment/secrets-manager), you can let the Ingestion client to pick up the JWT Token dynamically from the Secrets Manager at runtime. Let's show an example: We have an OpenMetadata server running with the `managed-aws` Secrets Manager. Since we used the `OPENMETADATA_CLUSTER_NAME` env var as `test`, our `ingestion-bot` JWT Token is safely stored under the secret ID ` /test/bot/ingestion-bot/config/jwttoken`. Now, we can use the following workflow config to run the ingestion without having to pass the token, but just pointing to the secret itself: ```yaml theme={null} workflowConfig: loggerLevel: INFO # DEBUG, INFO, WARNING or ERROR openMetadataServerConfig: hostPort: "http://localhost:8585/api" authProvider: openmetadata securityConfig: jwtToken: "secret:/test/bot/ingestion-bot/config/jwttoken" secretsManagerProvider: aws secretsManagerLoader: env ``` Notice how: 1. We specify the `secretsManagerProvider` pointing to `aws`, since that's the manager we are using. 2. We set `secretsManagerLoader` as `env`. Since we're running this from our local, we'll let the AWS credentials to be loaded from the local env vars. (When running this using the UI, note that the generated workflows will have this value set as `airflow`!) 3. We set the `jwtToken` value as `secret:/test/bot/ingestion-bot/config/jwttoken`, which tells the client that this value is a `secret` located under `/test/bot/ingestion-bot/config/jwttoken`. Those are our env vars: ``` export AWS_ACCESS_KEY_ID=... export AWS_SECRET_ACCESS_KEY=... export AWS_DEFAULT_REGION=... ``` And we can run this normally with `metadata ingest -c `. Note that **even if you are not using the Secrets Manager for the OpenMetadata Server**, you can still apply the same approach by storing the JWT token manually to the secrets manager, and let the Ingestion client pick it up from there automatically. ## 3. (Optional) Ingestion Pipeline Additionally, if you want to see your runs logged in the `Ingestions` tab of the connectors page in the UI as you would when running the connectors natively with OpenMetadata, you can add the following configuration on your YAMLs: ```yaml theme={null} source: type: mysql serviceName: mysql [...] workflowConfig: openMetadataServerConfig: hostPort: 'http://localhost:8585/api' authProvider: openmetadata securityConfig: jwtToken: ... ingestionPipelineFQN: . # E.g., mysql.marketing_metadata` ``` Adding the `ingestionPipelineFQN` - the Ingestion Pipeline Fully Qualified Name - will tell the Ingestion Framework to log the executions and update the ingestion status, which will appear on the UI. Note that the action buttons will be disabled, since OpenMetadata won't be able to interact with external systems. ## 4. (Optional) Disable the Pipeline Service Client If you want to run your workflows **ONLY externally** without relying on OpenMetadata for any workflow management or scheduling, you can update the following server configuration: ```yaml theme={null} pipelineServiceClientConfiguration: enabled: ${PIPELINE_SERVICE_CLIENT_ENABLED:-true} ``` by setting `enabled: false` or setting the `PIPELINE_SERVICE_CLIENT_ENABLED=false` as an environment variable. This will stop certain APIs and monitors related to the Pipeline Service Client (e.g., Airflow) from being operative. ## Examples This is not an exhaustive list, and it will keep growing over time. Not because the orchestrators X or Y are not supported, but just because we did not have the time yet to add it here. If you'd like to chip in and help us expand these guides and examples, don't hesitate to reach to us in [Slack](https://slack.open-metadata.org/) or directly open a PR in [GitHub](https://github.com/open-metadata/docs-v1/tree/main/content). Run the ingestion process externally from Airflow Run the ingestion process externally using AWS MWAA Run the ingestion process externally from GCP Composer Run the ingestion process externally from GitHub Actions For code examples running Metadata, Lineage, Usage, Profiler, and Data Quality workflows externally, see [Workflow Examples](/v2.0.x/deployment/ingestion/external/examples). # Run the ingestion from your Airflow Source: https://docs.open-metadata.org/v2.0.x/deployment/ingestion/external/airflow Deploy ingestion externally using Airflow for scalable orchestration of metadata pipelines across environments. This page is about running the Ingestion Framework **externally**! There are mainly 2 ways of running the ingestion: 1. Internally, by managing the workflows from OpenMetadata. 2. Externally, by using any other tool capable of running Python code. If you are looking for how to manage the ingestion process from OpenMetadata, you can follow this [doc](/deployment/ingestion/openmetadata). # Run the ingestion from your Airflow OpenMetadata integrates with Airflow to orchestrate ingestion workflows. You can use Airflow to [extract metadata](/v2.0.x/connectors/pipeline/airflow) and \[deploy workflows] (/deployment/ingestion/openmetadata) directly. This guide explains how to run ingestion workflows in Airflow using three different operators: 1. [Python Operator](#using-the-python-operator) 2. [Docker Operator](/v2.0.x/deployment/ingestion/external/airflow-docker-virtualenv#using-the-docker-operator) 3. [Python Virtualenv Operator](/v2.0.x/deployment/ingestion/external/airflow-docker-virtualenv#using-the-python-virtualenv-operator) ## Using the Python Operator ### Prerequisites Install the `openmetadata-ingestion` package in your Airflow environment. This approach works best if you have access to the Airflow host and can manage dependencies. #### Installation Command: ``` pip3 install openmetadata-ingestion[<plugin>]==x.y.z ``` -Replace [\](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/setup.py) with the sources to ingest, such as mysql, snowflake, or s3. -Replace x.y.z with the OpenMetadata version matching your server (e.g., 2.0.1.0). ### Example ``` pip3 install openmetadata-ingestion[mysql,snowflake,s3]==2.0.1.0 ``` ### Example DAG ```python theme={null} import yaml from datetime import timedelta from airflow import DAG try: from airflow.operators.python import PythonOperator except ModuleNotFoundError: from airflow.operators.python_operator import PythonOperator from metadata.config.common import load_config_file from metadata.workflow.metadata import MetadataWorkflow from airflow.utils.dates import days_ago default_args = { "owner": "user_name", "email": ["username@org.com"], "email_on_failure": False, "retries": 3, "retry_delay": timedelta(minutes=5), "execution_timeout": timedelta(minutes=60) } config = """ """ def metadata_ingestion_workflow(): workflow_config = yaml.safe_load(config) workflow = MetadataWorkflow.create(workflow_config) workflow.execute() workflow.raise_from_status() workflow.print_status() workflow.stop() with DAG( "sample_data", default_args=default_args, description="An example DAG which runs a OpenMetadata ingestion workflow", start_date=days_ago(1), is_paused_upon_creation=False, schedule_interval='*/5 * * * *', catchup=False, ) as dag: ingest_task = PythonOperator( task_id="ingest_using_recipe", python_callable=metadata_ingestion_workflow, ) ``` ### Key Notes * **Function Setup**: The `python_callable` argument in the `PythonOperator` executes the `metadata_ingestion_workflow` function, which instantiates the workflow and runs the ingestion process. * **Drawback**: This method requires pre-installed dependencies, which may not always be feasible. Consider using the **DockerOperator** or **PythonVirtualenvOperator** as alternatives. ## Next Steps Run ingestion using the Docker Operator or Python Virtualenv Operator for isolated, dependency-free execution. # Airflow Docker & Virtualenv Operators | OpenMetadata Ingestion Source: https://docs.open-metadata.org/v2.0.x/deployment/ingestion/external/airflow-docker-virtualenv Run OpenMetadata ingestion workflows in Airflow using the Docker Operator or Python Virtualenv Operator for isolated, dependency-free execution. # Docker & Virtualenv Operators These container-based operators let you run OpenMetadata ingestion without installing dependencies directly on the Airflow host. ## Using the Docker Operator For this operator, we can use the `openmetadata/ingestion-base` image. This is useful to prepare DAGs without any installation required on the environment, although it needs for the host to have access to the Docker commands. ### Prerequisites Ensure the Airflow host can run Docker commands. For Docker Compose setups, map the Docker socket as follows: ### Example ```yaml theme={null} volumes: - /var/run/docker.sock:/var/run/docker.sock:z # Need 666 permissions to run DockerOperator ``` ### Example DAG ```python theme={null} from datetime import datetime from airflow import models from airflow.providers.docker.operators.docker import DockerOperator config = """ """ with models.DAG( "ingestion-docker-operator", schedule_interval='*/5 * * * *', start_date=datetime(2021, 1, 1), catchup=False, tags=["OpenMetadata"], ) as dag: DockerOperator( command="python main.py", image="openmetadata/ingestion-base:2.0.1", environment={"config": config, "pipelineType": "metadata"}, docker_url="unix://var/run/docker.sock", # To allow to start Docker. Needs chmod 666 permissions tty=True, auto_remove="True", network_mode="host", # To reach the OM server task_id="ingest", dag=dag, ) ``` Make sure to tune out the DAG configurations (`schedule_interval`, `start_date`, etc.) as your use case requires. If you encounter issues such as missing task instances or Airflow failing to locate a deployed DAG (e.g., `Dag '' could not be found`), this may be due to a **timezone mismatch** in your Airflow configuration. To resolve this, set the following in your `airflow.cfg`: ```ini theme={null} default_timezone = system ``` This ensures that Airflow uses the system timezone, which is particularly important when OpenMetadata and Airflow are running on the same server. ### Key Notes * **Image Version**: Ensure the Docker image version matches your OpenMetadata server version (e.g., `openmetadata/ingestion-base:2.0.1`). * **Pipeline Types**: Set the `pipelineType` to `metadata`, `usage`, `lineage`, `profiler`, or other supported values. * **No Installation Required**: The `DockerOperator` eliminates the need to install dependencies directly on the Airflow host. Another important point here is making sure that the Airflow will be able to run Docker commands to create the task. As our example was done with Airflow in Docker Compose, that meant setting `docker_url="unix://var/run/docker.sock"`. The final important elements here are: * `command="python main.py"`: This does not need to be modified, as we are shipping the `main.py` script in the image, used to trigger the workflow. * `environment={"config": config, "pipelineType": "metadata"}`: Again, in most cases you will just need to update the `config` string to point to the right connector. Other supported values of `pipelineType` are `usage`, `lineage`, `profiler`, `dataInsight`, `elasticSearchReindex`, `dbt`, `application` or `TestSuite`. Pass the required flag depending on the type of workflow you want to execute. Make sure that the YAML config reflects what ingredients are required for your Workflow. ## Using the Python Virtualenv Operator ### Prerequisites As stated in Airflow's [docs](https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html#pythonvirtualenvoperator), install the `virtualenv` package on the Airflow host.If using a different Python version in the virtual environment (e.g., Python 3.9 while Airflow uses 3.7), install additional packages such as: ``` gcc python3.9-dev python3.9-distutils ``` ### Example DAG ```python theme={null} from datetime import timedelta from airflow import DAG try: from airflow.operators.python import PythonVirtualenvOperator except ModuleNotFoundError: from airflow.operators.python_operator import PythonVirtualenvOperator from airflow.utils.dates import days_ago default_args = { "owner": "user_name", "email": ["username@org.com"], "email_on_failure": False, "retries": 3, "retry_delay": timedelta(seconds=10), "execution_timeout": timedelta(minutes=60), } def metadata_ingestion_workflow(): from metadata.workflow.metadata import MetadataWorkflow import yaml config = """ source: type: postgres serviceName: local_postgres serviceConnection: config: type: Postgres username: openmetadata_user authType: password: openmetadata_password hostPort: localhost:5432 database: pagila sourceConfig: config: type: DatabaseMetadata sink: type: metadata-rest config: {} workflowConfig: # loggerLevel: INFO # DEBUG, INFO, WARN or ERROR openMetadataServerConfig: hostPort: http://localhost:8585/api authProvider: openmetadata securityConfig: jwtToken: "eyJraWQiOiJHYjM4OWEtOWY3Ni1nZGpzLWE5MmotMDI0MmJrOTQzNTYiLCJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhZG1pbiIsImlzQm90IjpmYWxzZSwiaXNzIjoib3Blbi1tZXRhZGF0YS5vcmciLCJpYXQiOjE2NjM5Mzg0NjIsImVtYWlsIjoiYWRtaW5Ab3Blbm1ldGFkYXRhLm9yZyJ9.tS8um_5DKu7HgzGBzS1VTA5 """ workflow_config = yaml.safe_load(config) workflow = MetadataWorkflow.create(workflow_config) workflow.execute() workflow.raise_from_status() workflow.print_status() workflow.stop() with DAG( "ingestion_dag", default_args=default_args, description="An example DAG which runs a OpenMetadata ingestion workflow", start_date=days_ago(1), is_paused_upon_creation=True, catchup=False, ) as dag: ingest_task = PythonVirtualenvOperator( task_id="ingest_using_recipe", requirements=[ 'openmetadata-ingestion[mysql]~=2.0.1.0', # Specify any additional Python package dependencies ], system_site_packages=False, # Set to True if you want to include system site-packages in the virtual environment python_version="3.9", # Remove if necessary python_callable=metadata_ingestion_workflow ) ``` ### Key Notes **Function Rules**: * Use a `def` function (not part of a class). * All imports must occur inside the function. * Avoid referencing variables outside the function's scope. ## Ingestion Workflow classes We have different classes for different types of workflows. The logic is always the same, but you will need to change your import path. The rest of the method calls will remain the same. For example, for the `Metadata` workflow we'll use: ```python theme={null} import yaml from metadata.workflow.metadata import MetadataWorkflow def run(): workflow_config = yaml.safe_load(CONFIG) workflow = MetadataWorkflow.create(workflow_config) workflow.execute() workflow.raise_from_status() workflow.print_status() workflow.stop() ``` The classes for each workflow type are: * `Metadata`: `from metadata.workflow.metadata import MetadataWorkflow` * `Lineage`: `from metadata.workflow.metadata import MetadataWorkflow` (same as metadata) * `Usage`: `from metadata.workflow.usage import UsageWorkflow` * `dbt`: `from metadata.workflow.metadata import MetadataWorkflow` * `Profiler`: `from metadata.workflow.profiler import ProfilerWorkflow` * `Data Quality`: `from metadata.workflow.data_quality import TestSuiteWorkflow` * `Data Insights`: `from metadata.workflow.data_insight import DataInsightWorkflow` * `Elasticsearch Reindex`: `from metadata.workflow.metadata import MetadataWorkflow` (same as metadata) # Managing Credentials Source: https://docs.open-metadata.org/v2.0.x/deployment/ingestion/external/credentials # Managing Credentials On the release 0.12 we updated how services credentials are handled from an Ingestion Workflow. We are covering now two scenarios: **1.** If we are running a metadata workflow for the first time, pointing to a service that **does not yet exist**, then the service will be created from the Metadata Ingestion pipeline. It does not matter if the workflow is run from the CLI or any other scheduler. **2.** If instead, there is an already existing service to which we are pointing with a Metadata Ingestion pipeline, then we will be using the **stored credentials**, not the ones incoming from the YAML config. ## Existing Services What this means is that once a service is created, the only way to update its connection credentials is via the **UI** or directly running an API call. This prevents the scenario where a new YAML config is created, using a name of a service that already exists, but pointing to a completely different source system. One of the main benefits of this approach is that if an admin in our organisation creates the service from the UI, then we can prepare any Ingestion Workflow without having to pass the connection details. For example, for an Athena YAML, instead of requiring the full set of credentials as below: ```yaml theme={null} source: type: athena serviceName: my_athena_service serviceConnection: config: type: Athena awsConfig: awsAccessKeyId: KEY awsSecretAccessKey: SECRET awsRegion: us-east-2 s3StagingDir: s3 directory for datasource workgroup: workgroup name sourceConfig: type: DatabaseMetadata config: markDeletedTables: true includeTables: true includeViews: true sink: type: metadata-rest config: {} workflowConfig: openMetadataServerConfig: hostPort: authProvider: ``` We can use a simplified version: ```yaml theme={null} source: type: athena serviceName: my_athena_service sourceConfig: config: type: DatabaseMetadata markDeletedTables: true includeTables: true includeViews: true sink: type: metadata-rest config: {} workflowConfig: openMetadataServerConfig: hostPort: authProvider: ``` The workflow will then dynamically pick up the service connection details for `my_athena_service` and ingest the metadata accordingly. If instead, you want to have the full source of truth in your DAGs or processes, you can keep reading on different ways to secure the credentials in your environment and not have them at plain sight. ## Securing Credentials Note that these are just a few examples. Any secure and automated approach to retrieve a string would work here, as our only requirement is to pass the string inside the YAML configuration. When running Workflow with the CLI or your favourite scheduler, it's safer to not have the services' credentials visible. For the CLI, the ingestion package can load sensitive information from environment variables. For example, if you are using the [Glue](/v2.0.x/connectors/database/glue) connector you could specify the AWS configurations as follows in the case of a JSON config file ```json theme={null} [...] "awsConfig": { "awsAccessKeyId": "${AWS_ACCESS_KEY_ID}", "awsSecretAccessKey": "${AWS_SECRET_ACCESS_KEY}", "awsRegion": "${AWS_REGION}", "awsSessionToken": "${AWS_SESSION_TOKEN}" }, [...] ``` Or ```yaml theme={null} [...] awsConfig: awsAccessKeyId: '${AWS_ACCESS_KEY_ID}' awsSecretAccessKey: '${AWS_SECRET_ACCESS_KEY}' awsRegion: '${AWS_REGION}' awsSessionToken: '${AWS_SESSION_TOKEN}' [...] ``` for a YAML configuration. ### AWS Credentials The AWS Credentials are based on the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/security/credentials/awsCredentials.json). Note that the only required field is the `awsRegion`. This configuration is rather flexible to allow installations under AWS that directly use instance roles for permissions to authenticate to whatever service we are pointing to without having to write the credentials down. #### AWS Vault If using [aws-vault](https://github.com/99designs/aws-vault), it gets a bit more involved to run the CLI ingestion as the credentials are not globally available in the terminal. In that case, you could use the following command after setting up the ingestion configuration file: ```bash theme={null} aws-vault exec -- $SHELL -c 'metadata ingest -c ' ``` ### GCP Credentials The GCP Credentials are based on the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/security/credentials/gcpCredentials.json). These are the fields that you can export when preparing a Service Account. Once the account is created, you can see the fields in the exported JSON file from: ``` IAM & Admin > Service Accounts > Keys ``` You can validate the whole Google service account setup [here](/v2.0.x/deployment/security/google). ### Using GitHub Actions Secrets If running the ingestion in a GitHub Action, you can create [encrypted secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets) to store sensitive information such as users and passwords. In the end, we'll map these secrets to environment variables in the process, that we can pick up with `os.getenv`, for example: ```python theme={null} import os import yaml from metadata.workflow.metadata import MetadataWorkflow CONFIG = f""" source: type: snowflake serviceName: snowflake_from_github_actions serviceConnection: config: type: Snowflake username: {os.getenv('SNOWFLAKE_USERNAME')} ... """ def run(): workflow_config = yaml.safe_load(CONFIG) workflow = MetadataWorkflow.create(workflow_config) workflow.execute() workflow.raise_from_status() workflow.print_status() workflow.stop() if __name__ == "__main__": run() ``` Make sure to update your step environment to pass the secrets as environment variables: ```yaml theme={null} - name: Run Ingestion run: | source env/bin/activate python ingestion-github-actions/snowflake_ingestion.py # Add the env vars we need to load the snowflake credentials env: SNOWFLAKE_USERNAME: ${{ secrets.SNOWFLAKE_USERNAME }} SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }} SNOWFLAKE_WAREHOUSE: ${{ secrets.SNOWFLAKE_WAREHOUSE }} SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }} ``` ## Next Steps For a step-by-step guide on using Airflow Connections to securely retrieve service credentials in your DAGs, see [Using Airflow Connections](/v2.0.x/deployment/ingestion/external/credentials-airflow). # Using Airflow Connections for Credentials Source: https://docs.open-metadata.org/v2.0.x/deployment/ingestion/external/credentials-airflow Retrieve and use Airflow Connections to securely pass service credentials in OpenMetadata ingestion pipelines. # Using Airflow Connections In any connector page, you might have seen an example on how to build a DAG to run the ingestion with Airflow (e.g., [Athena](/v2.0.x/connectors/database/athena/yaml)). A possible approach to retrieving sensitive information from Airflow would be using Airflow's [Connections](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html). Note that these connections can be stored as environment variables, to Airflow's underlying DB or to multiple external services such as Hashicorp Vault. Note that for external systems, you'll need to provide the necessary package and configure the [Secrets Backend](https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/index.html). The best way to choose how to store these credentials is to go through Airflow's [docs](https://airflow.apache.org/docs/apache-airflow/stable/concepts/connections.html). ## Example Let's go over an example on how to create a connection to extract data from MySQL and how a DAG would look like afterwards. ### Step 1 - Create the Connection From our Airflow host, (e.g., `docker exec -it openmetadata_ingestion bash` if testing in Docker), you can run: ```bash theme={null} airflow connections add 'my_mysql_db' \ --conn-uri 'mysql+pymysql://openmetadata_user:openmetadata_password@mysql:3306/openmetadata_db' ``` You will see an output like ``` Successfully added `conn_id`=my_mysql_db : mysql+pymysql://openmetadata_user:openmetadata_password@mysql:3306/openmetadata_db ``` Checking the credentials from the Airflow UI, we will see: Airflow Connection ### Step 2 - Understanding the shape of a Connection In the same host, we can open a Python shell to explore the Connection object with some more details. To do so, we first need to pick up the connection from Airflow. We will use the `BaseHook` for that as the connection is not stored in any external system. ```python theme={null} from airflow.hooks.base import BaseHook # Retrieve the connection connection = BaseHook.get_connection("my_mysql_db") # Access the connection details connection.host # 'mysql' connection.port # 3306 connection.login # 'openmetadata_user' connection.password # 'openmetadata_password' ``` Based on this information, we now know how to prepare the DAG! ### Step 3 - Write the DAG A full example on how to write a DAG to ingest data from our Connection can look like this: ```python theme={null} import pathlib import yaml from datetime import timedelta from airflow import DAG from airflow.utils.dates import days_ago try: from airflow.operators.python import PythonOperator except ModuleNotFoundError: from airflow.operators.python_operator import PythonOperator from metadata.config.common import load_config_file from metadata.workflow.metadata import MetadataWorkflow # Import the hook from airflow.hooks.base import BaseHook # Retrieve the connection connection = BaseHook.get_connection("my_mysql_db") # Use the connection details when setting the YAML # Note how we escaped the braces as {{}} to not be parsed by the f-string config = f""" source: type: mysql serviceName: mysql_from_connection serviceConnection: config: type: Mysql username: {connection.login} password: {connection.password} hostPort: {connection.host}:{connection.port} # databaseSchema: schema sourceConfig: config: markDeletedTables: true includeTables: true includeViews: true sink: type: metadata-rest config: {{}} workflowConfig: openMetadataServerConfig: hostPort: "" authProvider: "" """ def metadata_ingestion_workflow(): workflow_config = yaml.safe_load(config) workflow = MetadataWorkflow.create(workflow_config) workflow.execute() workflow.raise_from_status() workflow.print_status() workflow.stop() with DAG( "mysql_connection_ingestion", description="An example DAG which runs a OpenMetadata ingestion workflow", start_date=days_ago(1), is_paused_upon_creation=False, schedule_interval='*/5 * * * *', catchup=False, ) as dag: ingest_task = PythonOperator( task_id="ingest_using_recipe", python_callable=metadata_ingestion_workflow, ) ``` ### Option B - Reuse an existing Service As explained in the [Managing Credentials](/v2.0.x/deployment/ingestion/external/credentials#existing-services) guide, once a service exists in OpenMetadata its connection details are stored and can be reused — just omit the `serviceConnection` YAML entries in your DAG: ```python theme={null} import pathlib import yaml from datetime import timedelta from airflow import DAG from airflow.utils.dates import days_ago try: from airflow.operators.python import PythonOperator except ModuleNotFoundError: from airflow.operators.python_operator import PythonOperator from metadata.config.common import load_config_file from metadata.workflow.metadata import MetadataWorkflow config = """ source: type: mysql serviceName: existing_mysql_service sourceConfig: config: markDeletedTables: true includeTables: true includeViews: true sink: type: metadata-rest config: {} workflowConfig: openMetadataServerConfig: hostPort: "" authProvider: "" """ def metadata_ingestion_workflow(): workflow_config = yaml.safe_load(config) workflow = MetadataWorkflow.create(workflow_config) workflow.execute() workflow.raise_from_status() workflow.print_status() workflow.stop() with DAG( "mysql_connection_ingestion", description="An example DAG which runs a OpenMetadata ingestion workflow", start_date=days_ago(1), is_paused_upon_creation=False, schedule_interval='*/5 * * * *', catchup=False, ) as dag: ingest_task = PythonOperator( task_id="ingest_using_recipe", python_callable=metadata_ingestion_workflow, ) ``` # External Ingestion Workflow Examples Source: https://docs.open-metadata.org/v2.0.x/deployment/ingestion/external/examples Code examples for running OpenMetadata ingestion workflows externally — metadata, lineage, usage, profiler, and data quality. # External Ingestion Workflow Examples This page contains code examples for running each workflow type externally. For framework setup, configuration, and JWT token handling, see the [External Ingestion Overview](/v2.0.x/deployment/ingestion/external). Let's jump now into some examples on how you could create the function to run the different workflows. Note that this code can then be executed inside a DAG, a GitHub action, or a vanilla Python script. It will work for any environment. ### Testing You can easily test every YAML configuration using the `metadata` CLI from the Ingestion Framework. In order to install it, you just need to get it from [PyPI](https://pypi.org/project/openmetadata-ingestion/). In each of the examples below, we'll showcase how to run the CLI, assuming you have a YAML file that contains the workflow configuration. ### Metadata Workflow This is the first workflow you have to configure and run. It will take care of fetching the metadata from your sources, be it Database Services, Dashboard Services, Pipelines, etc. The rest of the workflows (Lineage, Profiler,...) will be executed on top of the metadata already available in the platform. **Adding the imports** The first step is to import the `MetadataWorkflow` class, which will take care of the full ingestion logic. We'll add the import for printing the results at the end. **Defining the YAML** Then, we need to pass the YAML configuration. For this simple example we are defining a variable, but you can read from a file, parse secrets from your environment, or any other approach you'd need. In the end, it's just Python code. You can find complete YAMLs in each connector [docs](/v2.0.x/connectors) and find more information about the available configurations. **Preparing the Workflow** Finally, we'll prepare a function that we can execute anywhere. It will take care of instantiating the workflow, executing it and giving us the results. ```python theme={null} import yaml from metadata.workflow.metadata import MetadataWorkflow CONFIG = """ source: type: snowflake serviceName: serviceConnection: config: type: Snowflake ... sourceConfig: config: type: DatabaseMetadata markDeletedTables: true includeTables: true ... sink: type: metadata-rest config: {} workflowConfig: openMetadataServerConfig: hostPort: "http://localhost:8585/api" authProvider: openmetadata securityConfig: jwtToken: "{bot_jwt_token}" """ def run(): workflow = MetadataWorkflow.create(yaml.safe_load(CONFIG)) workflow.execute() workflow.raise_from_status() workflow.print_status() workflow.stop() ``` You can test the workflow via `metadata ingest -c `. ### Lineage Workflow This workflow will take care of scanning your query history and defining lineage relationships between your tables. You can find more information about this workflow [here](/v2.0.x/connectors/ingestion/lineage). **Adding the imports** The first step is to import the `MetadataWorkflow` class, which will take care of the full ingestion logic. We'll add the import for printing the results at the end. Note that we are using the same class as in the Metadata Ingestion. **Defining the YAML** Then, we need to pass the YAML configuration. For this simple example we are defining a variable, but you can read from a file, parse secrets from your environment, or any other approach you'd need. Note how we have not added here the `serviceConnection`. Since the service would have been created during the metadata ingestion, we can let the Ingestion Framework dynamically fetch the Service Connection information. If, however, you are configuring the workflow with `storeServiceConnection: false`, you'll need to explicitly define the `serviceConnection`. You can find complete YAMLs in each connector [docs](/v2.0.x/connectors) and find more information about the available configurations. **Preparing the Workflow** Finally, we'll prepare a function that we can execute anywhere. It will take care of instantiating the workflow, executing it and giving us the results. ```python theme={null} import yaml from metadata.workflow.metadata import MetadataWorkflow CONFIG = """ source: type: snowflake-lineage serviceName: sourceConfig: config: type: DatabaseLineage queryLogDuration: 1 parsingTimeoutLimit: 300 ... sink: type: metadata-rest config: {} workflowConfig: openMetadataServerConfig: hostPort: "http://localhost:8585/api" authProvider: openmetadata securityConfig: jwtToken: "{bot_jwt_token}" """ def run(): workflow = MetadataWorkflow.create(yaml.safe_load(CONFIG)) workflow.execute() workflow.raise_from_status() workflow.print_status() workflow.stop() ``` You can test the workflow via `metadata ingest -c `. ### Usage Workflow As with the lineage workflow, we'll scan the query history for any DML statements. The goal is to ingest queries into the platform, figure out the relevancy of your assets and frequently joined tables. **Adding the imports** The first step is to import the `UsageWorkflow` class, which will take care of the full ingestion logic. We'll add the import for printing the results at the end. **Defining the YAML** Then, we need to pass the YAML configuration. For this simple example we are defining a variable, but you can read from a file, parse secrets from your environment, or any other approach you'd need. Note how we have not added here the `serviceConnection`. Since the service would have been created during the metadata ingestion, we can let the Ingestion Framework dynamically fetch the Service Connection information. If, however, you are configuring the workflow with `storeServiceConnection: false`, you'll need to explicitly define the `serviceConnection`. You can find complete YAMLs in each connector [docs](/v2.0.x/connectors) and find more information about the available configurations. **Preparing the Workflow** Finally, we'll prepare a function that we can execute anywhere. It will take care of instantiating the workflow, executing it and giving us the results. ```python theme={null} import yaml from metadata.workflow.usage import UsageWorkflow CONFIG = """ source: type: snowflake-usage serviceName: sourceConfig: config: type: DatabaseUsage queryLogDuration: 1 parsingTimeoutLimit: 300 ... processor: type: query-parser config: {} stage: type: table-usage config: filename: "/tmp/snowflake_usage" bulkSink: type: metadata-usage config: filename: "/tmp/snowflake_usage" workflowConfig: openMetadataServerConfig: hostPort: "http://localhost:8585/api" authProvider: openmetadata securityConfig: jwtToken: "{bot_jwt_token}" """ def run(): workflow = UsageWorkflow.create(yaml.safe_load(CONFIG)) workflow.execute() workflow.raise_from_status() workflow.print_status() workflow.stop() ``` You can test the workflow via `metadata usage -c `. ### Profiler Workflow This workflow will execute queries against your database and send the results into OpenMetadata. The goal is to compute metrics about your data and give you a high-level view of its shape, together with the sample data. This is an interesting previous step before creating Data Quality Workflows. You can find more information about this workflow [here](/v2.0.x/how-to-guides/data-quality-observability/profiler/profiler-workflow). **Adding the imports** The first step is to import the `ProfilerWorkflow` class, which will take care of the full ingestion logic. We'll add the import for printing the results at the end. **Defining the YAML** Then, we need to pass the YAML configuration. For this simple example we are defining a variable, but you can read from a file, parse secrets from your environment, or any other approach you'd need. Note how we have not added here the `serviceConnection`. Since the service would have been created during the metadata ingestion, we can let the Ingestion Framework dynamically fetch the Service Connection information. If, however, you are configuring the workflow with `storeServiceConnection: false`, you'll need to explicitly define the `serviceConnection`. You can find complete YAMLs in each connector [docs](/v2.0.x/connectors) and find more information about the available configurations. **Preparing the Workflow** Finally, we'll prepare a function that we can execute anywhere. It will take care of instantiating the workflow, executing it and giving us the results. ```python theme={null} import yaml from metadata.workflow.profiler import ProfilerWorkflow CONFIG = """ source: type: snowflake serviceName: sourceConfig: config: type: Profiler generateSampleData: true ... processor: type: orm-profiler config: {} sink: type: metadata-rest config: {} workflowConfig: openMetadataServerConfig: hostPort: "http://localhost:8585/api" authProvider: openmetadata securityConfig: jwtToken: "{bot_jwt_token}" """ def run(): workflow = ProfilerWorkflow.create(yaml.safe_load(CONFIG)) workflow.execute() workflow.raise_from_status() workflow.print_status() workflow.stop() ``` You can test the workflow via `metadata profile -c `. ### Data Quality Workflow This workflow will execute queries against your database and send the results into OpenMetadata. The goal is to compute metrics about your data and give you a high-level view of its shape, together with the sample data. This is an interesting previous step before creating Data Quality Workflows. You can find more information about this workflow [here](/v2.0.x/how-to-guides/data-quality-observability/quality/configure). **Adding the imports** The first step is to import the `TestSuiteWorkflow` class, which will take care of the full ingestion logic. We'll add the import for printing the results at the end. **Defining the YAML** Then, we need to pass the YAML configuration. For this simple example we are defining a variable, but you can read from a file, parse secrets from your environment, or any other approach you'd need. Note how we have not added here the `serviceConnection`. Since the service would have been created during the metadata ingestion, we can let the Ingestion Framework dynamically fetch the Service Connection information. If, however, you are configuring the workflow with `storeServiceConnection: false`, you'll need to explicitly define the `serviceConnection`. Moreover, see how we are not configuring any tests in the `processor`. You can configure them in the [advanced YAML example](/v2.0.x/how-to-guides/data-quality-observability/quality/data-quality-as-code/advanced-usage#yaml-file-structure), but even if nothing gets defined in the YAML, we will execute all the tests configured against the table. You can find complete YAMLs in each connector [docs](/v2.0.x/connectors) and find more information about the available configurations. **Preparing the Workflow** Finally, we'll prepare a function that we can execute anywhere. It will take care of instantiating the workflow, executing it and giving us the results. ```python theme={null} import yaml from metadata.workflow.data_quality import TestSuiteWorkflow CONFIG = """ source: type: TestSuite serviceName: sourceConfig: config: type: TestSuite entityFullyQualifiedName: processor: type: orm-test-runner config: {} sink: type: metadata-rest config: {} workflowConfig: openMetadataServerConfig: hostPort: "http://localhost:8585/api" authProvider: openmetadata securityConfig: jwtToken: "{bot_jwt_token}" """ def run(): workflow = TestSuiteWorkflow.create(yaml.safe_load(CONFIG)) workflow.execute() workflow.raise_from_status() workflow.print_status() workflow.stop() ``` You can test the workflow via `metadata test -c `. # Run the ingestion from GCP Composer | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/ingestion/external/gcp-composer Deploy external ingestion using GCP Composer to automate metadata and quality pipelines on Google Cloud environments. This page is about running the Ingestion Framework **externally**! There are mainly 2 ways of running the ingestion: 1. Internally, by managing the workflows from OpenMetadata. 2. Externally, by using any other tool capable of running Python code. If you are looking for how to manage the ingestion process from OpenMetadata, you can follow this [doc](/deployment/ingestion/openmetadata). # Run the ingestion from GCP Composer ## Requirements This approach has been last tested against: * Composer version 2.5.4 * Airflow version 2.6.3 It also requires the ingestion package to be at least `openmetadata-ingestion==2.0.1.0`. ## Using the Python Operator The most comfortable way to run the metadata workflows from GCP Composer is directly via a `PythonOperator`. Note that it will require you to install the packages and plugins directly on the host. ### Install the Requirements In your environment you will need to install the following packages: * `openmetadata-ingestion[]==x.y.z`. * `sqlalchemy==1.4.27`: This is needed to align OpenMetadata version with the Composer internal requirements. Where `x.y.z` is the version of the OpenMetadata ingestion package. Note that the version needs to match the server version. If we are using the server at 2.0.1, then the ingestion package needs to also be 2.0.1.0. The plugin parameter is a list of the sources that we want to ingest. An example would look like this `openmetadata-ingestion[mysql,snowflake,s3]==2.0.1.0`. ### Prepare the DAG! Note that this DAG is a usual connector DAG, just using the Airflow service with the `Backend` connection. As an example of a DAG pushing data to OpenMetadata under Google SSO, we could have: ```python theme={null} from datetime import timedelta import yaml from airflow import DAG try: from airflow.operators.python import PythonOperator except ModuleNotFoundError: from airflow.operators.python_operator import PythonOperator from airflow.utils.dates import days_ago from metadata.workflow.metadata import MetadataWorkflow default_args = { "owner": "user_name", "email": ["username@org.com"], "email_on_failure": False, "retries": 3, "retry_delay": timedelta(minutes=5), "execution_timeout": timedelta(minutes=60), } CONFIG = """ ... """ def metadata_ingestion_workflow(): workflow_config = yaml.safe_load(CONFIG) workflow = MetadataWorkflow.create(workflow_config) workflow.execute() workflow.raise_from_status() workflow.print_status() workflow.stop() with DAG( "airflow_metadata_extraction", default_args=default_args, description="An example DAG which pushes Airflow data to OM", start_date=days_ago(1), is_paused_upon_creation=True, schedule_interval="*/5 * * * *", catchup=False, ) as dag: ingest_task = PythonOperator( task_id="ingest_using_recipe", python_callable=metadata_ingestion_workflow, ) ``` ## Ingestion Workflow classes We have different classes for different types of workflows. The logic is always the same, but you will need to change your import path. The rest of the method calls will remain the same. For example, for the `Metadata` workflow we'll use: ```python theme={null} import yaml from metadata.workflow.metadata import MetadataWorkflow def run(): workflow_config = yaml.safe_load(CONFIG) workflow = MetadataWorkflow.create(workflow_config) workflow.execute() workflow.raise_from_status() workflow.print_status() workflow.stop() ``` The classes for each workflow type are: * `Metadata`: `from metadata.workflow.metadata import MetadataWorkflow` * `Lineage`: `from metadata.workflow.metadata import MetadataWorkflow` (same as metadata) * `Usage`: `from metadata.workflow.usage import UsageWorkflow` * `dbt`: `from metadata.workflow.metadata import MetadataWorkflow` * `Profiler`: `from metadata.workflow.profiler import ProfilerWorkflow` * `Data Quality`: `from metadata.workflow.data_quality import TestSuiteWorkflow` * `Data Insights`: `from metadata.workflow.data_insight import DataInsightWorkflow` * `Elasticsearch Reindex`: `from metadata.workflow.metadata import MetadataWorkflow` (same as metadata) ## Using the Kubernetes Pod Operator In this second approach we won't need to install absolutely anything to the GCP Composer environment. Instead, we will rely on the `KubernetesPodOperator` to use the underlying k8s cluster of Composer. Then, the code won't directly run using the hosts' environment, but rather inside a container that we created with only the `openmetadata-ingestion` package. **Note:** This approach only has the `openmetadata/ingestion-base` ready from version 0.12.1 or higher! ### Prepare the DAG! ```python theme={null} from datetime import datetime from airflow import models from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator CONFIG = """ ... """ with models.DAG( "ingestion-k8s-operator", schedule_interval="@once", start_date=datetime(2021, 1, 1), catchup=False, tags=["OpenMetadata"], ) as dag: KubernetesPodOperator( task_id="ingest", name="ingest", cmds=["python", "main.py"], image="openmetadata/ingestion-base:2.0.1", namespace='default', env_vars={"config": CONFIG, "pipelineType": "metadata"}, dag=dag, ) ``` Some remarks on this example code: #### Kubernetes Pod Operator You can name the task as you want (`task_id` and `name`). The important points here are the `cmds`, this should not be changed, and the `env_vars`. The `main.py` script that gets shipped within the image will load the env vars as they are shown, so only modify the content of the config YAML, but not this dictionary. Note that the example uses the image `openmetadata/ingestion-base:2.0.1`. The image version should be aligned with your OpenMetadata server version to avoid incompatibilities. ```python theme={null} KubernetesPodOperator( task_id="ingest", name="ingest", cmds=["python", "main.py"], image="openmetadata/ingestion-base:2.0.1", namespace='default', env_vars={"config": config, "pipelineType": "metadata"}, dag=dag, ) ``` You can find more information about the `KubernetesPodOperator` and how to tune its configurations [here](https://cloud.google.com/composer/docs/how-to/using/using-kubernetes-pod-operator). Note that depending on the kind of workflow you will be deploying, the YAML configuration will need to updated following the official OpenMetadata docs, and the value of the `pipelineType` configuration will need to hold one of the following values: * `metadata` * `usage` * `lineage` * `profiler` * `TestSuite` Which are based on the `PipelineType` [JSON Schema definitions](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/ingestionPipelines/ingestionPipeline.json#L14) # Run the ingestion from GitHub Actions Source: https://docs.open-metadata.org/v2.0.x/deployment/ingestion/external/github-actions Configure GitHub Actions for metadata ingestion to run workflows on commits, schedules, or triggers in your CI/CD pipelines. This page is about running the Ingestion Framework **externally**! There are mainly 2 ways of running the ingestion: 1. Internally, by managing the workflows from OpenMetadata. 2. Externally, by using any other tool capable of running Python code. If you are looking for how to manage the ingestion process from OpenMetadata, you can follow this [doc](/deployment/ingestion/openmetadata). # Run the ingestion from GitHub Actions The process to run the ingestion from GitHub Actions is the same as running it from anywhere else. 1. Get the YAML configuration, 2. Prepare the Python Script 3. Schedule the Ingestion ## 1. YAML Configuration For any connector and workflow, you can pick it up from its doc [page](/v2.0.x/connectors). ## 2. Prepare the Python Script In the GitHub Action we will just be triggering a custom Python script. This script will: * Load the secrets from environment variables (we don't want any security risks!), * Prepare the Workflow class from the Ingestion Framework that contains all the logic on how to run the metadata ingestion, * Execute the workflow and log the results. * A simplified version of such script looks like follows: ```python theme={null} import os import yaml from metadata.workflow.metadata import MetadataWorkflow CONFIG = f""" source: type: snowflake serviceName: snowflake_from_github_actions serviceConnection: config: type: Snowflake username: {os.getenv('SNOWFLAKE_USERNAME')} ... """ def run(): workflow_config = yaml.safe_load(CONFIG) workflow = MetadataWorkflow.create(workflow_config) workflow.execute() workflow.raise_from_status() workflow.print_status() workflow.stop() if __name__ == "__main__": run() ``` Note how we are securing the credentials using environment variables. You will need to create these env vars in your GitHub repository. Follow the GitHub [docs](https://docs.github.com/en/actions/security-guides/encrypted-secrets) for more information on how to create and use Secrets. In the end, we'll map these secrets to environment variables in the process, that we can pick up with `os.getenv`. ## 3. Schedule the Ingestion Now that we have all the ingredients, we just need to build a simple GitHub Actions with the following steps: * Install Python * Prepare virtual environment with the openmetadata-ingestion package * Run the script! * It is as simple as this. Internally the function run we created will be sending the results to the OpenMetadata server, so there's nothing else we need to do here. A first version of the action could be: ```yaml theme={null} name: ingest-snowflake on: # Any expression you'd like here schedule: - cron: '0 */2 * * *' # If you also want to execute it manually workflow_dispatch: permissions: id-token: write contents: read jobs: ingest: runs-on: ubuntu-latest steps: # Pick up the repository code, where the script lives - name: Checkout uses: actions/checkout@v3 # Prepare Python in the GitHub Agent - name: Set up Python 3.9 uses: actions/setup-python@v4 with: python-version: 3.9 # Install the dependencies. Make sure that the client version matches the server! - name: Install Deps run: | python -m venv env source env/bin/activate pip install "openmetadata-ingestion[snowflake]==2.0.1.0" - name: Run Ingestion run: | source env/bin/activate python ingestion-github-actions/snowflake_ingestion.py # Add the env vars we need to load the snowflake credentials env: SNOWFLAKE_USERNAME: ${{ secrets.SNOWFLAKE_USERNAME }} SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }} SNOWFLAKE_WAREHOUSE: ${{ secrets.SNOWFLAKE_WAREHOUSE }} SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }} SBX_JWT: ${{ secrets.SBX_JWT }} ``` ## \[Optional] - Getting Alerts in Slack A very interesting option that GitHub Actions provide is the ability to get alerts in Slack after our action fails. This can become specially useful if we want to be notified when our metadata ingestion is not working as expected. We can use the same setup as above with a couple of slight changes: ```yaml theme={null} - name: Run Ingestion id: ingestion continue-on-error: true run: | source env/bin/activate python ingestion-github-actions/snowflake_ingestion.py # Add the env vars we need to load the snowflake credentials env: SNOWFLAKE_USERNAME: ${{ secrets.SNOWFLAKE_USERNAME }} SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }} SNOWFLAKE_WAREHOUSE: ${{ secrets.SNOWFLAKE_WAREHOUSE }} SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }} SBX_JWT: ${{ secrets.SBX_JWT }} - name: Slack on Failure if: steps.ingestion.outcome != 'success' uses: slackapi/slack-github-action@v1.23.0 with: payload: | { "text": "🔥 Metadata ingestion failed! 🔥" } env: SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }} SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK - name: Force failure if: steps.ingestion.outcome != 'success' run: | exit 1 ``` We have: * Marked the `Run Ingestion` step with a specific `id` and with `continue-on-error: true`. If anything happens, we don't want the action to stop. * We added a step with `slackapi/slack-github-action@v1.23.0`. By passing a Slack Webhook link via a secret, we can send any payload to a * specific Slack channel. You can find more info on how to set up a Slack Webhook [here](https://api.slack.com/messaging/webhooks). * If our `ingestion` step fails, we still want to mark the action as failed, so we are forcing the failure we skipped before. # Run the ingestion from AWS MWAA | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/ingestion/external/mwaa Integrate with MWAA to schedule and execute ingestion workflows using managed Airflow on AWS infrastructure. This page is about running the Ingestion Framework **externally**! There are mainly 2 ways of running the ingestion: 1. Internally, by managing the workflows from OpenMetadata. 2. Externally, by using any other tool capable of running Python code. If you are looking for how to manage the ingestion process from OpenMetadata, you can follow this [doc](/deployment/ingestion/openmetadata). # Run the ingestion from AWS MWAA This page covers using MWAA as the **orchestrator** that runs OpenMetadata ingestion workflows for your other sources, such as databases and dashboards. To extract **MWAA's own pipeline metadata** (its DAGs and runs) into OpenMetadata, see the [MWAA connector documentation](/v2.0.x/connectors/pipeline/airflow/mwaa). In most cases the [REST API connection](/v2.0.x/connectors/pipeline/airflow/rest-api-connection) is the recommended approach and requires nothing installed in MWAA for metadata extraction. Table-level lineage additionally requires the OpenLineage provider configured in MWAA. When running ingestion workflows from MWAA we have three approaches: 1. Install the openmetadata-ingestion package as a requirement in the Airflow environment. We will then run the process using a `PythonOperator` 2. Configure an ECS cluster and run the ingestion as an `ECSOperator`. 3. Install a plugin and run the ingestion with the `PythonVirtualenvOperator`. We will now discuss pros and cons of each aspect and how to configure them. OpenMetadata does not support using Amazon MWAA (Managed Workflows for Apache Airflow) for internal ingestion. This limitation exists because MWAA does not allow the installation of the `openmetadata-ingestion-rest-apis` plugin, which is required to expose the necessary REST APIs for initiating workflows. ## Ingestion Workflows as a Python Operator ### PROs * It is the simplest approach * We don’t need to spin up any further infrastructure ### CONs * We need to install the [openmetadata-ingestion](https://pypi.org/project/openmetadata-ingestion/) package in the MWAA environment * The installation can clash with existing libraries * Upgrading the OM version will require to repeat the installation process To install the package, we need to update the `requirements.txt` file from the MWAA environment to add the following line: ``` openmetadata-ingestion[]==x.y.z ``` Where `x.y.z` is the version of the OpenMetadata ingestion package. Note that the version needs to match the server version. If we are using the server at 2.0.1, then the ingestion package needs to also be 2.0.1.0. The plugin parameter is a list of the sources that we want to ingest. An example would look like this `openmetadata-ingestion[mysql,snowflake,s3]==2.0.1.0`. A DAG deployed using a Python Operator would then look like follows ```python theme={null} import json from datetime import timedelta from airflow import DAG try: from airflow.operators.python import PythonOperator except ModuleNotFoundError: from airflow.operators.python_operator import PythonOperator from airflow.utils.dates import days_ago from metadata.workflow.metadata import MetadataWorkflow default_args = { "retries": 3, "retry_delay": timedelta(seconds=10), "execution_timeout": timedelta(minutes=60), } config = """ YAML config """ def metadata_ingestion_workflow(): workflow_config = json.loads(config) workflow = MetadataWorkflow.create(workflow_config) workflow.execute() workflow.raise_from_status() workflow.print_status() workflow.stop() with DAG( "redshift_ingestion", default_args=default_args, description="An example DAG which runs a OpenMetadata ingestion workflow", start_date=days_ago(1), is_paused_upon_creation=False, catchup=False, ) as dag: ingest_task = PythonOperator( task_id="ingest_redshift", python_callable=metadata_ingestion_workflow, ) ``` Where you can update the YAML configuration and workflow classes accordingly. accordingly. Further examples on how to run the ingestion can be found on the documentation (e.g., [Snowflake](/v2.0.x/connectors/database/snowflake)). ## Ingestion Workflow classes We have different classes for different types of workflows. The logic is always the same, but you will need to change your import path. The rest of the method calls will remain the same. For example, for the `Metadata` workflow we'll use: ```python theme={null} theme={null} theme={null} theme={null} theme={null} theme={null} import yaml from metadata.workflow.metadata import MetadataWorkflow def run(): workflow_config = yaml.safe_load(CONFIG) workflow = MetadataWorkflow.create(workflow_config) workflow.execute() workflow.raise_from_status() workflow.print_status() workflow.stop() ``` The classes for each workflow type are: * `Metadata`: `from metadata.workflow.metadata import MetadataWorkflow` * `Lineage`: `from metadata.workflow.metadata import MetadataWorkflow` (same as metadata) * `Usage`: `from metadata.workflow.usage import UsageWorkflow` * `dbt`: `from metadata.workflow.metadata import MetadataWorkflow` * `Profiler`: `from metadata.workflow.profiler import ProfilerWorkflow` * `Data Quality`: `from metadata.workflow.data_quality import TestSuiteWorkflow` * `Data Insights`: `from metadata.workflow.data_insight import DataInsightWorkflow` * `Elasticsearch Reindex`: `from metadata.workflow.metadata import MetadataWorkflow` (same as metadata) ## Ingestion Workflows as an ECS Operator ### PROs * Completely isolated environment * Easy to update each version ### CONs * We need to set up an ECS cluster and the required policies in MWAA to connect to ECS and handle Log Groups. We will now describe the steps, following the official AWS documentation. ### 1. Create an ECS Cluster & Task Definition * The cluster needs a task to run in `FARGATE` mode. * The required image is `docker.open-metadata.org/openmetadata/ingestion-base:x.y.z` * The same logic as above applies. The `x.y.z` version needs to match the server version. For example, `docker.open-metadata.org/openmetadata/ingestion-base:2.0.1` We have tested this process with a Task Memory of 512MB and Task CPU (unit) of 256. This can be tuned depending on the amount of metadata that needs to be ingested. When creating the Task Definition, take notes on the **log groups** assigned, as we will need them to prepare the MWAA Executor Role policies. For example, if in the JSON from the Task Definition we see: ```json theme={null} "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-create-group": "true", "awslogs-group": "/ecs/openmetadata", "awslogs-region": "us-east-2", "awslogs-stream-prefix": "ecs" }, "secretOptions": [] } ``` We'll need to use the `/ecs/openmetadata` below when configuring the policies. ### 2. Task Definition ARN & Networking 1. From the AWS Console, copy your task definition ARN. It will look something like this `arn:aws:ecs:::task-definition/:`. 2. Get the network details on where the task should execute. We will be using a JSON like: ```json theme={null} { "awsvpcConfiguration": { "subnets": [ "subnet-xxxyyyzzz", "subnet-xxxyyyzzz" ], "securityGroups": [ "sg-xxxyyyzzz" ], "assignPublicIp": "ENABLED" } } ``` If you want to extract MWAA metadata, add the **VPC**, **subnets** and **security groups** used when setting up MWAA. We need to be in the same network environment as MWAA to reach the underlying database. ### 3. Update MWAA Executor Role policies * Identify your MWAA executor role. This can be obtained from the details view of your MWAA environment. * Add the following two policies to the role, the first with ECS permissions: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "ecs:RunTask", "ecs:DescribeTasks" ], "Resource": "*" }, { "Action": "iam:PassRole", "Effect": "Allow", "Resource": [ "*" ], "Condition": { "StringLike": { "iam:PassedToService": "ecs-tasks.amazonaws.com" } } } ] } ``` And for the Log Group permissions ```json theme={null} { "Effect": "Allow", "Action": [ "logs:CreateLogStream", "logs:CreateLogGroup", "logs:PutLogEvents", "logs:GetLogEvents", "logs:GetLogRecord", "logs:GetLogGroupFields", "logs:GetQueryResults" ], "Resource": [ "arn:aws:logs:::log-group:*", "arn:aws:logs:*:*:log-group::*" ] } ``` Note how you need to replace the `region`, `account-id` and the `log group` names for your Airflow Environment and ECS. ### 4. Prepare the DAG A DAG created using the ECS Operator will then look like this: ```python theme={null} from airflow import DAG # If using Airflow < 2.5 # from airflow.providers.amazon.aws.operators.ecs import ECSOperator # If using Airflow > 2.5 from airflow.providers.amazon.aws.operators.ecs import EcsRunTaskOperator from airflow.utils.dates import days_ago CLUSTER_NAME="openmetadata-ingestion" # Replace value for CLUSTER_NAME with your information. CONTAINER_NAME="openmetadata-ingestion" # Replace value for CONTAINER_NAME with your information. LAUNCH_TYPE="FARGATE" TASK_DEFINITION = "arn:aws:ecs:::task-definition/:" NETWORK_CONFIG = { "awsvpcConfiguration": { "subnets": [ "subnet-xxxyyyzzz", "subnet-xxxyyyzzz" ], "securityGroups": [ "sg-xxxyyyzzz" ], "assignPublicIp": "ENABLED" } } config = """ YAML config """ with DAG( dag_id="ecs_fargate_dag", schedule_interval=None, catchup=False, start_date=days_ago(1), is_paused_upon_creation=True, ) as dag: ecs_operator_task = EcsRunTaskOperator( task_id = "ecs_ingestion_task", dag=dag, cluster=CLUSTER_NAME, task_definition=TASK_DEFINITION, launch_type=LAUNCH_TYPE, overrides={ "containerOverrides":[ { "name":CONTAINER_NAME, "command":["python", "main.py"], "environment": [ { "name": "config", "value": config }, { "name": "pipelineType", "value": "metadata" }, ], }, ], }, network_configuration=NETWORK_CONFIG, awslogs_group="/ecs/ingest", awslogs_stream_prefix=f"ecs/{CONTAINER_NAME}", ) ``` Note that depending on the kind of workflow you will be deploying, the YAML configuration will need to updated following the official OpenMetadata docs, and the value of the `pipelineType` configuration will need to hold one of the following values: * `metadata` * `usage` * `lineage` * `profiler` * `TestSuite` Which are based on the `PipelineType` [JSON Schema definitions](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/ingestionPipelines/ingestionPipeline.json#L14) Moreover, one of the imports will depend on the MWAA Airflow version you are using: * If using Airflow \< 2.5: `from airflow.providers.amazon.aws.operators.ecs import ECSOperator` * If using Airflow > 2.5: `from airflow.providers.amazon.aws.operators.ecs import EcsRunTaskOperator` Make sure to update the `ecs_operator_task` task call accordingly. For the Python VirtualenvOperator approach, see [MWAA with Python VirtualenvOperator](/v2.0.x/deployment/ingestion/external/mwaa/virtualenv). # MWAA Ingestion with Python VirtualenvOperator | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/ingestion/external/mwaa/virtualenv Configure the PythonVirtualenvOperator in AWS MWAA for isolated ingestion workflow execution without library conflicts. # Ingestion Workflows as a Python Virtualenv Operator ## PROs * Installation does not clash with existing libraries * Simpler than ECS ## CONs * We need to install an additional plugin in MWAA * DAGs take longer to run due to needing to set up the virtualenv from scratch for each run. We need to update the `requirements.txt` file from the MWAA environment to add the following line: ``` virtualenv ``` Then, we need to set up a custom plugin in MWAA. Create a file named virtual\_python\_plugin.py. Note that you may need to update the python version (eg, python3.7 -> python3.10) depending on what your MWAA environment is running. ```python theme={null} """ Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. """ from airflow.plugins_manager import AirflowPlugin import airflow.utils.python_virtualenv from typing import List import os def _generate_virtualenv_cmd(tmp_dir: str, python_bin: str, system_site_packages: bool) -> List[str]: cmd = ['python3', '/usr/local/airflow/.local/lib/python3.7/site-packages/virtualenv', tmp_dir] if system_site_packages: cmd.append('--system-site-packages') if python_bin is not None: cmd.append(f'--python={python_bin}') return cmd airflow.utils.python_virtualenv._generate_virtualenv_cmd = _generate_virtualenv_cmd os.environ["PATH"] = f"/usr/local/airflow/.local/bin:{os.environ['PATH']}" class VirtualPythonPlugin(AirflowPlugin): name = 'virtual_python_plugin' ``` This is modified from the [AWS sample](https://docs.aws.amazon.com/mwaa/latest/userguide/samples-virtualenv.html). Next, create the plugins.zip file and upload it according to [AWS docs](https://docs.aws.amazon.com/mwaa/latest/userguide/configuring-dag-import-plugins.html). You will also need to [disable lazy plugin loading in MWAA](https://docs.aws.amazon.com/mwaa/latest/userguide/samples-virtualenv.html#samples-virtualenv-airflow-config). A DAG deployed using the PythonVirtualenvOperator would then look like: ```python theme={null} from datetime import timedelta from airflow import DAG from airflow.operators.python import PythonVirtualenvOperator from airflow.utils.dates import days_ago default_args = { "retries": 3, "retry_delay": timedelta(seconds=10), "execution_timeout": timedelta(minutes=60), } def metadata_ingestion_workflow(): from metadata.workflow.metadata import MetadataWorkflow import yaml config = """ YAML config """ workflow_config = yaml.safe_load(config) workflow = MetadataWorkflow.create(workflow_config) workflow.execute() workflow.raise_from_status() workflow.print_status() workflow.stop() with DAG( "redshift_ingestion", default_args=default_args, description="An example DAG which runs a OpenMetadata ingestion workflow", start_date=days_ago(1), is_paused_upon_creation=False, catchup=False, ) as dag: ingest_task = PythonVirtualenvOperator( task_id="ingest_redshift", python_callable=metadata_ingestion_workflow, requirements=['openmetadata-ingestion[redshift]~=2.0.1.0', 'apache-airflow==2.4.3', # note, v2.4.3 is the first version that does not conflict with OpenMetadata's 'tabulate' requirements 'apache-airflow-providers-amazon==6.0.0', # Amazon Airflow provider is necessary for MWAA 'watchtower',], system_site_packages=False, dag=dag, ) ``` Where you can update the YAML configuration and workflow classes accordingly. Further examples on how to run the ingestion can be found on the documentation (e.g., [Snowflake](/v2.0.x/connectors/database/snowflake)). You will also need to determine the OpenMetadata ingestion extras and Airflow providers you need. The OpenMetadata ingestion package should match your server minor version. For example, use a `2.0.x` ingestion package with a `2.0.x` server, and include the connector extras required by your YAML, such as `openmetadata-ingestion[mysql,snowflake,s3]~=2.0.1.0`. For Airflow providers, you will want to pull the provider versions from [the matching constraints file](https://raw.githubusercontent.com/apache/airflow/constraints-2.4.3/constraints-3.7.txt). Since this example installs Airflow Providers v2.4.3 on Python 3.7, we use that constraints file. Also note that the ingestion workflow function must be entirely self-contained as it will run by itself in the virtualenv. Any imports it needs, including the configuration, must exist within the function itself. ## Ingestion Workflow classes We have different classes for different types of workflows. The logic is always the same, but you will need to change your import path. The rest of the method calls will remain the same. For example, for the `Metadata` workflow we'll use: ```python theme={null} import yaml from metadata.workflow.metadata import MetadataWorkflow def run(): workflow_config = yaml.safe_load(CONFIG) workflow = MetadataWorkflow.create(workflow_config) workflow.execute() workflow.raise_from_status() workflow.print_status() workflow.stop() ``` The classes for each workflow type are: * `Metadata`: `from metadata.workflow.metadata import MetadataWorkflow` * `Lineage`: `from metadata.workflow.metadata import MetadataWorkflow` (same as metadata) * `Usage`: `from metadata.workflow.usage import UsageWorkflow` * `dbt`: `from metadata.workflow.metadata import MetadataWorkflow` * `Profiler`: `from metadata.workflow.profiler import ProfilerWorkflow` * `Data Quality`: `from metadata.workflow.data_quality import TestSuiteWorkflow` * `Data Insights`: `from metadata.workflow.data_insight import DataInsightWorkflow` * `Elasticsearch Reindex`: `from metadata.workflow.metadata import MetadataWorkflow` (same as metadata) # Kubernetes Native Orchestrator Source: https://docs.open-metadata.org/v2.0.x/deployment/ingestion/kubernetes Run ingestion pipelines using native Kubernetes Jobs and CronJobs without requiring Apache Airflow. # Kubernetes Native Orchestrator Starting with OpenMetadata 1.12, you can run ingestion pipelines directly using **native Kubernetes**, eliminating the need for Apache Airflow. This is ideal for organizations that: * Already run workloads on Kubernetes and prefer native solutions * Don't need the full feature set of Apache Airflow ## Orchestration Modes The Kubernetes orchestrator supports two modes for running ingestion pipelines: ### Option 1: OMJob Operator (Recommended) Uses custom Kubernetes CRDs (`OMJob` and `CronOMJob`) managed by the OpenMetadata operator. | Resource | Description | | ------------- | ------------------------------------------------------ | | **CronOMJob** | Scheduled pipelines - runs on a cron schedule | | **OMJob** | On-demand pipelines - one-off execution when triggered | **Recommended for production.** The OMJob Operator provides guaranteed exit handler execution and failure diagnostics. **Advantages:** * **Exit Handler Guarantee**: Even if the ingestion pod crashes (OOMKilled, node failure, etc.), the operator ensures pipeline status is always reported back to OpenMetadata * **Failure Diagnostics**: Automatically collects detailed error context from pod logs and events when pipelines fail * **Pod Lifecycle Monitoring**: The operator watches pod events and updates pipeline status in real-time **Requirements:** * Elevated permissions to install Custom Resource Definitions (CRDs) * The OMJob Operator deployment running in your cluster ### Option 2: Native Kubernetes Jobs Uses standard Kubernetes resources (`Job` and `CronJob`) without any custom CRDs. | Resource | Description | | ----------- | ------------------------------------------------------ | | **CronJob** | Scheduled pipelines - runs on a cron schedule | | **Job** | On-demand pipelines - one-off execution when triggered | **Advantages:** * No CRD installation required - uses only built-in Kubernetes resources * Works in environments with restricted permissions * Simpler setup **Limitations:** * No guaranteed exit handler - if a pod is killed unexpectedly, status updates may not reach OpenMetadata * No automatic failure diagnostics ## Features Pipelines run as standard Kubernetes Jobs, making them easy to monitor with existing K8s tooling. Pipeline status is automatically reported back to OpenMetadata, including success/failure details. When pipelines fail, detailed diagnostics are collected from pod logs and events. **(OMJob Operator only)** Configure CPU, memory, node selectors, and security contexts for ingestion pods. *** ## Setup Option 1: OMJob Operator (Recommended) This setup uses custom CRDs for guaranteed exit handler execution and failure diagnostics. ### Prerequisites 1. **OpenMetadata deployed on Kubernetes** (Helm chart recommended) 2. **Permissions to install CRDs** in your cluster 3. **Ingestion image** accessible from your cluster (`docker.getcollate.io/openmetadata/ingestion-base`) ### Helm Values Configuration Always pin `ingestionImage` to the same version as your OpenMetadata server — never use `:latest`. The `:latest` tag moves whenever a new release is published; any Kubernetes node without the previous image cached will silently pull the newer version on the next scheduled job, causing a server/client version mismatch. The ingestion major version must match the server major version (for example, a 2.0.x server requires a 2.0.x ingestion image). ```yaml theme={null} # Enable the OMJob Operator omjobOperator: enabled: true image: repository: docker.getcollate.io/openmetadata/omjob-operator tag: "2.0.1" pullPolicy: IfNotPresent resources: requests: cpu: "100m" memory: "128Mi" limits: cpu: "500m" memory: "256Mi" openmetadata: config: pipelineServiceClientConfig: enabled: true type: "k8s" metadataApiEndpoint: http://openmetadata:8585/api k8s: # Use the OMJob Operator useOMJobOperator: true # Container image for ingestion jobs ingestionImage: "docker.getcollate.io/openmetadata/ingestion-base:2.0.1" imagePullPolicy: "IfNotPresent" imagePullSecrets: "" # Service account for ingestion jobs serviceAccountName: "openmetadata-ingestion" # Job lifecycle settings ttlSecondsAfterFinished: 86400 # Keep completed jobs for 24 hours activeDeadlineSeconds: 7200 # Max 2 hour runtime backoffLimit: 3 # Retry up to 3 times # Job history successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 3 # Pod security context securityContext: runAsUser: 1000 runAsGroup: 1000 fsGroup: 1000 runAsNonRoot: true # Resource limits resources: limits: cpu: "2" memory: "4Gi" requests: cpu: "500m" memory: "1Gi" # Enable failure diagnostics (only works with OMJob Operator) enableFailureDiagnostics: true # RBAC - set to false if managed externally rbac: enabled: true ``` ### Required RBAC Permissions When using the OMJob Operator, additional permissions are needed for the custom resources: ```yaml theme={null} rules: # Pod management for pipeline jobs and diagnostics - apiGroups: [""] resources: ["pods", "pods/log"] verbs: ["get", "list", "create", "update", "patch", "delete"] # ConfigMaps for pipeline configuration - apiGroups: [""] resources: ["configmaps"] verbs: ["get", "list", "create", "update", "patch", "delete"] # Secrets for pipeline credentials - apiGroups: [""] resources: ["secrets"] verbs: ["get", "list", "create", "update", "patch", "delete"] # Events for diagnostics - apiGroups: [""] resources: ["events"] verbs: ["get", "list"] # Jobs and CronJobs management - apiGroups: ["batch"] resources: ["jobs", "cronjobs"] verbs: ["get", "list", "create", "update", "patch", "delete"] # OMJob CRDs - apiGroups: ["pipelines.openmetadata.org"] resources: ["omjobs"] verbs: ["get", "list", "create", "update", "patch", "delete"] - apiGroups: ["pipelines.openmetadata.org"] resources: ["omjobs/status"] verbs: ["get", "patch"] - apiGroups: ["pipelines.openmetadata.org"] resources: ["cronomjobs"] verbs: ["get", "list", "create", "update", "patch", "delete"] - apiGroups: ["pipelines.openmetadata.org"] resources: ["cronomjobs/status"] verbs: ["get", "patch"] ``` *** ## Setup Option 2: Native Kubernetes Jobs This setup uses standard Kubernetes Jobs and CronJobs without any custom CRDs. ### Prerequisites 1. **OpenMetadata deployed on Kubernetes** (Helm chart recommended) 2. **RBAC permissions** for the OpenMetadata service account to manage Jobs, CronJobs, ConfigMaps, and Secrets 3. **Ingestion image** accessible from your cluster (`docker.getcollate.io/openmetadata/ingestion-base`) ### Helm Values Configuration Always pin `ingestionImage` to the same version as your OpenMetadata server — never use `:latest`. The `:latest` tag moves whenever a new release is published; any Kubernetes node without the previous image cached will silently pull the newer version on the next scheduled job, causing a server/client version mismatch. The ingestion major version must match the server major version (for example, a 2.0.x server requires a 2.0.x ingestion image). ```yaml theme={null} openmetadata: config: pipelineServiceClientConfig: enabled: true type: "k8s" metadataApiEndpoint: http://openmetadata:8585/api k8s: # Do NOT use the OMJob Operator (default) useOMJobOperator: false # Container image for ingestion jobs ingestionImage: "docker.getcollate.io/openmetadata/ingestion-base:2.0.1" imagePullPolicy: "IfNotPresent" imagePullSecrets: "" # Service account for ingestion jobs serviceAccountName: "openmetadata-ingestion" # Job lifecycle settings ttlSecondsAfterFinished: 86400 activeDeadlineSeconds: 7200 backoffLimit: 3 # Job history successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 3 # Pod security context securityContext: runAsUser: 1000 runAsGroup: 1000 fsGroup: 1000 runAsNonRoot: true # Resource limits resources: limits: cpu: "2" memory: "4Gi" requests: cpu: "500m" memory: "1Gi" # RBAC - set to false if managed externally rbac: enabled: true ``` ### Required RBAC Permissions ```yaml theme={null} rules: # Pod management for pipeline jobs - apiGroups: [""] resources: ["pods", "pods/log"] verbs: ["get", "list", "create", "update", "patch", "delete"] # ConfigMaps for pipeline configuration - apiGroups: [""] resources: ["configmaps"] verbs: ["get", "list", "create", "update", "patch", "delete"] # Secrets for pipeline credentials - apiGroups: [""] resources: ["secrets"] verbs: ["get", "list", "create", "update", "patch", "delete"] # Events for diagnostics - apiGroups: [""] resources: ["events"] verbs: ["get", "list"] # Jobs and CronJobs management - apiGroups: ["batch"] resources: ["jobs", "cronjobs"] verbs: ["get", "list", "create", "update", "patch", "delete"] ``` *** For validating your setup, viewing pipeline logs, troubleshooting, and migrating from Airflow, see the [Operations & Troubleshooting](/v2.0.x/deployment/ingestion/kubernetes/troubleshooting) guide. # Kubernetes Orchestrator Operations & Troubleshooting Source: https://docs.open-metadata.org/v2.0.x/deployment/ingestion/kubernetes/troubleshooting Validate, monitor, and troubleshoot the Kubernetes native orchestrator for OpenMetadata ingestion pipelines. # Kubernetes Orchestrator Operations & Troubleshooting This guide covers validating your Kubernetes orchestrator setup, viewing pipeline logs, troubleshooting common issues, and migrating from Airflow. For initial setup, see the [Kubernetes Native Orchestrator](/v2.0.x/deployment/ingestion/kubernetes) guide. ## Validating the Setup ### 1. Check Service Health Navigate to **Settings → Preferences → Health** in the OpenMetadata UI to verify the Kubernetes pipeline client is properly configured and can connect to the Kubernetes API. ### 2. Deploy a Test Pipeline Create a simple metadata ingestion pipeline from the OpenMetadata UI. The pipeline should: * Show "Deployed" status * Display the Kubernetes Job/CronJob name ### 3. Check Kubernetes Resources ```bash theme={null} # List ingestion ConfigMaps kubectl get configmaps -l app.kubernetes.io/managed-by=openmetadata # List ingestion Jobs kubectl get jobs -l app.kubernetes.io/managed-by=openmetadata # List ingestion CronJobs (native mode) kubectl get cronjobs -l app.kubernetes.io/managed-by=openmetadata # List CronOMJobs (operator mode) kubectl get cronomjobs -l app.kubernetes.io/managed-by=openmetadata # View pod logs kubectl logs -l app.kubernetes.io/component=ingestion -f ``` ## Pipeline Logs Pipeline logs are retrieved directly from Kubernetes pod logs. OpenMetadata implements log pagination for large log files, splitting them into \~1MB chunks for efficient retrieval. To view logs: 1. Navigate to **Settings → Services → Agents** 2. Select your pipeline 3. Click on Logs to view them directly on OpenMetadata UI Alternatively, view logs directly with kubectl: ```bash theme={null} kubectl logs job/ -c main ``` ## Troubleshooting ### Server/Client Version Mismatch If you see an error like `server version X does not match client version Y`, the ingestion job is running a different version of the ingestion library than the OpenMetadata server expects. **Most common cause:** `ingestionImage` is set to `:latest` in your Helm values. When a new OpenMetadata release is published, `:latest` moves to that version. Any Kubernetes node without the old image cached will pull the new one on the next job run, silently upgrading the ingestion client while your server stays on the previous version. **Fix:** Pin `ingestionImage` to the exact version of your server: ```yaml theme={null} k8s: ingestionImage: "docker.getcollate.io/openmetadata/ingestion-base:2.0.1" ``` Replace `2.0.1` with your actual server version. The ingestion major version must match the server major version — for example, a 2.0.x server requires a 2.0.x ingestion image. After updating, redeploy OpenMetadata: ```bash theme={null} helm upgrade openmetadata open-metadata/openmetadata -f values.yaml -n openmetadata ``` ### Pipeline stuck in "Queued" state If the pipeline cannot start and remains in "Queued" state, check if the pod can be scheduled: ```bash theme={null} kubectl get pods -l app.kubernetes.io/pipeline= kubectl describe pod ``` Common causes: * Image pull errors (check `imagePullSecrets`) * Insufficient cluster resources (increase CPU/memory limits or add nodes) * Node selector constraints ### Permission Denied Errors If you see RBAC-related errors: ```bash theme={null} kubectl auth can-i create jobs --as=system:serviceaccount::openmetadata ``` Ensure the service account has the required permissions. ### Ingestion Pod Crashes (OOMKilled) Increase memory limits in the Helm values: ```yaml theme={null} k8s: resources: limits: memory: "8Gi" requests: memory: "2Gi" ``` ### CronJob Not Triggering Check CronJob status and events: ```bash theme={null} kubectl get cronjob -o yaml kubectl describe cronjob ``` Common issues: * Invalid cron expression * `startingDeadlineSeconds` too short * Concurrency policy blocking execution ## Migrating from Airflow If you're migrating from Airflow to the Kubernetes orchestrator: 1. **Stop existing Airflow-managed pipelines** - Disable or delete pipelines managed by Airflow 2. **Update Helm values** - Switch `type: "airflow"` to `type: "k8s"` 3. **Redeploy OpenMetadata** - Apply the new Helm configuration 4. **Re-deploy pipelines** - Navigate to each pipeline and click "Deploy" to create the Kubernetes resources The migration does not automatically transfer pipeline schedules. You'll need to re-configure and deploy each pipeline after switching to the Kubernetes orchestrator. ## Comparison: Airflow vs Kubernetes Orchestrator | Feature | Airflow | K8s Native | K8s with OMJob Operator | | -------------------------- | --------------------------- | ------------------------- | ------------------------- | | **Infrastructure** | Requires Airflow deployment | Uses existing K8s cluster | Uses existing K8s cluster | | **CRD Installation** | N/A | Not required | Required | | **Exit Handler Guarantee** | ✅ Airflow handles | ❌ Best effort | ✅ Guaranteed | | **Failure Diagnostics** | ❌ | ❌ | ✅ | | **UI for DAGs** | ✅ Airflow UI | OpenMetadata UI | OpenMetadata UI | | **Resource efficiency** | Always running | Jobs on-demand | Jobs on-demand | | **K8s-native monitoring** | Extra setup | ✅ Native | ✅ Native | # Run the ingestion from the OpenMetadata UI Source: https://docs.open-metadata.org/v2.0.x/deployment/ingestion/openmetadata Learn how to deploy and configure OpenMetadata Ingestion pipelines. Complete setup guide with connectors, scheduling, and best practices. # Run the ingestion from the OpenMetadata UI When you create and manage ingestion workflows from OpenMetadata, under the hood we need to communicate with an orchestration system. It does not matter which one, but we need it to have a set of APIs to create, run, fetch the logs, etc. of our workflows. openmetadata-orchestration OpenMetadata supports two orchestration backends: | Orchestrator | Description | | --------------------- | ----------------------------------------------------------------------------- | | **Apache Airflow** | The traditional approach - uses Airflow DAGs to manage pipelines | | **Kubernetes Native** | **New in 1.12** - Uses native K8s Jobs and CronJobs without requiring Airflow | Continue below for Airflow configuration Use native K8s Jobs (no Airflow required) *** ## Airflow as Orchestrator Out of the box, OpenMetadata comes with integration for Airflow. In this guide, we will show you how to manage ingestions from OpenMetadata by linking it to an Airflow service. Advanced note for developers: We have an [interface](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/java/org/openmetadata/service/clients/pipeline/PipelineServiceClient.java) that can be extended to bring support to any other orchestrator. You can follow the implementation we have for [Airflow](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/clients/pipeline/airflow/AirflowRESTClient.java) or [Kubernetes](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/clients/pipeline/k8s/K8sPipelineClient.java) as starting points. 1. **If you do not have an Airflow service** up and running on your platform, we provide a custom [Docker](https://hub.docker.com/r/openmetadata/ingestion) image, which already contains the OpenMetadata ingestion packages and custom [Airflow APIs](https://github.com/open-metadata/openmetadata-airflow-apis) to deploy Workflows from the UI as well. **This is the simplest approach**. 2. If you already have Airflow up and running and want to use it for the metadata ingestion, you will need to install the ingestion modules to the host. You can find more information on how to do this in the Custom Airflow Installation section. ## Airflow permissions These are the permissions required by the user that will manage the communication between the OpenMetadata Server and Airflow's Webserver: ``` [ (permissions.ACTION_CAN_DELETE, permissions.RESOURCE_DAG), (permissions.ACTION_CAN_CREATE, permissions.RESOURCE_DAG), (permissions.ACTION_CAN_EDIT, permissions.RESOURCE_DAG), (permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG), ] ``` `User` permissions is enough for these requirements. You can find more information on Airflow's Access Control [here](https://airflow.apache.org/docs/apache-airflow/stable/security/access-control.html). ## Shared Volumes The Airflow Webserver, Scheduler and Workers - if using a distributed setup - need to have access to the same shared volumes with RWX permissions. We have specific instructions on how to set up the shared volumes in Kubernetes depending on your cloud deployment [here](/v2.0.x/deployment/kubernetes). ## Using the OpenMetadata Ingestion Image If you are using our `openmetadata/ingestion` Docker image, there is just one thing to do: Configure the OpenMetadata server. The OpenMetadata server takes all its configurations from a YAML file. You can find them in our [repo](https://github.com/open-metadata/OpenMetadata/tree/main/conf). In `openmetadata.yaml`, update the `pipelineServiceClientConfiguration` section accordingly. ```yaml theme={null} # For Bare Metal Installations [...] pipelineServiceClientConfiguration: className: ${PIPELINE_SERVICE_CLIENT_CLASS_NAME:-"org.openmetadata.service.clients.pipeline.airflow.AirflowRESTClient"} apiEndpoint: ${PIPELINE_SERVICE_CLIENT_ENDPOINT:-http://localhost:8080} metadataApiEndpoint: ${SERVER_HOST_API_URL:-http://localhost:8585/api} hostIp: ${PIPELINE_SERVICE_CLIENT_HOST_IP:-""} verifySSL: ${PIPELINE_SERVICE_CLIENT_VERIFY_SSL:-"no-ssl"} # Possible values are "no-ssl", "ignore", "validate" sslConfig: certificatePath: ${PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH:-""} # Local path for the Pipeline Service Client # Default required parameters for Airflow as Pipeline Service Client parameters: username: ${AIRFLOW_USERNAME:-admin} password: ${AIRFLOW_PASSWORD:-admin} timeout: ${AIRFLOW_TIMEOUT:-10} [...] ``` If using Docker, make sure that you are passing the correct environment variables: ```env theme={null} PIPELINE_SERVICE_CLIENT_ENDPOINT: ${PIPELINE_SERVICE_CLIENT_ENDPOINT:-http://ingestion:8080} SERVER_HOST_API_URL: ${SERVER_HOST_API_URL:-http://openmetadata-server:8585/api} ``` If using Kubernetes, make sure that you are passing the correct values to Helm Chart: ```yaml theme={null} # Custom OpenMetadata Values.yaml openmetadata: config: pipelineServiceClientConfig: enabled: true # endpoint url for airflow apiEndpoint: http://openmetadata-dependencies-web.default.svc.cluster.local:8080 auth: username: admin password: secretRef: airflow-secrets secretKey: openmetadata-airflow-password ``` ## Custom Airflow Installation * The `openmetadata-ingestion` package requires Python 3.10 or later. * The `airflow` extra installs Apache Airflow 3.3.1 for the 2.0 release line. Use this extra when creating a new Airflow environment. * If you already run Airflow, do not install the `airflow` extra. Install only the connector extras you need. The `all` extra intentionally excludes Airflow, so it does not install or upgrade Airflow on an existing host. You will need to follow three steps: 1. Install the `openmetadata-ingestion` package with the connector plugins that you need. 2. Install the `openmetadata-managed-apis` to deploy our custom APIs on top of Airflow. 3. Configure the Airflow environment. 4. Configure the OpenMetadata server. ### 1. Install the Connector Modules The current approach we are following here is preparing the metadata ingestion DAGs as `PythonOperators`. This means that the packages need to be present in the Airflow instances. You will need to install: ```python theme={null} pip3 install "openmetadata-ingestion[]==x.y.z" ``` And then run the DAG as explained in each [Connector](/v2.0.x/connectors), where `x.y.z` is the same version of your OpenMetadata server. For example, if you are on version 1.0.0, then you can install the `openmetadata-ingestion` with versions `1.0.0.*`, e.g., `1.0.0.0`, `1.0.0.1`, etc., but not `1.0.1.x`. You can also install `openmetadata-ingestion[all]==x.y.z`, which will bring the requirements to run any connector. This extra intentionally excludes the separate `airflow` extra; install `openmetadata-ingestion[airflow]==x.y.z` only when you want the packaged Airflow environment. You can check the [Connector Modules](/v2.0.x/connectors) guide above to learn how to install the `openmetadata-ingestion` package with the necessary plugins. They are necessary because even if we install the APIs, the Airflow instance needs to have the required libraries to connect to each source. ### 2. Install the Airflow APIs The `openmetadata-ingestion-apis` has a dependency on `apache-airflow>=2.2.2`. Please make sure that your host satisfies such requirement. Only installing the `openmetadata-ingestion-apis` won't result in a proper full Airflow installation. For that, please follow the Airflow [docs](https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html). The goal of this module is to add some HTTP endpoints that the UI calls for deploying the Airflow DAGs. The first step can be achieved by running: ```python theme={null} pip3 install "openmetadata-managed-apis==x.y.z" ``` Here, the same versioning logic applies: `x.y.z` is the same version of your OpenMetadata server. For example, if you are on version 1.0.0, then you can install the `openmetadata-managed-apis` with versions `1.0.0.*`, e.g., `1.0.0.0`, `1.0.0.1`, etc., but not `1.0.1.x`. ### 3. Configure the Airflow environment The ingestion image is built on Airflow's base image, ensuring it includes all necessary requirements to run Airflow. For Kubernetes deployments, the setup uses community Airflow charts with a modified base image, enabling it to function seamlessly as a **scheduler**, **webserver**, and **worker**. We need a couple of settings: #### AIRFLOW\_HOME The APIs will look for the `AIRFLOW_HOME` environment variable to place the dynamically generated DAGs. Make sure that the variable is set and reachable from Airflow. #### Airflow APIs Basic Auth Note that the integration of OpenMetadata with Airflow requires Basic Auth in the APIs. Make sure that your Airflow configuration supports that. You can read more about it [here](https://airflow.apache.org/docs/apache-airflow/stable/security/api.html). A possible approach here is to update your `airflow.cfg` entries for Airflow 3.x: ``` [api] auth_backends = airflow.api_fastapi.auth.backend.basic_auth ``` #### DAG Generated Configs Every time a DAG is created from OpenMetadata, it will also create a JSON file with some information about the workflow that needs to be executed. By default, these files live under `${AIRFLOW_HOME}/dag_generated_configs`, which in most environments translates to `/opt/airflow/dag_generated_configs`. You can change this directory by specifying the environment variable `AIRFLOW__OPENMETADATA_AIRFLOW_APIS__DAG_GENERATED_CONFIGS` or updating the `airflow.cfg` with: ```cfg theme={null} [openmetadata_airflow_apis] dag_generated_configs=/opt/airflow/dag_generated_configs ``` A safe way to validate if the configuration is properly set in Airflow is to run: ```bash theme={null} airflow config get-value openmetadata_airflow_apis dag_generated_configs ``` ### 4. Configure in the OpenMetadata Server After installing the Airflow APIs, you will need to update your OpenMetadata Server. The OpenMetadata server takes all its configurations from a YAML file. You can find them in our [repo](https://github.com/open-metadata/OpenMetadata/tree/main/conf). In `openmetadata.yaml`, update the `pipelineServiceClientConfiguration` section accordingly. ```yaml theme={null} # For Bare Metal Installations [...] pipelineServiceClientConfiguration: className: ${PIPELINE_SERVICE_CLIENT_CLASS_NAME:-"org.openmetadata.service.clients.pipeline.airflow.AirflowRESTClient"} apiEndpoint: ${PIPELINE_SERVICE_CLIENT_ENDPOINT:-http://localhost:8080} metadataApiEndpoint: ${SERVER_HOST_API_URL:-http://localhost:8585/api} hostIp: ${PIPELINE_SERVICE_CLIENT_HOST_IP:-""} verifySSL: ${PIPELINE_SERVICE_CLIENT_VERIFY_SSL:-"no-ssl"} # Possible values are "no-ssl", "ignore", "validate" sslConfig: certificatePath: ${PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH:-""} # Local path for the Pipeline Service Client # Default required parameters for Airflow as Pipeline Service Client parameters: username: ${AIRFLOW_USERNAME:-admin} password: ${AIRFLOW_PASSWORD:-admin} timeout: ${AIRFLOW_TIMEOUT:-10} [...] ``` If using Docker, make sure that you are passing the correct environment variables: ```env theme={null} PIPELINE_SERVICE_CLIENT_ENDPOINT: ${PIPELINE_SERVICE_CLIENT_ENDPOINT:-http://ingestion:8080} SERVER_HOST_API_URL: ${SERVER_HOST_API_URL:-http://openmetadata-server:8585/api} ``` If using Kubernetes, make sure that you are passing the correct values to Helm Chart: ```yaml theme={null} # Custom OpenMetadata Values.yaml openmetadata: config: pipelineServiceClientConfig: enabled: true # endpoint url for airflow apiEndpoint: http://openmetadata-dependencies-web.default.svc.cluster.local:8080 auth: username: admin password: secretRef: airflow-secrets secretKey: openmetadata-airflow-password ``` *** For installation validation, Git Sync guidance, SSL configuration, and troubleshooting Airflow pipeline issues, see the [Airflow Troubleshooting & Advanced](/v2.0.x/deployment/ingestion/openmetadata/troubleshooting) guide. # Airflow Troubleshooting & Advanced | OpenMetadata Ingestion Source: https://docs.open-metadata.org/v2.0.x/deployment/ingestion/openmetadata/troubleshooting Validate your Airflow installation, configure Git Sync and SSL, and troubleshoot common ingestion pipeline deployment issues. # Airflow Troubleshooting & Advanced This page covers installation validation, Git Sync guidance, SSL configuration, and troubleshooting for Airflow-based ingestion pipelines. For setup and configuration, see the [OpenMetadata Ingestion Overview](/v2.0.x/deployment/ingestion/openmetadata). ## Validating the installation What we need to verify here is that the OpenMetadata server can reach the Airflow APIs endpoints (wherever they live: bare metal, containers, k8s pods...). One way to ensure that is to connect to the deployment hosting your OpenMetadata server and running a query against the `/health` endpoint. For example: ```bash theme={null} $ curl -XGET ${PIPELINE_SERVICE_CLIENT_ENDPOINT}/api/v1/openmetadata/health {"status": "healthy", "version": "x.y.z"} ``` It is important to do this validation passing the command as is (i.e., `curl -XGET ${PIPELINE_SERVICE_CLIENT_ENDPOINT}/api/v1/openmetadata/health`) and allowing the environment to do the substitution for you. That's the only way we can be sure that the setup is correct. #### More validations in the installation If you have an existing DAG in Airflow, you can further test your setup by running the following: ```bash theme={null} curl -XPOST http://localhost:8080/api/v1/openmetadata/enable --data-raw '{"dag_id": "example_bash_operator"}' -u "admin:admin" --header 'Content-Type: application/json' ``` Note that in this example we are assuming: * There is an Airflow instance running at `localhost:8080`, * There is a user `admin` with password `admin` * There is a DAG named `example_bash_operator`. A generic call would look like: ```bash theme={null} curl -XPOST /api/v1/openmetadata/enable --data-raw '{"dag_id": ""}' -u ":" --header 'Content-Type: application/json' ``` Please update it accordingly. ## Git Sync? One recurrent question when setting up Airflow is the possibility of using [git-sync](https://airflow.apache.org/docs/helm-chart/1.7.0/manage-dags-files.html#mounting-dags-from-a-private-github-repo-using-git-sync-sidecar) to manage the ingestion DAGs. Let's remark the differences between `git-sync` and what we want to achieve by installing our custom API plugins: 1. `git-sync` will use Git as the source of truth for your DAGs. Meaning, any DAG you have on Git will eventually be used and scheduled in Airflow. 2. With the `openmetadata-managed-apis` we are using the OpenMetadata server as the source of truth. We are enabling dynamic DAG creation from the OpenMetadata into your Airflow instance every time that you create a new Ingestion Workflow. Then, should you use `git-sync`? * If you have an existing Airflow instance, and you want to build and maintain your own ingestion DAGs then you can go for it. Check a DAG example [here](/v2.0.x/deployment/ingestion/external/airflow#example). * If instead, you want to use the full deployment process from OpenMetadata, `git-sync` would not be the right tool, since the DAGs won't be backed up by Git, but rather created from OpenMetadata. Note that if anything would to happen where you might lose the Airflow volumes, etc. You can just redeploy the DAGs from OpenMetadata. ## SSL If you want to learn how to set up Airflow using SSL, you can learn more here: Learn how to configure Airflow with SSL. ## Troubleshooting ## Ingestion Pipeline deployment issues ### Airflow APIs Not Found Validate the installation, making sure that from the OpenMetadata server you can reach the Airflow host, and the call to `/health` gives us the proper response: ```bash theme={null} $ curl -XGET ${PIPELINE_SERVICE_CLIENT_ENDPOINT}/api/v1/openmetadata/health {"status": "healthy", "version": "x.y.z"} ``` Also, make sure that the version of your OpenMetadata server matches the `openmetadata-ingestion` client version installed in Airflow. ### GetServiceException: Could not get service from type XYZ In this case, the OpenMetadata client running in the Airflow host had issues getting the service you are trying to deploy from the API. Note that once pipelines are deployed, the auth happens via the `ingestion-bot`. Here there are a couple of points to validate: 1. The JWT of the ingestion bot is valid. You can check services such as [https://jwt.io/](https://jwt.io/) to help you review if the token is expired or if there are any configuration issues. 2. The `ingestion-bot` does not have the proper role. If you go to `/bots/ingestion-bot`, the bot should present the `Ingestion bot role`. You can validate the role policies as well to make sure they were not updated and the bot can indeed view and access services from the API. 3. Run an API call for your service to verify the issue. An example trying to get a database service would look like follows: ``` curl -XGET 'http://:8585/api/v1/services/databaseServices/name/' \ -H 'Accept: application/json' -H 'Authorization: Bearer ' ``` If, for example, you have an issue with the roles you would be getting a message similar to: ``` {"code":403,"message":"Principal: CatalogPrincipal{name='ingestion-bot'} operations [ViewAll] not allowed"} ``` ### AirflowException: Dag 'XYZ' could not be found If you're seeing a similar error to ``` [...] task_run _dag = get_dag(args.subdir, args.dag_id) File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/cli.py", line 235, in get_dag raise AirflowException( airflow.exceptions.AirflowException: Dag '...' could not be found; either it does not exist or it failed to parse. ``` This is a common situation where you have not properly enabled the shared volumes between Webserver \<> Scheduler \<> Worker in your distributed environment. We have specific instructions on how to set up the shared volumes in Kubernetes depending on your cloud deployment [here](/v2.0.x/deployment/kubernetes). ### ClientInitializationError The main root cause here is a version mismatch between the server and the client. Make sure that the `openmetadata-ingestion` python package you installed on the Airflow host has the same version as the OpenMetadata server. For example, to set up OpenMetadata server 2.0.1 you will need to install `openmetadata-ingestion~=2.0.1.0`. Note that we are validating the version as in `x.y.z`. Any differences after the PATCH versioning are not taken into account, as they are usually small bugfixes on existing functionalities. ### 401 Unauthorized If you get this response during a `Test Connection` or `Deploy`: ``` airflow API returned Unauthorized and response { "detail": null, "status": 401, "title": "Unauthorized", "type": "https://airflow.apache.org/docs/apache-airflow/2.3.3/stable-rest-api-ref.html#section/Errors/Unauthenticated" } ``` This is a communication issue between the OpenMetadata Server and the Airflow instance. You are able to reach the Airflow host, but your provided user and password are not correct. Note the following section of the server configuration: ```yaml theme={null} pipelineServiceClientConfiguration: [...] parameters: username: ${AIRFLOW_USERNAME:-admin} password: ${AIRFLOW_PASSWORD:-admin} ``` You should validate if the content of the environment variables `AIRFLOW_USERNAME` and `AIRFLOW_PASSWORD` allow you to authenticate to the instance. ### CentOS / Debian - The name 'template\_blueprint' is already registered If you are using a CentOS / Debian system to install the `openmetadata-managed-apis` you might encounter the following issue when starting Airflow: ```bash theme={null} airflow standalone standalone | Starting Airflow Standalone standalone | Checking database is initialized INFO [alembic.runtime.migration] Context impl SQLiteImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. WARNI [airflow.models.crypto] empty cryptography key - values will not be stored encrypted. standalone | Database ready [2023-08-11 05:39:28,851] {manager.py:508} INFO - Created Permission View: can create on DAGs [2023-08-11 05:39:28,910] {manager.py:508} INFO - Created Permission View: menu access on REST API Plugin [2023-08-11 05:39:28,916] {manager.py:568} INFO - Added Permission menu access on REST API Plugin to role Admin Traceback (most recent call last): File "/home/pmcevoy/airflow233/bin/airflow", line 8, in sys.exit(main()) File "/home/pmcevoy/airflow233/lib64/python3.9/site-packages/airflow/__main__.py", line 38, in main args.func(args) File "/home/pmcevoy/airflow233/lib64/python3.9/site-packages/airflow/cli/cli_parser.py", line 51, in command return func(*args, **kwargs) File "/home/pmcevoy/airflow233/lib64/python3.9/site-packages/airflow/cli/commands/standalone_command.py", line 48, in entrypoint StandaloneCommand().run() File "/home/pmcevoy/airflow233/lib64/python3.9/site-packages/airflow/cli/commands/standalone_command.py", line 64, in run self.initialize_database() File "/home/pmcevoy/airflow233/lib64/python3.9/site-packages/airflow/cli/commands/standalone_command.py", line 180, in initialize_database appbuilder = cached_app().appbuilder File "/home/pmcevoy/airflow233/lib64/python3.9/site-packages/airflow/www/app.py", line 158, in cached_app app = create_app(config=config, testing=testing) File "/home/pmcevoy/airflow233/lib64/python3.9/site-packages/airflow/www/app.py", line 140, in create_app init_plugins(flask_app) File "/home/pmcevoy/airflow233/lib64/python3.9/site-packages/airflow/www/extensions/init_views.py", line 141, in init_plugins app.register_blueprint(blue_print["blueprint"]) File "/home/pmcevoy/airflow233/lib64/python3.9/site-packages/flask/scaffold.py", line 56, in wrapper_func return f(self, *args, **kwargs) File "/home/pmcevoy/airflow233/lib64/python3.9/site-packages/flask/app.py", line 1028, in register_blueprint blueprint.register(self, options) File "/home/pmcevoy/airflow233/lib64/python3.9/site-packages/flask/blueprints.py", line 305, in register raise ValueError( ValueError: The name 'template_blueprint' is already registered for this blueprint. Use 'name=' to provide a unique name. ``` The issue occurs because a symlink exists inside the `venv` ```bash theme={null} (airflow233) [pmcevoy@lab1 airflow233]$ ls -la total 28 drwxr-xr-x 6 pmcevoy pmcevoy 4096 Aug 14 00:34 . drwx------ 6 pmcevoy pmcevoy 4096 Aug 14 00:32 .. drwxr-xr-x 3 pmcevoy pmcevoy 4096 Aug 14 00:34 bin drwxr-xr-x 3 pmcevoy pmcevoy 4096 Aug 14 00:33 include drwxr-xr-x 3 pmcevoy pmcevoy 4096 Aug 14 00:32 lib lrwxrwxrwx 1 pmcevoy pmcevoy 3 Aug 14 00:32 lib64 -> lib -rw-r--r-- 1 pmcevoy pmcevoy 70 Aug 14 00:32 pyvenv.cfg drwxr-xr-x 3 pmcevoy pmcevoy 4096 Aug 14 00:34 share ``` ```bash theme={null} (airflow233) [pmcevoy@lab1 airflow233]$ grep -r template_blueprint * lib/python3.9/site-packages/openmetadata_managed_apis/plugin.py:template_blueprint = Blueprint( lib/python3.9/site-packages/openmetadata_managed_apis/plugin.py: "template_blueprint", lib/python3.9/site-packages/openmetadata_managed_apis/plugin.py: flask_blueprints = [template_blueprint, api_blueprint] grep: lib/python3.9/site-packages/openmetadata_managed_apis/__pycache__/plugin.cpython-39.pyc: binary file matches lib64/python3.9/site-packages/openmetadata_managed_apis/plugin.py:template_blueprint = Blueprint( lib64/python3.9/site-packages/openmetadata_managed_apis/plugin.py: "template_blueprint", lib64/python3.9/site-packages/openmetadata_managed_apis/plugin.py: flask_blueprints = [template_blueprint, api_blueprint] grep: lib64/python3.9/site-packages/openmetadata_managed_apis/__pycache__/plugin.cpython-39.pyc: binary file matches ``` A workaround is to remove the `lib64` symlink: `rm lib64`. # Kubernetes Deployment | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/kubernetes Deploy the OpenMetadata on Kubernetes using Helm, custom values, and supported cloud configurations for scalable containerized environments. # Kubernetes Deployment OpenMetadata supports the Installation and Running of Application on kubernetes through Helm Charts. ## Kubernetes Deployment Architecture Below is the expected Kubernetes Deployment Architecture for OpenMetadata Application in **Production**. Kubernetes Deployment Architecture In the above architecture diagram, OpenMetadata Application is deployed using Helm Charts. The various kubernetes manifests that supports the installation. With the above architecture, OpenMetadata Application Connects with external dependencies which is Database, ElasticSearch and Orchestration tools like airflow. The OpenMetadata Helm Charts Exposes the Application from Kubernetes Service at Port `8585` and `8586`. The Health Checks and Metrics endpoints are available on port `8586`. Network Policies and Ingresses are optional manifests and disabled by default. These can be installed / enabled using the [Helm Values](/v2.0.x/deployment/kubernetes/on-prem). ## Links For customizing OpenMetadata Helm Deployments Run ingestion pipelines as native K8s Jobs (no Airflow required) Deploy OpenMetadata in AWS Kubernetes Deploy OpenMetadata in GCP Kubernetes Deploy OpenMetadata in Azure Kubernetes Deploy OpenMetadata in On Premises Kubernetes # AKS Deployment: Prerequisites & Kubernetes Orchestrator Source: https://docs.open-metadata.org/v2.0.x/deployment/kubernetes/aks Configure OpenMetadata on Azure Kubernetes Service using the recommended Kubernetes-native orchestrator with Helm charts and scalable configuration templates. # OpenMetadata Deployment on Azure Kubernetes Service Cluster OpenMetadata can be deployed on Azure Kubernetes Service. This guide covers both the recommended Kubernetes orchestrator (new in 1.12) and the alternative Airflow-based orchestrator. ## Prerequisites ### Azure Services for Database and Search Engine as Elastic Cloud It is recommended to use [Azure SQL](https://azure.microsoft.com/en-in/products/azure-sql/database) and [Elastic Cloud on Azure](https://www.elastic.co/partners/microsoft-azure) for Production Deployments. We support: * Azure SQL (MySQL) engine version 8 or higher * Azure SQL (PostgreSQL) engine version 15 or higher * Elastic Cloud (ElasticSearch version 9.x, minimum 9.0.0, recommended 9.3.0) We recommend: * Azure SQL to be Multi Zone Available and Production Workload Environment * Elastic Cloud Environment with multiple zones and minimum 2 nodes ### Step 1 - Create an AKS Cluster If you are deploying on a new cluster set the `EnableAzureDiskFileCSIDriver=true` to enable container storage interface storage drivers. ```azure-cli theme={null} az aks create --resource-group MyResourceGroup \ --name MyAKSClusterName \ --nodepool-name agentpool \ --outbound-type loadbalancer \ --location YourPreferredLocation \ --generate-ssh-keys \ --enable-addons monitoring \ EnableAzureDiskFileCSIDriver=true ``` For existing cluster it is important to enable the CSI storage drivers: ```azure-cli theme={null} az aks update -n MyAKSCluster -g MyResourceGroup --enable-disk-driver --enable-file-driver ``` ### Step 2 - Create a Namespace (optional) ```azure-cli theme={null} kubectl create namespace openmetadata ``` ### Step 3 - Add the Helm OpenMetadata Repo ```azure-cli theme={null} helm repo add open-metadata https://helm.open-metadata.org/ helm repo update ``` *** ## Kubernetes Orchestrator Configuration (Recommended) Starting with OpenMetadata 1.12, we recommend using the **Kubernetes native orchestrator** for running ingestion pipelines. This eliminates the need for Apache Airflow and simplifies your deployment. The Kubernetes orchestrator runs ingestion pipelines as native K8s Jobs and CronJobs. For full documentation on features, configuration options, and troubleshooting, see the [Kubernetes Orchestrator Guide](/v2.0.x/deployment/ingestion/kubernetes). The recommended OMJob Operator approach requires installing Custom Resource Definitions (CRDs), which needs elevated cluster permissions. If your cluster policies don't allow CRDs, you can disable the operator by setting `useOMJobOperator: false` and `omjobOperator.enabled: false` in your values file to use native K8s Jobs instead. ### Create Kubernetes Secrets Create the required secrets for your database and search engine: ```azure-cli theme={null} # Database secret (for MySQL) kubectl create secret generic mysql-secrets \ --namespace openmetadata \ --from-literal=openmetadata-mysql-password= # ElasticSearch secret kubectl create secret generic elasticsearch-secrets \ --namespace openmetadata \ --from-literal=openmetadata-elasticsearch-password= ``` ### OpenMetadata Values Configuration Create your `openmetadata-values.yaml` with the following configuration: ```yaml theme={null} # openmetadata-values.yaml openmetadata: config: # Database configuration elasticsearch: host: searchType: elasticsearch port: 443 scheme: https connectionTimeoutSecs: 5 socketTimeoutSecs: 60 keepAliveTimeoutSecs: 600 batchSize: 10 auth: enabled: true username: password: secretRef: elasticsearch-secrets secretKey: openmetadata-elasticsearch-password database: host: port: 3306 driverClass: com.mysql.cj.jdbc.Driver dbScheme: mysql dbUseSSL: true databaseName: auth: username: password: secretRef: mysql-secrets secretKey: openmetadata-mysql-password # Kubernetes Orchestrator configuration pipelineServiceClientConfig: enabled: true type: "k8s" metadataApiEndpoint: http://openmetadata.openmetadata.svc.cluster.local:8585/api k8s: ingestionImage: "docker.getcollate.io/openmetadata/ingestion-base:2.0.1" useOMJobOperator: true # Enable the OMJob Operator (recommended for production) omjobOperator: enabled: true image: repository: docker.getcollate.io/openmetadata/omjob-operator tag: "2.0.1" image: tag: "2.0.1" ``` For advanced configuration options such as resource limits, job lifecycle settings, failure diagnostics, RBAC, and security contexts, see the [Kubernetes Orchestrator Guide](/v2.0.x/deployment/ingestion/kubernetes). For Database as PostgreSQL, use the below config for database values: ```yaml theme={null} database: host: port: 5432 driverClass: org.postgresql.Driver dbScheme: postgresql dbUseSSL: true databaseName: auth: username: password: secretRef: postgresql-secret secretKey: postgresql-password ``` ### Deploy OpenMetadata ```bash theme={null} # Install OpenMetadata (no dependencies chart needed with K8s orchestrator) helm install openmetadata open-metadata/openmetadata \ --namespace openmetadata \ --values openmetadata-values.yaml ``` With the Kubernetes orchestrator, you don't need to deploy the `openmetadata-dependencies` chart that includes Airflow. This significantly simplifies your deployment. ### Verify the Deployment ```bash theme={null} # Check pods are running kubectl get pods -n openmetadata # Check the K8s orchestrator health in OpenMetadata UI # Navigate to Settings → Preferences → Health ``` ### Access OpenMetadata ```azure-cli theme={null} kubectl port-forward service/openmetadata 8585:8585 -n openmetadata ``` # AKS Deployment: Airflow Orchestrator & Troubleshooting Source: https://docs.open-metadata.org/v2.0.x/deployment/kubernetes/aks-airflow Configure OpenMetadata on Azure Kubernetes Service using Apache Airflow as the ingestion orchestrator, including persistent volume setup and troubleshooting. # AKS Deployment: Airflow Orchestrator & Troubleshooting This page covers the Airflow-based orchestrator setup for AKS. For the simpler recommended Kubernetes-native orchestrator, see [AKS Deployment: Prerequisites & Kubernetes Orchestrator](/v2.0.x/deployment/kubernetes/aks). ## Using Airflow Orchestrator (Alternative) If you prefer to use Apache Airflow as the orchestrator (e.g., for existing Airflow investments or complex DAG requirements), follow the configuration below. Using Airflow requires additional infrastructure: persistent volumes with ReadWriteMany access, the openmetadata-dependencies Helm chart, and more complex configuration. ### Create Persistent Volumes OpenMetadata helm chart depends on Airflow and Airflow expects a persistent disk that support ReadWriteMany (the volume can be mounted as read-write by many nodes). The Azure CSI storage drivers we enabled earlier support the provisioning of the disks in ReadWriteMany mode. ```yaml theme={null} # logs_dags_pvc.yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: openmetadata-dependencies-dags-pvc namespace: openmetadata spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi storageClassName: azurefile-csi --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: openmetadata-dependencies-logs-pvc namespace: openmetadata spec: accessModes: - ReadWriteMany resources: requests: storage: 5Gi storageClassName: azurefile-csi ``` Create the volume claims by applying the manifest: ```azure-cli theme={null} kubectl apply -f logs_dags_pvc.yaml ``` ### Change Owner and Update Permission for Persistent Volumes Airflow pods run as non-root user and lack write access to our persistent volumes. To fix this we create a job permissions\_pod.yaml that runs a pod that mounts volumes into the persistent volume claim and updates the owner of the mounted folders /airflow-dags and /airflow-logs to user id 50000, which is the default linux user id of Airflow pods. ```yaml theme={null} # permissions_pod.yaml apiVersion: batch/v1 kind: Job metadata: labels: run: my-permission-pod name: my-permission-pod namespace: openmetadata spec: template: spec: containers: - image: busybox name: my-permission-pod volumeMounts: - name: airflow-dags mountPath: /airflow-dags - name: airflow-logs mountPath: /airflow-logs command: ["/bin/sh", "-c", "chown -R 50000 /airflow-dags /airflow-logs", "chmod -R a+rwx /airflow-dags"] restartPolicy: Never volumes: - name: airflow-logs persistentVolumeClaim: claimName: openmetadata-dependencies-logs-pvc - name: airflow-dags persistentVolumeClaim: claimName: openmetadata-dependencies-dags-pvc ``` Start the job by applying the manifest: ```azure-cli theme={null} kubectl apply -f permissions_pod.yaml ``` ### Create Airflow Secrets ```azure-cli theme={null} kubectl create secret generic airflow-secrets \ --namespace openmetadata \ --from-literal=openmetadata-airflow-password= ``` For production deployments connecting external postgresql database: ```azure-cli theme={null} kubectl create secret generic postgresql-secret \ --namespace openmetadata \ --from-literal=postgresql-password= ``` ### Install OpenMetadata Dependencies Create `values-dependencies.yaml` to configure Airflow with persistent volumes: ```yaml theme={null} # values-dependencies.yaml airflow: airflow: extraVolumeMounts: - mountPath: /airflow-logs name: aks-airflow-logs - mountPath: /airflow-dags/dags name: aks-airflow-dags extraVolumes: - name: aks-airflow-logs persistentVolumeClaim: claimName: openmetadata-dependencies-logs-pvc - name: aks-airflow-dags persistentVolumeClaim: claimName: openmetadata-dependencies-dags-pvc config: AIRFLOW__OPENMETADATA_AIRFLOW_APIS__DAG_GENERATED_CONFIGS: "/airflow-dags/dags" dags: path: /airflow-dags/dags persistence: enabled: false logs: path: /airflow-logs persistence: enabled: false externalDatabase: type: postgres # default mysql host: Host_db_address database: Airflow_metastore_dbname user: db_userName port: 5432 dbUseSSL: true passwordSecret: postgresql-secret passwordSecretKey: postgresql-password ``` Install the dependencies: ```azure-cli theme={null} helm install openmetadata-dependencies open-metadata/openmetadata-dependencies \ --values values-dependencies.yaml \ --namespace openmetadata \ --set mysql.enabled=false ``` It takes a few minutes for all the pods to be correctly set-up and running: ```azure-cli theme={null} kubectl get pods -n openmetadata ``` ### Install OpenMetadata with Airflow Create `openmetadata-values.yaml` for Airflow-based deployment: ```yaml theme={null} # openmetadata-values.yaml global: pipelineServiceClientConfig: apiEndpoint: http://openmetadata-dependencies-web.openmetadata.svc.cluster.local:8080 metadataApiEndpoint: http://openmetadata.openmetadata.svc.cluster.local:8585/api openmetadata: config: elasticsearch: host: searchType: elasticsearch port: 443 scheme: https auth: enabled: true username: password: secretRef: elasticsearch-secrets secretKey: openmetadata-elasticsearch-password database: host: port: 5432 driverClass: org.postgresql.Driver dbScheme: postgresql databaseName: openmetadata_db auth: username: password: secretRef: postgresql-secret secretKey: postgresql-password image: tag: "2.0.1" ``` ```azure-cli theme={null} helm install openmetadata open-metadata/openmetadata \ --values openmetadata-values.yaml \ --namespace openmetadata ``` ## Troubleshooting ### Troubleshooting Airflow ### JSONDecodeError: Unterminated string starting If you are using Airflow with Azure Blob Storage as `PersistentVolume` as explained in [Storage class using blobfuse](https://learn.microsoft.com/en-us/azure/aks/azure-csi-blob-storage-provision?tabs=mount-nfs%2Csecret), you may encounter the following error after a few days: ```bash theme={null} {dagbag.py:346} ERROR - Failed to import: /airflow-dags/dags/...py json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 3552 ``` Moreover, the Executor pods would actually be using old files. This behaviour is caused by the recommended config by the mentioned documentation: ```yaml theme={null} - -o allow_other - --file-cache-timeout-in-seconds=120 - --use-attr-cache=true - --cancel-list-on-mount-seconds=10 # prevent billing charges on mounting - -o attr_timeout=120 - -o entry_timeout=120 - -o negative_timeout=120 - --log-level=LOG_WARNING # LOG_WARNING, LOG_INFO, LOG_DEBUG - --cache-size-mb=1000 # Default will be 80% of available memory, eviction will happen beyond that. ``` **Disabling the cache** will help here. In this case it won't have any negative impact, since the `.py` and `.json` files are small enough and not heavily used. The same configuration without cache: ```yaml theme={null} - --o direct_io - --file-cache-timeout-in-seconds=0 - --use-attr-cache=false - --cancel-list-on-mount-seconds=10 - --o attr_timeout=0 - --o entry_timeout=0 - --o negative_timeout=0 - --log-level=LOG_WARNING - --cache-size-mb=0 ``` You can find more information about this error [here](https://github.com/open-metadata/OpenMetadata/issues/15321), and similar discussions [here](https://github.com/Azure/azure-storage-fuse/issues/1171) and [here](https://github.com/Azure/azure-storage-fuse/issues/1139). ## FAQs ## Java Memory Heap Issue If your openmetadata pods are not in ready state at any point in time and the openmetadata pod logs speaks about the below issue - ``` Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "AsyncAppender-Worker-async-file-appender" Exception in thread "pool-5-thread-1" java.lang.OutOfMemoryError: Java heap space Exception in thread "AsyncAppender-Worker-async-file-appender" java.lang.OutOfMemoryError: Java heap space Exception in thread "dw-46" java.lang.OutOfMemoryError: Java heap space Exception in thread "AsyncAppender-Worker-async-console-appender" java.lang.OutOfMemoryError: Java heap space ``` This is due to the default JVM Heap Space configuration (1 GiB) being not enough for your workloads. In order to resolve this issue, head over to your custom openmetadata helm values and append the below environment variable ```yaml theme={null} extraEnvs: - name: OPENMETADATA_HEAP_OPTS value: "-Xmx2G -Xms2G" ``` The flag `Xmx` specifies the maximum memory allocation pool for a Java virtual machine (JVM), while `Xms` specifies the initial memory allocation pool. Upgrade the helm charts with the above changes using the following command `helm upgrade --install openmetadata open-metadata/openmetadata --values --namespace `. Update this command your `values.yml` filename and `namespaceName` where you have deployed OpenMetadata in Kubernetes. ## PostgreSQL Issue permission denied to create extension "pgcrypto" If you are facing the below issue with PostgreSQL as Database Backend for OpenMetadata Application, ``` Message: ERROR: permission denied to create extension "pgcrypto" Hint: Must be superuser to create this extension. ``` It seems the Database User does not have sufficient privileges. In order to resolve the above issue, grant usage permissions to the PSQL User. ```sql theme={null} GRANT USAGE ON SCHEMA schema_name TO ; GRANT CREATE ON EXTENSION pgcrypto TO ; ``` In the above command, replace `` with the sql user used by OpenMetadata Application to connect to PostgreSQL Database. ## How to extend and use custom docker images with OpenMetadata Helm Charts ? ## Extending OpenMetadata Server Docker Image ### 1. Create a `Dockerfile` based on `docker.open-metadata.org/openmetadata/server` OpenMetadata helm charts uses official published docker images from [DockerHub](https://hub.docker.com/u/openmetadata). A typical scenario will be to install organization certificates for connecting with inhouse systems. For Example - ``` FROM docker.open-metadata.org/openmetadata/server:x.y.z WORKDIR /home/ COPY . RUN update-ca-certificates ``` where `docker.open-metadata.org/openmetadata/server:x.y.z` needs to point to the same version of the OpenMetadata server, for example `docker.open-metadata.org/openmetadata/server:1.3.1`. This image needs to be built and published to the container registry of your choice. ### 2. Update your openmetadata helm values yaml The OpenMetadata Application gets installed as part of `openmetadata` helm chart. In this step, update the custom helm values using YAML file to point the image created in the previous step. For example, create a helm values file named `values.yaml` with the following contents - ```yaml theme={null} ... image: repository: # Overrides the image tag whose default is the chart appVersion. tag: ... ``` ### 3. Install / Upgrade your helm release Upgrade/Install your openmetadata helm charts with the below single command: ```bash theme={null} helm upgrade --install openmetadata open-metadata/openmetadata--values values.yaml ``` ## Extending OpenMetadata Ingestion Docker Image One possible use case for a custom ingestion image is a custom connector. Build and test the package with the same `openmetadata-ingestion` version as your deployment. After your code is ready, follow these steps: ### 1. Create a `Dockerfile` based on `docker.open-metadata.org/openmetadata/ingestion`: For example - ``` FROM docker.open-metadata.org/openmetadata/ingestion:x.y.z USER airflow # Let's use the home directory of airflow user WORKDIR /home/airflow # Install our custom connector COPY COPY setup.py . RUN pip install --no-deps . ``` where `docker.open-metadata.org/openmetadata/ingestion:x.y.z` needs to point to the same version of the OpenMetadata server, for example `docker.open-metadata.org/openmetadata/ingestion:1.3.1`. This image needs to be built and published to the container registry of your choice. ### 2. Update the airflow in openmetadata dependencies values YAML The ingestion containers (which is the one shipping Airflow) gets installed in the `openmetadata-dependencies` helm chart. In this step, we use our own custom values YAML file to point to the image we just created on the previous step. You can create a file named `values.deps.yaml` with the following contents: ```yaml theme={null} airflow: airflow: image: repository: # by default, openmetadata/ingestion tag: # by default, the version you are deploying, e.g., 1.1.0 pullPolicy: "IfNotPresent" ``` ### 3. Install / Upgrade helm release Upgrade/Install your openmetadata-dependencies helm charts with the below single command: ```bash theme={null} helm upgrade --install openmetadata-dependencies open-metadata/openmetadata-dependencies --values values.deps.yaml ``` ## How to disable MySQL and ElasticSearch from OpenMetadata Dependencies Helm Charts ? If you are using MySQL and ElasticSearch externally, you would want to disable the local installation of mysql and elasticsearch while installing OpenMetadata Dependencies Helm Chart. You can disable the MySQL and ElasticSearch Helm Dependencies by setting `enabled: false` value for each dependency. Below is the command to set helm values from Helm CLI - ```commandline theme={null} helm upgrade --install openmetadata-dependencies open-metadata/openmetadata-dependencies --set mysql.enabled=false --set elasticsearch.enabled=false ``` Alternatively, you can create a custom YAML file named `values.deps.yaml` to disable installation of MySQL and Elasticsearch . ```yaml theme={null} mysql: enabled: false ... elasticsearch: enabled: false ... ... ``` ## How to configure external database like PostgreSQL with OpenMetadata Helm Charts ? OpenMetadata Supports PostgreSQL as one of the Database Dependencies. OpenMetadata Helm Charts by default does not include PostgreSQL as Database Dependencies. In order to configure Helm Charts with External Database like PostgreSQL, follow the below guide to make the helm values change and upgrade / install OpenMetadata helm charts with the same. ## Upgrade Airflow Helm Dependencies Helm Charts to connect to External Database like PostgreSQL We ship [airflow-helm](https://github.com/airflow-helm/charts/tree/main/charts/airflow) as one of OpenMetadata Dependencies with default values to connect to MySQL Database as part of `externalDatabase` configurations. You can find more information on setting the `externalDatabase` as part of helm values [here](https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/database/external-database.md). With OpenMetadata Dependencies Helm Charts, your helm values would look something like below - ```yaml theme={null} ... airflow: externalDatabase: type: postgresql host: port: 5432 database: user: passwordSecret: airflow-postgresql-secrets passwordSecretKey: airflow-postgresql-password ... ``` For the above code, it is assumed you are creating a kubernetes secret for storing Airflow Database login Credentials. A sample command to create the secret will be `kubectl create secret generic airflow-postgresql-secrets --from-literal=airflow-postgresql-password=`. ## Upgrade OpenMetadata Helm Charts to connect to External Database like PostgreSQL Update the `openmetadata.config.database.*` helm values for OpenMetadata Application to connect to External Database like PostgreSQL. With OpenMetadata Helm Charts, your helm values would look something like below - ```yaml theme={null} openmetadata: config: ... database: host: port: 5432 driverClass: org.postgresql.Driver dbScheme: postgresql dbUseSSL: true databaseName: auth: username: password: secretRef: openmetadata-postgresql-secrets secretKey: openmetadata-postgresql-password ``` For the above code, it is assumed you are creating a kubernetes secret for storing OpenMetadata Database login Credentials. A sample command to create the secret will be `kubectl create secret generic openmetadata-postgresql-secrets --from-literal=openmetadata-postgresql-password=`. Once you make the above changes to your helm values, run the below command to install/upgrade helm charts - ```commandline theme={null} helm upgrade --install openmetadata-dependencies open-metadata/openmetadata-dependencies --values <> --namespace helm upgrade --install openmetadata open-metadata/openmetadata --values <> --namespace ``` ## How to customize OpenMetadata Dependencies Helm Chart with custom helm values Our OpenMetadata Dependencies Helm Charts are internally depends on three sub-charts - * [Bitnami MySQL](https://artifacthub.io/packages/helm/bitnami/mysql/9.7.2) (helm chart version 9.7.2) * [OpenSearch](https://artifacthub.io/packages/helm/opensearch-project-helm-charts/opensearch/2.12.2) (helm chart version 2.12.2) * [Airflow](https://artifacthub.io/packages/helm/airflow-helm/airflow/8.8.0) (helm chart version 8.8.0) If you are looking to customize the deployments of any of the above dependencies, please refer to the above links for customizations of helm values for further references. By default, OpenMetadata Dependencies helm chart provides initial generic customization of these helm values in order to get you started quickly. You can refer to the openmetadata-dependencies helm charts default values [here](https://github.com/open-metadata/openmetadata-helm-charts/blob/main/charts/deps/values.yaml). # AWS EKS Deployment | OpenMetadata Kubernetes Guide Source: https://docs.open-metadata.org/v2.0.x/deployment/kubernetes/eks Deploy the OpenMetadata on Amazon EKS for cloud-native scalability with secure identity integration and managed infrastructure support. # EKS on Amazon Web Services Deployment OpenMetadata supports the Installation and Running of Application on Elastic Kubernetes Services (EKS) through Helm Charts. However, there are some additional configurations which needs to be done as prerequisites for the same. All the code snippets in this section assume the `default` namespace for kubernetes. This guide presumes you have AWS EKS Cluster already available. ## Prerequisites ### AWS Services for Database as RDS and Search Engine as ElasticSearch It is recommended to use [Amazon RDS](https://docs.aws.amazon.com/rds/index.html) and [Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/?id=docs_gateway) for Production Deployments. We support * Amazon RDS (MySQL) engine version 8 or higher * Amazon RDS (PostgreSQL) engine version 15 or higher * Amazon OpenSearch engine version 3.x (minimum 3.0.0, recommended 3.3.0) When using AWS Services the SearchType Configuration for elastic search should be `opensearch`, for both cases ElasticSearch and OpenSearch, as you can see in the ElasticSearch configuration example below. We recommend * Amazon RDS to be in Multiple Availability Zones. * Amazon OpenSearch (or ElasticSearch) Service with Multiple Availability Zones with minimum 2 Nodes. Make sure to increase `sort_buffer_size` (for MySQL) or `work_mem` (for PostgreSQL) to the recommended value of **20MB** or more using the [database parameter group setting](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html). This is especially important when running migrations to prevent **Out of Sort Memory Error**. You can revert the setting once the migrations are complete. ## Kubernetes Orchestrator Configuration (Recommended) Starting with OpenMetadata 1.12, we recommend using the **Kubernetes native orchestrator** for running ingestion pipelines. This eliminates the need for Apache Airflow and simplifies your deployment. The Kubernetes orchestrator runs ingestion pipelines as native K8s Jobs and CronJobs. For full documentation on features, configuration options, and troubleshooting, see the [Kubernetes Orchestrator Guide](/v2.0.x/deployment/ingestion/kubernetes). The recommended OMJob Operator approach requires installing Custom Resource Definitions (CRDs), which needs elevated cluster permissions. If your cluster policies don't allow CRDs, you can disable the operator by setting `useOMJobOperator: false` and `omjobOperator.enabled: false` in your values file to use native K8s Jobs instead. ### OpenMetadata Values Configuration Create your `openmetadata-values.yaml` with the following configuration: ```yaml theme={null} # openmetadata-values.yaml openmetadata: config: # Database configuration elasticsearch: host: searchType: opensearch port: 443 scheme: https connectionTimeoutSecs: 5 socketTimeoutSecs: 60 keepAliveTimeoutSecs: 600 batchSize: 10 auth: enabled: true username: password: secretRef: elasticsearch-secrets secretKey: openmetadata-elasticsearch-password database: host: port: 3306 driverClass: com.mysql.cj.jdbc.Driver dbScheme: mysql dbUseSSL: true databaseName: auth: username: password: secretRef: mysql-secrets secretKey: openmetadata-mysql-password # Kubernetes Orchestrator configuration pipelineServiceClientConfig: enabled: true type: "k8s" metadataApiEndpoint: http://openmetadata:8585/api k8s: ingestionImage: "docker.getcollate.io/openmetadata/ingestion-base:2.0.1" useOMJobOperator: true # Enable the OMJob Operator (recommended for production) omjobOperator: enabled: true image: repository: docker.getcollate.io/openmetadata/omjob-operator tag: "2.0.1" ``` For advanced configuration options such as resource limits, job lifecycle settings, failure diagnostics, RBAC, and security contexts, see the [Kubernetes Orchestrator Guide](/v2.0.x/deployment/ingestion/kubernetes). ### Create Kubernetes Secrets Create the required secrets for RDS and OpenSearch: ```bash theme={null} # Database secret kubectl create secret generic mysql-secrets \ --from-literal=openmetadata-mysql-password= # OpenSearch secret kubectl create secret generic elasticsearch-secrets \ --from-literal=openmetadata-elasticsearch-password= ``` ### Deploy OpenMetadata ```bash theme={null} # Add the OpenMetadata Helm repository helm repo add open-metadata https://helm.open-metadata.org/ helm repo update # Install OpenMetadata (no dependencies chart needed with K8s orchestrator) helm install openmetadata open-metadata/openmetadata \ --values openmetadata-values.yaml ``` With the Kubernetes orchestrator, you don't need to deploy the `openmetadata-dependencies` chart that includes Airflow. This significantly simplifies your deployment. ### Verify the Deployment ```bash theme={null} # Check pods are running kubectl get pods # Check the K8s orchestrator health in OpenMetadata UI # Navigate to Settings → Preferences → Health ``` *** If you prefer to use Apache Airflow as the orchestrator for your EKS deployment, see the [Airflow on EKS](/v2.0.x/deployment/kubernetes/eks/airflow) guide. # Airflow Orchestrator on EKS | OpenMetadata Kubernetes Guide Source: https://docs.open-metadata.org/v2.0.x/deployment/kubernetes/eks/airflow Configure Apache Airflow as the orchestrator for OpenMetadata on Amazon EKS with EFS persistent storage. # Airflow Orchestrator on EKS If you prefer to use Apache Airflow as the orchestrator (e.g., for existing Airflow investments or complex DAG requirements), follow the configuration below. Using Airflow requires additional infrastructure: persistent volumes with ReadWriteMany access, the openmetadata-dependencies Helm chart, and more complex configuration. ### Create Elastic File System in AWS You can follow official AWS Guides [here](https://docs.aws.amazon.com/efs/latest/ug/gs-step-two-create-efs-resources.html) to provision EFS File System in the same VPC which is associated with your EKS Cluster. ### Persistent Volumes with ReadWriteMany Access Modes OpenMetadata helm chart depends on Airflow and Airflow expects a persistent disk that support ReadWriteMany (the volume can be mounted as read-write by many nodes). In AWS, this is achieved by Elastic File System (EFS) service. AWS Elastic Block Store (EBS) does not provide ReadWriteMany Volume access mode as EBS will only be attached to one Kubernetes Node at any given point of time. In order to provision persistent volumes from AWS EFS, you will need to setup and install [aws-efs-csi-driver](https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html). Note that this is required for Airflow as One OpenMetadata Dependencies. Also, [aws-ebs-csi-driver](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html) might be required for Persistent Volumes that are to be used for MySQL and ElasticSearch as OpenMetadata Dependencies. The below guide provides Persistent Volumes provisioning as static volumes (meaning you will be responsible to create, maintain and destroy Persistent Volumes). ### Provision EFS backed PVs, PVCs for Airflow DAGs and Airflow Logs Please note that we are using one AWS Elastic File System (EFS) service with subdirectories as `airflow-dags` and `airflow-logs` with the reference in this documentation. Also, it is presumed that `airflow-dags` and `airflow-logs` directories are already available on that file system. In order to create directories inside the AWS Elastic File System (EFS) you would need to follow these [steps](https://docs.aws.amazon.com/efs/latest/ug/accessing-fs-nfs-permissions-per-user-subdirs.html). #### Code Samples for PV and PVC for Airflow DAGs ```yaml theme={null} # dags_pv_pvc.yml apiVersion: v1 kind: PersistentVolume metadata: name: openmetadata-dependencies-dags-pv labels: app: airflow-dags spec: capacity: storage: 10Gi storageClassName: "" accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Retain csi: driver: efs.csi.aws.com volumeHandle: :/airflow-dags # Replace with EFS File System Id --- apiVersion: v1 kind: PersistentVolumeClaim metadata: labels: app: airflow-dags name: openmetadata-dependencies-dags-pvc namespace: default spec: accessModes: - ReadWriteMany storageClassName: "" resources: requests: storage: 10Gi ``` Create Persistent Volumes and Persistent Volume claims with the below command. ```commandline theme={null} kubectl create -f dags_pv_pvc.yml ``` #### Code Samples for PV and PVC for Airflow Logs ```yaml theme={null} # logs_pv_pvc.yml apiVersion: v1 kind: PersistentVolume metadata: name: openmetadata-dependencies-logs-pv labels: app: airflow-logs spec: capacity: storage: 5Gi storageClassName: "" accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Retain csi: driver: efs.csi.aws.com volumeHandle: :/airflow-logs # Replace with EFS File System Id --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: openmetadata-dependencies-logs-pvc namespace: default labels: app: airflow-dags spec: accessModes: - ReadWriteMany storageClassName: "" resources: requests: storage: 5Gi ``` Create Persistent Volumes and Persistent Volume claims with the below command. ```commandline theme={null} kubectl create -f logs_pv_pvc.yml ``` ### Change owner and permission manually on disks Since airflow pods run as non root users, they would not have write access on the nfs server volumes. In order to fix the permission here, spin up a pod with persistent volumes attached and run it once. You can find more reference on AWS EFS permissions in docs [here](https://docs.aws.amazon.com/efs/latest/ug/using-fs.html). ```yaml theme={null} # permissions_pod.yml apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: my-permission-pod name: my-permission-pod spec: containers: - image: nginx name: my-permission-pod volumeMounts: - name: airflow-dags mountPath: /airflow-dags - name: airflow-logs mountPath: /airflow-logs command: - "chown -R 50000 /airflow-dags /airflow-logs" # if needed - "chmod -R a+rwx /airflow-dags" volumes: - name: airflow-logs persistentVolumeClaim: claimName: openmetadata-dependencies-logs-pvc - name: airflow-dags persistentVolumeClaim: claimName: openmetadata-dependencies-dags-pvc dnsPolicy: ClusterFirst restartPolicy: Always ``` Airflow runs the pods with linux user name as airflow and linux user id as 50000. Run the below command to create the pod and fix the permissions ```commandline theme={null} kubectl create -f permissions_pod.yml ``` ### Create OpenMetadata dependencies Values Override openmetadata dependencies airflow helm values to bind the efs persistent volumes for DAGs and logs. ```yaml theme={null} # values-dependencies.yml airflow: airflow: extraVolumeMounts: - mountPath: /airflow-logs name: efs-airflow-logs - mountPath: /airflow-dags/dags name: efs-airflow-dags extraVolumes: - name: efs-airflow-logs persistentVolumeClaim: claimName: openmetadata-dependencies-logs-pvc - name: efs-airflow-dags persistentVolumeClaim: claimName: openmetadata-dependencies-dags-pvc config: AIRFLOW__OPENMETADATA_AIRFLOW_APIS__DAG_GENERATED_CONFIGS: "/airflow-dags/dags" dags: path: /airflow-dags/dags persistence: enabled: false logs: path: /airflow-logs persistence: enabled: false ``` For more information on airflow helm chart values, please refer to [airflow-helm](https://artifacthub.io/packages/helm/airflow-helm/airflow/8.5.3). When deploying openmetadata dependencies helm chart, use the below command - ```commandline theme={null} helm install openmetadata-dependencies open-metadata/openmetadata-dependencies --values values-dependencies.yaml ``` The above command uses configurations defined [here](https://raw.githubusercontent.com/open-metadata/openmetadata-helm-charts/main/charts/deps/values.yaml). You can modify any configuration and deploy by passing your own `values.yaml` ```commandline theme={null} helm install openmetadata-dependencies open-metadata/openmetadata-dependencies --values ``` ### Deploy OpenMetadata with Airflow Configuration Create your OpenMetadata values file with Airflow configuration: ```yaml theme={null} # openmetadata-values-airflow.yaml openmetadata: config: elasticsearch: host: searchType: opensearch port: 443 scheme: https connectionTimeoutSecs: 5 socketTimeoutSecs: 60 keepAliveTimeoutSecs: 600 batchSize: 10 auth: enabled: true username: password: secretRef: elasticsearch-secrets secretKey: openmetadata-elasticsearch-password database: host: port: 3306 driverClass: com.mysql.cj.jdbc.Driver dbScheme: mysql dbUseSSL: true databaseName: auth: username: password: secretRef: mysql-secrets secretKey: openmetadata-mysql-password # Airflow configuration pipelineServiceClientConfig: enabled: true type: "airflow" metadataApiEndpoint: http://openmetadata:8585/api airflow: apiEndpoint: http://openmetadata-dependencies-web:8080 auth: username: admin password: secretRef: airflow-secrets secretKey: openmetadata-airflow-password ``` Once the openmetadata dependencies helm chart deployed, you can then run the below command to install the openmetadata helm chart - ```commandline theme={null} helm install openmetadata open-metadata/openmetadata --values openmetadata-values-airflow.yaml ``` Create RDS and OpenSearch credentials as Kubernetes Secrets, as described [here](/v2.0.x/quick-start/local-kubernetes-deployment#2-create-kubernetes-secrets-required-for-helm-charts). Also, disable MySQL and Elasticsearch from OpenMetadata Dependencies Helm Charts as described in the [FAQ](#how-to-disable-mysql-and-elasticsearch-from-openmetadata-dependencies-helm-charts-). ## FAQs ## Getting an error when install OpenMetadata Dependencies Helm Charts on EKS with EFS If you are facing the below issue - ``` MountVolume.SetUp failed for volume "openmetadata-dependencies-dags-pv" : rpc error: code = Internal desc = Could not mount "fs-012345abcdef:/airflow-dags" at "/var/lib/kubelet/pods/xyzabc-123-0062-44c3-b0e9-fa193c19f41c/volumes/kubernetes.io~csi/openmetadata-dependencies-dags-pv/mount": mount failed: exit status 1 Mounting command: mount Mounting arguments: -t efs -o tls fs-012345abcdef:/airflow-dags /var/lib/kubelet/pods/xyzabc-123-0062-44c3-b0e9-fa193c19f41c/volumes/kubernetes.io~csi/openmetadata-dependencies-dags-pv/mount Output: Failed to locate an available port in the range [20049, 20449], try specifying a different port range in /etc/amazon/efs/efs-utils.conf ``` This error is typically related to EKS Cluster not able to reach to EFS File systems. You can check the security groups associated between the connectivity EFS and EKS. [Here is an article](https://github.com/kubernetes-sigs/aws-efs-csi-driver/blob/master/docs/efs-create-filesystem.md) which further describes the steps required to create Security Group Rules for EKS to use EFS over `port 2049`. It can also happen if the mount targets are already available for EKS Nodes but the Nodes do not pick that up. In such cases, you can do an [AWS AutoScaling Group instance refresh](https://docs.aws.amazon.com/autoscaling/ec2/userguide/start-instance-refresh.html) in order for EKS nodes to get the available mount targets. ## Java Memory Heap Issue If your openmetadata pods are not in ready state at any point in time and the openmetadata pod logs speaks about the below issue - ``` Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "AsyncAppender-Worker-async-file-appender" Exception in thread "pool-5-thread-1" java.lang.OutOfMemoryError: Java heap space Exception in thread "AsyncAppender-Worker-async-file-appender" java.lang.OutOfMemoryError: Java heap space Exception in thread "dw-46" java.lang.OutOfMemoryError: Java heap space Exception in thread "AsyncAppender-Worker-async-console-appender" java.lang.OutOfMemoryError: Java heap space ``` This is due to the default JVM Heap Space configuration (1 GiB) being not enough for your workloads. In order to resolve this issue, head over to your custom openmetadata helm values and append the below environment variable ```yaml theme={null} extraEnvs: - name: OPENMETADATA_HEAP_OPTS value: "-Xmx2G -Xms2G" ``` The flag `Xmx` specifies the maximum memory allocation pool for a Java virtual machine (JVM), while `Xms` specifies the initial memory allocation pool. Upgrade the helm charts with the above changes using the following command `helm upgrade --install openmetadata open-metadata/openmetadata --values --namespace `. Update this command your `values.yml` filename and `namespaceName` where you have deployed OpenMetadata in Kubernetes. ## PostgreSQL Issue permission denied to create extension "pgcrypto" If you are facing the below issue with PostgreSQL as Database Backend for OpenMetadata Application, ``` Message: ERROR: permission denied to create extension "pgcrypto" Hint: Must be superuser to create this extension. ``` It seems the Database User does not have sufficient privileges. In order to resolve the above issue, grant usage permissions to the PSQL User. ```sql theme={null} GRANT USAGE ON SCHEMA schema_name TO ; GRANT CREATE ON EXTENSION pgcrypto TO ; ``` In the above command, replace `` with the sql user used by OpenMetadata Application to connect to PostgreSQL Database. ## How to extend and use custom docker images with OpenMetadata Helm Charts ? ## Extending OpenMetadata Server Docker Image ### 1. Create a `Dockerfile` based on `docker.open-metadata.org/openmetadata/server` OpenMetadata helm charts uses official published docker images from [DockerHub](https://hub.docker.com/u/openmetadata). A typical scenario will be to install organization certificates for connecting with inhouse systems. For Example - ``` FROM docker.open-metadata.org/openmetadata/server:x.y.z WORKDIR /home/ COPY . RUN update-ca-certificates ``` where `docker.open-metadata.org/openmetadata/server:x.y.z` needs to point to the same version of the OpenMetadata server, for example `docker.open-metadata.org/openmetadata/server:1.3.1`. This image needs to be built and published to the container registry of your choice. ### 2. Update your openmetadata helm values yaml The OpenMetadata Application gets installed as part of `openmetadata` helm chart. In this step, update the custom helm values using YAML file to point the image created in the previous step. For example, create a helm values file named `values.yaml` with the following contents - ```yaml theme={null} ... image: repository: # Overrides the image tag whose default is the chart appVersion. tag: ... ``` ### 3. Install / Upgrade your helm release Upgrade/Install your openmetadata helm charts with the below single command: ```bash theme={null} helm upgrade --install openmetadata open-metadata/openmetadata--values values.yaml ``` ## Extending OpenMetadata Ingestion Docker Image One possible use case for a custom ingestion image is a custom connector. Build and test the package with the same `openmetadata-ingestion` version as your deployment. After your code is ready, follow these steps: ### 1. Create a `Dockerfile` based on `docker.open-metadata.org/openmetadata/ingestion`: For example - ``` FROM docker.open-metadata.org/openmetadata/ingestion:x.y.z USER airflow # Let's use the home directory of airflow user WORKDIR /home/airflow # Install our custom connector COPY COPY setup.py . RUN pip install --no-deps . ``` where `docker.open-metadata.org/openmetadata/ingestion:x.y.z` needs to point to the same version of the OpenMetadata server, for example `docker.open-metadata.org/openmetadata/ingestion:1.3.1`. This image needs to be built and published to the container registry of your choice. ### 2. Update the airflow in openmetadata dependencies values YAML The ingestion containers (which is the one shipping Airflow) gets installed in the `openmetadata-dependencies` helm chart. In this step, we use our own custom values YAML file to point to the image we just created on the previous step. You can create a file named `values.deps.yaml` with the following contents: ```yaml theme={null} airflow: airflow: image: repository: # by default, openmetadata/ingestion tag: # by default, the version you are deploying, e.g., 1.1.0 pullPolicy: "IfNotPresent" ``` ### 3. Install / Upgrade helm release Upgrade/Install your openmetadata-dependencies helm charts with the below single command: ```bash theme={null} helm upgrade --install openmetadata-dependencies open-metadata/openmetadata-dependencies --values values.deps.yaml ``` ## How to disable MySQL and ElasticSearch from OpenMetadata Dependencies Helm Charts ? If you are using MySQL and ElasticSearch externally, you would want to disable the local installation of mysql and elasticsearch while installing OpenMetadata Dependencies Helm Chart. You can disable the MySQL and ElasticSearch Helm Dependencies by setting `enabled: false` value for each dependency. Below is the command to set helm values from Helm CLI - ```commandline theme={null} helm upgrade --install openmetadata-dependencies open-metadata/openmetadata-dependencies --set mysql.enabled=false --set elasticsearch.enabled=false ``` Alternatively, you can create a custom YAML file named `values.deps.yaml` to disable installation of MySQL and Elasticsearch . ```yaml theme={null} mysql: enabled: false ... elasticsearch: enabled: false ... ... ``` ## How to configure external database like PostgreSQL with OpenMetadata Helm Charts ? OpenMetadata Supports PostgreSQL as one of the Database Dependencies. OpenMetadata Helm Charts by default does not include PostgreSQL as Database Dependencies. In order to configure Helm Charts with External Database like PostgreSQL, follow the below guide to make the helm values change and upgrade / install OpenMetadata helm charts with the same. ## Upgrade Airflow Helm Dependencies Helm Charts to connect to External Database like PostgreSQL We ship [airflow-helm](https://github.com/airflow-helm/charts/tree/main/charts/airflow) as one of OpenMetadata Dependencies with default values to connect to MySQL Database as part of `externalDatabase` configurations. You can find more information on setting the `externalDatabase` as part of helm values [here](https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/database/external-database.md). With OpenMetadata Dependencies Helm Charts, your helm values would look something like below - ```yaml theme={null} ... airflow: externalDatabase: type: postgresql host: port: 5432 database: user: passwordSecret: airflow-postgresql-secrets passwordSecretKey: airflow-postgresql-password ... ``` For the above code, it is assumed you are creating a kubernetes secret for storing Airflow Database login Credentials. A sample command to create the secret will be `kubectl create secret generic airflow-postgresql-secrets --from-literal=airflow-postgresql-password=`. ## Upgrade OpenMetadata Helm Charts to connect to External Database like PostgreSQL Update the `openmetadata.config.database.*` helm values for OpenMetadata Application to connect to External Database like PostgreSQL. With OpenMetadata Helm Charts, your helm values would look something like below - ```yaml theme={null} openmetadata: config: ... database: host: port: 5432 driverClass: org.postgresql.Driver dbScheme: postgresql dbUseSSL: true databaseName: auth: username: password: secretRef: openmetadata-postgresql-secrets secretKey: openmetadata-postgresql-password ``` For the above code, it is assumed you are creating a kubernetes secret for storing OpenMetadata Database login Credentials. A sample command to create the secret will be `kubectl create secret generic openmetadata-postgresql-secrets --from-literal=openmetadata-postgresql-password=`. Once you make the above changes to your helm values, run the below command to install/upgrade helm charts - ```commandline theme={null} helm upgrade --install openmetadata-dependencies open-metadata/openmetadata-dependencies --values <> --namespace helm upgrade --install openmetadata open-metadata/openmetadata --values <> --namespace ``` ## How to customize OpenMetadata Dependencies Helm Chart with custom helm values Our OpenMetadata Dependencies Helm Charts are internally depends on three sub-charts - * [Bitnami MySQL](https://artifacthub.io/packages/helm/bitnami/mysql/9.7.2) (helm chart version 9.7.2) * [OpenSearch](https://artifacthub.io/packages/helm/opensearch-project-helm-charts/opensearch/2.12.2) (helm chart version 2.12.2) * [Airflow](https://artifacthub.io/packages/helm/airflow-helm/airflow/8.8.0) (helm chart version 8.8.0) If you are looking to customize the deployments of any of the above dependencies, please refer to the above links for customizations of helm values for further references. By default, OpenMetadata Dependencies helm chart provides initial generic customization of these helm values in order to get you started quickly. You can refer to the openmetadata-dependencies helm charts default values [here](https://github.com/open-metadata/openmetadata-helm-charts/blob/main/charts/deps/values.yaml). # Terraform AWS Deployment | OpenMetadata Source: https://docs.open-metadata.org/v2.0.x/deployment/kubernetes/eks/terraform Deploy OpenMetadata on Amazon EKS using the official Terraform module with support for RDS, OpenSearch Service, and multiple provisioner modes. # Deploy OpenMetadata on AWS with Terraform The [OpenMetadata Terraform module for AWS](https://registry.terraform.io/modules/open-metadata/openmetadata/aws) deploys OpenMetadata and all its dependencies on an existing EKS cluster. Each component (database and search engine) can be independently configured using one of three provisioners: deploy it inside the cluster via Helm, provision a managed AWS service, or connect to an existing resource you already operate. ## Prerequisites Before using this module, ensure you have: * **Terraform** `~> 1.0` * **An existing EKS cluster** with `kubectl` configured to access it * **Helm and Kubernetes Terraform providers** configured to point to your cluster * **AWS provider** `~> 6.0` with permissions to create the resources required by your chosen provisioners (see [IAM permissions](#iam-permissions) below) The module manages OpenMetadata and its dependencies only (it does not create the EKS cluster, VPC, or node groups). See the [complete example](https://github.com/open-metadata/terraform-aws-openmetadata/tree/main/examples/complete) for a reference that provisions the full AWS infrastructure from scratch. ### IAM Permissions The following permissions are required depending on which provisioners you use: | Provisioner | Required AWS permissions | | :------------------- | :-------------------------------------------------------------- | | `db = "aws"` | RDS: create/manage DB instances, subnet groups, security groups | | `opensearch = "aws"` | OpenSearch Service: create/manage domains, security groups | | `kms_key_id` | KMS: use the specified key for encryption | ### Provider Configuration Your Terraform configuration must include the AWS, Kubernetes, and Helm providers: ```hcl theme={null} provider "aws" { region = "us-east-1" } provider "kubernetes" { host = aws_eks_cluster.this.endpoint cluster_ca_certificate = base64decode(aws_eks_cluster.this.certificate_authority[0].data) token = data.aws_eks_cluster_auth.this.token } provider "helm" { kubernetes { host = aws_eks_cluster.this.endpoint cluster_ca_certificate = base64decode(aws_eks_cluster.this.certificate_authority[0].data) token = data.aws_eks_cluster_auth.this.token } } ``` *** ## Choosing a Provisioner Each component supports a different set of provisioners. Mix and match to fit your infrastructure: | Component | `helm` | `aws` | `existing` | | :------------------------ | :----: | :---: | :--------: | | **OpenMetadata** | ✅ | N/A | N/A | | **OpenMetadata database** | ✅ | ✅ | ✅ | | **OpenSearch** | ✅ | ✅ | ✅ | | Provisioner | When to use | | :---------- | :-------------------------------------------------------------------------------------------------------------------------------- | | `helm` | Development, testing, or when you want everything self-contained inside the cluster. | | `aws` | Production. Creates a managed AWS resource (RDS or OpenSearch Service) with high availability, automated backups, and encryption. | | `existing` | You already have a database or search engine running. The module connects OpenMetadata to it without creating anything new. | *** ## Quick Start - Helm The simplest deployment. All components run inside your cluster via Helm. Suitable for development and evaluation: ```hcl theme={null} module "omd" { source = "open-metadata/openmetadata/aws" version = "1.13" app_namespace = "openmetadata" eks_nodes_sg_ids = ["sg-1234abcd5678efgh"] subnet_ids = ["subnet-1a2b3c4d", "subnet-5e6f7g8h", "subnet-9i0j1k2l"] vpc_id = "vpc-1a2b3c4d" } ``` ```bash theme={null} terraform init ``` ```bash theme={null} terraform plan ``` ```bash theme={null} terraform apply ``` *** ## Production Deployment - AWS Managed Services Use the `aws` provisioner for the database and OpenSearch to get production-grade infrastructure. This creates: * **RDS PostgreSQL** instance (Multi-AZ, `db.t4g.medium`) for OpenMetadata * **OpenSearch Service** domain (2 nodes, `t3.small.search`) for search * **Security groups** allowing traffic from your EKS nodes to each resource * **Kubernetes secrets** with auto-generated credentials in your application namespace The `aws` provisioner creates billable AWS resources. Run `terraform destroy` when you no longer need them. ```hcl theme={null} module "omd" { source = "open-metadata/openmetadata/aws" version = "1.13" app_namespace = "openmetadata" eks_nodes_sg_ids = ["sg-1234abcd5678efgh"] kms_key_id = "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012" subnet_ids = ["subnet-1a2b3c4d", "subnet-5e6f7g8h", "subnet-9i0j1k2l"] vpc_id = "vpc-1a2b3c4d" db = { provisioner = "aws" } opensearch = { provisioner = "aws" } } ``` Credentials for RDS and OpenSearch are generated automatically and stored as Kubernetes secrets in your application namespace. You do not need to manage passwords manually. ### Customizing AWS Resources Override the defaults for any AWS-managed resource using the `aws` sub-object: ```hcl theme={null} db = { provisioner = "aws" aws = { instance_class = "db.t4g.large" multi_az = true backup_retention_period = 14 deletion_protection = true skip_final_snapshot = false } } opensearch = { provisioner = "aws" aws = { instance_type = "m6g.large.search" instance_count = 3 availability_zone_count = 3 engine_version = "OpenSearch_3.3" } } ``` *** ## Bring Your Own Infrastructure - Existing Connect OpenMetadata to a database and search engine you already operate. No new AWS resources are created: ```hcl theme={null} module "omd" { source = "open-metadata/openmetadata/aws" version = "1.13" app_namespace = "openmetadata" db = { provisioner = "existing" host = "omd-db.postgres.example" port = 5432 db_name = "openmetadata_db" engine = { name = "postgres" } credentials = { username = "dbadmin" password = { secret_ref = "db-secrets" secret_key = "password" } } } opensearch = { provisioner = "existing" host = "opensearch.example" port = "443" scheme = "https" } } ``` The `secret_ref` and `secret_key` values reference a Kubernetes secret that must already exist in your application namespace before `terraform apply`. *** ## Kubernetes Orchestrator (No Airflow) This is the default mode. The module deploys OpenMetadata without Airflow and configures it to run ingestion pipelines as native Kubernetes Jobs via the OMJob operator. No extra configuration is needed: ```hcl theme={null} module "omd" { source = "open-metadata/openmetadata/aws" version = "1.13" app_namespace = "openmetadata" eks_nodes_sg_ids = ["sg-1234abcd5678efgh"] subnet_ids = ["subnet-1a2b3c4d", "subnet-5e6f7g8h", "subnet-9i0j1k2l"] vpc_id = "vpc-1a2b3c4d" } ``` No Airflow deployment, Airflow database, or EFS volumes are created. OpenMetadata is configured automatically to use the OMJob operator: * `pipelineServiceClientConfig.type` is set to `k8s` * `pipelineServiceClientConfig.k8s.useOMJobOperator` is set to `true` * `omjobOperator.enabled` is set to `true` The OMJob operator installs Custom Resource Definitions (CRDs) on your cluster, which requires elevated permissions during the first `terraform apply`. *** ## Advanced Configuration ### Extra Environment Variables Inject arbitrary environment variables into the OpenMetadata pod: ```hcl theme={null} extra_envs = { "ELASTICSEARCH_BATCH_SIZE" = "250" "PIPELINE_SERVICE_IP_INFO_ENABLED" = "false" } ``` Or load them from an existing Kubernetes secret: ```hcl theme={null} env_from = ["my-app-secrets", "another-secret"] ``` Both can be used together. `env_from` secrets are mounted before `extra_envs`, so individual values in `extra_envs` can override keys from a secret. ### Overriding Helm Values Pass arbitrary values to any Helm chart using the `*_helm_values` variables. These are merged on top of the values generated by the module, so they can override defaults or configure options not exposed as Terraform variables: | Variable | Helm chart | | :------------------------- | :--------------------------------- | | `openmetadata_helm_values` | OpenMetadata | | `opensearch_helm_values` | OpenSearch (inside the deps chart) | ```hcl theme={null} openmetadata_helm_values = { "replicaCount" = "2" } ``` *** ## Accessing Your Deployment ```bash theme={null} kubectl port-forward service/openmetadata 8585:8585 -n ``` Open `http://localhost:8585` in your browser. Keep the terminal session with `kubectl port-forward` open while accessing OpenMetadata. If port 8585 is already in use on your machine, change the local port number (the first number in `local:remote`, e.g. `9585:8585`). *** ## Complete AWS Example The [complete example](https://github.com/open-metadata/terraform-aws-openmetadata/tree/main/examples/complete) provisions a full AWS environment from scratch, including: * VPC with public/private subnets, Internet Gateway, and NAT Gateway * EKS cluster with EBS and EFS CSI driver addons * KMS key for encrypting all resources * RDS instance for OpenMetadata (Multi-AZ, deletion protection enabled) * OpenSearch domain with a security group allowing inbound traffic from EKS nodes * Kubernetes namespace, storage classes, and secrets It is a good reference for production deployments and for understanding how to wire together the AWS, Kubernetes, and Helm providers. *** ## Next Steps Run ingestion pipelines as native Kubernetes Jobs Manual Helm-based deployment on Amazon EKS Full reference for OpenMetadata Helm chart values Store and rotate credentials securely using AWS Secrets Manager # Kubernetes GKE Deployment | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/kubernetes/gke Run your deployment on Google Kubernetes Engine (GKE) for a reliable, managed Kubernetes experience with secure configurations. # GKE on Google Cloud Platform Deployment OpenMetadata supports the Installation and Running of Application on Google Kubernetes Engine through Helm Charts. However, there are some additional configurations which needs to be done as prerequisites for the same. Google Kubernetes Engine (GKE) Auto Pilot Mode is not compatible with one of OpenMetadata Dependencies - ElasticSearch. The reason being that ElasticSearch Pods require Elevated permissions to run initContainers for changing configurations which is not allowed by GKE AutoPilot PodSecurityPolicy. All the code snippets in this section assume the `default` namespace for kubernetes. ## Prerequisites ### Cloud Database with CloudSQL and ElasticCloud for GCP as Search Engine It is recommended to use GCP [Cloud SQL](https://cloud.google.com/sql/) services for Database and [Elastic Cloud GCP](https://www.elastic.co/partners/google-cloud) for Search Engine for Production. We support - * Cloud SQL (MySQL) engine version 8 or higher * Cloud SQL (postgreSQL) engine version 15 or higher * ElasticSearch version 9.x (minimum 9.0.0, recommended 9.3.0) We recommend - * CloudSQL to be Multi Zone Available * Elastic Cloud Environment with multiple zones and minimum 2 nodes Make sure to increase `sort_buffer_size` ([for MySQL](https://cloud.google.com/sql/docs/mysql/flags)) or `work_mem` ([for PostgreSQL](https://cloud.google.com/sql/docs/postgres/flags)) to the recommended value of **20MB** or more using flags. This is especially important when running migrations to prevent **Out of Sort Memory Error**. You can revert the setting once the migrations are complete. ## Kubernetes Orchestrator Configuration (Recommended) Starting with OpenMetadata 1.12, we recommend using the **Kubernetes native orchestrator** for running ingestion pipelines. This eliminates the need for Apache Airflow and simplifies your deployment. The Kubernetes orchestrator runs ingestion pipelines as native K8s Jobs and CronJobs. For full documentation on features, configuration options, and troubleshooting, see the [Kubernetes Orchestrator Guide](/v2.0.x/deployment/ingestion/kubernetes). The recommended OMJob Operator approach requires installing Custom Resource Definitions (CRDs), which needs elevated cluster permissions. If your cluster policies don't allow CRDs, you can disable the operator by setting `useOMJobOperator: false` and `omjobOperator.enabled: false` in your values file to use native K8s Jobs instead. ### OpenMetadata Values Configuration Create your `openmetadata-values.yaml` with the following configuration: ```yaml theme={null} # openmetadata-values.yaml openmetadata: config: # Database configuration elasticsearch: host: searchType: elasticsearch port: 443 scheme: https connectionTimeoutSecs: 5 socketTimeoutSecs: 60 keepAliveTimeoutSecs: 600 batchSize: 10 auth: enabled: true username: password: secretRef: elasticsearch-secrets secretKey: openmetadata-elasticsearch-password database: host: port: 3306 driverClass: com.mysql.cj.jdbc.Driver dbScheme: mysql dbUseSSL: true databaseName: auth: username: password: secretRef: mysql-secrets secretKey: openmetadata-mysql-password # Kubernetes Orchestrator configuration pipelineServiceClientConfig: enabled: true type: "k8s" metadataApiEndpoint: http://openmetadata:8585/api k8s: useOMJobOperator: true # Enable the OMJob Operator (recommended for production) omjobOperator: enabled: true ``` For advanced configuration options such as resource limits, job lifecycle settings, failure diagnostics, RBAC, and security contexts, see the [Kubernetes Orchestrator Guide](/v2.0.x/deployment/ingestion/kubernetes). For Database as PostgreSQL, use the below config for database values: ```yaml theme={null} database: host: port: 5432 driverClass: org.postgresql.Driver dbScheme: postgresql dbUseSSL: true databaseName: auth: username: password: secretRef: sql-secrets secretKey: openmetadata-sql-password ``` ### Create Kubernetes Secrets Create the required secrets for CloudSQL and ElasticSearch: ```bash theme={null} # Database secret kubectl create secret generic mysql-secrets \ --from-literal=openmetadata-mysql-password= # ElasticSearch secret kubectl create secret generic elasticsearch-secrets \ --from-literal=openmetadata-elasticsearch-password= ``` ### Deploy OpenMetadata ```bash theme={null} # Add the OpenMetadata Helm repository helm repo add open-metadata https://helm.open-metadata.org/ helm repo update # Install OpenMetadata (no dependencies chart needed with K8s orchestrator) helm install openmetadata open-metadata/openmetadata \ --values openmetadata-values.yaml ``` With the Kubernetes orchestrator, you don't need to deploy the `openmetadata-dependencies` chart that includes Airflow. This significantly simplifies your deployment. ### Verify the Deployment ```bash theme={null} # Check pods are running kubectl get pods # Check the K8s orchestrator health in OpenMetadata UI # Navigate to Settings → Preferences → Health ``` For deployments using Apache Airflow as the orchestrator, see the [GKE Airflow Orchestrator](/v2.0.x/deployment/kubernetes/gke/airflow) guide. # GKE with Apache Airflow Orchestrator | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/kubernetes/gke/airflow Configure Apache Airflow as the ingestion orchestrator on Google Kubernetes Engine with NFS persistent volumes and ReadWriteMany access. # GKE with Apache Airflow Orchestrator If you prefer to use Apache Airflow as the orchestrator (e.g., for existing Airflow investments or complex DAG requirements), follow the configuration below. Using Airflow requires additional infrastructure: persistent volumes with ReadWriteMany access, the openmetadata-dependencies Helm chart, and more complex configuration. ## Persistent Volumes with ReadWriteMany Access Modes OpenMetadata helm chart depends on Airflow and Airflow expects a persistent disk that support ReadWriteMany (the volume can be mounted as read-write by many nodes). The workaround is to create nfs-server disk on Google Kubernetes Engine and use that as the persistent claim and deploy OpenMetadata by implementing the following steps in order. ### Create NFS Share #### Provision GCP Persistent Disk for Google Kubernetes Engine Run the below command to create a gcloud compute zonal disk. For more information on Google Cloud Disk Options, please visit [here](https://cloud.google.com/compute/docs/disks). ```commandline theme={null} gcloud compute disks create --size=100GB --zone= nfs-disk ``` #### Deploy NFS Server in GKE ```yaml theme={null} # nfs-server-deployment.yml apiVersion: apps/v1 kind: Deployment metadata: name: nfs-server spec: replicas: 1 selector: matchLabels: role: nfs-server template: metadata: labels: role: nfs-server spec: initContainers: - name: init-airflow-directories image: busybox command: ['sh', '-c', 'mkdir -p /exports/airflow-dags /exports/airflow-logs'] volumeMounts: - mountPath: /exports name: nfs-pvc containers: - name: nfs-server image: itsthenetwork/nfs-server-alpine env: - name: SHARED_DIRECTORY value: /exports ports: - name: nfs containerPort: 2049 securityContext: privileged: true volumeMounts: - mountPath: /exports name: nfs-pvc volumes: - name: nfs-pvc gcePersistentDisk: pdName: nfs-disk fsType: ext4 --- # nfs-cluster-ip-service.yml apiVersion: v1 kind: Service metadata: name: nfs-server spec: ports: - name: nfs port: 2049 selector: role: nfs-server ``` Run the commands below and ensure the pods are running. ```commandline theme={null} kubectl create -f nfs-server-deployment.yml kubectl create -f nfs-cluster-ip-service.yml ``` We create a ClusterIP Service for pods to access NFS within the cluster at a fixed IP/DNS. #### Provision NFS backed PV and PVC for Airflow DAGs and Airflow Logs Update `` with the NFS Service Cluster IP Address for below code snippets. You can get the clusterIP using the following command ```commandline theme={null} kubectl get service nfs-server -o jsonpath='{.spec.clusterIP}' ``` #### Code Samples for PV and PVC for Airflow DAGs ```yaml theme={null} # dags_pv_pvc.yml apiVersion: v1 kind: PersistentVolume metadata: name: openmetadata-dependencies-dags-pv spec: capacity: storage: 10Gi accessModes: - ReadWriteMany nfs: server: path: "/airflow-dags" --- apiVersion: v1 kind: PersistentVolumeClaim metadata: labels: app: airflow release: openmetadata-dependencies name: openmetadata-dependencies-dags namespace: default spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi storageClassName: "" ``` Create Persistent Volumes and Persistent Volume claims with the below command. ```commandline theme={null} kubectl create -f dags_pv_pvc.yml ``` #### Code Samples for PV and PVC for Airflow Logs ```yaml theme={null} # logs_pv_pvc.yml apiVersion: v1 kind: PersistentVolume metadata: name: openmetadata-dependencies-logs-pv spec: capacity: storage: 10Gi accessModes: - ReadWriteMany nfs: server: path: "/airflow-logs" --- apiVersion: v1 kind: PersistentVolumeClaim metadata: labels: app: airflow name: openmetadata-dependencies-logs namespace: default spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi storageClassName: "" ``` Create Persistent Volumes and Persistent Volume claims with the below command. ```commandline theme={null} kubectl create -f logs_pv_pvc.yml ``` ### Change owner and permission manually on disks Since airflow pods run as non root users, they would not have write access on the nfs server volumes. In order to fix the permission here, spin up a pod with persistent volumes attached and run it once. ```yaml theme={null} # permissions_pod.yml apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: my-permission-pod name: my-permission-pod spec: containers: - image: nginx name: my-permission-pod volumeMounts: - name: airflow-dags mountPath: /airflow-dags - name: airflow-logs mountPath: /airflow-logs volumes: - name: airflow-logs persistentVolumeClaim: claimName: openmetadata-dependencies-logs - name: airflow-dags persistentVolumeClaim: claimName: openmetadata-dependencies-dags dnsPolicy: ClusterFirst restartPolicy: Always ``` Airflow runs the pods with linux user name as airflow and linux user id as 50000. Run the below command to create the pod and fix the permissions ```commandline theme={null} kubectl create -f permissions_pod.yml ``` Once the permissions pod is up and running, execute the below commands within the container. ```commandline theme={null} kubectl exec --tty my-permission-pod --container my-permission-pod -- chown -R 50000 /airflow-dags /airflow-logs # If needed kubectl exec --tty my-permission-pod --container my-permission-pod -- chmod -R a+rwx /airflow-dags ``` ### Create OpenMetadata dependencies Values Override openmetadata dependencies airflow helm values to bind the nfs persistent volumes for DAGs and logs. ```yaml theme={null} # values-dependencies.yml airflow: airflow: extraVolumeMounts: - mountPath: /airflow-logs name: nfs-airflow-logs - mountPath: /airflow-dags/dags name: nfs-airflow-dags extraVolumes: - name: nfs-airflow-logs persistentVolumeClaim: claimName: openmetadata-dependencies-logs - name: nfs-airflow-dags persistentVolumeClaim: claimName: openmetadata-dependencies-dags config: AIRFLOW__OPENMETADATA_AIRFLOW_APIS__DAG_GENERATED_CONFIGS: "/airflow-dags/dags" dags: path: /airflow-dags/dags persistence: enabled: false logs: path: /airflow-logs persistence: enabled: false ``` For more information on airflow helm chart values, please refer to [airflow-helm](https://artifacthub.io/packages/helm/airflow-helm/airflow/8.8.0). When deploying openmeteadata dependencies helm chart, use the below command - ```commandline theme={null} helm install openmetadata-dependencies open-metadata/openmetadata-dependencies --values values-dependencies.yaml ``` The above command uses configurations defined [here](https://raw.githubusercontent.com/open-metadata/openmetadata-helm-charts/main/charts/deps/values.yaml). You can modify any configuration and deploy by passing your own `values.yaml` ```commandline theme={null} helm install openmetadata-dependencies open-metadata/openmetadata-dependencies --values ``` Once the openmetadata dependencies helm chart deployed, you can then run the below command to install the openmetadata helm chart - ```commandline theme={null} helm install openmetadata open-metadata/openmetadata --values ``` Create CloudSQL and Elasticsearch credentials as Kubernetes Secrets, as described [here](/v2.0.x/quick-start/local-kubernetes-deployment#2-create-kubernetes-secrets-required-for-helm-charts). Also, disable MySQL and Elasticsearch from OpenMetadata Dependencies Helm Charts as described in the [FAQ](#how-to-disable-mysql-and-elasticsearch-from-openmetadata-dependencies-helm-charts-). ## Troubleshooting ### Pods are stuck in Pending State due to Persistent Volume Creation Failure If you came across `invalid access type while creating the pvc`, and the permission pod is stuck in "pending" state. The above error might have occurred due to the pvc volumes not setup or pvc volumes are not mounted properly. dag-log permission-pod-events Please validate: * all the prerequisites mentioned in this [section](/v2.0.x/deployment/kubernetes/gke#prerequisites) * the configuration of `dags_pv_pvc.yml` file * `storageClassName` field in YAML file ## FAQs ## Java Memory Heap Issue If your openmetadata pods are not in ready state at any point in time and the openmetadata pod logs speaks about the below issue - ``` Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "AsyncAppender-Worker-async-file-appender" Exception in thread "pool-5-thread-1" java.lang.OutOfMemoryError: Java heap space Exception in thread "AsyncAppender-Worker-async-file-appender" java.lang.OutOfMemoryError: Java heap space Exception in thread "dw-46" java.lang.OutOfMemoryError: Java heap space Exception in thread "AsyncAppender-Worker-async-console-appender" java.lang.OutOfMemoryError: Java heap space ``` This is due to the default JVM Heap Space configuration (1 GiB) being not enough for your workloads. In order to resolve this issue, head over to your custom openmetadata helm values and append the below environment variable ```yaml theme={null} extraEnvs: - name: OPENMETADATA_HEAP_OPTS value: "-Xmx2G -Xms2G" ``` The flag `Xmx` specifies the maximum memory allocation pool for a Java virtual machine (JVM), while `Xms` specifies the initial memory allocation pool. Upgrade the helm charts with the above changes using the following command `helm upgrade --install openmetadata open-metadata/openmetadata --values --namespace `. Update this command your `values.yml` filename and `namespaceName` where you have deployed OpenMetadata in Kubernetes. ## PostgreSQL Issue permission denied to create extension "pgcrypto" If you are facing the below issue with PostgreSQL as Database Backend for OpenMetadata Application, ``` Message: ERROR: permission denied to create extension "pgcrypto" Hint: Must be superuser to create this extension. ``` It seems the Database User does not have sufficient privileges. In order to resolve the above issue, grant usage permissions to the PSQL User. ```sql theme={null} GRANT USAGE ON SCHEMA schema_name TO ; GRANT CREATE ON EXTENSION pgcrypto TO ; ``` In the above command, replace `` with the sql user used by OpenMetadata Application to connect to PostgreSQL Database. ## How to extend and use custom docker images with OpenMetadata Helm Charts ? ## Extending OpenMetadata Server Docker Image ### 1. Create a `Dockerfile` based on `docker.open-metadata.org/openmetadata/server` OpenMetadata helm charts uses official published docker images from [DockerHub](https://hub.docker.com/u/openmetadata). A typical scenario will be to install organization certificates for connecting with inhouse systems. For Example - ``` FROM docker.open-metadata.org/openmetadata/server:x.y.z WORKDIR /home/ COPY . RUN update-ca-certificates ``` where `docker.open-metadata.org/openmetadata/server:x.y.z` needs to point to the same version of the OpenMetadata server, for example `docker.open-metadata.org/openmetadata/server:1.3.1`. This image needs to be built and published to the container registry of your choice. ### 2. Update your openmetadata helm values yaml The OpenMetadata Application gets installed as part of `openmetadata` helm chart. In this step, update the custom helm values using YAML file to point the image created in the previous step. For example, create a helm values file named `values.yaml` with the following contents - ```yaml theme={null} ... image: repository: # Overrides the image tag whose default is the chart appVersion. tag: ... ``` ### 3. Install / Upgrade your helm release Upgrade/Install your openmetadata helm charts with the below single command: ```bash theme={null} helm upgrade --install openmetadata open-metadata/openmetadata--values values.yaml ``` ## Extending OpenMetadata Ingestion Docker Image One possible use case for a custom ingestion image is a custom connector. Build and test the package with the same `openmetadata-ingestion` version as your deployment. After your code is ready, follow these steps: ### 1. Create a `Dockerfile` based on `docker.open-metadata.org/openmetadata/ingestion`: For example - ``` FROM docker.open-metadata.org/openmetadata/ingestion:x.y.z USER airflow # Let's use the home directory of airflow user WORKDIR /home/airflow # Install our custom connector COPY COPY setup.py . RUN pip install --no-deps . ``` where `docker.open-metadata.org/openmetadata/ingestion:x.y.z` needs to point to the same version of the OpenMetadata server, for example `docker.open-metadata.org/openmetadata/ingestion:1.3.1`. This image needs to be built and published to the container registry of your choice. ### 2. Update the airflow in openmetadata dependencies values YAML The ingestion containers (which is the one shipping Airflow) gets installed in the `openmetadata-dependencies` helm chart. In this step, we use our own custom values YAML file to point to the image we just created on the previous step. You can create a file named `values.deps.yaml` with the following contents: ```yaml theme={null} airflow: airflow: image: repository: # by default, openmetadata/ingestion tag: # by default, the version you are deploying, e.g., 1.1.0 pullPolicy: "IfNotPresent" ``` ### 3. Install / Upgrade helm release Upgrade/Install your openmetadata-dependencies helm charts with the below single command: ```bash theme={null} helm upgrade --install openmetadata-dependencies open-metadata/openmetadata-dependencies --values values.deps.yaml ``` ## How to disable MySQL and ElasticSearch from OpenMetadata Dependencies Helm Charts ? If you are using MySQL and ElasticSearch externally, you would want to disable the local installation of mysql and elasticsearch while installing OpenMetadata Dependencies Helm Chart. You can disable the MySQL and ElasticSearch Helm Dependencies by setting `enabled: false` value for each dependency. Below is the command to set helm values from Helm CLI - ```commandline theme={null} helm upgrade --install openmetadata-dependencies open-metadata/openmetadata-dependencies --set mysql.enabled=false --set elasticsearch.enabled=false ``` Alternatively, you can create a custom YAML file named `values.deps.yaml` to disable installation of MySQL and Elasticsearch . ```yaml theme={null} mysql: enabled: false ... elasticsearch: enabled: false ... ... ``` ## How to configure external database like PostgreSQL with OpenMetadata Helm Charts ? OpenMetadata Supports PostgreSQL as one of the Database Dependencies. OpenMetadata Helm Charts by default does not include PostgreSQL as Database Dependencies. In order to configure Helm Charts with External Database like PostgreSQL, follow the below guide to make the helm values change and upgrade / install OpenMetadata helm charts with the same. ## Upgrade Airflow Helm Dependencies Helm Charts to connect to External Database like PostgreSQL We ship [airflow-helm](https://github.com/airflow-helm/charts/tree/main/charts/airflow) as one of OpenMetadata Dependencies with default values to connect to MySQL Database as part of `externalDatabase` configurations. You can find more information on setting the `externalDatabase` as part of helm values [here](https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/database/external-database.md). With OpenMetadata Dependencies Helm Charts, your helm values would look something like below - ```yaml theme={null} ... airflow: externalDatabase: type: postgresql host: port: 5432 database: user: passwordSecret: airflow-postgresql-secrets passwordSecretKey: airflow-postgresql-password ... ``` For the above code, it is assumed you are creating a kubernetes secret for storing Airflow Database login Credentials. A sample command to create the secret will be `kubectl create secret generic airflow-postgresql-secrets --from-literal=airflow-postgresql-password=`. ## Upgrade OpenMetadata Helm Charts to connect to External Database like PostgreSQL Update the `openmetadata.config.database.*` helm values for OpenMetadata Application to connect to External Database like PostgreSQL. With OpenMetadata Helm Charts, your helm values would look something like below - ```yaml theme={null} openmetadata: config: ... database: host: port: 5432 driverClass: org.postgresql.Driver dbScheme: postgresql dbUseSSL: true databaseName: auth: username: password: secretRef: openmetadata-postgresql-secrets secretKey: openmetadata-postgresql-password ``` For the above code, it is assumed you are creating a kubernetes secret for storing OpenMetadata Database login Credentials. A sample command to create the secret will be `kubectl create secret generic openmetadata-postgresql-secrets --from-literal=openmetadata-postgresql-password=`. Once you make the above changes to your helm values, run the below command to install/upgrade helm charts - ```commandline theme={null} helm upgrade --install openmetadata-dependencies open-metadata/openmetadata-dependencies --values <> --namespace helm upgrade --install openmetadata open-metadata/openmetadata --values <> --namespace ``` ## How to customize OpenMetadata Dependencies Helm Chart with custom helm values Our OpenMetadata Dependencies Helm Charts are internally depends on three sub-charts - * [Bitnami MySQL](https://artifacthub.io/packages/helm/bitnami/mysql/9.7.2) (helm chart version 9.7.2) * [OpenSearch](https://artifacthub.io/packages/helm/opensearch-project-helm-charts/opensearch/2.12.2) (helm chart version 2.12.2) * [Airflow](https://artifacthub.io/packages/helm/airflow-helm/airflow/8.8.0) (helm chart version 8.8.0) If you are looking to customize the deployments of any of the above dependencies, please refer to the above links for customizations of helm values for further references. By default, OpenMetadata Dependencies helm chart provides initial generic customization of these helm values in order to get you started quickly. You can refer to the openmetadata-dependencies helm charts default values [here](https://github.com/open-metadata/openmetadata-helm-charts/blob/main/charts/deps/values.yaml). # Kubernetes On Premises Deployment | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/kubernetes/on-prem Set up the OpenMetadata on-premises with Kubernetes for full control over infrastructure, security, and compliance requirements. # On Premises Kubernetes Deployment OpenMetadata supports the Installation and Running of application on OnPremises Kubernetes through Helm Charts. However, there are some additional configurations which needs to be done as prerequisites for the same. This guide presumes you have an on premises Kubernetes cluster setup, and you are installing OpenMetadata in `default` namespace. ## Prerequisites ### External Database and Search Engine as ElasticSearch / OpenSearch We support * MySQL engine version 8 or higher * PostgreSQL engine version 15 or higher * ElasticSearch version 9.x (minimum 9.0.0, recommended 9.3.0) or OpenSearch version 3.x (minimum 3.0.0, recommended 3.3.0) Once you have the External Database and Search Engine configured, you can update the environment variables below for OpenMetadata kubernetes deployments to connect with Database and ElasticSearch. ```yaml theme={null} # openmetadata-values.prod.yaml ... openmetadata: config: elasticsearch: host: searchType: elasticsearch # or `opensearch` if Search Engine is OpenSearch port: 443 scheme: https connectionTimeoutSecs: 5 socketTimeoutSecs: 60 keepAliveTimeoutSecs: 600 batchSize: 10 auth: enabled: true username: password: secretRef: elasticsearch-secrets secretKey: openmetadata-elasticsearch-password database: host: port: 3306 driverClass: com.mysql.cj.jdbc.Driver dbScheme: mysql dbUseSSL: true databaseName: auth: username: password: secretRef: mysql-secrets secretKey: openmetadata-mysql-password ... ``` Create database and search engine credentials as Kubernetes Secrets, as described [here](/v2.0.x/quick-start/local-kubernetes-deployment#2-create-kubernetes-secrets-required-for-helm-charts). Also, disable MySQL and Elasticsearch from OpenMetadata Dependencies Helm Charts as described in the [FAQ](/v2.0.x/deployment/kubernetes/on-prem/airflow#how-to-disable-mysql-and-elasticsearch-from-openmetadata-dependencies-helm-charts-). ### Persistent Volumes with ReadWriteMany Access Modes OpenMetadata helm chart depends on Airflow and Airflow expects a persistent disk that support ReadWriteMany (the volume can be mounted as read-write by many nodes). The workaround is to create nfs-share and use that as the persistent claim to deploy OpenMetadata by implementing the following steps in order. This guide assumes you have NFS Server already setup with Hostname or IP Address which is reachable from your on premises Kubernetes cluster, and you have configured a path to be used for OpenMetadata Airflow Helm Dependency. ### Dynamic Provisioning using StorageClass To provision PersistentVolume dynamically using the StorageClass, you need to install the NFS provisioner. It is recommended to use [nfs-subdir-external-provisioner](https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner) helm charts for this case. ```commandline theme={null} helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \ --create-namespace \ --namespace nfs-provisioner \ --set nfs.server= \ --set nfs.path=/airflow ``` Replace the `NFS_HOSTNAME_OR_IP` with your NFS Server value and run the commands. This will create a new StorageClass with `nfs-subdir-external-provisioner`. You can view the same using the kubectl command `kubectl get storageclass -n nfs-provisioner`. *** Continue to [On-Prem Airflow Storage Setup](/v2.0.x/deployment/kubernetes/on-prem/airflow) to provision NFS-backed persistent volumes, configure Airflow dependencies, and deploy OpenMetadata. # On-Prem Airflow Storage Setup | OpenMetadata Kubernetes Guide Source: https://docs.open-metadata.org/v2.0.x/deployment/kubernetes/on-prem/airflow Provision NFS-backed persistent volumes for Airflow DAGs and logs on on-premises Kubernetes and deploy OpenMetadata. # On-Prem Airflow Storage Setup This guide walks through provisioning NFS-backed PVCs for Airflow DAGs and logs on an on-premises Kubernetes cluster, then deploying OpenMetadata. For prerequisites and StorageClass setup, see the [On-Prem Kubernetes Deployment](/v2.0.x/deployment/kubernetes/on-prem) page. ## Provision NFS backed PVC for Airflow DAGs and Airflow Logs ### Code Samples for PVC for Airflow DAGs ```yaml theme={null} # dags_pvc.yml apiVersion: v1 kind: PersistentVolumeClaim metadata: namespace: default name: openmetadata-dependencies-dags labels: storage.k8s.io/name: nfs app: airflow spec: accessModes: - ReadWriteMany storageClassName: nfs-client resources: requests: storage: 1Gi ``` Create Persistent Volumes and Persistent Volume claims with the below command. ```commandline theme={null} kubectl create -f dags_pvc.yml ``` ### Code Samples for PVC for Airflow Logs ```yaml theme={null} # logs_pvc.yml apiVersion: v1 kind: PersistentVolumeClaim metadata: namespace: default name: openmetadata-dependencies-logs labels: storage.k8s.io/name: nfs app: airflow spec: accessModes: - ReadWriteMany storageClassName: nfs-client resources: requests: storage: 10Gi ``` Create Persistent Volumes and Persistent Volume claims with the below command. ```commandline theme={null} kubectl create -f logs_pvc.yml ``` ## Change owner and permission manually on disks Since airflow pods run as non-root users, they would not have write access on the nfs server volumes. In order to fix the permission here, spin up a pod with persistent volumes attached and run it once. ```yaml theme={null} # permissions_pod.yml apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: my-permission-pod name: my-permission-pod spec: containers: - image: busybox name: my-permission-pod volumeMounts: - name: airflow-dags mountPath: /airflow-dags - name: airflow-logs mountPath: /airflow-logs command: - "chown -R 50000 /airflow-dags /airflow-logs" # if needed - "chmod -R a+rwx /airflow-dags" volumes: - name: airflow-logs persistentVolumeClaim: claimName: openmetadata-dependencies-logs - name: airflow-dags persistentVolumeClaim: claimName: openmetadata-dependencies-dags dnsPolicy: ClusterFirst restartPolicy: Always ``` Airflow runs the pods with linux username as airflow and linux user id as 50000. Run the below command to create the pod and fix the permissions ```commandline theme={null} kubectl create -f permissions_pod.yml ``` ## Create OpenMetadata dependencies Values Override openmetadata dependencies airflow helm values to bind the nfs persistent volumes for DAGs and logs. ```yaml theme={null} # values-dependencies.yml airflow: airflow: extraVolumeMounts: - mountPath: /airflow-logs name: nfs-airflow-logs - mountPath: /airflow-dags/dags name: nfs-airflow-dags extraVolumes: - name: nfs-airflow-logs persistentVolumeClaim: claimName: openmetadata-dependencies-logs - name: nfs-airflow-dags persistentVolumeClaim: claimName: openmetadata-dependencies-dags config: AIRFLOW__OPENMETADATA_AIRFLOW_APIS__DAG_GENERATED_CONFIGS: "/airflow-dags/dags" dags: path: /airflow-dags/dags persistence: enabled: false logs: path: /airflow-logs persistence: enabled: false ``` For more information on airflow helm chart values, please refer to [airflow-helm](https://artifacthub.io/packages/helm/airflow-helm/airflow/8.8.0). When deploying openmetadata dependencies helm chart, use the below command - ```commandline theme={null} helm install openmetadata-dependencies open-metadata/openmetadata-dependencies --values values-dependencies.yaml ``` The above command uses configurations defined [here](https://raw.githubusercontent.com/open-metadata/openmetadata-helm-charts/main/charts/deps/values.yaml). You can modify any configuration and deploy by passing your own `values.yaml` ```commandline theme={null} helm install openmetadata-dependencies open-metadata/openmetadata-dependencies --values ``` Once the openmetadata dependencies helm chart deployed, you can then run the below command to install the openmetadata helm chart - ```commandline theme={null} helm install openmetadata open-metadata/openmetadata ``` Again, this uses the values defined [here](https://github.com/open-metadata/openmetadata-helm-charts/blob/main/charts/openmetadata/values.yaml). Use the `--values` flag to point to your own YAML configuration if needed. ## Troubleshooting Starting with **OpenMetadata v1.12.4**, the dependency Helm chart no longer supports passing database passwords using individual Kubernetes secret keys (for example, `passwordSecret` and `passwordSecretKey`). Instead, database credentials must be provided via a **single Kubernetes Secret** referenced using `metadataSecretName`. This secret must contain the **full database connection string**, including the password. ```yaml theme={null} data: metadataSecretName: your-connection-string-secret ``` This change applies **only to the dependency `values.yml` configuration** and aligns with the database configuration approach used by the **Airflow Helm chart**. ## FAQs ## Java Memory Heap Issue If your openmetadata pods are not in ready state at any point in time and the openmetadata pod logs speaks about the below issue - ``` Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "AsyncAppender-Worker-async-file-appender" Exception in thread "pool-5-thread-1" java.lang.OutOfMemoryError: Java heap space Exception in thread "AsyncAppender-Worker-async-file-appender" java.lang.OutOfMemoryError: Java heap space Exception in thread "dw-46" java.lang.OutOfMemoryError: Java heap space Exception in thread "AsyncAppender-Worker-async-console-appender" java.lang.OutOfMemoryError: Java heap space ``` This is due to the default JVM Heap Space configuration (1 GiB) being not enough for your workloads. In order to resolve this issue, head over to your custom openmetadata helm values and append the below environment variable ```yaml theme={null} extraEnvs: - name: OPENMETADATA_HEAP_OPTS value: "-Xmx2G -Xms2G" ``` The flag `Xmx` specifies the maximum memory allocation pool for a Java virtual machine (JVM), while `Xms` specifies the initial memory allocation pool. Upgrade the helm charts with the above changes using the following command `helm upgrade --install openmetadata open-metadata/openmetadata --values --namespace `. Update this command your `values.yml` filename and `namespaceName` where you have deployed OpenMetadata in Kubernetes. ## PostgreSQL Issue permission denied to create extension "pgcrypto" If you are facing the below issue with PostgreSQL as Database Backend for OpenMetadata Application, ``` Message: ERROR: permission denied to create extension "pgcrypto" Hint: Must be superuser to create this extension. ``` It seems the Database User does not have sufficient privileges. In order to resolve the above issue, grant usage permissions to the PSQL User. ```sql theme={null} GRANT USAGE ON SCHEMA schema_name TO ; GRANT CREATE ON EXTENSION pgcrypto TO ; ``` In the above command, replace `` with the sql user used by OpenMetadata Application to connect to PostgreSQL Database. ## How to extend and use custom docker images with OpenMetadata Helm Charts ? ## Extending OpenMetadata Server Docker Image ### 1. Create a `Dockerfile` based on `docker.open-metadata.org/openmetadata/server` OpenMetadata helm charts uses official published docker images from [DockerHub](https://hub.docker.com/u/openmetadata). A typical scenario will be to install organization certificates for connecting with inhouse systems. For Example - ``` FROM docker.open-metadata.org/openmetadata/server:x.y.z WORKDIR /home/ COPY . RUN update-ca-certificates ``` where `docker.open-metadata.org/openmetadata/server:x.y.z` needs to point to the same version of the OpenMetadata server, for example `docker.open-metadata.org/openmetadata/server:1.3.1`. This image needs to be built and published to the container registry of your choice. ### 2. Update your openmetadata helm values yaml The OpenMetadata Application gets installed as part of `openmetadata` helm chart. In this step, update the custom helm values using YAML file to point the image created in the previous step. For example, create a helm values file named `values.yaml` with the following contents - ```yaml theme={null} ... image: repository: # Overrides the image tag whose default is the chart appVersion. tag: ... ``` ### 3. Install / Upgrade your helm release Upgrade/Install your openmetadata helm charts with the below single command: ```bash theme={null} helm upgrade --install openmetadata open-metadata/openmetadata--values values.yaml ``` ## Extending OpenMetadata Ingestion Docker Image One possible use case for a custom ingestion image is a custom connector. Build and test the package with the same `openmetadata-ingestion` version as your deployment. After your code is ready, follow these steps: ### 1. Create a `Dockerfile` based on `docker.open-metadata.org/openmetadata/ingestion`: For example - ``` FROM docker.open-metadata.org/openmetadata/ingestion:x.y.z USER airflow # Let's use the home directory of airflow user WORKDIR /home/airflow # Install our custom connector COPY COPY setup.py . RUN pip install --no-deps . ``` where `docker.open-metadata.org/openmetadata/ingestion:x.y.z` needs to point to the same version of the OpenMetadata server, for example `docker.open-metadata.org/openmetadata/ingestion:1.3.1`. This image needs to be built and published to the container registry of your choice. ### 2. Update the airflow in openmetadata dependencies values YAML The ingestion containers (which is the one shipping Airflow) gets installed in the `openmetadata-dependencies` helm chart. In this step, we use our own custom values YAML file to point to the image we just created on the previous step. You can create a file named `values.deps.yaml` with the following contents: ```yaml theme={null} airflow: airflow: image: repository: # by default, openmetadata/ingestion tag: # by default, the version you are deploying, e.g., 1.1.0 pullPolicy: "IfNotPresent" ``` ### 3. Install / Upgrade helm release Upgrade/Install your openmetadata-dependencies helm charts with the below single command: ```bash theme={null} helm upgrade --install openmetadata-dependencies open-metadata/openmetadata-dependencies --values values.deps.yaml ``` ## How to disable MySQL and ElasticSearch from OpenMetadata Dependencies Helm Charts ? If you are using MySQL and ElasticSearch externally, you would want to disable the local installation of mysql and elasticsearch while installing OpenMetadata Dependencies Helm Chart. You can disable the MySQL and ElasticSearch Helm Dependencies by setting `enabled: false` value for each dependency. Below is the command to set helm values from Helm CLI - ```commandline theme={null} helm upgrade --install openmetadata-dependencies open-metadata/openmetadata-dependencies --set mysql.enabled=false --set elasticsearch.enabled=false ``` Alternatively, you can create a custom YAML file named `values.deps.yaml` to disable installation of MySQL and Elasticsearch . ```yaml theme={null} mysql: enabled: false ... elasticsearch: enabled: false ... ... ``` ## How to configure external database like PostgreSQL with OpenMetadata Helm Charts ? OpenMetadata Supports PostgreSQL as one of the Database Dependencies. OpenMetadata Helm Charts by default does not include PostgreSQL as Database Dependencies. In order to configure Helm Charts with External Database like PostgreSQL, follow the below guide to make the helm values change and upgrade / install OpenMetadata helm charts with the same. ## Upgrade Airflow Helm Dependencies Helm Charts to connect to External Database like PostgreSQL We ship [airflow-helm](https://github.com/airflow-helm/charts/tree/main/charts/airflow) as one of OpenMetadata Dependencies with default values to connect to MySQL Database as part of `externalDatabase` configurations. You can find more information on setting the `externalDatabase` as part of helm values [here](https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/database/external-database.md). With OpenMetadata Dependencies Helm Charts, your helm values would look something like below - ```yaml theme={null} ... airflow: externalDatabase: type: postgresql host: port: 5432 database: user: passwordSecret: airflow-postgresql-secrets passwordSecretKey: airflow-postgresql-password ... ``` For the above code, it is assumed you are creating a kubernetes secret for storing Airflow Database login Credentials. A sample command to create the secret will be `kubectl create secret generic airflow-postgresql-secrets --from-literal=airflow-postgresql-password=`. ## Upgrade OpenMetadata Helm Charts to connect to External Database like PostgreSQL Update the `openmetadata.config.database.*` helm values for OpenMetadata Application to connect to External Database like PostgreSQL. With OpenMetadata Helm Charts, your helm values would look something like below - ```yaml theme={null} openmetadata: config: ... database: host: port: 5432 driverClass: org.postgresql.Driver dbScheme: postgresql dbUseSSL: true databaseName: auth: username: password: secretRef: openmetadata-postgresql-secrets secretKey: openmetadata-postgresql-password ``` For the above code, it is assumed you are creating a kubernetes secret for storing OpenMetadata Database login Credentials. A sample command to create the secret will be `kubectl create secret generic openmetadata-postgresql-secrets --from-literal=openmetadata-postgresql-password=`. Once you make the above changes to your helm values, run the below command to install/upgrade helm charts - ```commandline theme={null} helm upgrade --install openmetadata-dependencies open-metadata/openmetadata-dependencies --values <> --namespace helm upgrade --install openmetadata open-metadata/openmetadata --values <> --namespace ``` ## How to customize OpenMetadata Dependencies Helm Chart with custom helm values Our OpenMetadata Dependencies Helm Charts are internally depends on three sub-charts - * [Bitnami MySQL](https://artifacthub.io/packages/helm/bitnami/mysql/9.7.2) (helm chart version 9.7.2) * [OpenSearch](https://artifacthub.io/packages/helm/opensearch-project-helm-charts/opensearch/2.12.2) (helm chart version 2.12.2) * [Airflow](https://artifacthub.io/packages/helm/airflow-helm/airflow/8.8.0) (helm chart version 8.8.0) If you are looking to customize the deployments of any of the above dependencies, please refer to the above links for customizations of helm values for further references. By default, OpenMetadata Dependencies helm chart provides initial generic customization of these helm values in order to get you started quickly. You can refer to the openmetadata-dependencies helm charts default values [here](https://github.com/open-metadata/openmetadata-helm-charts/blob/main/charts/deps/values.yaml). # Enable Security Source: https://docs.open-metadata.org/v2.0.x/deployment/kubernetes/security # Kubernetes Security Follow the steps for setting up the SSO, and then check the specific `Kubernetes` section of your chosen SSO. By default, Basic Authentication will be enabled as authentication mechanism. Configure Basic Authentication to access the UI and APIs Configure Ldap Authentication to access the UI and APIs Configure Auth0 SSO to access the UI and APIs Configure Azure SSO to access the UI and APIs Configure a Custom OIDC SSO to access the UI and APIs Configure Google SSO to access the UI and APIs Configure Okta SSO to access the UI and APIs # Configuring OpenMetadata to Run Under a Subpath Source: https://docs.open-metadata.org/v2.0.x/deployment/kubernetes/subpath ## Subpath in OpenMetadata To configure **OpenMetadata** to operate under a subpath (for example `/openmetadata`), useful when deploying behind a reverse proxy or load balancer, you need to adjust specific settings in the `openmetadata.yaml` configuration file. **`BASE_PATH` must not have a trailing slash**, but `basePath` in `openmetadata.yaml` needs one. OpenMetadata builds static asset URLs by appending directly to `basePath` with no separator, so if it's missing the trailing slash, requests resolve to `/openmetadataassets/...` instead of `/openmetadata/assets/...` and every static asset 404s. ## Configuration Steps ### 1. Set the Base Path Define the `basePath` parameter to configure the application's root context, and ensure that the `publicKeyUrl` is updated accordingly to reflect the new base path. This sets the root context for the application. Note the trailing slash on `basePath`: ```yaml theme={null} basePath: ${BASE_PATH:-/openmetadata}/ ``` This configuration sets the base path to /openmetadata by default. You can override it by setting the BASE\_PATH environment variable: set `BASE_PATH` itself **without** a trailing slash (e.g., `BASE_PATH=/openmetadata`), since the `/` above is appended for you. ### 2. Configure Web Paths Configure the web application and API endpoint paths to align with the specified base path. These settings live under the top-level `server:` block: ```yaml theme={null} server: applicationContextPath: ${BASE_PATH:-/openmetadata} rootPath: ${BASE_PATH:-/openmetadata}api/* ``` * `applicationContextPath`: Defines the context path for the web application. * `rootPath`: Specifies the root path for API endpoints. [GitHub](https://github.com/open-metadata/OpenMetadata/discussions/17954) ### 3. Set Asset Paths Ensure that asset paths are correctly prefixed with the base path. ```yaml theme={null} assets: resourcePath: /openmetadata/assets/ uriPath: ${BASE_PATH:-/openmetadata} ``` * `resourcePath`: Path to static resources. * `uriPath`: URI path prefix for assets. Subpath ## Example Configuration Here's how the relevant section of your `openmetadata.yaml` might look: ```yaml theme={null} basePath: ${BASE_PATH:-/openmetadata}/ publicKeyUrl: ${BASE_PATH:-/}api/v1/system/config/jwks server: applicationContextPath: ${BASE_PATH:-/openmetadata} rootPath: ${BASE_PATH:-/openmetadata}api/* assets: resourcePath: /openmetadata/assets/ uriPath: ${BASE_PATH:-/openmetadata} ``` ## Deployment Considerations * **Reverse Proxy Configuration**: Ensure that your reverse proxy (e.g., NGINX, Apache) is configured to forward requests to the OpenMetadata application with the correct subpath. * **Environment Variables**: You can override the default base path by setting the BASE\_PATH environment variable in your deployment environment. Ensure that related parameters such as basePath, applicationContextPath, rootPath, and publicKeyUrl are updated to reflect this change. * **Static Assets**: Verify that static assets are accessible under the new subpath to prevent broken links or missing resources. # Kubernetes Helm Values | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/kubernetes/values Customize your Helm values for Kubernetes deployments to control services, authentication, storage, and resource tuning. # Kubernetes Helm Values This page list all the supported helm values for OpenMetadata Helm Charts. ## Openmetadata Config Chart Values | Key | Type | Default | Environment Variable from openmetadata.yaml | | ------------------------------------------------------------------------------------------------------------------------------- | ------ | --------------------------------------------------------------------- | -------------------------------------------------- | | openmetadata.config.authentication.enabled | bool | `true` | | | openmetadata.config.authentication.clientType | string | `public` | AUTHENTICATION\_CLIENT\_TYPE | | openmetadata.config.authentication.provider | string | `basic` | AUTHENTICATION\_PROVIDER | | openmetadata.config.authentication.publicKeys | list | `[http://openmetadata:8585/api/v1/system/config/jwks]` | AUTHENTICATION\_PUBLIC\_KEYS | | openmetadata.config.authentication.authority | string | `https://accounts.google.com` | AUTHENTICATION\_AUTHORITY | | openmetadata.config.authentication.clientId | string | `Empty String` | AUTHENTICATION\_CLIENT\_ID | | openmetadata.config.authentication.callbackUrl | string | `Empty String` | AUTHENTICATION\_CALLBACK\_URL | | openmetadata.config.authentication.enableSelfSignup | bool | `true` | AUTHENTICATION\_ENABLE\_SELF\_SIGNUP | | openmetadata.config.authentication.jwtPrincipalClaims | list | `[email,preferred_username,sub]` | AUTHENTICATION\_JWT\_PRINCIPAL\_CLAIMS | | openmetadata.config.authentication.ldapConfiguration.host | string | `localhost` | AUTHENTICATION\_LDAP\_HOST | | openmetadata.config.authentication.ldapConfiguration.port | int | 10636 | AUTHENTICATION\_LDAP\_PORT | | openmetadata.config.authentication.ldapConfiguration.dnAdminPrincipal | string | `cn=admin,dc=example,dc=com` | AUTHENTICATION\_LOOKUP\_ADMIN\_DN | | openmetadata.config.authentication.ldapConfiguration.dnAdminPassword.secretRef | string | `ldap-secret` | AUTHENTICATION\_LOOKUP\_ADMIN\_PWD | | openmetadata.config.authentication.ldapConfiguration.dnAdminPassword.secretKey | string | `openmetadata-ldap-secret` | AUTHENTICATION\_LOOKUP\_ADMIN\_PWD | | openmetadata.config.authentication.ldapConfiguration.userBaseDN | string | `ou=people,dc=example,dc=com` | AUTHENTICATION\_USER\_LOOKUP\_BASEDN | | openmetadata.config.authentication.ldapConfiguration.groupBaseDN | string | `Empty String` | AUTHENTICATION\_GROUP\_LOOKUP\_BASEDN | | openmetadata.config.authentication.ldapConfiguration.roleAdminName | string | `Empty String` | AUTHENTICATION\_USER\_ROLE\_ADMIN\_NAME | | openmetadata.config.authentication.ldapConfiguration.allAttributeName | string | `Empty String` | AUTHENTICATION\_USER\_ALL\_ATTR | | openmetadata.config.authentication.ldapConfiguration.usernameAttributeName | string | `Empty String` | AUTHENTICATION\_USER\_NAME\_ATTR | | openmetadata.config.authentication.ldapConfiguration.groupAttributeName | string | `Empty String` | AUTHENTICATION\_USER\_GROUP\_ATTR | | openmetadata.config.authentication.ldapConfiguration.groupAttributeValue | string | `Empty String` | AUTHENTICATION\_USER\_GROUP\_ATTR\_VALUE | | openmetadata.config.authentication.ldapConfiguration.groupMemberAttributeName | string | `Empty String` | AUTHENTICATION\_USER\_GROUP\_MEMBER\_ATTR | | openmetadata.config.authentication.ldapConfiguration.authRolesMapping | string | `Empty String` | AUTH\_ROLES\_MAPPING | | openmetadata.config.authentication.ldapConfiguration.authReassignRoles | string | `Empty String` | AUTH\_REASSIGN\_ROLES | | openmetadata.config.authentication.ldapConfiguration.mailAttributeName | string | `email` | AUTHENTICATION\_USER\_MAIL\_ATTR | | openmetadata.config.authentication.ldapConfiguration.maxPoolSize | int | 3 | AUTHENTICATION\_LDAP\_POOL\_SIZE | | openmetadata.config.authentication.ldapConfiguration.sslEnabled | bool | `true` | AUTHENTICATION\_LDAP\_SSL\_ENABLED | | openmetadata.config.authentication.ldapConfiguration.truststoreConfigType | string | `TrustAll` | AUTHENTICATION\_LDAP\_TRUSTSTORE\_TYPE | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.customTrustManagerConfig.trustStoreFilePath | string | `Empty String` | AUTHENTICATION\_LDAP\_TRUSTSTORE\_PATH | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.customTrustManagerConfig.trustStoreFilePassword.secretRef | string | `Empty String` | AUTHENTICATION\_LDAP\_KEYSTORE\_PASSWORD | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.customTrustManagerConfig.trustStoreFilePassword.secretKey | string | `Empty String` | AUTHENTICATION\_LDAP\_KEYSTORE\_PASSWORD | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.customTrustManagerConfig.trustStoreFileFormat | string | `Empty String` | AUTHENTICATION\_LDAP\_SSL\_KEY\_FORMAT | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.customTrustManagerConfig.verifyHostname | string | `Empty String` | AUTHENTICATION\_LDAP\_SSL\_VERIFY\_CERT\_HOST | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.customTrustManagerConfig.examineValidityDate | bool | `true` | AUTHENTICATION\_LDAP\_EXAMINE\_VALIDITY\_DATES | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.hostNameConfig.allowWildCards | bool | `false` | AUTHENTICATION\_LDAP\_ALLOW\_WILDCARDS | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.hostNameConfig.acceptableHostNames | string | `[Empty String]` | AUTHENTICATION\_LDAP\_ALLOWED\_HOSTNAMES | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.jvmDefaultConfig.verifyHostname | string | `Empty String` | AUTHENTICATION\_LDAP\_SSL\_VERIFY\_CERT\_HOST | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.trustAllConfig.examineValidityDates | bool | `true` | AUTHENTICATION\_LDAP\_EXAMINE\_VALIDITY\_DATES | | openmetadata.config.authentication.oidcConfiguration.callbackUrl | string | `http://openmetadata:8585/callback` | OIDC\_CALLBACK | | openmetadata.config.authentication.oidcConfiguration.clientAuthenticationMethod | string | `client_secret_post` | OIDC\_CLIENT\_AUTH\_METHOD | | openmetadata.config.authentication.oidcConfiguration.clientId.secretKey | string | `openmetadata-oidc-client-id` | OIDC\_CLIENT\_ID | | openmetadata.config.authentication.oidcConfiguration.clientId.secretRef | string | `oidc-secrets` | OIDC\_CLIENT\_ID | | openmetadata.config.authentication.oidcConfiguration.clientSecret.secretKey | string | `openmetadata-oidc-client-secret` | OIDC\_CLIENT\_SECRET | | openmetadata.config.authentication.oidcConfiguration.clientSecret.secretRef | string | `oidc-secrets` | OIDC\_CLIENT\_SECRET | | openmetadata.config.authentication.oidcConfiguration.customParams | string | `Empty` | OIDC\_CUSTOM\_PARAMS | | openmetadata.config.authentication.oidcConfiguration.disablePkce | bool | true | OIDC\_DISABLE\_PKCE | | openmetadata.config.authentication.oidcConfiguration.discoveryUri | string | `Empty` | OIDC\_DISCOVERY\_URI | | openmetadata.config.authentication.oidcConfiguration.enabled | bool | false | | | openmetadata.config.authentication.oidcConfiguration.maxClockSkew | string | `Empty` | OIDC\_MAX\_CLOCK\_SKEW | | openmetadata.config.authentication.oidcConfiguration.oidcType | string | `Empty` | OIDC\_TYPE | | openmetadata.config.authentication.oidcConfiguration.preferredJwsAlgorithm | string | `RS256` | OIDC\_PREFERRED\_JWS | | openmetadata.config.authentication.oidcConfiguration.responseType | string | `code` | OIDC\_RESPONSE\_TYPE | | openmetadata.config.authentication.oidcConfiguration.scope | string | `openid email profile` | OIDC\_SCOPE | | openmetadata.config.authentication.oidcConfiguration.serverUrl | string | `http://openmetadata:8585` | OIDC\_SERVER\_URL | | openmetadata.config.authentication.oidcConfiguration.tenant | string | `Empty` | OIDC\_TENANT | | openmetadata.config.authentication.oidcConfiguration.useNonce | bool | `true` | OIDC\_USE\_NONCE | | openmetadata.config.authentication.saml.debugMode | bool | false | SAML\_DEBUG\_MODE | | openmetadata.config.authentication.saml.idp.entityId | string | `Empty` | SAML\_IDP\_ENTITY\_ID | | openmetadata.config.authentication.saml.idp.ssoLoginUrl | string | `Empty` | SAML\_IDP\_SSO\_LOGIN\_URL | | openmetadata.config.authentication.saml.idp.idpX509Certificate.secretRef | string | `Empty` | SAML\_IDP\_CERTIFICATE | | openmetadata.config.authentication.saml.idp.idpX509Certificate.secretKey | string | `Empty` | SAML\_IDP\_CERTIFICATE | | openmetadata.config.authentication.saml.idp.authorityUrl | string | `http://openmetadata:8585/api/v1/saml/login` | SAML\_AUTHORITY\_URL | | openmetadata.config.authentication.saml.idp.nameId | string | `urn:oasis:names:tc:SAML:2.0:nameid-format:emailAddress` | SAML\_IDP\_NAME\_ID | | openmetadata.config.authentication.saml.sp.entityId | string | `http://openmetadata:8585/api/v1/saml/acs` | SAML\_SP\_ENTITY\_ID | | openmetadata.config.authentication.saml.sp.acs | string | `http://openmetadata:8585/api/v1/saml/acs` | SAML\_SP\_ACS | | openmetadata.config.authentication.saml.sp.spX509Certificate.secretRef | string | `Empty` | SAML\_SP\_CERTIFICATE | | openmetadata.config.authentication.saml.sp.spX509Certificate.secretKey | string | `Empty` | SAML\_SP\_CERTIFICATE | | openmetadata.config.authentication.saml.sp.callback | string | `http://openmetadata:8585/saml/callback` | SAML\_SP\_CALLBACK | | openmetadata.config.authentication.saml.security.strictMode | bool | false | SAML\_STRICT\_MODE | | openmetadata.config.authentication.saml.security.tokenValidity | int | 3600 | SAML\_SP\_TOKEN\_VALIDITY | | openmetadata.config.authentication.saml.security.sendEncryptedNameId | bool | false | SAML\_SEND\_ENCRYPTED\_NAME\_ID | | openmetadata.config.authentication.saml.security.sendSignedAuthRequest | bool | false | SAML\_SEND\_SIGNED\_AUTH\_REQUEST | | openmetadata.config.authentication.saml.security.signSpMetadata | bool | false | SAML\_SIGNED\_SP\_METADATA | | openmetadata.config.authentication.saml.security.wantMessagesSigned | bool | false | SAML\_WANT\_MESSAGE\_SIGNED | | openmetadata.config.authentication.saml.security.wantAssertionsSigned | bool | false | SAML\_WANT\_ASSERTION\_SIGNED | | openmetadata.config.authentication.saml.security.wantAssertionEncrypted | bool | false | SAML\_WANT\_ASSERTION\_ENCRYPTED | | openmetadata.config.authentication.saml.security.wantNameIdEncrypted | bool | false | SAML\_WANT\_NAME\_ID\_ENCRYPTED | | openmetadata.config.authentication.saml.security.keyStoreFilePath | string | `Empty` | SAML\_KEYSTORE\_FILE\_PATH | | openmetadata.config.authentication.saml.security.keyStoreAlias.secretRef | string | `Empty` | SAML\_KEYSTORE\_ALIAS | | openmetadata.config.authentication.saml.security.keyStoreAlias.secretKey | string | `Empty` | SAML\_KEYSTORE\_ALIAS | | openmetadata.config.authentication.saml.security.keyStorePassword.secretRef | string | `Empty` | SAML\_KEYSTORE\_PASSWORD | | openmetadata.config.authentication.saml.security.keyStorePassword.secretKey | string | `Empty` | SAML\_KEYSTORE\_PASSWORD | | openmetadata.config.authorizer.enabled | bool | `true` | | | openmetadata.config.authorizer.allowedEmailRegistrationDomains | list | `[all]` | AUTHORIZER\_ALLOWED\_REGISTRATION\_DOMAIN | | openmetadata.config.authorizer.className | string | `org.openmetadata.service.security.DefaultAuthorizer` | AUTHORIZER\_CLASS\_NAME | | openmetadata.config.authorizer.containerRequestFilter | string | `org.openmetadata.service.security.JwtFilter` | AUTHORIZER\_REQUEST\_FILTER | | openmetadata.config.authorizer.enforcePrincipalDomain | bool | `false` | AUTHORIZER\_ENFORCE\_PRINCIPAL\_DOMAIN | | openmetadata.config.authorizer.enableSecureSocketConnection | bool | `false` | AUTHORIZER\_ENABLE\_SECURE\_SOCKET | | openmetadata.config.authorizer.initialAdmins | list | `[admin]` | AUTHORIZER\_ADMIN\_PRINCIPALS | | openmetadata.config.authorizer.principalDomain | string | `open-metadata.org` | AUTHORIZER\_PRINCIPAL\_DOMAIN | | openmetadata.config.airflow\.auth.password.secretRef | string | `airflow-secrets` | AIRFLOW\_PASSWORD | | openmetadata.config.airflow\.auth.password.secretKey | string | `openmetadata-airflow-password` | AIRFLOW\_PASSWORD | | openmetadata.config.airflow\.auth.username | string | `admin` | AIRFLOW\_USERNAME | | openmetadata.config.airflow\.enabled | bool | `true` | | | openmetadata.config.airflow\.host | string | `http://openmetadata-dependencies-web:8080` | PIPELINE\_SERVICE\_CLIENT\_ENDPOINT | | openmetadata.config.airflow\.openmetadata.serverHostApiUrl | string | `http://openmetadata:8585/api` | SERVER\_HOST\_API\_URL | | openmetadata.config.airflow\.sslCertificatePath | string | `/no/path` | PIPELINE\_SERVICE\_CLIENT\_SSL\_CERT\_PATH | | openmetadata.config.airflow\.verifySsl | string | `no-ssl` | PIPELINE\_SERVICE\_CLIENT\_VERIFY\_SSL | | openmetadata.config.clusterName | string | `openmetadata` | OPENMETADATA\_CLUSTER\_NAME | | openmetadata.config.database.enabled | bool | `true` | | | openmetadata.config.database.auth.password.secretRef | string | `mysql-secrets` | DB\_USER\_PASSWORD | | openmetadata.config.database.auth.password.secretKey | string | `openmetadata-mysql-password` | DB\_USER\_PASSWORD | | openmetadata.config.database.auth.username | string | `openmetadata_user` | DB\_USER | | openmetadata.config.database.databaseName | string | `openmetadata_db` | OM\_DATABASE | | openmetadata.config.database.dbParams | string | `allowPublicKeyRetrieval=true&useSSL=false&serverTimezone=UTC` | DB\_PARAMS | | openmetadata.config.database.dbScheme | string | `mysql` | DB\_SCHEME | | openmetadata.config.database.driverClass | string | `com.mysql.cj.jdbc.Driver` | DB\_DRIVER\_CLASS | | openmetadata.config.database.host | string | `mysql` | DB\_HOST | | openmetadata.config.database.port | int | 3306 | DB\_PORT | | openmetadata.config.elasticsearch.enabled | bool | `true` | | | openmetadata.config.elasticsearch.auth.enabled | bool | `false` | | | openmetadata.config.elasticsearch.auth.username | string | `elasticsearch` | ELASTICSEARCH\_USER | | openmetadata.config.elasticsearch.auth.password.secretRef | string | `elasticsearch-secrets` | ELASTICSEARCH\_PASSWORD | | openmetadata.config.elasticsearch.auth.password.secretKey | string | `openmetadata-elasticsearch-password` | ELASTICSEARCH\_PASSWORD | | openmetadata.config.elasticsearch.host | string | `opensearch` | ELASTICSEARCH\_HOST | | openmetadata.config.elasticsearch.keepAliveTimeoutSecs | int | `600` | ELASTICSEARCH\_KEEP\_ALIVE\_TIMEOUT\_SECS | | openmetadata.config.elasticsearch.port | int | 9200 | ELASTICSEARCH\_PORT | | openmetadata.config.elasticsearch.searchType | string | `opensearch` | SEARCH\_TYPE | | openmetadata.config.elasticsearch.scheme | string | `http` | ELASTICSEARCH\_SCHEME | | openmetadata.config.elasticsearch.clusterAlias | string | `Empty String` | ELASTICSEARCH\_CLUSTER\_ALIAS | | openmetadata.config.elasticsearch.searchIndexMappingLanguage | string | `EN` | ELASTICSEARCH\_INDEX\_MAPPING\_LANG | | openmetadata.config.elasticsearch.trustStore.enabled | bool | `false` | | | openmetadata.config.elasticsearch.trustStore.path | string | `Empty String` | ELASTICSEARCH\_TRUST\_STORE\_PATH | | openmetadata.config.elasticsearch.trustStore.password.secretRef | string | `elasticsearch-truststore-secrets` | ELASTICSEARCH\_TRUST\_STORE\_PASSWORD | | openmetadata.config.elasticsearch.trustStore.password.secretKey | string | `openmetadata-elasticsearch-truststore-password` | ELASTICSEARCH\_TRUST\_STORE\_PASSWORD | | openmetadata.config.eventMonitor.enabled | bool | `true` | | | openmetadata.config.eventMonitor.type | string | `prometheus` | EVENT\_MONITOR | | openmetadata.config.eventMonitor.batchSize | int | `10` | EVENT\_MONITOR\_BATCH\_SIZE | | openmetadata.config.eventMonitor.pathPattern | list | `[/api/v1/tables/*,/api/v1/health-check]` | EVENT\_MONITOR\_PATH\_PATTERN | | openmetadata.config.eventMonitor.latency | list | `[]` | EVENT\_MONITOR\_LATENCY | | openmetadata.config.fernetkey.value | string | `jJ/9sz0g0OHxsfxOoSfdFdmk3ysNmPRnH3TUAbz3IHA=` | FERNET\_KEY | | openmetadata.config.fernetkey.secretRef | string | \`\` | FERNET\_KEY | | openmetadata.config.fernetkey.secretKef | string | \`\` | FERNET\_KEY | | openmetadata.config.jwtTokenConfiguration.enabled | bool | `true` | | | openmetadata.config.jwtTokenConfiguration.rsapublicKeyFilePath | string | `./conf/public_key.der` | RSA\_PUBLIC\_KEY\_FILE\_PATH | | openmetadata.config.jwtTokenConfiguration.rsaprivateKeyFilePath | string | `./conf/private_key.der` | RSA\_PRIVATE\_KEY\_FILE\_PATH | | openmetadata.config.jwtTokenConfiguration.jwtissuer | string | `open-metadata.org` | JWT\_ISSUER | | openmetadata.config.jwtTokenConfiguration.keyId | string | `Gb389a-9f76-gdjs-a92j-0242bk94356` | JWT\_KEY\_ID | | openmetadata.config.logLevel | string | `INFO` | LOG\_LEVEL | | openmetadata.config.openmetadata.adminPort | int | 8586 | SERVER\_ADMIN\_PORT | | openmetadata.config.openmetadata.host | string | `openmetadata` | OPENMETADATA\_SERVER\_URL | | openmetadata.config.openmetadata.port | int | 8585 | SERVER\_PORT | | openmetadata.config.pipelineServiceClientConfig.auth.password.secretRef | string | `airflow-secrets` | AIRFLOW\_PASSWORD | | openmetadata.config.pipelineServiceClientConfig.auth.password.secretKey | string | `openmetadata-airflow-password` | AIRFLOW\_PASSWORD | | openmetadata.config.pipelineServiceClientConfig.auth.username | string | `admin` | AIRFLOW\_USERNAME | | openmetadata.config.pipelineServiceClientConfig.auth.trustStorePath | string | \`\` | AIRFLOW\_TRUST\_STORE\_PATH | | openmetadata.config.pipelineServiceClientConfig.auth.trustStorePassword.secretRef | string | \`\` | AIRFLOW\_TRUST\_STORE\_PASSWORD | | openmetadata.config.pipelineServiceClientConfig.auth.trustStorePassword.secretKey | string | \`\` | AIRFLOW\_TRUST\_STORE\_PASSWORD | | openmetadata.config.pipelineServiceClientConfig.apiEndpoint | string | `http://openmetadata-dependencies-web:8080` | PIPELINE\_SERVICE\_CLIENT\_ENDPOINT | | openmetadata.config.pipelineServiceClientConfig.className | string | `org.openmetadata.service.clients.pipeline.airflow.AirflowRESTClient` | PIPELINE\_SERVICE\_CLIENT\_CLASS\_NAME | | openmetadata.config.pipelineServiceClientConfig.enabled | bool | `true` | PIPELINE\_SERVICE\_CLIENT\_ENABLED | | openmetadata.config.pipelineServiceClientConfig.healthCheckInterval | int | `300` | PIPELINE\_SERVICE\_CLIENT\_HEALTH\_CHECK\_INTERVAL | | openmetadata.config.pipelineServiceClientConfig.ingestionIpInfoEnabled | bool | `false` | PIPELINE\_SERVICE\_IP\_INFO\_ENABLED | | openmetadata.config.pipelineServiceClientConfig.metadataApiEndpoint | string | `http://openmetadata:8585/api` | SERVER\_HOST\_API\_URL | | openmetadata.config.pipelineServiceClientConfig.sslCertificatePath | string | `/no/path` | PIPELINE\_SERVICE\_CLIENT\_SSL\_CERT\_PATH | | openmetadata.config.pipelineServiceClientConfig.verifySsl | string | `no-ssl` | PIPELINE\_SERVICE\_CLIENT\_VERIFY\_SSL | | openmetadata.config.pipelineServiceClientConfig.hostIp | string | `Empty` | PIPELINE\_SERVICE\_CLIENT\_HOST\_IP | | openmetadata.config.pipelineServiceClientConfig.type | string | `airflow` | Orchestrator type: `airflow` or `k8s` | ### Kubernetes Native Orchestrator Values The following values are used when `pipelineServiceClientConfig.type` is set to `k8s`. See the [Kubernetes Orchestrator](/v2.0.x/deployment/ingestion/kubernetes) guide for full documentation. | Key | Type | Default | Description | | -------------------------------------------------------------------------------- | ------ | ----------------------------------------------------------------- | ------------------------------------------- | | openmetadata.config.pipelineServiceClientConfig.k8s.className | string | `org.openmetadata.service.clients.pipeline.k8s.K8sPipelineClient` | K8s client class | | openmetadata.config.pipelineServiceClientConfig.k8s.ingestionImage | string | `docker.getcollate.io/openmetadata/ingestion-base:latest` | Ingestion container image | | openmetadata.config.pipelineServiceClientConfig.k8s.imagePullPolicy | string | `IfNotPresent` | Image pull policy | | openmetadata.config.pipelineServiceClientConfig.k8s.imagePullSecrets | string | \`\` | Comma-separated image pull secrets | | openmetadata.config.pipelineServiceClientConfig.k8s.serviceAccountName | string | `openmetadata-ingestion` | Service account for ingestion jobs | | openmetadata.config.pipelineServiceClientConfig.k8s.ttlSecondsAfterFinished | int | `86400` | Time to keep completed jobs (seconds) | | openmetadata.config.pipelineServiceClientConfig.k8s.activeDeadlineSeconds | int | `7200` | Maximum job runtime (seconds) | | openmetadata.config.pipelineServiceClientConfig.k8s.backoffLimit | int | `3` | Maximum retry attempts | | openmetadata.config.pipelineServiceClientConfig.k8s.successfulJobsHistoryLimit | int | `3` | Successful jobs to retain | | openmetadata.config.pipelineServiceClientConfig.k8s.failedJobsHistoryLimit | int | `3` | Failed jobs to retain | | openmetadata.config.pipelineServiceClientConfig.k8s.nodeSelector | string | \`\` | Node selector (comma-separated key=value) | | openmetadata.config.pipelineServiceClientConfig.k8s.securityContext.runAsUser | int | `1000` | Run as user ID | | openmetadata.config.pipelineServiceClientConfig.k8s.securityContext.runAsGroup | int | `1000` | Run as group ID | | openmetadata.config.pipelineServiceClientConfig.k8s.securityContext.fsGroup | int | `1000` | Filesystem group ID | | openmetadata.config.pipelineServiceClientConfig.k8s.securityContext.runAsNonRoot | bool | `true` | Require non-root | | openmetadata.config.pipelineServiceClientConfig.k8s.resources.limits.cpu | string | `2` | CPU limit | | openmetadata.config.pipelineServiceClientConfig.k8s.resources.limits.memory | string | `4Gi` | Memory limit | | openmetadata.config.pipelineServiceClientConfig.k8s.resources.requests.cpu | string | `500m` | CPU request | | openmetadata.config.pipelineServiceClientConfig.k8s.resources.requests.memory | string | `1Gi` | Memory request | | openmetadata.config.pipelineServiceClientConfig.k8s.podAnnotations | string | \`\` | Pod annotations (comma-separated key=value) | | openmetadata.config.pipelineServiceClientConfig.k8s.extraEnvVars | list | `[]` | Extra environment variables | | openmetadata.config.pipelineServiceClientConfig.k8s.enableFailureDiagnostics | bool | `true` | Enable failure diagnostics | | openmetadata.config.pipelineServiceClientConfig.k8s.useOMJobOperator | bool | `false` | Use OMJob operator for exit handlers | | openmetadata.config.pipelineServiceClientConfig.k8s.rbac.enabled | bool | `true` | Create RBAC resources | ### OMJob Operator Values The OMJob Operator provides guaranteed exit handler execution. Required when `k8s.useOMJobOperator: true`. | Key | Type | Default | Description | | ---------------------------------------- | ------ | -------------------------------------------------- | ----------------------------------------------- | | omjobOperator.enabled | bool | `false` | Install OMJob CRD and operator | | omjobOperator.image.repository | string | `docker.getcollate.io/openmetadata/omjob-operator` | Operator image | | omjobOperator.image.tag | string | `""` (defaults to the chart `appVersion`) | Operator image tag | | omjobOperator.image.pullPolicy | string | `IfNotPresent` | Image pull policy | | omjobOperator.resources.requests.cpu | string | `100m` | CPU request | | omjobOperator.resources.requests.memory | string | `128Mi` | Memory request | | omjobOperator.resources.limits.cpu | string | `500m` | CPU limit | | omjobOperator.resources.limits.memory | string | `256Mi` | Memory limit | | omjobOperator.env.logLevel | string | `INFO` | Log level | | omjobOperator.env.reconciliationThreads | string | `5` | Reconciliation threads | | omjobOperator.env.healthCheckPort | string | `8080` | Health check port | | omjobOperator.env.metricsPort | string | `8081` | Metrics port | | omjobOperator.env.pollingIntervalSeconds | string | `10` | Pod status polling interval | | omjobOperator.env.requeueDelaySeconds | string | `30` | Requeue delay after errors | | omjobOperator.env.watchNamespaces | string | \`\` | Namespaces to watch (comma-separated, or "ALL") | \| openmetadata.config.secretsManager.enabled | bool | `true` | | \| openmetadata.config.secretsManager.provider | string | `Empty String` | SECRET\_MANAGER | \| openmetadata.config.secretsManager.prefix | string | `Empty String` | SECRET\_MANAGER\_PREFIX | \| openmetadata.config.secretsManager.tags | list | `[]` | SECRET\_MANAGER\_TAGS | \| openmetadata.config.secretsManager.additionalParameters.enabled | bool | `false` | | \| openmetadata.config.secretsManager.additionalParameters.accessKeyId.secretRef | string | `aws-access-key-secret` | OM\_SM\_ACCESS\_KEY\_ID | \| openmetadata.config.secretsManager.additionalParameters.accessKeyId.secretKey | string | `aws-key-secret` | OM\_SM\_ACCESS\_KEY\_ID | \| openmetadata.config.secretsManager.additionalParameters.clientId.secretRef | string | `azure-client-id-secret` | OM\_SM\_CLIENT\_ID | \| openmetadata.config.secretsManager.additionalParameters.clientId.secretKey | string | `azure-key-secret` | OM\_SM\_CLIENT\_ID | \| openmetadata.config.secretsManager.additionalParameters.clientSecret.secretRef | string | `azure-client-secret` | OM\_SM\_CLIENT\_SECRET | \| openmetadata.config.secretsManager.additionalParameters.clientSecret.secretKey | string | `azure-key-secret` | OM\_SM\_CLIENT\_SECRET | \| openmetadata.config.secretsManager.additionalParameters.tenantId.secretRef | string | `azure-tenant-id-secret` | OM\_SM\_TENANT\_ID | \| openmetadata.config.secretsManager.additionalParameters.tenantId.secretKey | string | `azure-key-secret` | OM\_SM\_TENANT\_ID | \| openmetadata.config.secretsManager.additionalParameters.vaultName.secretRef | string | `azure-vault-name-secret` | OM\_SM\_VAULT\_NAME | \| openmetadata.config.secretsManager.additionalParameters.vaultName.secretKey | string | `azure-key-secret` | OM\_SM\_VAULT\_NAME | \| openmetadata.config.secretsManager.additionalParameters.region | string | `Empty String` | OM\_SM\_REGION | \| openmetadata.config.secretsManager.additionalParameters.secretAccessKey.secretRef | string | `aws-secret-access-key-secret` | OM\_SM\_ACCESS\_KEY | \| openmetadata.config.secretsManager.additionalParameters.secretAccessKey.secretKey | string | `aws-key-secret` | OM\_SM\_ACCESS\_KEY | \| openmetadata.config.smtpConfig.enableSmtpServer | bool | `false` | AUTHORIZER\_ENABLE\_SMTP | \| openmetadata.config.smtpConfig.emailingEntity | string | `OpenMetadata` | OM\_EMAIL\_ENTITY | \| openmetadata.config.smtpConfig.openMetadataUrl | string | `Empty String` | OPENMETADATA\_SERVER\_URL | \| openmetadata.config.smtpConfig.password.secretKey | string | `Empty String` | SMTP\_SERVER\_PWD | \| openmetadata.config.smtpConfig.password.secretRef | string | `Empty String` | SMTP\_SERVER\_PWD | \| openmetadata.config.smtpConfig.serverEndpoint | string | `Empty String` | SMTP\_SERVER\_ENDPOINT | \| openmetadata.config.smtpConfig.serverPort | string | `Empty String` | SMTP\_SERVER\_PORT | \| openmetadata.config.smtpConfig.supportUrl | string | `https://slack.open-metadata.org` | OM\_SUPPORT\_URL | \| openmetadata.config.smtpConfig.transportationStrategy | string | `SMTP_TLS` | SMTP\_SERVER\_STRATEGY | \| openmetadata.config.smtpConfig.username | string | `Empty String` | SMTP\_SERVER\_USERNAME | \| openmetadata.config.upgradeMigrationConfigs.debug | bool | `false` | | \| openmetadata.config.upgradeMigrationConfigs.additionalArgs | string | `Empty String` | | \| openmetadata.config.web.enabled | bool | `true` | | \| openmetadata.config.web.contentTypeOptions.enabled | bool | `false` | WEB\_CONF\_CONTENT\_TYPE\_OPTIONS\_ENABLED | \| openmetadata.config.web.csp.enabled | bool | `false` | WEB\_CONF\_XSS\_CSP\_ENABLED | \| openmetadata.config.web.csp.policy | string | `default-src 'self` | WEB\_CONF\_XSS\_CSP\_POLICY | \| openmetadata.config.web.csp.reportOnlyPolicy | string | `Empty String` | WEB\_CONF\_XSS\_CSP\_REPORT\_ONLY\_POLICY | \| openmetadata.config.web.frameOptions.enabled | bool | `false` | WEB\_CONF\_FRAME\_OPTION\_ENABLED | \| openmetadata.config.web.frameOptions.option | string | `SAMEORIGIN` | WEB\_CONF\_FRAME\_OPTION | \| openmetadata.config.web.frameOptions.origin | string | `Empty String` | WEB\_CONF\_FRAME\_ORIGIN | \| openmetadata.config.web.hsts.enabled | bool | `false` | WEB\_CONF\_HSTS\_ENABLED | \| openmetadata.config.web.hsts.includeSubDomains | bool | `true` | WEB\_CONF\_HSTS\_INCLUDE\_SUBDOMAINS | \| openmetadata.config.web.hsts.maxAge | string | `365 days` | WEB\_CONF\_HSTS\_MAX\_AGE | \| openmetadata.config.web.hsts.preload | bool | `true` | WEB\_CONF\_HSTS\_PRELOAD | \| openmetadata.config.web.uriPath | string | `/api` | WEB\_CONF\_URI\_PATH | \| openmetadata.config.web.xssProtection.block | bool | `true` | WEB\_CONF\_XSS\_PROTECTION\_BLOCK | \| openmetadata.config.web.xssProtection.enabled | bool | `false` | WEB\_CONF\_XSS\_PROTECTION\_ENABLED | \| openmetadata.config.web.xssProtection.onXss | bool | `true` | WEB\_CONF\_XSS\_PROTECTION\_ON | \| openmetadata.config.web.referrer-policy.enabled | bool | `false` | WEB\_CONF\_REFERRER\_POLICY\_ENABLED | \| openmetadata.config.web.referrer-policy.option | string | `SAME_ORIGIN'` | WEB\_CONF\_REFERRER\_POLICY\_OPTION | \| openmetadata.config.web.permission-policy.enabled | bool | `false` | WEB\_CONF\_PERMISSION\_POLICY\_ENABLED | \| openmetadata.config.web.permission-policy.option | string | `Empty String` | WEB\_CONF\_PERMISSION\_POLICY\_OPTION | ## Chart Values | Key | Type | Default | | ----------------------------------------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------ | | affinity | object | `{}` | | commonLabels | object | `{}` | | extraEnvs | Extra \[environment variables]\[] which will be appended to the `env:` definition for the container | `[]` | | extraInitContainers | Templatable string of additional `initContainers` to be passed to `tpl` function | `[]` | | extraVolumes | Templatable string of additional `volumes` to be passed to the `tpl` function | `[]` | | extraVolumeMounts | Templatable string of additional `volumeMounts` to be passed to the `tpl` function | `[]` | | fullnameOverride | string | `"openmetadata"` | | image.pullPolicy | string | `"Always"` | | image.repository | string | `"docker.open-metadata.org/openmetadata/server"` | | image.tag | string | `1.3.4` | | imagePullSecrets | list | `[]` | | ingress.annotations | object | `{}` | | ingress.className | string | `""` | | ingress.enabled | bool | `false` | | ingress.hosts\[0].host | string | `"open-metadata.local"` | | ingress.hosts\[0].paths\[0].path | string | `"/"` | | ingress.hosts\[0].paths\[0].pathType | string | `"ImplementationSpecific"` | | ingress.tls | list | `[]` | | livenessProbe.initialDelaySeconds | int | `60` | | livenessProbe.periodSeconds | int | `30` | | livenessProbe.failureThreshold | int | `5` | | livenessProbe.httpGet.path | string | `/healthcheck` | | livenessProbe.httpGet.port | string | `http-admin` | | nameOverride | string | `""` | | nodeSelector | object | `{}` | | podAnnotations | object | `{}` | | podSecurityContext | object | `{}` | | readinessProbe.initialDelaySeconds | int | `60` | | readinessProbe.periodSeconds | int | `30` | | readinessProbe.failureThreshold | int | `5` | | readinessProbe.httpGet.path | string | `/` | | readinessProbe.httpGet.port | string | `http` | | replicaCount | int | `1` | | resources | object | `{}` | | securityContext | object | `{}` | | service.adminPort | string | `8586` | | service.annotations | object | `{}` | | service.port | int | `8585` | | service.type | string | `"ClusterIP"` | | serviceAccount.annotations | object | `{}` | | serviceAccount.create | bool | `true` | | serviceAccount.name | string | `nil` | | automountServiceAccountToken | bool | `true` | | serviceMonitor.annotations | object | `{}` | | serviceMonitor.enabled | bool | `false` | | serviceMonitor.interval | string | `30s` | | serviceMonitor.labels | object | `{}` | | sidecars | list | `[]` | | startupProbe.periodSeconds | int | `60` | | startupProbe.failureThreshold | int | `5` | | startupProbe.httpGet.path | string | `/healthcheck` | | startupProbe.httpGet.port | string | `http-admin` | | startupProbe.successThreshold | int | `1` | | tolerations | list | `[]` | | networkPolicy.enabled | bool | `false` | | podDisruptionBudget.enabled | bool | `false` | | podDisruptionBudget.config.maxUnavailable | String | `1` | | podDisruptionBudget.config.minAvailable | String | `1` | # Minimum Requirements | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/minimum-requirements Review system requirements including OS, CPU, memory, and dependencies to ensure a successful and stable deployment. # Minimum Hardware Requirements OpenMetadata requires either MySQL or PostgreSQL as Backend Database and ElasticSearch or OpenSearch as Search Engine. Please refer to the [architecture section](/v2.0.x/developers/architecture) for more details. We recommend the following for the Production Grade OpenMetadata Installation - ## MySQL or PostgreSQL as Database Our minimum specs recommendation for MySQL / PostgreSQL as database deployment is * 4 vCPUs * 16 GiB Memory * Storage: * **Minimum**: 100 GB * **Recommended**: 150–200 GB (depending on workload) For enterprise use cases with heavy workloads, we recommend 150–200 GB of storage to accommodate larger datasets and transaction volumes. These settings apply as well when using managed instances, such as AWS RDS or GCP CloudSQL or Azure Flexible Servers. ### Software Requirements OpenMetadata currently supports - * MySQL version 8.0.42 or higher * PostgreSQL version 15 or higher ## ElasticSearch or OpenSearch as Search Instance Our minimum specs recommendation for ElasticSearch / OpenSearch deployment is * 2 vCPUs * 8 GiB Memory * 100 GiB Storage (per node) * Master / Worker Nodes with atleast 1 Master and 2 Worker Nodes ### Software Requirements OpenMetadata currently supports - * ElasticSearch version 9.x (minimum 9.0.0, recommended 9.3.0) * OpenSearch version 3.x (minimum 3.0.0, recommended 3.3.0) These settings apply as well when using managed instances, such as AWS OpenSearch Service or Elastic Cloud on GCP, AWS, Azure. ## Apache Airflow as Ingestion Instance Our minimum specs recommendation for Apache Airflow is * 4 vCPU * 16 GiB Memory * 100 GiB Storage for Airflow Dags and Logs ### Software Requirements OpenMetadata currently supports - * Airflow version 2.10.5 Learn more about how to deploy and manage the ingestion workflows [here](/v2.0.x/deployment/ingestion). # Production-Ready Requirements for OpenMetadata Deployment Source: https://docs.open-metadata.org/v2.0.x/deployment/production-ready-requirements # Production-Ready Requirements for OpenMetadata Deployment This section outlines the minimum hardware and resource specifications required for deploying OpenMetadata and its dependencies. These recommendations ensure optimal performance and scalability for your deployment. ## OpenMetadata Server * **vCPUs**: Minimum of 4 vCPUs * **Memory**: 16 GiB * **Storage Volume**: 100 GiB ## External Services OpenMetadata depends on the following external services, each with specific resource requirements: ### Version Compatibility Matrix Before deploying OpenMetadata in a production environment, ensure that all required external services meet the minimum supported versions listed below. Running unsupported versions may lead to ingestion failures, search issues, or upgrade incompatibilities. ## **Note:** OpenMetadata relies on these external services for metadata storage and search indexing. Using versions lower than the minimum supported versions may result in unexpected behavior or deployment failures. ### Database (e.g., PostgreSQL, MySQL) * **vCPUs**: Minimum of 4 vCPUs per instance * **Memory**: 16 GiB RAM per instance * **Buffer Size Setting**: 20 MB (sort\_buffer\_size) * **Storage Volume**: * 100 GiB (minimum) * Dynamic expansion up to 200 GiB ### Elasticsearch * **vCPUs**: Minimum of 2 vCPUs per instance * **Memory**: 8 GiB RAM per instance * **Storage Volume**: 100 GiB These specifications are also applicable for managed services like **AWS RDS**, **GCP CloudSQL**, or **AWS OpenSearch**. ## Summary Recommendations For a typical OpenMetadata deployment (one replica): * **OpenMetadata Server**: 4 vCPUs, 16 GiB RAM, 100 GiB persistent storage * **Database**: 4 vCPUs, 16 GiB RAM, 30 GiB storage (expandable to 100 GiB) * **Elasticsearch**: 2 vCPUs, 8 GiB RAM, 100 GiB storage Ensure these resources are allocated adequately to prevent performance bottlenecks or scalability issues. Managed services with equivalent specifications are supported. # Enable RDF (Knowledge Graph) (Beta) | OpenMetadata Deployment Guide Source: https://docs.open-metadata.org/v2.0.x/deployment/rdf-knowledge-graph Enable the RDF Knowledge Graph in OpenMetadata by deploying Apache Jena Fuseki and pointing the server at the triplestore. Beta ## Overview RDF Knowledge Graph represents your metadata as RDF triples in a Fuseki triplestore, alongside the relational data in your primary database. This unlocks: * **SPARQL querying**: A query language for graphs, similar in spirit to SQL - lets you ask questions that span many related entities at once, which are hard to write as SQL joins. * **Ontology-based reasoning**: OpenMetadata loads a formal description of its own data model (an ontology) and a set of validation rules (SHACL shapes) into the graph, so relationships can be checked and inferred automatically. * A **standardized, linked-data representation** (JSON-LD) of every entity, which makes it easier to connect your catalog with external knowledge graph and governance tools. RDF Knowledge Graph is an optional feature, **disabled by default**. Turning it on for a catalog with a large number of existing assets adds meaningful load while the initial index runs. See [Performance Considerations](#performance-considerations) below. Test in a non-production environment first, and schedule the initial full index for a low-traffic window. ## Prerequisites Before enabling RDF, make sure the following are in place. * An **Apache Jena Fuseki** triplestore reachable from the OpenMetadata server. You can run the one shipped in the OpenMetadata repository (`docker/rdf-store`) or point at your own Fuseki deployment. * Persistent storage for Fuseki sized for your catalog. The RDF dataset grows independently of your primary database and search index, so plan capacity the same way you would for Elasticsearch/OpenSearch storage. * Network access from the OpenMetadata server (and from any container that runs schema migrations) to the Fuseki endpoint. * **OpenSearch** (not Elasticsearch) if you also want to use the semantic-search-over-graph endpoint (`/api/v1/rdf/search/semantic`), which delegates to OpenSearch's vector search. Core RDF indexing, SPARQL, and graph exploration do not require OpenSearch. ## How It Works RDF support layers on top of your existing OpenMetadata deployment in three steps. 1. **Storage Backend**: Apache Jena Fuseki stores your metadata as RDF triples (small subject-predicate-object statements -- the basic building blocks of a graph) in a dataset named `openmetadata` by default. OpenMetadata talks to Fuseki using SPARQL, the standard protocol for querying and updating graph data over HTTP. 2. **Ontology & Shapes**: On startup, OpenMetadata loads two reference documents into Fuseki: an ontology that defines what kinds of things exist in the graph and how they relate, and a set of shapes that define what valid data looks like. Together, these keep the graph consistent as new data flows in. 3. **Continuous Sync**: As entities are created, updated, or deleted, an RDF updater keeps the triplestore in sync - the same way OpenMetadata already keeps its search index in sync. ## Configuration RDF is configured in `openmetadata.yaml` under the `rdf` section. All settings can be overridden with environment variables. | Environment Variable | Default | Description | | --------------------- | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `RDF_ENABLED` | `false` | Master switch. Every RDF code path is a no-op until this is `true`. | | `RDF_STORAGE_TYPE` | `FUSEKI` | Triplestore backend. `FUSEKI` is the only implemented option today (see warning below). | | `RDF_ENDPOINT` | `http://localhost:3030/openmetadata` | The full URL OpenMetadata uses to reach Fuseki, ending in the dataset name -- for example, `.../openmetadata`. Keep this in sync with `RDF_DATASET` below; OpenMetadata reads the dataset name back out of this URL. | | `RDF_REMOTE_USERNAME` | `admin` | Fuseki basic-auth username. | | `RDF_REMOTE_PASSWORD` | `admin` | Fuseki basic-auth password. | | `RDF_BASE_URI` | `https://open-metadata.org/` | Base URI used when minting RDF resource URIs. | | `RDF_DATASET` | `openmetadata` | Fuseki dataset name. Must match the path segment in `RDF_ENDPOINT`. | ```yaml theme={null} rdf: enabled: ${RDF_ENABLED:-false} baseUri: ${RDF_BASE_URI:-"https://open-metadata.org/"} storageType: ${RDF_STORAGE_TYPE:-"FUSEKI"} remoteEndpoint: ${RDF_ENDPOINT:-"http://localhost:3030/openmetadata"} username: ${RDF_REMOTE_USERNAME:-"admin"} password: ${RDF_REMOTE_PASSWORD:-"admin"} dataset: ${RDF_DATASET:-"openmetadata"} ``` **Important**: `storageType` currently only supports `FUSEKI`. `QLEVER` is defined in the configuration schema for future use but is not implemented -- setting it will fail at startup. **Note**: Some RDF settings that appear in `docker-compose` override files you may find in the OpenMetadata repository -- for example `RDF_JSONLD_ENABLED`, `RDF_SPARQL_ENABLED`, `RDF_AUTO_GENERATE`, `RDF_SYNC_BATCH_SIZE` -- are not wired to any configuration key and have no effect. Only the environment variables in the table above are read by the server. If you're copying an example compose file, verify it sets `RDF_ENDPOINT` (not `RDF_REMOTE_ENDPOINT`). Set these environment variables on **both** the OpenMetadata server and the migrations container -- not just one. The migrations container is what actually loads the RDF ontology/shapes and registers the `RdfIndexApp` application, so if only one of the two containers has RDF configured, installation fails with an error like `RdfRepository not initialized`. ## Docker Deployment Docker is the easiest way to try RDF locally or in a small non-production environment, since Fuseki and its configuration are already wired into the compose files shipped with OpenMetadata. There are two ways to run RDF with Docker - pick whichever matches your situation, you don't need both. Runs the full stack (MySQL, Elasticsearch, Fuseki, migrations, server, ingestion) with RDF enabled from the start. This is **not** the default quickstart -- RDF support is opt-in via this separate compose file. ```bash theme={null} git clone https://github.com/open-metadata/OpenMetadata.git cd OpenMetadata/docker/docker-compose-quickstart docker compose -f docker-compose-rdf.yml up -d ``` Layers Fuseki and the required environment variables onto an already-running MySQL or Postgres development stack. ```bash theme={null} # MySQL-backed stack docker compose -f docker-compose.yml -f docker-compose-fuseki.yml up -d # Postgres-backed stack docker compose -f docker-compose-postgres.yml -f docker-compose-postgres-fuseki.yml up -d ``` To configure RDF manually in a `docker-compose` override or `.env` file: ```yaml theme={null} environment: RDF_ENABLED: "true" RDF_STORAGE_TYPE: "FUSEKI" RDF_ENDPOINT: "http://fuseki:3030/openmetadata" RDF_REMOTE_USERNAME: "admin" RDF_REMOTE_PASSWORD: "admin" RDF_DATASET: "openmetadata" ``` ## Kubernetes Deployment The official OpenMetadata Helm charts can deploy Fuseki for you as an optional add-on. You then tell OpenMetadata where to find it using a dedicated `rdf` section in the chart's values file, rather than the generic `extraEnvs` list used for other settings. Fuseki ships as an optional dependency in the `openmetadata-dependencies` Helm chart. Deploy it as its own release, and disable the chart's other dependencies (MySQL, OpenSearch, Airflow) if you're already running them through another release -- otherwise you'll end up with duplicates. ### Step 1: Create the Fuseki Admin Credentials Secret OpenMetadata uses this same password when it connects to Fuseki, so store it in a secret the Fuseki chart reads by default: ```bash theme={null} kubectl create secret generic fuseki-admin-credentials \ --from-literal=admin-password= \ --namespace ``` ### Step 2: Add the OpenMetadata Helm Repository Skip this if you've already added it while installing OpenMetadata itself: ```bash theme={null} helm repo add open-metadata https://helm.open-metadata.org/ helm repo update ``` ### Step 3: Deploy Fuseki Prepare a values file that enables only Fuseki. The `resources`, `jvmArgs`, and `persistence` values below are the chart's production defaults -- scale them to your catalog size: ```yaml theme={null} # fuseki-values.yaml mysql: enabled: false opensearch: enabled: false airflow: enabled: false fuseki: enabled: true image: repository: daschswiss/apache-jena-fuseki tag: "6.0.0" resources: requests: cpu: "1500m" memory: "10Gi" limits: cpu: "2500m" memory: "12Gi" jvmArgs: "-Xmx8g -Xms8g" persistence: enabled: true size: 100Gi storageClass: "" # empty string uses the cluster default StorageClass adminPassword: secretName: "fuseki-admin-credentials" secretKey: "admin-password" ``` Deploy it as a dedicated release: ```bash theme={null} helm upgrade --install fuseki open-metadata/openmetadata-dependencies \ --values fuseki-values.yaml \ --namespace ``` This creates a Fuseki `Deployment`, a `ClusterIP` `Service` named `fuseki` on port `3030`, and -- since `persistence.enabled` is `true` -- a PVC. OpenMetadata will reach it in-cluster at `http://fuseki:3030`. Confirm it came up cleanly before moving on: ```bash theme={null} kubectl get pods -n -l app=fuseki kubectl logs -n deploy/fuseki ``` You don't need to create the Fuseki dataset yourself: on startup, the OpenMetadata server checks for the configured dataset and creates it through the Fuseki admin API if it doesn't already exist. ### Step 4: Point OpenMetadata at Fuseki Add an `rdf` block under `openmetadata.config` in your OpenMetadata chart's values file. Each key here maps to one of the environment variables from the [Configuration](#configuration) - `enabled` to `RDF_ENABLED`, `storageType` to `RDF_STORAGE_TYPE`, `remoteEndpoint` to `RDF_ENDPOINT`, `username`/`password` to `RDF_REMOTE_USERNAME`/`RDF_REMOTE_PASSWORD`, `dataset` to `RDF_DATASET`, and `baseUri` to `RDF_BASE_URI`: ```yaml theme={null} # openmetadata-values.yaml openmetadata: config: rdf: enabled: true storageType: "FUSEKI" remoteEndpoint: "http://fuseki:3030/openmetadata" username: "admin" password: secretRef: fuseki-admin-credentials secretKey: admin-password dataset: "openmetadata" baseUri: "https://open-metadata.org/" ``` Apply the change with a Helm upgrade against your existing release: ```bash theme={null} helm upgrade --install openmetadata open-metadata/openmetadata \ --values openmetadata-values.yaml \ --namespace ``` Once the server pod restarts, confirm it picked up the RDF configuration and connected to Fuseki without errors: ```bash theme={null} kubectl logs -n deploy/openmetadata | grep -i rdf ``` From here, move on to [Running the Initial Index](#running-the-initial-index) below to populate the graph. ## Running the Initial Index Once RDF is enabled and the server has started successfully, run the **RDF Knowledge Graph Indexing** application to index existing entities into the triplestore: * **From the OpenMetadata UI**: 1. Navigate to **Settings** > **Applications**, and click the **RDF Knowledge Graph Indexing** application. 2. Click **Run Now**. * **Via the API**: ```bash theme={null} curl -X POST "$OM_HOST/api/v1/apps/trigger/RdfIndexApp" -H "Authorization: Bearer $TOKEN" ``` **Check Status**: Run the following command: ```bash theme={null} curl "$OM_HOST/api/v1/apps/name/RdfIndexApp/status" -H "Authorization: Bearer $TOKEN" ``` If a large indexing run needs to be stopped (for example, before a maintenance window ends), use: ```bash theme={null} curl -X POST "$OM_HOST/api/v1/apps/stop/RdfIndexApp" -H "Authorization: Bearer $TOKEN" ``` RDF indexing runs distributed across available server instances by default. Only one reindex job can be active per cluster at a time. ## Performance Considerations Indexing a large catalog (hundreds of thousands of assets) into RDF for the first time adds meaningful load to both the OpenMetadata server and Fuseki, and can noticeably slow down concurrent operations -- entity updates such as adding a tag have been observed taking 30-40 seconds while a full reindex is in progress. * Schedule the initial full index, and any subsequent full reindex, during a low-traffic window (for example, overnight or over a weekend). * Give Fuseki adequate JVM heap for your catalog size (`-Xmx`/`-Xms`, 8G is a reasonable starting point for mid-size catalogs) and persistent storage sized for growth -- the RDF dataset is a separate volume from your primary database. * Incremental updates (create/update/delete of individual entities) are lightweight and do not require the same scheduling care as a full reindex. ## Troubleshooting Common failure modes when enabling RDF or running a reindex, and how to recover from them. ### `RdfRepository not initialized` / RdfIndexApp fails to install at startup This means RDF was turned on (`RDF_ENABLED=true`) but the server couldn't finish setting up its connection to Fuseki \-- usually because the server and migrations container ended up with different RDF settings, or because Fuseki wasn't reachable yet when the server started. * Confirm the same RDF environment variables are set on both the migrations container and the main server container. On Kubernetes, older Helm chart versions had a bug where the `rdf` block was only applied to the migrations init container and not the main server container -- make sure you're on a chart version that applies it to both. * Confirm `RDF_ENDPOINT` is reachable from the server pod/container (not just from your local machine). * Check the server startup logs for the underlying `JenaFusekiStorage` / `RdfRepository` log lines -- they indicate which step failed (dataset creation, ontology load, connection). ### Another RDF reindex job is already active Distributed indexing blocks a new run while an existing job is in `READY`, `RUNNING`, or `STOPPING` state. If a previous job is stuck (for example, after a server restart mid-index), restarting the server pods alone does not clear this state. * Stop the stuck job explicitly: `POST /api/v1/apps/stop/RdfIndexApp`. * If that doesn't clear it, wait for the distributed lock to go stale (a few minutes) before retrying. ### Reindex fails with "Failed to clear RDF data" or a SPARQL update times out This typically indicates the Fuseki dataset was left in an inconsistent state by a restart that happened mid-write (for example, the OpenMetadata server or Fuseki pod restarting while a clear/reindex was in progress). Recovery steps: 1. **Scale Fuseki down.** Scale the Fuseki deployment to 0 replicas. 2. **Delete only the RDF dataset directory.** On Fuseki's persistent volume, delete just the `openmetadata` dataset directory (leave any other datasets on the same Fuseki instance untouched). 3. **Scale Fuseki back up.** Scale Fuseki back to 1 replica and confirm it starts cleanly. 4. **Restart the OpenMetadata server.** Restart the OpenMetadata server pod(s)/container(s) so the ontology and shapes graphs get reloaded into the fresh dataset. 5. **Retrigger the reindex.** Run `RdfIndexApp` again. This deletes all RDF data. Since the RDF graph is a derived index of your existing metadata (not a source of truth), this is safe -- but you will need to run a full reindex afterward. ### RDF indexing is slowing down the rest of the platform See [Performance Considerations](#performance-considerations) above -- run full reindexes during low-traffic windows, and confirm Fuseki has adequate CPU/memory/storage for your catalog size. ### Fuseki pod is OOMKilled during indexing A full reindex of a large catalog can push memory usage past Fuseki's configured limits. * Increase `fuseki.resources.requests`/`limits` and `fuseki.jvmArgs` (`-Xmx`/`-Xms`) proportionally to your catalog size, keeping the JVM heap comfortably below the container memory limit. * Re-run the reindex after resizing. ## API Reference RDF exposes a set of REST endpoints under `/api/v1/rdf`, including: | Endpoint | Description | | ---------------------------------------------------- | ----------------------------------------------- | | `GET /api/v1/rdf/status` | RDF enabled/inference status. | | `GET /api/v1/rdf/entity/{entityType}/{id}` | RDF representation of a single entity. | | `GET /api/v1/rdf/sparql` / `POST /api/v1/rdf/sparql` | Run a SPARQL query against the metadata graph. | | `POST /api/v1/rdf/sparql/update` | Run a SPARQL update against the metadata graph. | | `GET /api/v1/rdf/graph/explore` | Explore the graph around an entity. | | `GET /api/v1/rdf/search/semantic` | Semantic search over the graph. | # Enable Secrets Manager | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/secrets-manager Learn how to manage secrets and credentials used by connectors and services through centralized secrets manager setup. # Enable Secrets Manager Secret Manager integrations allow you to use your existing third-party **Key Management Store** (KMS) with OpenMetadata. Your credentials and sensitive information are stored in a tool that you control, and the KMS will mediate between any OpenMetadata internal requirement and sensitive information. Without a secret manager configured in OpenMetadata, all your sensitive data, any password field of a service connection parameters, bot credentials configuration or dbt configuration of an ingestion pipeline, were stored in MySQL (or Postgres) encrypted. The following diagram shows how is the process between the OM server and Airflow workflows: om-secrets-manager-disabled As you can see, the `Workflow` consumed by Airflow contains the service information as an `EntityReference`. We use that reference to read the Service information, including its connection details. This information goes from `Database > OM > Airflow`. When the Secrets Manager is enabled, sensitive information stop being stored in any system from OpenMetadata. Instead, the KMS will act as a mediator, as we can observe in the diagram below: om-secrets-manager-enabled In 0.13 and up, OpenMetadata will communicate through an interface to read/write sensitive information -- removing the need to store sensitive data in OM systems. This new interface works whether users keep using the underlying database of OpenMetadata to store credentials (as it was set up thus far) or any external system such as AWS Secrets Manager or AWS SSM Parameter Store. In future releases, we will add support for additional Key Management Stores, such as Azure Key Vault or Kubernetes Secrets. If you’d like to contribute by creating the interface, check the implementation guide, or if you want to see a new one on the supported list, please reach out to us on [Slack](https://slack.open-metadata.org/). If you are interested in enabling the secrets' manager feature, this is our list of supported Secrets Manager implementations: * [AWS Secrets Manager](/v2.0.x/deployment/secrets-manager/supported-implementations/aws-secrets-manager) * [AWS Systems Manager Parameter Store](/v2.0.x/deployment/secrets-manager/supported-implementations/aws-ssm-parameter-store) Things to take into account when enabling the Secrets Manager feature: 1. The migration of all the sensitive data will be done automatically after restarting the OpenMetadata server, which can not be undone for the time being. 2. Only users with permissions can edit and retrieve the service connections. The connection parameters will be hidden for all other users. ## How it works There are two types of secrets manager implementations. ### Managed secrets manager All the sensitive data will be held automatically in the configured secrets manager, i.e., any password field stored in the connection parameters of a service, in a bot credentials configuration, or a dbt configuration of an ingestion pipeline. For example, suppose we create a MySQL service with the name `mysql-test`. In that case, the connection password will be stored in the secrets manager using the secret id `/openmetadata/database/mysql-test/password`. When we retrieve the connection parameters from the service, the password field will have the value `secret:/openmetadata/database/mysql-test/password`. We can also use secrets already stored in our secrets vault using the same convention `secret:{secret_id}`. All the sensitive data (the secrets ids in this case) values will be encrypted using the Fernet algorithm as extra security protection. ### Non-managed secrets manager On the other hand, the non-managed configuration allows flexibility on how we want to use our secrets vault. Instead of automatically storing all the sensitive data, we can use the secrets ids from our secrets vault following the convention `secret:{secret_id}` when filling in password fields of the connection parameters of a service, in a bot configuration, or a dbt configuration of an ingestion pipeline. The rest of the values which don't follow the convention for using a secret will be encrypted using the Fernet algorithm as extra security protection. # Secrets Manager Source: https://docs.open-metadata.org/v2.0.x/deployment/secrets-manager/how-to-add-a-new-implementation # How to add a new implementation If we want to create our implementation of a Secrets Manager, we can do it in 3 simple steps. ## 1. Update the JSON schema Create a new entry in the JSON schema definition of the Secrets Manager provider inside the `enum` property. ```json theme={null} { "$id": "https://open-metadata.org/schema/security/secrets/secretsManagerProvider.json", "$schema": "http://json-schema.org/draft-07/schema#", "title": "Secrets Manager Provider", "description": "OpenMetadata Secrets Manager Provider. Make sure to configure the same secrets manager providers as the ones configured on the OpenMetadata server.", "type": "string", "javaType": "org.openmetadata.schema.services.connections.metadata.SecretsManagerProvider", "enum": ["noop", "managed-aws","aws", "managed-aws-ssm", "aws-ssm", "in-memory", "awesome-sm"], "additionalProperties": false } ``` You can find [this](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/security/secrets/secretsManagerProvider.json) file here in the repository. ## 2. Update OM Server code Once we have updated the JSON Schema, we can start implementing our Secrets Manager, extending the `ExternalSecretsManager.java` abstract class located [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/secrets/ExternalSecretsManager.java). For example: ```java theme={null} public abstract class AwesomeSecretsManager extends ExternalSecretsManager { protected AwesomeSecretsManager(String clusterPrefix) { super(SecretsManagerProvider.AWESOME_SM, clusterPrefix); } void storeSecret(String secretName, String secretValue) { // your implementation } void updateSecret(String secretName, String secretValue) { // your implementation } String getSecret(String secretName) { // your implementation } } ``` After this, we can update `SecretsManagerFactory.java` which is a factory class. We can find this file [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/secrets/SecretsManagerFactory.java). ```java theme={null} ... case AWESOME_SM: return AwesomeSecretsManager.getInstance(config, clusterName); ... ``` ## 3. Update Python SDK code The steps are similar to the Java ones. We have to extend the [following](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/utils/secrets/external_secrets_manager.py) `ExternalSecretsManager` abstract class as it is shown below: ```python theme={null} class AwesomeSecretsManager(ExternalSecretsManager, ABC): def __init__( self, cluster_prefix: str, ): super().__init__(cluster_prefix, SecretsManagerProvider.awesome-sm) @abstractmethod def get_string_value(self, name: str) -> str: # your implementation pass ``` Similar to what we did in step 2, we have to add our implementation to the factory class `ExternalSecretsManager` that can be found [here](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/utils/secrets/secrets_manager_factory.py): ```json theme={null} ... elif secrets_manager_provider == SecretsManagerProvider.awesome-sm: return AwesomeSecretsManager(cluster_name) ... ``` If you need support while implementing your Secret Manager client, do not hesitate to reach out to us on [Slack](https://slack.open-metadata.org/). # Secrets Manager | OpenMetadata Deployment Integration Source: https://docs.open-metadata.org/v2.0.x/deployment/secrets-manager/supported-implementations Review supported secrets manager implementations to securely manage sensitive credentials across cloud and hybrid environments. # Supported implementations This is our list of supported Secrets Manager implementations: AWS Secrets Manager AWS Systems Manager Parameter Store Azure Key Vault GCP Secrets Manager # AWS Secrets Manager Source: https://docs.open-metadata.org/v2.0.x/deployment/secrets-manager/supported-implementations/aws-secrets-manager Use AWS Secrets Manager to integrate secret storage with your OpenMetadata, supporting encrypted access to service credentials. # AWS Secrets Manager ## Setup The setup steps covers the use of the managed version of the AWS Secrets Manager as secrets manager but for the non-managed follow only the steps related to the Airflow server and CLI. ### 1. Permissions needed These are the permissions required in the IAM policy to enable the AWS Secrets Manager in OpenMetadata. ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "secretsmanager:GetSecretValue", "secretsmanager:PutSecretValue", "secretsmanager:CreateSecret", "secretsmanager:UpdateSecret" ], "Resource": "*" } ] } ``` ### 2. Update configuration We have to set up the secret manager provider we want to use, that in our case is `aws`, and the credentials for our AWS account. The changes to be done in `openmetadata.yaml` file of the OpenMetadata server are: ```yaml theme={null} ... secretsManagerConfiguration: secretsManager: managed-aws # or env var SECRET_MANAGER. For non-managed use 'aws'. prefix: ${SECRET_MANAGER_PREFIX:-""} # Define the secret key ID as /// tags: ${SECRET_MANAGER_TAGS:-[]} # Add tags to the created resource, e.g., in AWS. Format is `[key1:value1,key2:value2,...]` parameters: region: # or env var OM_SM_REGION accessKeyId: # or env var OM_SM_ACCESS_KEY_ID secretAccessKey: # or env var OM_SM_ACCESS_KEY pipelineServiceClientConfiguration: # ... # Secrets Manager Loader: specify to the Ingestion Framework how to load the SM credentials from its env # Supported: noop, airflow, env secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-"noop"} ... ``` And these are the changes required in `airflow.cfg` of our Airflow instance: ```properties theme={null} ... [openmetadata_secrets_manager] aws_region = aws_access_key_id = aws_secret_access_key = ... ``` As an alternative to editing the `airflow.cfg` file, we can also set the following environment variables: ```bash theme={null} AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_REGION= AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_ACCESS_KEY_ID= AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_SECRET_ACCESS_KEY= ``` If no parameters are provided for the AWS account, or only ``, it will use the default credentials. The default credential will look for credentials in: 1. **Environment variables** - `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. 2. **Shared credential file** - `~/.aws/credentials` 3. **AWS config file** - `~/.aws/config` 4. **Assume Role provider** 5. Instance metadata service on an Amazon EC2 instance that has an IAM role configured More info in [AWS SDK for Java](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html) and [Boto3 Docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) ### 3. Migrate Secrets & restart both servers After updating the configuration files, we are ready to migrate the secrets and restart both services. In order to ensure that the current sensitive information is properly migrated to the Secrets Manager, you need to run the following command: ```bash theme={null} ./bootstrap/openmetadata-ops.sh migrate-secrets ``` Make sure you are running it with the same environment variables required by the server. If everything goes as planned, all the data would be displayed using the secrets names which starts with `/openmetadata/...` in your AWS Secrets Manager console. The following image shows what it should look like: secrets-manager-console **Note:** If we want to change the starting path for our secrets names from `openmetadata` to a different one, we have to change the property `clusterName` in our `openmetadata.yaml`. Also, if you inform the `prefix` value, it will be added before the `clusterName`, i.e., `///`. You can inform the `tags` as well as a list of strings `[key1:value1,key2:value2,...]`. These tags will be added to the resource created in AWS. ## CLI After enabling the Secret Manager, we also have to make a slight change in our workflows YAML files. In the `workflowConfig` we have to add the secret manager configuration: ```yaml theme={null} workflowConfig: openMetadataServerConfig: secretsManagerProvider: aws secretsManagerLoader: env hostPort: authProvider: ``` Then, in the environment running the CLI make sure to have an environment variable `AWS_DEFAULT_REGION` with the rest of the required configurations from [AWS](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#using-environment-variables). ## Airflow If you enabled the Secret Manager and you are using your own Airflow to run the ingestions, make sure to configure your YAML files as: ```yaml theme={null} workflowConfig: openMetadataServerConfig: secretsManagerProvider: aws secretsManagerLoader: airflow hostPort: authProvider: ``` and follow the same environment variables to set up the Airflow configuration: ```bash theme={null} AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_REGION= AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_ACCESS_KEY_ID= AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_SECRET_ACCESS_KEY= ``` # AWS Systems Manager Parameter Store Source: https://docs.open-metadata.org/v2.0.x/deployment/secrets-manager/supported-implementations/aws-ssm-parameter-store Set up AWS SSM Parameter Store for secrets management to store and retrieve credentials securely in your deployment. # AWS Systems Manager Parameter Store The setup steps covers the use of the managed version of the AWS Systems Manager Parameter Store as secrets manager but for the non-managed follow only the steps related to the Airflow server and CLI. ## Setup ### 1. Permissions needed These are the permissions required in the IAM policy to enable the AWS Systems Manager Parameter Store in OpenMetadata. ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ssm:PutParameter", "ssm:GetParameter" ], "Resource": "*" } ] } ``` ### 2. Update configuration We have to set up the secret manager provider we want to use, that in our case is `aws-ssm`, and the credentials for our AWS account. The changes to be done in `openmetadata.yaml` file of the OpenMetadata server are: ```yaml theme={null} ... secretsManagerConfiguration: secretsManager: managed-aws-ssm # or env var SECRET_MANAGER. For non-managed use 'aws-ssm'. prefix: ${SECRET_MANAGER_PREFIX:-""} # Define the secret key ID as /// tags: ${SECRET_MANAGER_TAGS:-[]} # Add tags to the created resource, e.g., in AWS. Format is `[key1:value1,key2:value2,...]` parameters: region: # or env var OM_SM_REGION accessKeyId: # or env var OM_SM_ACCESS_KEY_ID secretAccessKey: # or env var OM_SM_ACCESS_KEY pipelineServiceClientConfiguration: # ... # Secrets Manager Loader: specify to the Ingestion Framework how to load the SM credentials from its env # Supported: noop, airflow, env secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-"noop"} ... ``` And these are the changes required in `airflow.cfg` of our Airflow instance: ```properties theme={null} ... [openmetadata_secrets_manager] aws_region = aws_access_key_id = aws_secret_access_key = ... ``` As an alternative to editing the `airflow.cfg` file, we can also set the following environment variables: ```bash theme={null} AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_REGION= AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_ACCESS_KEY_ID= AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_SECRET_ACCESS_KEY= ``` If no parameters are provided for the AWS account, or only ``, it will use the default credentials. The default credential will look for credentials in: 1. **Environment variables** - `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. 2. **Shared credential file** - `~/.aws/credentials` 3. **AWS config file** - `~/.aws/config` 4. **Assume Role provider** 5. Instance metadata service on an Amazon EC2 instance that has an IAM role configured More info in [AWS SDK for Java](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html) and [Boto3 Docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) ### 3. Migrate Secrets & restart both servers After updating the configuration files, we are ready to migrate the secrets and restart both services. In order to ensure that the current sensitive information is properly migrated to the Secrets Manager, you need to run the following command: ```bash theme={null} ./bootstrap/openmetadata-ops.sh migrate-secrets ``` Make sure you are running it with the same environment variables required by the server. If everything goes as planned, all the data would be displayed using the parameters names which starts with `/openmetadata/...` in your AWS Systems Manager Parameter Store console. The following image shows what it should look like: ssm-parameter-store-console **Note:** If we want to change the starting path for our secrets names from `openmetadata` to a different one, we have to change the property `clusterName` in our `openmetadata.yaml`. Also, if you inform the `prefix` value, it will be added before the `clusterName`, i.e., `///` You can inform the `tags` as well as a list of strings `[key1:value1,key2:value2,...]`. These tags will be added to the resource created in AWS. ## CLI After enabling the Secret Manager, we also have to make a slight change in our workflows YAML files. In the `workflowConfig` we have to add the secret manager configuration: ```yaml theme={null} workflowConfig: openMetadataServerConfig: secretsManagerProvider: aws-ssm secretsManagerLoader: env hostPort: authProvider: ``` Then, in the environment running the CLI make sure to have an environment variable `AWS_DEFAULT_REGION` with the rest of the required configurations from [AWS](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#using-environment-variables). ## Airflow If you enabled the Secret Manager and you are using your own Airflow to run the ingestions, make sure to configure your YAML files as: ```yaml theme={null} workflowConfig: openMetadataServerConfig: secretsManagerProvider: aws-ssm secretsManagerLoader: airflow hostPort: authProvider: ``` and follow the same environment variables to set up the Airflow configuration: ```bash theme={null} AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_REGION= AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_ACCESS_KEY_ID= AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_SECRET_ACCESS_KEY= ``` # Azure Key Vault | OpenMetadata Secrets Manager Guide Source: https://docs.open-metadata.org/v2.0.x/deployment/secrets-manager/supported-implementations/azure-key-vault Get started with azure key vault. Setup instructions, features, and configuration details inside. Refer to the official documentation for the latest updates. # Azure Key Vault The setup steps covers the use of the managed version of the Azure Key Vault as secrets manager but for the non-managed follow only the steps related to the Airflow server and CLI. ## Setup ### 1. Create Principal #### Service Principal 1. Go to `Microsoft Entra ID` and create an [App Registration](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app). 2. Inside the App Registration, go to `Certificates & Secrets` and create a `Client secret`. Note down the `Value`, it will be our `clientSecret` configuration. 3. From the App Registration overview page, note down the `Application (client) ID` and the `Directory (tenant) ID`. #### Managed Identity (recommnded) 1. In your Azure subscription create [Manged Identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview) 2. Use this created identity - for AKS users this means you need to use [Pod Identity](https://learn.microsoft.com/en-us/azure/aks/use-azure-ad-pod-identity) or [Workload Identity (recommnded)](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview?tabs=dotnet). Note that the using Managed Identity require using [default Authentication Credential](https://learn.microsoft.com/en-us/python/api/overview/azure/identity-readme?view=azure-python#defaultazurecredential). ### 2. Add RBAC roles It if possible to use different Principals for OpenMetadata Server and the Ingestion. In that case the server needs higher privileges - `Key Vault Secrets Officer` - to be able to create/read/update secrets in the Vault. While the Airflow part only needs to read the secrets hence the role `Key Vault Secrets Officer`. #### Open Metadata server 1. In your Key Vault overview page, note down the `Vault URI`. 2. Go to `Access Control (IAM)` and click on `Add Role Assignment`. 3. Give the permission `Key Vault Secrets Officer` to your Principal. #### Airflow 1. In your Key Vault overview page, note down the `Vault URI`. 2. Go to `Access Control (IAM)` and click on `Add Role Assignment`. 3. Give the permission `Key Vault Secrets User` to your Principal. ### 3. Update configuration We have to set up the secret manager provider we want to use, that in our case is `azure-kv`, and the credentials. The changes to be done in `openmetadata.yaml` file of the OpenMetadata server are: #### Default Azure Credential ```yaml theme={null} --- secretsManagerConfiguration: secretsManager: managed-azure-kv # or env var SECRET_MANAGER. For non-managed use 'azure-kv'. prefix: ${SECRET_MANAGER_PREFIX:-""} # Define the secret key ID as -- tags: ${SECRET_MANAGER_TAGS:-[]} # Add tags to the created resource. Format is `[key1:value1,key2:value2,...]` parameters: enabled: true vaultName: ${OM_SM_VAULT_NAME:-""} pipelineServiceClientConfiguration: secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-airflow} ``` For Helm Values, you will need to add `PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER` as part of `extraEnvs`. This will look like below - ```yaml theme={null} --- ... extraEnvs: - name: PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER value: airflow ... ``` #### Client Secret Credential ```yaml theme={null} --- secretsManagerConfiguration: secretsManager: managed-azure-kv # or env var SECRET_MANAGER. For non-managed use 'azure-kv'. prefix: ${SECRET_MANAGER_PREFIX:-""} # Define the secret key ID as -- tags: ${SECRET_MANAGER_TAGS:-[]} # Add tags to the created resource. Format is `[key1:value1,key2:value2,...]` parameters: enabled: true clientId: ${OM_SM_CLIENT_ID:-""} clientSecret: ${OM_SM_CLIENT_SECRET:-""} tenantId: ${OM_SM_TENANT_ID:-""} vaultName: ${OM_SM_VAULT_NAME:-""} pipelineServiceClientConfiguration: secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-airflow} ``` For Helm Values, you will need to add `PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER` as part of `extraEnvs`. This will look like below - ```yaml theme={null} --- ... extraEnvs: - name: PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER value: airflow ... ``` The changes to be done in `airflow.yaml` file of the Airflow are: Note that the **Key Vault Name** parameter is MANDATORY for the system to know where to store and retrieve the secrets. And these are the changes required in `airflow.cfg` of our Airflow instance: ```properties theme={null} ... [openmetadata_secrets_manager] azure_key_vault_name = azure_tenant_id = azure_client_id = azure_client_secret = ... ``` As an alternative to editing the `airflow.cfg` file, we can also set the following environment variables: ```bash theme={null} AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_KEY_VAULT_NAME= AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_TENANT_ID= AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_CLIENT_ID= AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_CLIENT_SECRET= ``` If only the ``, parameter is provided, we will use Azure's [default Authentication Credential](https://learn.microsoft.com/en-us/python/api/overview/azure/identity-readme?view=azure-python#defaultazurecredential). Also if you are using [Microsoft Entra Workload ID](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview) with [Service Account Token Volume Projection](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#serviceaccount-token-volume-projection) then you need also to use projected service account instead one created by Airflow and OpenMetadata: airflow\.yaml: ```yaml theme={null} --- serviceAccount: create: false name: "name-of-your-service-account" ``` openmetadata.yaml: ```yaml theme={null} --- serviceAccount: create: false name: "name-of-your-service-account" ``` ### 3. Migrate Secrets & restart both servers After updating the configuration files, we are ready to migrate the secrets and restart both services. In order to ensure that the current sensitive information is properly migrated to the Secrets Manager, you need to run the following command: ```bash theme={null} ./bootstrap/openmetadata-ops.sh migrate-secrets ``` Make sure you are running it with the same environment variables required by the server. If everything goes as planned, all the data would be displayed using the parameters names which starts with `openmetadata-...` in your Key Vault console. **Note:** If we want to change the starting path for our secrets names from `openmetadata` to a different one, we have to change the property `clusterName` in our `openmetadata.yaml`. Also, if you inform the `prefix` value, it will be added before the `clusterName`, i.e., `--` You can inform the `tags` as well as a list of strings `[key1:value1,key2:value2,...]`. These tags will be added to the created secret. ## CLI After enabling the Secret Manager, we also have to make a slight change in our workflows YAML files. In the `workflowConfig` we have to add the secret manager configuration: ```yaml theme={null} workflowConfig: openMetadataServerConfig: secretsManagerProvider: azure-kv secretsManagerLoader: env hostPort: authProvider: ``` Make sure to follow the steps [here](https://learn.microsoft.com/en-us/python/api/overview/azure/identity-readme?view=azure-python#defaultazurecredential) to allow the Python client to authenticate to Azure. Note that the `AZURE_KEY_VAULT_NAME` variable is **REQUIRED** to know against which Key Vault service to point to. You can specify as well the environment variables of your App Registration if you're running the ingestion outside of Azure: [docs](https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.environmentcredential?view=azure-python). ## Airflow If you enabled the Secret Manager and you are using your own Airflow to run the ingestions, make sure to configure your YAML files as: ```yaml theme={null} workflowConfig: openMetadataServerConfig: secretsManagerProvider: azure-kv secretsManagerLoader: airflow hostPort: authProvider: ``` and follow the same environment variables to set up the Airflow configuration: ```bash theme={null} AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_KEY_VAULT_NAME= AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_TENANT_ID= AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_CLIENT_ID= AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_CLIENT_SECRET= ``` Note that the `AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_KEY_VAULT_NAME` variable is **REQUIRED** to know against which Key Vault service to point to. # GCP Secret Manager Parameter Store | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/secrets-manager/supported-implementations/gcp-secret-manager Connect Gcp Secret Manager to enable streamlined access, monitoring, or search of enterprise data using secure and scalable integrations. # GCP Secret Manager The setup steps covers the use of the managed version of the GCP Secret Manager as secrets manager but for the non-managed follow only the steps related to the Airflow server and CLI. ## Setup ### 1. Permissions needed These are the permissions required in the service account to enable the GCP Secret Manager in OpenMetadata. We recommend to use the role named `roles/secretmanager.secretAccessor` to grant necessary permissions. * resourcemanager.projects.get * resourcemanager.projects.list * secretmanager.versions.access ### 2. Update configuration We have to set up the secret manager provider we want to use, that in our case is `gcp`, and the credentials for our GCP information. The changes to be done in `openmetadata.yaml` file of the OpenMetadata server are: ```yaml theme={null} ... secretsManagerConfiguration: secretsManager: gcp # or env var SECRET_MANAGER. prefix: ${SECRET_MANAGER_PREFIX:-""} # Define the secret key ID as /// parameters: projectId: # or env var OM_SM_PROJECT_ID pipelineServiceClientConfiguration: # ... # Secrets Manager Loader: specify to the Ingestion Framework how to load the SM credentials from its env # Supported: noop, airflow, env secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-"noop"} ... ``` And these are the changes required in `airflow.cfg` of our Airflow instance: ```properties theme={null} ... [openmetadata_secrets_manager] gcp_project_id = ... ``` As an alternative to editing the `airflow.cfg` file, we can also set the following environment variables: ```bash theme={null} AIRFLOW__OPENMETADATA_SECRETS_MANAGER__GCP_PROJECT_ID= ``` If no parameters are provided for the GCP account, it will use Application Default Credentials (ADC). ADC will look for credentials in: 1. Local development environment 2. Cloud Shell or other Google Cloud cloud-based development environments 3. Compute Engine or other Google Cloud services that support attaching a service account 4. Google Kubernetes Engine or GKE Enterprise 5. On-premises or another cloud provider More info in [Set up Application Default Credentials](https://cloud.google.com/docs/authentication/provide-credentials-adc) ### 3. Migrate Secrets & restart both servers After updating the configuration files, we are ready to migrate the secrets and restart both services. In order to ensure that the current sensitive information is properly migrated to the Secrets Manager, you need to run the following command: ```bash theme={null} ./bootstrap/openmetadata-ops.sh migrate-secrets ``` Make sure you are running it with the same environment variables required by the server. If everything goes as planned, all the data would be displayed using the parameters names which starts with `/openmetadata/...` in your GCP Secret Manager console. The following image shows what it should look like: gcp-secret-manager-console **Note:** If we want to change the starting path for our secrets names from `openmetadata` to a different one, we have to change the property `clusterName` in our `openmetadata.yaml`. Also, if you inform the `prefix` value, it will be added before the `clusterName`, i.e., `///` You can inform the `tags` as well as a list of strings `[key1:value1,key2:value2,...]`. These tags will be added to the resource created in GCP. ## Airflow If you enabled the Secret Manager and you are using your own Airflow to run the ingestions, make sure to configure your YAML files as: ```yaml theme={null} workflowConfig: openMetadataServerConfig: secretsManagerProvider: gcp secretsManagerLoader: airflow hostPort: authProvider: ``` and follow the same environment variables to set up the Airflow configuration: ```bash theme={null} AIRFLOW__OPENMETADATA_SECRETS_MANAGER__GCP_PROJECT_ID= ``` # Kubernetes Secrets Manager Source: https://docs.open-metadata.org/v2.0.x/deployment/secrets-manager/supported-implementations/kubernetes-secrets-manager Use Kubernetes Secrets to integrate secret storage with your OpenMetadata, supporting secure access to service credentials. # Kubernetes Secrets Manager OpenMetadata can use Kubernetes Secrets as its secrets manager backend, storing sensitive values (passwords, tokens, keys, etc.) as native K8s Secret objects instead of encrypted fields in the database. ## 1. Permissions (Kubernetes RBAC) The OpenMetadata Server needs RBAC access to Kubernetes Secret objects in the target namespace. The Python ingestion runtime only needs **read** access. ### Required verbs | Component | Verbs | | ---------------------------- | ----------------------------------- | | OpenMetadata Server (Java) | `create`, `get`, `update`, `delete` | | Ingestion / Airflow (Python) | `get` | ### Example: Role + RoleBinding (namespace-scoped) ```yaml theme={null} apiVersion: v1 kind: ServiceAccount metadata: name: openmetadata-secrets-sa namespace: --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: openmetadata-secrets-role namespace: rules: - apiGroups: [""] resources: ["secrets"] verbs: ["create", "get", "update", "delete"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: openmetadata-secrets-rb namespace: subjects: - kind: ServiceAccount name: openmetadata-secrets-sa namespace: roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: openmetadata-secrets-role ``` If the ingestion runtime runs in a separate ServiceAccount, grant it a read-only Role with only `get`. > If OpenMetadata and the target secrets live in the same namespace, `` and `` are the same. *** ## 2. Update configuration ### 2.1 OpenMetadata Server (`openmetadata.yaml`) ```yaml theme={null} secretsManagerConfiguration: secretsManager: kubernetes # or env var SECRET_MANAGER prefix: ${SECRET_MANAGER_PREFIX:-""} # Optional prefix for secret names tags: ${SECRET_MANAGER_TAGS:-[]} # Labels added to K8s Secrets, format: [key1:value1,key2:value2,...] parameters: namespace: ${OM_SM_NAMESPACE:-"default"} kubeconfigPath: ${OM_SM_KUBECONFIG_PATH:-""} inCluster: ${OM_SM_IN_CLUSTER:-"false"} ``` | Parameter | Env var | Description | | ---------------- | ----------------------- | ----------------------------------------------------------------------------------------------- | | `secretsManager` | `SECRET_MANAGER` | Set to `kubernetes` | | `prefix` | `SECRET_MANAGER_PREFIX` | Optional prefix prepended to all secret names | | `tags` | `SECRET_MANAGER_TAGS` | Key-value pairs added as K8s labels on created Secrets. Format: `[key1:value1,key2:value2,...]` | | `namespace` | `OM_SM_NAMESPACE` | Namespace where secrets are stored (default: `default`) | | `kubeconfigPath` | `OM_SM_KUBECONFIG_PATH` | Path to a kubeconfig file (out-of-cluster only) | | `inCluster` | `OM_SM_IN_CLUSTER` | Use in-cluster ServiceAccount auth (default: `false`) | #### Choosing `inCluster` vs `kubeconfigPath` * **In-cluster (recommended for K8s deployments):** set `OM_SM_IN_CLUSTER=true`. Uses the pod's ServiceAccount and the RBAC from section 1. Leave `kubeconfigPath` empty. * **Out-of-cluster:** set `OM_SM_IN_CLUSTER=false` and `OM_SM_KUBECONFIG_PATH` to a kubeconfig file readable by the OpenMetadata process. If the kubeconfig path is also empty, the default kubeconfig (`~/.kube/config`) is used. ### 2.2 Pipeline Service Client In the `pipelineServiceClientConfiguration` section of `openmetadata.yaml`, set the secrets manager loader so the ingestion framework knows how to authenticate: ```yaml theme={null} pipelineServiceClientConfiguration: secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-"noop"} ``` Supported values: `noop`, `airflow`, `env`. ### 2.3 Airflow configuration (self-managed Airflow) If you run ingestion via your own Airflow deployment, configure it to resolve K8s secrets. **Option A: `airflow.cfg`** ```ini theme={null} [openmetadata_secrets_manager] kubernetes_namespace = kubernetes_kubeconfig_path = kubernetes_in_cluster = true ``` **Option B: Airflow environment variables** Airflow auto-maps env vars with the pattern `AIRFLOW__
__`: ```bash theme={null} AIRFLOW__OPENMETADATA_SECRETS_MANAGER__KUBERNETES_NAMESPACE= AIRFLOW__OPENMETADATA_SECRETS_MANAGER__KUBERNETES_KUBECONFIG_PATH= AIRFLOW__OPENMETADATA_SECRETS_MANAGER__KUBERNETES_IN_CLUSTER=true ``` If Airflow runs inside the cluster, use `KUBERNETES_IN_CLUSTER=true` and ensure Airflow's ServiceAccount has `get` permissions on Secrets in the target namespace. ### 2.4 Non-Airflow ingestion (env loader) When using the `env` secrets manager loader (e.g., standalone ingestion containers), configure via plain environment variables: ```bash theme={null} KUBERNETES_NAMESPACE= KUBERNETES_IN_CLUSTER=true KUBERNETES_KUBECONFIG_PATH= # leave empty for in-cluster ``` > The Python `env` loader also auto-detects the current namespace from `/var/run/secrets/kubernetes.io/serviceaccount/namespace` when running in-cluster, falling back to `default`. *** ## 3. Migrate secrets and restart services After updating configuration, migrate existing sensitive values from database encryption to Kubernetes Secrets: ```bash theme={null} ./bootstrap/openmetadata-ops.sh migrate-secrets ``` This migrates secrets from the DB to the configured Secrets Manager. It does **not** support migrating between external Secrets Managers (e.g., from AWS SSM to Kubernetes). Then restart: 1. **OpenMetadata Server** — so it reads/writes secrets to K8s. 2. **Airflow / ingestion runtime** — so it picks up the secrets manager settings. *** ## 4. Workflow YAML (self-managed Airflow) If you use your own Airflow to run ingestion workflows, configure the workflow YAML: ```yaml theme={null} workflowConfig: openMetadataServerConfig: secretsManagerProvider: kubernetes secretsManagerLoader: airflow # or "env" for non-Airflow hostPort: authProvider: ``` *** ## 5. How secrets are stored ### Naming convention Secret names use **hyphens** as separators (not slashes) to comply with Kubernetes DNS naming rules: ``` --- ``` * `prefix` comes from `secretsManagerConfiguration.prefix` * `clusterName` comes from the top-level `clusterName` in `openmetadata.yaml` (default: `openmetadata`) * The remaining path components are derived from the entity/connection being stored **Examples** (assuming default `clusterName: openmetadata`, no prefix): ``` openmetadata-bot-name-config-jwttoken openmetadata-database-myservice-password ``` Names are sanitized for Kubernetes compatibility: * Only lowercase alphanumeric characters and hyphens are allowed * Consecutive hyphens are collapsed to a single hyphen * Leading and trailing hyphens are stripped * Truncated to 253 characters (K8s Secret name limit) ### Data format Each secret is stored as a Kubernetes `Secret` object with: * A single data key: `value` — containing the secret as UTF-8 bytes * Labels: * `app: openmetadata` * `managed-by: openmetadata-secrets-manager` * Plus any custom labels from the `tags` configuration ```yaml theme={null} apiVersion: v1 kind: Secret metadata: name: openmetadata-database-myservice-password namespace: default labels: app: openmetadata managed-by: openmetadata-secrets-manager data: value: ``` # Enable Security | OpenMetadata Deployment Security Source: https://docs.open-metadata.org/v2.0.x/deployment/security Learn about authentication, encryption, secret management, and provider configurations for securing your platform deployment. # Enable Security This section provides detailed instructions to secure the REST endpoints of the OpenMetadata Server. OpenMetadata has support for Google SSO, Okta SSO, custom OIDC, Auth0, Azure SSO, Amazon Cognito, and OneLogin as identity providers. Please see the next sections about how to configure them. Enabling Security is only required for your **Production** installation. If you are testing OpenMetadata, it will be easier and faster to set up without security. To get up and running quickly with OpenMetadata (without security), please follow the [Quickstart](/v2.0.x/quick-start) guide. OpenMetadata currently does not support the simultaneous use of multiple authentication mechanisms, such as combining SSO and Basic Authentication. Configure Auth0 SSO to access the UI and APIs Configure Azure SSO to access the UI and APIs Configure a Custom OIDC SSO to access the UI and APIs Configure Google SSO to access the UI and APIs Configure Okta SSO to access the UI and APIs Configure Amazon Cognito SSO to access the UI and APIs Configure OneLogin SSO to access the UI and APIs Configure Keycloak SSO to access the UI and APIs # Amazon Cognito SSO | OpenMetadata Authentication Setup Source: https://docs.open-metadata.org/v2.0.x/deployment/security/amazon-cognito-sso Enable Amazon Cognito for managing user authentication, token issuance, and identity pools in distributed or serverless systems. # Amazon Cognito SSO Follow the sections in this guide to set up Amazon Cognito SSO. Security requirements for your **production** environment: * **DELETE** the admin default account shipped by OM in case you had [Basic Authentication](/v2.0.x/deployment/security/basic-auth) enabled before configuring the authentication with Amazon Cognito SSO. * **UPDATE** the Private / Public keys used for the [JWT Tokens](/v2.0.x/deployment/security/enable-jwt-tokens) in case it is enabled. ## Create Server Credentials ### Step 1: Login to AWS Portal * Login to [Amazon AWS Portal](https://aws.amazon.com/). * Search for `Cognito` in the search box and select Cognito Service from the dropdown menu. create-account ### Step 2: Setup User Pool * Click on the "Create user pool" button if you do not have any user pools configured yet. Skip this step if you already have a user pool available. * Select the type of ID providers you want to configure for your users and click "Next" create-account * Configure the security requirements in Step 2 as per your organizational needs and proceed to Step 3 * Configure the Sign-up experience in Step 3. Make sure to add email as a required attribute before proceeding to step 4 create-account * Configure message delivery as per your organizational needs and proceed to Step 5 * In Step 5, add a name for the user pool and check the "Use the Cognito Hosted UI" option and provide a Cognito domain as shown in the screenshot below create-account * In the same step, select "Public client" for the Initial App client type and configure the Allowed callback URLs with `http://localhost:8585/callback` as shown in the screenshot below. Note: For production deployments, the Allowed callback URLs should be updated with the appropriate domain name. create-account * The last step is to Review and create the User Pool. ### Step 3: Where to find the Credentials * The `User Pool ID` can be found in the User Pool summary page as seen in the screenshot below create-account * The App client ID can be found under the "App Integration" tab of the User Pool page. There will be a section that lists all the App clients with client name and client ID as shown below create-account create-account After the applying these steps, you can update the configuration of your deployment: Configure Amazon Cognito SSO for Docker deployment. Configure Amazon Cognito SSO for Kubernetes deployment. Configure Amazon Cognito SSO for Bare Metal deployment. ## Configure Ingestion Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](/deployment/security/jwt-troubleshooting) to ensure seamless authentication. # Auth0 SSO | OpenMetadata Security Integration Source: https://docs.open-metadata.org/v2.0.x/deployment/security/auth0 Set up Auth0 as an identity provider to manage secure, token-based authentication across web apps, APIs, and user-facing services. # Auth0 SSO Follow the sections in this guide to set up Auth0 SSO. Security requirements for your **production** environment: * **DELETE** the admin default account shipped by OM in case you had [Basic Authentication](/v2.0.x/deployment/security/basic-auth) enabled before configuring the authentication with Auth0 SSO. * **UPDATE** the Private / Public keys used for the [JWT Tokens](/v2.0.x/deployment/security/enable-jwt-tokens). The keys we provide by default are aimed only for quickstart and testing purposes. They should NEVER be used in a production installation. ## Create Server Credentials ### Step 1: Create the Account * If you don't have an account, [Sign up](https://auth0.com/signup) to create one. * Select the Account Type, i.e., Company or Personal * Click I need advanced settings and click next. create-account * Provide the Tenant Domain, select the region and click on Create Account. create-account * Once done, you will land on the dashboard page. create-account ## Step 2: Create Server Credentials ## Choose Your Authentication Flow After creating the account, choose the authentication flow you want to use: * [Implicit Flow](/v2.0.x/deployment/security/auth0/implicit-flow) (Public) * [Auth Code Flow](/v2.0.x/deployment/security/auth0/auth-code-flow) (Confidential) - **SPA (Single Page Application):** This type is designed for implicit flows. In this case, providing both the client ID and client secret will result in a failure because the implicit flow only requires the client ID for authentication. - **Web:** This type is intended for confidential clients. If you select this option, you must provide both the client ID and client secret. Simply passing the client ID will cause the authorization process to fail, as the Authorization Code flow requires both credentials for successful authentication. The [OIDC Authorization Code Flow](/v2.0.x/deployment/security/oidc) is used in this case, where the client secret is required to securely exchange the authorization code for tokens. ### Recommendation: * Use the **Web** type for confidential clients that require both a client ID and secret. * Use the **SPA** type for applications using implicit flows where only a client ID is needed. ## Configure Ingestion Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](/deployment/security/jwt-troubleshooting) to ensure seamless authentication. # Azure SSO | OpenMetadata Authentication Integration Source: https://docs.open-metadata.org/v2.0.x/deployment/security/azure Deploy Azure authentication to manage secure token-based access and identity roles in cloud-native or hybrid environments. # Azure SSO Follow the sections in this guide to set up Azure SSO. Security requirements for your **production** environment: * **DELETE** the admin default account shipped by OM in case you had [Basic Authentication](/v2.0.x/deployment/security/basic-auth) enabled before configuring the authentication with Azure SSO. * **UPDATE** the Private / Public keys used for the [JWT Tokens](/v2.0.x/deployment/security/enable-jwt-tokens). The keys we provide by default are aimed only for quickstart and testing purposes. They should NEVER be used in a production installation. ## Create Server Credentials ### Step 1: Login to Azure Active Directory * Sign in to Azure. For help navigating the portal, see the [Azure portal overview](https://learn.microsoft.com/en-us/azure/azure-portal/azure-portal-overview). * Navigate to the Azure Active Directory. Admin permissions are required to register the application on the Azure portal. ### Step 2: Create a New Application * From the Azure Active Directory, navigate to the `App Registrations` section from the left nav bar. create-app * Click on `New Registration`. This step is for registering the OpenMetadata UI. create-app * Provide an Application Name for registration. * Provide a redirect URL as a `Single Page Application`. * Click on `Register`. ## Choose Your Authentication Flow After creating the account, choose the authentication flow you want to use: * [Implicit Flow](/v2.0.x/deployment/security/azure/implicit-flow) (Public) * [Auth Code Flow](/v2.0.x/deployment/security/azure/auth-code-flow) (Confidential) - **SPA (Single Page Application):** This type is designed for implicit flows. In this case, providing both the client ID and client secret will result in a failure because the implicit flow only requires the client ID for authentication. - **Web:** This type is intended for confidential clients. If you select this option, you must provide both the client ID and client secret. Simply passing the client ID will cause the authorization process to fail, as the Authorization Code flow requires both credentials for successful authentication. The [OIDC Authorization Code Flow](/v2.0.x/deployment/security/oidc) is used in this case, where the client secret is required to securely exchange the authorization code for tokens. ### Recommendation: * Use the **Web** type for confidential clients that require both a client ID and secret. * Use the **SPA** type for applications using implicit flows where only a client ID is needed. # Basic Authentication | OpenMetadata Security Setup Source: https://docs.open-metadata.org/v2.0.x/deployment/security/basic-auth Configure Basic Auth to secure service access with username and password authentication for lightweight, internal use cases. # UserName/Password Login Out of the box, OpenMetadata comes with a Username & Password Login Mechanism. The default Username and Password for Login are: ```commandline theme={null} Username - admin@open-metadata.org Password - admin ``` When using a custom domain, configure the principal domain as follows: ```yaml theme={null} config: authorizer: adminPrincipals: [admin] principalDomain: "yourdomain.com" ``` With this setup, the default Username will be `admin@yourdomain.com`. Security requirements for your **production** environment: * **DELETE** the admin default account shipped by OM. * **UPDATE** the Private / Public keys used for the [JWT Tokens](/v2.0.x/deployment/security/enable-jwt-tokens) in case it is enabled. ## Setting up Basic Auth Manually Below are the required steps to set up the Basic Login: ## Set up Configurations in openmetadata.yaml ### Authentication Configuration The following configuration controls the auth mechanism for OpenMetadata. Update the mentioned fields as required. ```yaml theme={null} authenticationConfiguration: provider: ${AUTHENTICATION_PROVIDER:-basic} publicKeyUrls: ${AUTHENTICATION_PUBLIC_KEYS:-[`{your domain}`/api/v1/system/config/jwks]} # Update with your Domain and Make sure this "/api/v1/system/config/jwks" is always configured to enable JWT tokens authority: ${AUTHENTICATION_AUTHORITY:-https://accounts.google.com} enableSelfSignup : ${AUTHENTICATION_ENABLE_SELF_SIGNUP:-true} ``` For the Basic auth we need to set: * `provider`: basic * `publicKeyUrls`: `{http|https}`://`{your_domain}`:`{port}`/api/v1/system/config/jwks * `authority`: `{your_domain}` * `enableSelfSignup`: This flag indicates if users can come and signup by themselves on the OM ### Authorizer Configuration This configuration controls the authorizer for OpenMetadata: ```yaml theme={null} authorizerConfiguration: adminPrincipals: ${AUTHORIZER_ADMIN_PRINCIPALS:-[admin]} allowedEmailRegistrationDomains: ${AUTHORIZER_ALLOWED_REGISTRATION_DOMAIN:-["all"]} principalDomain: ${AUTHORIZER_PRINCIPAL_DOMAIN:-"open-metadata.org"} ``` For the Basic auth we need to set: * `adminPrincipals`: admin usernames to bootstrap the server with, comma-separated values. * `allowedEmailRegistrationDomains`: This controls what all domain are allowed for email registration can be your as well, for example gmail.com, outlook.comm etc. * `principalDomain`: This controls what all domain are allowed for email registration, for example gmail.com, outlook.comm etc. When `AUTHORIZER_ENFORCE_PRINCIPAL_DOMAIN` is set to `true`, only users with email addresses from the `AUTHORIZER_PRINCIPAL_DOMAIN` can log in. Please note the following are the formats to bootstrap admins on server startup: `[admin1,admin2,admin3]` This works for SMTP-enabled servers, Login Password for these are generated randomly and sent to the mail `adminName`@`principalDomain`. If SMTP is not enabled for OpenMetadata, please use the method below to create admin users: `[admin1, admin2, admin3]`. The default password for all admin users will be admin. After logging into the OpenMetadata UI, admin users can change their default password by navigating to `Settings > Members > Admins`. ## Metadata Ingestion For ingesting metadata when Basic Auth is enabled, it is mandatory to configure the `ingestion-bot` account with the JWT configuration. To know how to enable it, you can follow the documentation of [Enable JWT Tokens](/v2.0.x/deployment/security/enable-jwt-tokens). ### Setting up SMTP Server Basic Authentication is successfully set. For a better login experience, we can also set up the SMTP server to allow the users to Reset Password, Account Status Updates, etc. as well. ```yaml theme={null} email: emailingEntity: ${OM_EMAIL_ENTITY:-"OpenMetadata"} -> Company Name (Optional) supportUrl: ${OM_SUPPORT_URL:-"https://slack.open-metadata.org"} -> SupportUrl (Optional) enableSmtpServer : ${AUTHORIZER_ENABLE_SMTP:-false} -> True/False senderMail: ${OPENMETADATA_SMTP_SENDER_MAIL:-""} -> Sender's email serverEndpoint: ${SMTP_SERVER_ENDPOINT:-""} -> (Ex :- smtp.gmail.com) serverPort: ${SMTP_SERVER_PORT:-""} -> (SSL/TLS port) username: ${SMTP_SERVER_USERNAME:-""} -> (SMTP Server Username) password: ${SMTP_SERVER_PWD:-""} -> (SMTP Server Password) transportationStrategy: ${SMTP_SERVER_STRATEGY:-"SMTP_TLS"} ``` Following are valid value for transportation strategy: * `SMTP`: If SMTP port is 25 use this * `SMTPS`: If SMTP port is 465 use this * `SMTP_TLS`: If SMTP port is 587 use this ## Configure Ingestion Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](/v2.0.x/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](/v2.0.x/deployment/security/jwt-troubleshooting) to ensure seamless authentication. # OIDC Based Authentication Source: https://docs.open-metadata.org/v2.0.x/deployment/security/configuration-parameters # Configuration Reference Parameters ## Public Key Url (publicKeyUrls): This needs to be updated as per different SSO providers. The default value is `http://localhost:8585/api/v1/system/config/jwks`. This is the URL where the public keys are stored. The public keys are used to verify the signature of the JWT token. **Google**: [https://www.googleapis.com/oauth2/v3/certs](https://www.googleapis.com/oauth2/v3/certs) **Okta**: [https://dev-19259000.okta.com/oauth2/aus5836ihy7o8ivuJ5d7/v1/keys](https://dev-19259000.okta.com/oauth2/aus5836ihy7o8ivuJ5d7/v1/keys) **Auth0**: [https://dev-3e0nwcqx.us.auth0.com/.well-known/jwks.json](https://dev-3e0nwcqx.us.auth0.com/.well-known/jwks.json) **Azure**: `https://login.microsoftonline.com/{tenant}/discovery/v2.0/keys` Also if you have enabled [JWT Tokens](/v2.0.x/deployment/security/enable-jwt-tokens) then `http://localhost:8585/api/v1/system/config/jwks` also needs to be there in the list with proper server url. ## Client ID (id): The client ID provided by your OIDC provider. This is typically obtained when you register your application with the OIDC provider. ## Type (type): Specify the type of OIDC provider you are using (e.g., google, azure). This value is same as `provider` in `authenticationConfiguration`. ## Client Secret (secret): Replace with the client secret provided by your OIDC provider. ## Scope (scope): Define the scopes that your application requests during authentication. Update `${OIDC_SCOPE:-"openid email profile"}` with the desired scopes. It does not need to be changed in most cases. The default scopes are `openid email profile`. The openid scope is required for OIDC authentication. The email and profile scopes are used to retrieve the user's email address and profile information. Although, some provider only give Refresh Token if `offline_access` scope is provided. So, if you want to use Refresh Token, you need to add `offline_access` scope, like below: `offline_access openid email profile`. ## Discovery URI (discoveryUri): Provide the URL of the OIDC provider's discovery document. This document contains metadata about the provider's configuration. It is mostly in the format as below: [https://accounts.google.com/.well-known/openid-configuration](https://accounts.google.com/.well-known/openid-configuration) **Google**: [https://accounts.google.com/.well-known/openid-configuration](https://accounts.google.com/.well-known/openid-configuration) **Okta**: [https://dev-19259000.okta.com/oauth2/aus5836ihy7o8ivuJ5d7/.well-known/openid-configuration](https://dev-19259000.okta.com/oauth2/aus5836ihy7o8ivuJ5d7/.well-known/openid-configuration) **Auth0**: [https://dev-3e0nwcqx.us.auth0.com/.well-known/openid-configuration](https://dev-3e0nwcqx.us.auth0.com/.well-known/openid-configuration) **Azure**: `https://login.microsoftonline.com/{tenant}/v2.0/.well-known/openid-configuration` Normally it's some initial SSO provider URL followed by `.well-known/openid-configuration` ## Use Nonce (useNonce): Set to true by Default, if you want to use nonce for replay attack protection during authentication. This does not need to be changed. ## Preferred JWS Algorithm (preferredJwsAlgorithm): Specify the preferred JSON Web Signature (JWS) algorithm. Default is RS256 and need not be changed . ## Response Type (responseType): Define the response type for the authentication request. Default is code and need not be changed. ## Disable PKCE (disablePkce): Set `${OIDC_DISABLE_PKCE:-true}` to true if you want to disable Proof Key for Code Exchange (PKCE). If you want to send CodeVerifier and CodeChallenge in the request, set it to false. ## Callback URL (callbackUrl): Provide the callback URL where the OIDC provider redirects after authentication. Update `${OIDC_CALLBACK:-"http://localhost:8585/callback"}` with your actual callback URL. The only initial part of the URL should be changed, the rest of the URL should be the same as the default one. The default URL is `http://localhost:8585/callback`. Also, this should match what you have configured in your OIDC provider. ## Server URL (serverUrl): Specify the URL of your OM Server. Default is `http://localhost:8585`. ## Client Authentication Method (clientAuthenticationMethod): Define the method used for client authentication. Default is client\_secret\_post. This does not need to be changed in most cases. The default value is `client_secret_post`. This method is used to send the client ID and client secret in the request body. Another possible value is `client_secret_basic`, which sends the client ID and client secret in the Authorization header. Depending on the OIDC provider, you may need to change this value if only one of them is supported. ## Tenant (tenant): If applicable, specify the tenant ID for multi-tenant applications. Example in case of Azure. This is only applicable for multi-tenant applications. If you are using a single tenant application, you can leave this field empty. For Azure SSO Provider this may be needed. ## Max Clock Skew (maxClockSkew): Define the maximum acceptable clock skew between your application server and the OIDC server. ## Custom Parameters (customParams): If you have any additional custom parameters required for OIDC configuration, specify them here. ## Config (config): The central configuration block for OpenMetadata. ## Provider (provider): Specifies the authentication method to be used. The default is `ldap`, but you can change it to another supported provider. Example: `google`, `azure`. ## Entity Id (entityId): The unique identifier for the SAML Identity Provider. Example: `"https://mocksaml.com/api/saml/sso"` ## SSO Login URL (ssoLoginUrl): The URL to which users are redirected for Single Sign-On (SSO) authentication. Example: `"https://saml.example.com/entityid"` ## IPDX509 Certificate (idpX509Certificate): The public certificate used by the IdP to sign SAML assertions. Example: `""` (empty string means no certificate provided, needs to be set with actual certificate) ## Authority URL (authorityUrl): The URL used for SAML login, typically a custom endpoint for your SAML provider. Example: `"http://localhost:8585/api/v1/saml/login"` ## Name ID (nameId): The format for the NameID element in the SAML response, usually representing the unique identifier of the user. Example: `"urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress"` ## ACS (acs): The Assertion Consumer Service (ACS) URL, where the IdP sends the SAML response after authentication. Example: `"http://localhost:8585/api/v1/saml/acs"` ## SPX509 Certificate (spX509Certificate): The public certificate used by the Service Provider to verify the IdP's SAML response. Example: `""` (empty string means no certificate provided, needs to be set with actual certificate) ## Strict Mode (strictMode): Whether to enforce strict compliance with the SAML standard, ensuring the response is fully validated. Default: `false` ## Token Validity (tokenValidity): The validity period of the SAML token in seconds. Default: `"3600"` (1 hour) ## Send Encrypted Name ID (sendEncryptedNameId): Whether to send the NameID in an encrypted format in the SAML response. Default: `false` ## Send Signed Auth Request (sendSignedAuthRequest): Whether to sign the authentication request sent to the IdP. Default: `false` ## Sign SP Metadata (signSpMetadata): Whether to sign the Service Provider's metadata when exchanging SAML metadata with the IdP. Default: `false` ## Want Messages Signed (wantMessagesSigned): Whether the Service Provider expects SAML messages to be signed. Default: `false` ## Want Assertions Signed (wantAssertionsSigned): Whether the Service Provider expects SAML assertions to be signed. Default: `false` ## Want Assertion Encrypted (wantAssertionEncrypted): Whether to encrypt the SAML assertion before sending it to the Service Provider. Default: `false` ## Want Name ID Encrypted (wantNameIdEncrypted): Whether to encrypt the NameID element in the SAML response. Default: `false` ## Key Store File Path (keyStoreFilePath): The file path to the keystore file containing certificates and private keys used for signing and encryption. Example: `""` (empty string means no keystore file provided) ## KeyStore Alias (keyStoreAlias): The alias used to refer to the key inside the keystore file. Example: `""` (empty string means no alias provided) ## KeyStore Password (keyStorePassword): The password used to access the keystore file. Example: `""` (empty string means no password provided) ## Class Name (className): Specifies the class that handles the authorization logic. Default: `"org.openmetadata.service.security.DefaultAuthorizer"` ## Container Request Filter (containerRequestFilter): Specifies the request filter used to process authentication, especially for handling JWT tokens. Default: `"org.openmetadata.service.security.JwtFilter"` ## Initial Admins (initialAdmins): A list of users who will be granted administrative privileges during the initial setup. Example: `["suresh"]` ## Principal Domain (principalDomain): The domain that is associated with user accounts. Default: `"open-metadata.org"` ## Authority (authority): The base URL of the OIDC authority. Example: Replace `{IssuerUrl}` with the URL of your custom OIDC provider. ## Client ID (clientId): The client ID for the application registered with the custom OIDC provider. Replace `{client id}` with the actual client ID. ## Host (host): The hostname of the LDAP server. Defaults to `localhost`. ## Port (port): The port number to connect to the LDAP server. Defaults to `10636`. ## DN Admin Principal (dnAdminPrincipal): The distinguished name (DN) of the admin user used for lookup operations in LDAP. Defaults to `"cn=admin,dc=example,dc=com"`. ## DN Admin Password (dnAdminPassword): The password for the admin user. Defaults to `"secret"`. ## Userbase DN (userBaseDN): The base DN for user lookup in LDAP. Defaults to `"ou=people,dc=example,dc=com"`. ## Mail Attribute Name (mailAttributeName): The attribute name in LDAP that stores user email addresses. Defaults to `email`. ## Maximum Pool Size (maxPoolSize) (Optional): Defines the maximum number of connections in the LDAP connection pool. Defaults to `3`. ## SSL Enabled (sslEnabled): Indicates if SSL is enabled for connecting to the LDAP server. Defaults to `true`. ## Custom Trust Manager Configuration (customTrustManagerConfig): * ### TrustStore FilePath (trustStoreFilePath): Path to the custom trust store file. Default is empty. * ### TrustStore File Password (trustStoreFilePassword): Password for the trust store file. Default is empty. * ### TrustStore File Format (trustStoreFileFormat): Format of the trust store file. Default is empty. * ### Verify Host Name (verifyHostname): If hostname verification is enabled. Default is empty. * ### Examine Validity Dates (examineValidityDates): Whether to check validity dates for certificates. Default is empty. ## Host Name Configuration (hostNameConfig): * ### Allow Wild Cards (allowWildCards): Allows wildcard certificates in hostnames. Default is empty. * ### Acceptable Host Names (acceptableHostNames): A list of acceptable hostnames. Default is an empty list. ## JVM Default Configurations (jvmDefaultConfig): * ### Verify Host Name (verifyHostname): Enables hostname verification using JVM defaults. Default is empty. ## Trust All Configurations (trustAllConfig): * ### Examine Validity Dates (examineValidityDates): Checks the validity dates of certificates when using `TrustAll` mode. Defaults to `true`. ## Enforce Principal Domain (enforcePrincipalDomain): Whether to enforce user principal matching with the defined principal domain ## Enable Secure Socket Connection (enableSecureSocketConnection): If true, enables secure connections (SSL/TLS) ## Use Roles From Provider (useRolesFromProvider): Whether to derive roles from the authentication provider ## Initial Admins (initialAdmins): List of initial admin users for the system ## JWT Principal Claims (jwtPrincipalClaims): **Definition:** The JWT claims OpenMetadata uses to identify the authenticated user. **Example:** ```yaml theme={null} ["preferred_username", "upn", "email", "sub"] ``` **Why it matters:** OpenMetadata evaluates this list in order and uses the first matching claim to identify the user. If none match, authentication fails. **Note:** At least one claim in this list must correspond to the user's **email address** — OpenMetadata uses this to identify and match authenticated users to their accounts. Use `email` for consistency and compatibility; for domain scoping, use the `hd` claim. **Note:** If you are using Keycloak or Azure, avoid changing the order of claims in `jwtPrincipalClaims` for existing deployments. The system relies on claim order to match users in tokens to existing user records — changing the order can cause authentication failures. **Common claim values by provider:** | Provider | Recommended claims | | -------------- | ---------------------------------------- | | Okta | `["preferred_username", "email", "sub"]` | | Auth0 | `["preferred_username","email", "sub"]` | | Azure | `["preferred_username","email", "sub"]` | | Amazon Cognito | `["cognito:username", "email", "sub"]` | | Custom OIDC | `["email", "preferred_username", "sub"]` | | Keycloak | `["preferred_username","email", "sub"]` | | OneLogin | `["preferred_username","email", "sub"]` | | SAML | `["preferred_username","email", "sub"]` | | LDAP | `["preferred_username","email", "sub"]` | ## JWT Principal Claims Mapping (jwtPrincipalClaimsMapping): **Definition:** Maps JWT claims to OpenMetadata user profile fields. **Supported keys:** Only `email` and `username` are valid mapping targets in `jwtPrincipalClaimsMapping`. **Example:** ```yaml theme={null} ["email:email", "username:preferred_username"] ``` **Why it matters:** Controls how SSO login data maps to user profiles in OpenMetadata. **Format:** `openmetadata_field:jwt_claim` (for example, `"email:email"`). **Note:** The display name is derived automatically from standard OIDC/JWT claims — you don't need to configure it using `jwtPrincipalClaimsMapping`. If you need richer name handling, make sure your identity provider is configured to include `given_name` and `family_name` as claims in the ID token — OpenMetadata will pick them up automatically. Refer to your identity provider's documentation for instructions: | Provider | How to enable `given_name` and `family_name` | | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Okta | Ensure `profile` scope is requested. See [OpenID Connect & OAuth 2.0 — Scopes and Claims](https://developer.okta.com/docs/api/openapi/okta-oauth/guides/overview) | | Auth0 | Included automatically when `profile` scope is requested. See [ID Token Structure](https://auth0.com/docs/secure/tokens/id-tokens/id-token-structure) | | Azure | Must be added as optional claims in App Registration → Token configuration. See [Configure optional claims](https://learn.microsoft.com/en-us/entra/identity-platform/optional-claims) | | Google | Included automatically when `profile` scope is requested. See [OpenID Connect — Sign in with Google](https://developers.google.com/identity/openid-connect/openid-connect) | | Amazon Cognito | Must be enabled as standard attributes in the User Pool. See [Working with user attributes](https://docs.aws.amazon.com/cognito/latest/developerguide/user-pool-settings-attributes.html) | | Keycloak | Included automatically when `profile` scope is requested. See [Securing Applications and Services Guide](https://www.keycloak.org/docs/latest/server_admin/index.html#_client_scopes) | | OneLogin | Included automatically when `profile` scope is requested. See [Scopes — OpenID Connect](https://developers.onelogin.com/openid-connect/scopes) | | SAML | Refer to your IdP's documentation to add `given_name` and `family_name` as assertion attributes | | LDAP | Ensure `givenName` and `sn` attributes are populated on user entries in your directory | **Important:** Using any other key (for example, `name` or `firstName`) will cause the service to fail on startup with a validation error. ## Enable Self Signup (enableSelfSignup): Allows users to sign up themselves if not already registered ## Preferred JWT Algorithms (preferredJwsAlgorithm): Preferred algorithm for JWT signature validation ## Allowed Email Registration Domains (allowedEmailRegistrationDomains): Specifies allowed domains for email registration # Custom OIDC SSO | OpenMetadata Security Integration Source: https://docs.open-metadata.org/v2.0.x/deployment/security/custom-oidc Set up a custom OIDC provider for authentication, allowing flexible federation and secure access across deployments. # Custom OIDC SSO Follow the sections in this guide to set up Custom OIDC SSO. Security requirements for your **production** environment: * **DELETE** the admin default account shipped by OM in case you had [Basic Authentication](/v2.0.x/deployment/security/basic-auth) enabled before configuring the authentication with Custom OIDC SSO. * **UPDATE** the Private / Public keys used for the [JWT Tokens](/v2.0.x/deployment/security/enable-jwt-tokens). The keys we provide by default are aimed only for quickstart and testing purposes. They should NEVER be used in a production installation. ## Create Server Credentials * Go to the console of your preferred custom OIDC SSO provider * Create an OIDC client application with implicit flow enabled to get a client ID. ### Create Client ID and Secret Key * Navigate to your preferred OIDC provider console and create an OIDC client application. * Generate client ID and secret key in JSON format. After the applying these steps, you can update the configuration of your deployment: Configure Custom OIDC SSO for Docker deployment. Configure Custom OIDC SSO for Kubernetes deployment. Configure Custom OIDC SSO for Bare Metal deployment. ## Configure Ingestion Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](/deployment/security/jwt-troubleshooting) to ensure seamless authentication. # Enable JWT Tokens | OpenMetadata Security Features Source: https://docs.open-metadata.org/v2.0.x/deployment/security/enable-jwt-tokens Enable JWT-based security for user authentication, session management, and platform API access using tokens. # Enable JWT Tokens When we [enable SSO security](/v2.0.x/deployment/security) on OpenMetadata, it will restrict access to all the APIs. Users who want to access the UI will be redirected to configured SSO to log in, and SSO will provide the token to continue to make OpenMetadata REST API calls. However, metadata ingestion or any other services which use OpenMetadata APIs to create entities or update them requires a token as well to authenticate. Typically, SSO offers service accounts for this very reason. OpenMetadata supports service accounts that the SSO provider supports. Please read the [docs](/v2.0.x/deployment/security) to enable them. In some cases, either creating a service account is not feasible, or the SSO provider itself doesn't support the service account. To address this gap, we shipped JWT token generation and authentication within OpenMetadata. Security requirements for your **production** environment: * **DELETE** the admin default account shipped by OM in case you have [Basic Authentication](/v2.0.x/deployment/security/basic-auth) enabled. * **UPDATE** the Private / Public keys used for the [JWT Tokens](/v2.0.x/deployment/security/enable-jwt-tokens). The keys we provide by default are aimed only for quickstart and testing purposes. They should NEVER be used in a production installation. ## Create Private / Public key ### For local/testing deployment You can work with the existing configuration or generate private/public keys. By default, the `jwtTokenConfiguration` is shipped with OM. ### For production deployment It is a **MUST** to update the JWT configuration. To create private/public key use the following commands can be used: ```commandline theme={null} openssl genrsa -out private_key.pem 2048 openssl pkcs8 -topk8 -inform PEM -outform DER -in private_key.pem -out private_key.der -nocrypt openssl rsa -in private_key.pem -pubout -outform DER -out public_key.der ``` Copy the `private_key.der` and `public_key.der` in OpenMetadata server `conf` directory. Make sure the permissions can only be readable by the user who is starting OpenMetadata server. ## Configure OpenMetadata Server To enable JWT token generation. Please add the following to the OpenMetadata server ```yaml theme={null} jwtTokenConfiguration: rsapublicKeyFilePath: ${RSA_PUBLIC_KEY_FILE_PATH:-"/opt/openmetadata/conf/public_key.der"} rsaprivateKeyFilePath: ${RSA_PRIVATE_KEY_FILE_PATH:-"/opt/openmetadata/conf/private_key.der"} jwtissuer: ${JWT_ISSUER:-"open-metadata.org"} keyId: ${JWT_KEY_ID:-"Gb389a-9f76-gdjs-a92j-0242bk94356"} ``` If you are using helm charts or docker use the env variables to override the configs above. Please use absolute path for public and private key files that we generated in previous steps. Update the `JWT_ISSUER` to be the domain where you are running the OpenMetadata server. Generate `UUID64` id to configure `JWT_KEY_ID`. This should be generated once and keep it static even when you are updating the versions. Any change in this id will result in all the tokens issued so far to be invalid. ### Add public key URIS ```yaml theme={null} authenticationConfiguration: provider: ${AUTHENTICATION_PROVIDER:-no-auth} # This will only be valid when provider type specified is customOidc providerName: ${CUSTOM_OIDC_AUTHENTICATION_PROVIDER_NAME:-""} publicKeyUrls: ${AUTHENTICATION_PUBLIC_KEYS:-[{your SSO public keys URL}]} authority: ${AUTHENTICATION_AUTHORITY:-https://accounts.google.com} clientId: ${AUTHENTICATION_CLIENT_ID:-""} callbackUrl: ${AUTHENTICATION_CALLBACK_URL:-""} jwtPrincipalClaims: ${AUTHENTICATION_JWT_PRINCIPAL_CLAIMS:-[email,preferred_username,sub]} ``` **Note:** At least one claim in `jwtPrincipalClaims` must correspond to the user's **email address** — OpenMetadata uses this to identify and match authenticated users to their accounts. add `{your domain}/api/v1/system/config/jwks` to `publicKeyUrls`. You should append to the existing configuration such that your SSO and JWTToken auth verification will work. ```yaml theme={null} publicKeyUrls: ${AUTHENTICATION_PUBLIC_KEYS:-[{your SSO public keys URL}, {your domain}/api/v1/system/config/jwks]} ``` Once you configure the above settings, restart OpenMetadata server .

Note on JWKS url Network Reachbility

Make sure the above JWKS URI - `{your domain}/api/v1/system/config/jwks` is reachable from OpenMetadata Server Instance (VM or Docker Container or Kubernetes Pod). You can run the below command from the OpenMetadata Server to test it's reachility - ``` wget -O - {your domain}/api/v1/system/config/jwks ``` ## Generate Token **Note**: The **Bots** tile under **Settings** is only visible to users with Admin privileges. If you don't see it, ask your organization's OpenMetadata Admin to generate a bot token for you or grant you Admin access. Once the above configuration is updated, the server is restarted. Admin can go to Settings -> Bots page. Settings Page Bot settings page Click on the `ingestion-bot`. The current token can be revoked, or you can create a new one. Bot credentials edition ## Configure Ingestion The generated token from the above page should pass onto the ingestion framework so that the ingestion can make calls securely to OpenMetadata. Make sure this token is not shared and stored securely. ### Running Ingestion from CLI If you are running the ingestion from CLI. Add the below configuration to the workflow configuration you pass: ```yaml theme={null} workflowConfig: openMetadataServerConfig: hostPort: http://localhost:8585/api authProvider: openmetadata securityConfig: jwtToken: ``` In the above section, under the `workflowConfig`, configure `authProvider` to be "openmetadata" and under `securityConfig` section, add `jwtToken` and its value from the ingestion bot page. ## Configure JWT Key Pairs for Docker Following the above documentation, you will have a private and public key pair available, as mentioned [here](#create-private-/-public-key). Next, configure JWT tokens in the Docker environment. ### Create docker compose host volume mappings Create a host directory which will be mapped as docker volumes to docker compose. This step will require you to update existing docker compose files that comes up with [OpenMetadata Releases](https://github.com/open-metadata/OpenMetadata/releases). ```yaml theme={null} services: ... openmetadata-server: volumes: - ./docker-volume/jwtkeys:/etc/openmetadata/jwtkeys ... ``` It is presumed with the above code snippet that you have `docker-volume` directory available on host where the docker-compose file is. ### Update the docker compose environment variables with jwtkeys Update the docker environment variables either directly in the docker-compose files or in a separate docker env files. Below is a code snippet for how the docker env file will look like. ```bash theme={null} # openmetadata.prod.env RSA_PUBLIC_KEY_FILE_PATH="/etc/openmetadata/jwtkeys/public_key.der" RSA_PRIVATE_KEY_FILE_PATH="/etc/openmetadata/jwtkeys/private_key.der" JWT_ISSUER="open-metadata.org" # update this as per your environment JWT_KEY_ID="c8ec220c-be7d-4e47-97c7-098bf6a57ce1" # update this to a unique uuid4 ``` ### Run the docker compose command to start the services Run the docker compose CLI command to start the docker services with the configured jwt keys. ``` docker compose -f docker-compose.yml --env-file openmetadata.prod.env up -d ``` ## Configure JWT Key Pairs for Kubernetes Following the above documentation, you will have a private and public key pair available, as mentioned [here](#create-private-/-public-key). Next, configure JWT tokens in the Kubernetes environment. ### Create Kubernetes Secrets for the Key Pairs Create Kubernetes Secrets from file using the kubernetes imperative commands below. ```bash theme={null} kubectl create secret generic openmetadata-jwt-keys --from-file private_key.der --from-file public_key.der --namespace default ``` ### Update Helm Values to mount Kubernetes secrets and configure JWT Token Configuration Update your helm values to mount Kubernetes Secrets as Volumes and update the Jwt Token Configuration to point the Key File Paths to mounted path (absolute file path). ```yaml theme={null} # openmetadata.prod.values.yml openmetadata: config: ... jwtTokenConfiguration: rsapublicKeyFilePath: "/etc/openmetadata/jwtkeys/public_key.der" rsaprivateKeyFilePath: "/etc/openmetadata/jwtkeys/private_key.der" jwtissuer: "open-metadata.org" # update this as per your environment keyId: "c8ec220c-be7d-4e47-97c7-098bf6a57ce1" # update this to a unique uuid4 ... extraVolumes: - name: openmetadata-jwt-vol secret: secretName: openmetadata-jwt-keys extraVolumeMounts: - name: openmetadata-jwt-vol mountPath: "/etc/openmetadata/jwtkeys" readOnly: true ``` It is recommended to consider new directory paths for mounting the secrets as volumes to OpenMetadata Server Pod. With OpenMetadata Helm Charts, you will be able to add volumes and volumeMounts with `extraVolumes` and `extraVolumeMounts` helm values. ### Install / Upgrade Helm Chart Release Run the below command to make sure the update helm values are available to OpenMetadata. ``` helm upgrade --install openmetadata open-metadata/openmetadata --values openmetadata.prod.values.yml ``` # Enable SSL | OpenMetadata Security Configuration Source: https://docs.open-metadata.org/v2.0.x/deployment/security/enable-ssl Enable SSL to secure platform communication, data transfers, and authentication across endpoints and services. # Enable SSL In this section we will guide you through adding SSL to your OpenMetadata deployment with two different approaches: Use Nginx to enable SSL. This is the simplest solution. Set SSL directly at the OpenMetadata server. # Enable SSL in Airflow | OpenMetadata Security Guide Source: https://docs.open-metadata.org/v2.0.x/deployment/security/enable-ssl/airflow-ssl Enable SSL for Airflow-based deployments to secure metadata transport, authentication, and configuration endpoints. # Configure OpenMetadata certificates in Airflow Follow this section if you added SSL certs in the OpenMetadata server. The OpenMetadata configuration related to Airflow (or in general, the Pipeline Service Client) is the following: ```yaml theme={null} pipelineServiceClientConfiguration: # ... # This SSL information is about the OpenMetadata server. # It will be picked up from the pipelineServiceClient to use/ignore SSL when connecting to the OpenMetadata server. verifySSL: ${PIPELINE_SERVICE_CLIENT_VERIFY_SSL:-"no-ssl"} # Possible values are "no-ssl", "ignore", "validate" sslConfig: certificatePath: ${PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH:-""} # Local path for the Pipeline Service Client ``` Then, in order to add this, you can either update the `openmetadata.yaml` config if your deployment is Bare Metal, or update the following environment variables: * `PIPELINE_SERVICE_CLIENT_VERIFY_SSL=validate` * `PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH="path/to/cert` Note that the `PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH` should be the path to the certificate you generated [here](/v2.0.x/deployment/security/enable-ssl), and it should be the local path in your Airflow deployment. ## Enable SSL in Airflow Follow this section if you want to add SSL certificates in Airflow. This will secure the connection from the OpenMetadata to Airflow. Airflow has two [configurations](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#web-server-ssl-cert) to be added in `airflow.cfg` to enable SSL: * `AIRFLOW__WEBSERVER__WEB_SERVER_SSL_CERT` * `AIRFLOW__WEBSERVER__WEB_SERVER_SSL_KEY` Those are files that will need to be local to the Airflow deployment. ## Generate Certs We can generate these files following this [SO](https://stackoverflow.com/questions/47883769/how-to-enable-ssl-on-apache-airflow) thread: ```bash theme={null} openssl req \ -newkey rsa:2048 -nodes -keyout airflow.key \ -x509 -days 365 -out airflow.crt ``` and we can provide the following answers to try this locally: ``` Country Name (2 letter code) []:US State or Province Name (full name) []:CA Locality Name (eg, city) []:San Francisco Organization Name (eg, company) []:OpenMetadata Organizational Unit Name (eg, section) []:OpenMetadata Common Name (eg, fully qualified host name) []:localhost Email Address []:local@open-metadata.org ``` It is important that the `Common Name` is the host name that will be hosting Airflow. This command will generate the pair `airflow.key` and `airflow.crt`. ## Include Certificates Once the files are generated we need to add them to the Airflow deployment. For example, if using the `openmetadata-ingestion` image, you can update it to add the following lines: ```dockerfile theme={null} # SET SSL COPY --chown=airflow:0 ingestion/airflow.key /opt/airflow COPY --chown=airflow:0 ingestion/airflow.crt /opt/airflow ENV AIRFLOW__WEBSERVER__WEB_SERVER_SSL_CERT=/opt/airflow/airflow.crt ENV AIRFLOW__WEBSERVER__WEB_SERVER_SSL_KEY=/opt/airflow/airflow.key ``` If you now start Airflow with these changes, it will be running at `https://localhost:8080`. ## Update the OpenMetadata configuration Since Airflow will be using SSL, we need to update the OpenMetadata Server configuration to use the certificates when preparing the connection to the Airflow Webserver. The `pipelineServiceClientConfiguration` will look like the following: ```yaml theme={null} pipelineServiceClientConfiguration: [...] parameters: username: ${AIRFLOW_USERNAME:-admin} password: ${AIRFLOW_PASSWORD:-admin} timeout: ${AIRFLOW_TIMEOUT:-10} # If we need to use SSL to reach Airflow truststorePath: ${AIRFLOW_TRUST_STORE_PATH:-""} truststorePassword: ${AIRFLOW_TRUST_STORE_PASSWORD:-""} ``` Update the `truststorePath` and `truststorePassword` accordingly, pointing to the `keystore` in your server host holding the certificates we created. For docker deployments, you will provide OpenMetadata Server Application with the self signed certificates of Airflow bundled in JVM keystore. These will be passed to the application using `AIRFLOW_TRUST_STORE_PATH` and `AIRFLOW_TRUST_STORE_PASSWORD` environment variable. ``` AIRFLOW_TRUST_STORE_PATH="" AIRFLOW_TRUST_STORE_PASSWORD="" ``` Please make sure to have the the truststore file mounted and available as part of Docker Deployments. For kubernetes deployments, update the helm values as below - ```yaml theme={null} extraEnvs: - name: AIRFLOW_TRUST_STORE_PASSWORD valueFrom: secretKeyRef: name: truststore-password-secret key: password - name: AIRFLOW_TRUST_STORE_PATH value: "/etc/openmetadata/certs/truststore.jks>" extraVolumes: - name: jks-vol secret: secretName: jks-certs extraVolumeMounts: - name: jks-vol mountPath: /etc/openmetadata/certs readOnly: true ``` In the above code snippet, we are mounting the volumes of truststore file from a kubernetes secret. You can create the secret from `truststore.jks` file from the below `kubectl` command - ```bash theme={null} kubectl create secret generic jks-certs --from-file truststore.jks --namespace kubectl create secret generic truststore-password-secret --from-literal password= --namespace ``` Next, restart or redeploy openmetadata application to take the above configs in effect. ### Example: Setting it locally For example, if we are running the server locally, we need to add the certificate to the JVM `cacerts` store: ```bash theme={null} sudo keytool -import -trustcacerts -keystore cacerts -storepass changeit -noprompt -alias localhost -file /path/to/airflow.crt ``` Then, the values of the YAML config would be something similar to: ```yaml theme={null} truststorePath: "/Library/Java/JavaVirtualMachines/amazon-corretto-11.jdk/Contents/Home/lib/security/cacerts" truststorePassword: "changeit" ``` Make sure to update these values to the ones in your host. Also, it's always preferred to use environment variables instead of hardcoding sensitive information. # Enable SSL with Nginx | OpenMetadata Security Setup Source: https://docs.open-metadata.org/v2.0.x/deployment/security/enable-ssl/nginx Enable SSL using NGINX to encrypt traffic and protect sensitive metadata exchanges across service endpoints and UI access. # Enable SSL with Nginx Nginx can be used as a load balancer or an SSL termination point for OpenMetadata. In this section, we will look at how to use Nginx and Certbot to deploy SSL. The below instructions are for Ubuntu 20 and any other flavor of Linux please find similar instructions. ## Install Nginx Nginx can be installed to a completely different host where you are running OpenMetadata Server or on the same host. For simplicity, we will do this on the same host as the OpenMetadata server. ```commandline theme={null} sudo apt update sudo apt install nginx sudo systemctl start nginx ``` ## Configure Nginx to redirect requests to OpenMetadata For Nginx to serve this content, it’s necessary to create a server block with the correct directives. Instead of modifying the default configuration file directly, let’s make a new one at `/etc/nginx/sites-available/openmetadata`: ```commandline theme={null} sudo vi /etc/nginx/sites-available/openmetadata ``` And add the below content ```commandline theme={null} server { access_log /var/log/nginx/sandbox-access.log; error_log /var/log/nginx/sandbox-error.log; server_name sandbox.open-metadata.org; location / { proxy_pass http://127.0.0.1:8585; } } ``` In the above configuration, please ensure that the `server_name` matches the domain where you are hosting the OpenMetadata server. Also, the `proxy_pass` configuration should point to the OpenMetadata server port. Then, link the configuration to `sites-enabled` and restart nginx: ```commandline theme={null} sudo ln -s /etc/nginx/sites-available/openmetadata /etc/nginx/sites-enabled/openmetadata sudo systemctl restart nginx ``` The above configuration will serve at port 80, so if you configured a domain like `sandbox.open-metadata.org` one can start accessing OpenMetadata server by just pointing the browser to [http://sandbox.open-metadata.org](http://sandbox.open-metadata.org). ## Enable SSL using Certbot Certbot, [https://certbot.eff.org/](https://certbot.eff.org/), is a non-profit org that distributes the certified X509 certs and renews them as well. ```commandline theme={null} sudo apt install certbot python3-certbot-nginx sudo systemctl reload nginx ``` ## Obtaining an SSL Certificate Certbot provides a variety of ways to obtain SSL certificates through plugins. The Nginx plugin will take care of reconfiguring Nginx and reloading the config whenever necessary. To use this plugin, type the following: ```commandline theme={null} sudo certbot --nginx -d sandbox.open-metadata.org ``` Replace` sandbox.open-metadata.org` with your domain for OpenMetadata. If this is your first time running certbot, you will be prompted to enter an email address and agree to the terms of service. After doing so, certbot will communicate with the `Let's Encrypt` server, then run a challenge to verify that you control the domain you’re requesting a certificate for. If that’s successful, certbot will ask how you’d like to configure your HTTPS settings. ## Verifying Certbot Auto-Renewal `Let's Encrypt`'s certificates are only valid for ninety days. This is to encourage users to automate their certificate renewal process. The certbot package we installed takes care of this for us by adding a `systemd` timer that will run twice a day and automatically renew any certificate that’s within thirty days of expiration. You can query the status of the timer with `systemctl`: ```commandline theme={null} sudo systemctl status certbot.timer ``` to renew, you can run the following command ```commandline theme={null} sudo certbot renew --dry-run ``` ## Summary In this tutorial, we walked through the setup of Nginx to serve the requests to OpenMetadata and used Certbot to enable SSL on Nginx. Do keep in mind that we secured the external connection to Nginx, and Nginx terminates the SSL connections, and the rest of the transport Nginx to the OpenMetadata server is on Plaintext. However, OpenMetadata server should be configured to listen to only localhost requests, i.e., It cannot be reached directly from outside traffic except for Nginx on that host. This makes it a secure SSL. # Enable SSL at the OpenMetadata Server Source: https://docs.open-metadata.org/v2.0.x/deployment/security/enable-ssl/openmetadata-server Configure SSL for the OpenMetadata server to enable HTTPS, protect authentication flows, and secure backend communications. # Enable SSL at the OpenMetadata Server The OpenMetadata Server is built using **Dropwizard** and **Jetty**. In this section, we will go through the steps involved in setting up SSL for Jetty. If you would like a simple way to set up SSL, please refer to the guide using [Nginx](/v2.0.x/deployment/security/enable-ssl/nginx). However, this step can be treated as an additional layer of adding SSL to OpenMetadata. In cases where one would use Nginx as a load balancer or AWS LB, you can set up SSL at the OpenMetadata server level such that traffic from the load balancer to OpenMetadata is going through an encrypted channel. ## Create Self-Signed Certificate A self-signed certificate should only be used for POC (demo) or `localhost` installation. For production scenarios, please reach out to your DevOps team to issue an X509 certificate which you can import into a Keystore. Run the below command to generate an X509 Certificate and import it into keystore: ```commandline theme={null} keytool -keystore openmetadata.keystore.jks -alias localhost -keyalg RSA -keysize 2048 -sigalg SHA256withRSA -genkey -validity 365 ``` keystore For this example, we are configuring the password to be `test12`. Copy the generated `openmetadata.keystore.jks` to OpenMetadata installation path under the `conf` directory. keystore ## Configure openmetadata.yaml Add the below section to your `openmetadata.yaml` under the `conf` directory. Please add the password you set for the Keystore generated above in the config below. ```yaml theme={null} server: rootPath: '/api/*' applicationConnectors: - type: https port: ${SERVER_PORT:-8585} keyStorePath: ./conf/openmetadata.keystore.jks keyStorePassword: test12 keyStoreType: JKS supportedProtocols: [TLSv1.2, TLSv1.5] excludedProtocols: [SSL, SSLv2, SSLv2Hello, SSLv3] ``` ## Access OpenMetadata server in the browser These steps are not necessary if you used proper X509 certificated signed by trusted CA Authority. Since we used self-signed certificates, browsers such as Chrome or Brave will not allow you to visit `https://localhost:8585`. You'll get the following error page and there is no way to proceed. browser However, the Safari browser allows you to visit if you click advanced and click proceed. To work around this issue, on OS X, you can import the certificate into the keychain and trust it so that browsers can trust and allow you to access OpenMetadata. ### Export X509 certificate from Keystore Run the below command to export the X509 cert. ```commandline theme={null} keytool -export -alias localhost -keystore openmetadata.keystore.jks -rfc -file public.cert ``` ### Import public cert into Keychain - OS X only Open the KeyChain app in OS X, drag and drop the `public.cert` file generated in the previous command into the Keychain: import Double-click on `localhost`: import Click on `Trust` to open and set `Always Trust`: import Once the above steps are finished, all the browsers will allow you to visit the OpenMetadata server using HTTPS. However, you'll still a warning in the address bar. All of these steps are not necessary with an X509 certificate issued by a trusted authority and one should always use that in production. # SSL Troubleshooting | OpenMetadata Security Guide Source: https://docs.open-metadata.org/v2.0.x/deployment/security/enable-ssl/ssl-troubleshooting Troubleshoot SSL errors and misconfigurations affecting secure access to services, ingestion, and the UI. # SSL Troubleshooting In this section we comment common issues that the user can face when enabling SSL in OpenMetadata. ## Bot using JWT as authentication mechanism After enabling SSL on the OM server, we have to update also the public keys URL for the validation of the JWT tokens by updating to the secured URL: `https://{server_domain}:{port}/api/v1/system/config/jwks`. In case we are using a self-signed certificate, it will fail with the error below: 500-error-ssl To avoid this error, you must import your public certificate into the Java Keystore of the OM server. If your OM deployment is done with Docker or Kubernetes, you must copy the cert into the `openmetadata_server` container or pod. After that, you can proceed with the following steps from your terminal: 1. Go to your \$JAVA\_HOME/lib/security where the **cacerts** keystore is located. 2. Run the following command once in the directory: ```bash theme={null} keytool -import -trustcacerts -keystore cacerts -storepass changeit -noprompt -alias localhost -file /path/to/public.cert ``` After that, you can restart the server, and the error 500 will disappear. ## Deploying workflows in Airflow One common issue after enabling SSL with a self-signed certificate is that our workflows in Airflow will fail or will not be deployed. We can notice it because the following error will be shown in the UI when deploying or re-deploying: handshake-error-ssl This can be solved in two different ways: #### 1. Validate the certificate using the public certificate (recommended): We specify which public certificate must be used to validate the OM server connection. 1. Copy the public certificate into our Airflow instance. 2. Update the configuration of our OM server so that each time a workflow is deployed, we send the new configuration. * In **docker**: ```yaml theme={null} PIPELINE_SERVICE_CLIENT_VERIFY_SSL=validate PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH=/path/to/certificate/in/airflow ``` * In **bare metal**: Edit the `conf/openmetadata.yaml` file: ```yaml theme={null} pipelineServiceClientConfiguration: verifySSL: "validate" sslConfig: certificatePath: "/path/to/certificate/in/airflow" ``` * In **K8s**: We have to update in the `values.yaml` file with: ```yaml theme={null} openmetadata: config: pipelineServiceClientConfig: verifySsl: "validate" sslCertificatePath: "/path/to/certificate/in/airflow" ``` #### 2. Ignore the certification validation (not recommended for production): When doing any call to the secured OM server, the certificate validation will be ignored. * In **docker**: ```yaml theme={null} PIPELINE_SERVICE_CLIENT_VERIFY_SSL=ignore ``` * In **bare metal**: Edit the `conf/openmetadata.yaml` file: ```yaml theme={null} pipelineServiceClientConfiguration: verifySSL: "ignore" ``` * In **K8s**: We have to update in the `values.yaml` file with: ```yaml theme={null} openmetadata: config: pipelineServiceClientConfig: verifySsl: "ignore" ``` Once one of the configurations is set, we can restart our OM server and deploy or redeploy without any issues. ## Ingesting from CLI Similar to what happens when deploying workflows in Airflow, we have to update our workflow config file with one of these options: * To validate our certificate: ```yaml theme={null} workflowConfig: openMetadataServerConfig: verifySSL: validate sslConfig: caCertificate: /local/path/to/certificate ``` * To ignore certificate validation: ```yaml theme={null} workflowConfig: openMetadataServerConfig: verifySSL: ignore ``` ## Providing a single keystore that has all the cacerts required This can be achieved using the `OPENMETADATA_OPTS` environment variable configuration across all the deployments. However, for Production, we recommend you to bundle your cacerts separately for each components (like ElasticSearch/Opensearch and Airflow) and provide that to each individual configs for [openmetadata.yaml](https://github.com/open-metadata/OpenMetadata/blob/main/conf/openmetadata.yaml). You can use this environment variable to also provide extra JVM parameters to tune the application as per your infrastructure needs. Below is an example values to be set for the `OPENMETADATA_OPTS` environment variable to use cacerts truststore which is bundled for an organization issued certificates - ```bash theme={null} OPENMETADATA_OPTS="-Djavax.net.ssl.trustStore= -Djavax.net.ssl.trustStorePassword=" ``` It is expected to have the keystore file either mounted as external volume or to be available over the filesystem where openmetadata server application will be running. # Google SSO | OpenMetadata Authentication Integration Source: https://docs.open-metadata.org/v2.0.x/deployment/security/google Set up Google OAuth as your identity provider to enable secure login, token exchange, and scoped user access in deployments. # Google SSO Follow the sections in this guide to set up Google SSO. Security requirements for your **production** environment: * **DELETE** the admin default account shipped by OM in case you had [Basic Authentication](/v2.0.x/deployment/security/basic-auth) enabled before configuring the authentication with Google SSO. * **UPDATE** the Private / Public keys used for the [JWT Tokens](/v2.0.x/deployment/security/enable-jwt-tokens). The keys we provide by default are aimed only for quickstart and testing purposes. They should NEVER be used in a production installation. ## Create Server Credentials ### Step 1: Create the Account * Go to [Create Google Cloud Account](https://console.cloud.google.com/) * Click on `Create Project` create-account ### Step 2: Create a New Project Enter the **Project name**. Enter the parent organization or folder in the **Location box**. That resource will be the hierarchical parent of the new project. Click **Create**. create-project ### Step 3: How to Configure OAuth Consent * Select the project you created above and click on **APIs & Services** on the left-side panel. configure-oauth-consent * Click on the **OAuth Consent Screen** available on the left-hand side panel. * Choose User Type **Internal**. select-user-type * Once the user type is selected, provide the **App Information** and other details. * Click **Save and Continue**. save-app-information * On the **Scopes Screen**, Click on **ADD OR REMOVE SCOPES** and select the scopes. * Once done click on **Update**. scopes-screen * Click **Save and Continue**. save-edit-app-registration * Click on **Back to Dashboard**. back-to-dashboard back-to-dashboard ### Step 4: Create Credentials for the Project * Once the OAuth Consent is configured, click on **Credentials** available on the left-hand side panel. create-credentials * Click on **Create Credentials** * Select **OAuth client ID** from the dropdown. cselect-outh-client-id * Once selected, you will be asked to select the **Application type**. Select **Web application**. select-web-application After selecting the **Application Type**, name your project and give the authorized URIs: * domain/callback * domain/silent-callback authorized-urls * Click **Create** * You will get the credentials get-the-credentials ### Step 5: Where to Find the Credentials * Go to **Credentials** * Click on the **pencil icon (Edit OAuth Client)** on the right side of the screen find-credentials * You will find the **Client ID** in the top right corner find-clientid-and-secret After the applying these steps, you can update the configuration of your deployment: Configure Google SSO for Docker deployment. Configure Google SSO for Kubernetes deployment. Configure Google SSO for Bare Metal deployment. ## Configure Ingestion Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](/deployment/security/jwt-troubleshooting) to ensure seamless authentication. # JWT validation Troubleshooting | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/security/jwt-troubleshooting Fix JWT-based authentication issues with deployment and ingestion by adjusting token validation and decoding settings. # JWT Troubleshooting Add the `{domain}:{port}/api/v1/sytem/config/jwks` in the list of publicKeys ```yaml theme={null} authentication: provider: "google" publicKeys: - "https://www.googleapis.com/oauth2/v3/certs" - "http://localhost:8585/api/v1/system/config/jwks" (your domain and port) ``` This config with `"http://localhost:8585/api/v1/system/config/jwks"` is the default behavior. If you are configuring and expecting a JWT token to work, configuring with that extra URL is required. JWT Tokens are issued by private certificates. We need public keys to decrypt it and get that token's user name, expiry time, etc. In OpenMetadata users can enable SSO for users to login and use JWT tokens issued by OpenMetadata for bots The way OpenMetadata issues a JWT Token is using this [config](https://github.com/open-metadata/OpenMetadata/blob/main/conf/openmetadata.yaml#L155). It uses the `rsapublicKeyFilePath` file to generate a token. When the ingestion workflow uses this token, we use `rsapublicKeyPath` to decrypt it. The way we do this is using the response from this endpoint `http://localhost:8585/api/v1/system/config/jwks`. ## Get JWT token from UI. **Note**: The **Bots** tile under **Settings** is only visible to users with Admin privileges. If you don't see it, ask your organization's OpenMetadata Admin to generate a bot token for you or grant you Admin access. First Open Open-Metadata UI than go to settings > Bots > Ingestion Bot jwt-token You can validate that in [jwt.io](https://jwt.io/). if there's something wrong on how the JWT token was generated. jwt.io ### Resolving the "Failed in filtering request: Not Authorized! Token not present" Error If you encounter the error message **"Failed in filtering request: Not Authorized! Token not present"**, verify the **`enableSecureSocketConnection`** environment setting. Ensure that **`enableSecureSocketConnection: ${AUTHORIZER_ENABLE_SECURE_SOCKET:-false}`** is set to `false` if it is currently set to `true`. # Keycloak SSO | OpenMetadata Security Integration Source: https://docs.open-metadata.org/v2.0.x/deployment/security/keycloak Use Keycloak as your authentication server to centralize identity management with support for OIDC and SSO integrations. # Keycloak SSO Follow the sections in this guide to set up Keycloak SSO. Security requirements for your **production** environment: * **DELETE** the admin default account shipped by OM in case you had [Basic Authentication](/v2.0.x/deployment/security/basic-auth) enabled before configuring the authentication with Keycloak SSO. * **UPDATE** the Private / Public keys used for the [JWT Tokens](/v2.0.x/deployment/security/enable-jwt-tokens). The keys we provide by default are aimed only for quickstart and testing purposes. They should NEVER be used in a production installation. ## Create Server Credentials ### Step 1: Access the Keycloak Admin Console * You need an administrator account. If you don't have, see [Creating the first administrator](https://www.keycloak.org/docs/latest/server_admin/#creating-first-admin_server_administration_guide). * Go to the URL for the Admin Console. For example, for localhost, use this URL: `http://localhost:8080/admin/` login-page * Enter the username and password you created. ### Step 2: Change Realm selected * The Keycloak use Realms as the primary form of organization, we can't use the realm "master" for new clients (apps), only for administration, so change for your specific realm or create a new. * In this example we are used an existing one called "Data-sec". change-realm ## Create Server Credentials ## Choose Your Authentication Flow After creating the account, choose the authentication flow you want to use: * [Implicit Flow](/v2.0.x/deployment/security/keycloak/implicit-flow) (Public) * [Auth Code Flow](/v2.0.x/deployment/security/keycloak/auth-code-flow) (Confidential) - **SPA (Single Page Application):** This type is designed for implicit flows. In this case, providing both the client ID and client secret will result in a failure because the implicit flow only requires the client ID for authentication. - **Web:** This type is intended for confidential clients. If you select this option, you must provide both the client ID and client secret. Simply passing the client ID will cause the authorization process to fail, as the Authorization Code flow requires both credentials for successful authentication. The [OIDC Authorization Code Flow](/v2.0.x/deployment/security/oidc) is used in this case, where the client secret is required to securely exchange the authorization code for tokens. ### Recommendation: * Use the **Web** type for confidential clients that require both a client ID and secret. * Use the **SPA** type for applications using implicit flows where only a client ID is needed. ## Configure Ingestion Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](/deployment/security/jwt-troubleshooting) to ensure seamless authentication. # Keycloak SSO for Bare Metal Source: https://docs.open-metadata.org/v2.0.x/deployment/security/keycloak/bare-metal # Keycloak SSO for Bare Metal ## Update conf/openmetadata.yaml In `openmetadata.yaml` file and use the following example as a reference. Replace the placeholder values with the details generated during your keycloak account and application credentials setup. Check the more information about environment variable [here](/v2.0.x/deployment/security/configuration-parameters). ``` # Implicit Flow authorizerConfiguration: className: "org.openmetadata.service.security.DefaultAuthorizer" containerRequestFilter: "org.openmetadata.service.security.JwtFilter" adminPrincipals: # Your `name` from name@domain.com - "admin" - "user1" - "user2" principalDomain: "open-metadata.org" # Update with your Domain,The primary domain for the organization (your domain.com from name@domain.com). authenticationConfiguration: provider: "custom-oidc" publicKeyUrls: # Update with your keycloak Domain and OMD server URL. - "{OMD-server-domain}/api/v1/system/config/jwks" # Update with your Domain and Make sure this "/api/v1/system/config/jwks" is always configured to enable JWT tokens - "{Keycloak-server-URL}/auth/realms/{your-realm-name}/protocol/openid-connect/certs" # Update your Keycloak Url And Realm. authority: "{Keycloak-server-URL}/auth/realms/{your-realm-name}" clientId: "{Client ID}" callbackUrl: "http://localhost:8585/callback" clientType: "public" ``` ``` # Auth Code Flow authorizerConfiguration: className: "org.openmetadata.service.security.DefaultAuthorizer" containerRequestFilter: "org.openmetadata.service.security.JwtFilter" adminPrincipals: # Your `name` from name@domain.com - "admin" - "user1" - "user2" principalDomain: "open-metadata.org" # Update with your Domain,The primary domain for the organization (your domain.com from name@domain.com). authenticationConfiguration: provider: "custom-oidc" publicKeyUrls: # Update with your keycloak Domain and OMD server URL. - "{OMD-server-domain}/api/v1/system/config/jwks" # Update with your Domain and Make sure this "/api/v1/system/config/jwks" is always configured to enable JWT tokens - "{Keycloak-server-URL}/auth/realms/{your-realm-name}/protocol/openid-connect/certs" # Update your Keycloak Url And Realm. authority: "{Keycloak-server-URL}/auth/realms/{your-realm-name}" clientId: "{Client ID}" callbackUrl: "http://localhost:8585/callback" clientType: "confidential" oidcConfiguration: id: "{Client ID}" # Update you keycloak Client ID type: "keycloak" secret: "{Client Secret}" # Update with keycloak Client Secret discoveryUri:"{Keycloak-server-URL}/realms/{your-realm-name}/.well-known/openid-configuration" # Keycloak's discovery URI Update your Keycloak's Domain and Realm callbackUrl: http://localhost:8585/callback" serverUrl: "http://localhost:8585" ``` Altering the order of claims in `jwtPrincipalClaims` may lead to problems when matching a user from a token with an existing user in the system. The mapping process relies on the specific order of claims, so changing it can result in inconsistencies or authentication failures, as the system cannot ensure correct user mapping with a new claim order. ## Configure Ingestion Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](/deployment/security/jwt-troubleshooting) to ensure seamless authentication. Go to KeyCloak Configuration # Keycloak SSO for Docker Source: https://docs.open-metadata.org/v2.0.x/deployment/security/keycloak/docker # Keycloak SSO for Docker To enable security for the Docker deployment, follow the next steps: ## 1. Create an .env file Create an `openmetadata_keycloak.env` file and add the following contents as an example. Use the information generated when setting up the account. Check the more information about environment variable [here](/v2.0.x/deployment/security/configuration-parameters). ``` # Implicit Flow AUTHORIZER_CLASS_NAME= org.openmetadata.service.security.DefaultAuthorizer AUTHORIZER_REQUEST_FILTER= org.openmetadata.service.security.JwtFilter AUTHORIZER_ADMIN_PRINCIPALS=[admin] # john.doe from john.doe@example.com AUTHORIZER_PRINCIPAL_DOMAIN=open-metadata.org # Update with your Domain,The primary domain for the organization (example.com from john.doe@example.com). AUTHENTICATION_PROVIDER=custom-oidc # Use "custom-oidc" for Keycloak AUTHENTICATION_CLIENT_ID=testsso CUSTOM_OIDC_AUTHENTICATION_PROVIDER_NAME=KeyCloak # Name of the OIDC provider AUTHENTICATION_PUBLIC_KEYS=[http://localhost:8081/auth/realms/data-sec/protocol/openid-connect/certs, https://{your domain}/api/v1/system/config/jwks] # Update with your Domain and Make sure this "/api/v1/system/config/jwks" is always configured to enable JWT tokens AUTHENTICATION_CALLBACK_URL="http://localhost:8585/callback" AUTHENTICATION_AUTHORITY={http://localhost:8080/realms/{your-realm}} AUTHENTICATION_CLIENT_TYPE=public ``` ``` # Auth Code Flow AUTHORIZER_CLASS_NAME= org.openmetadata.service.security.DefaultAuthorizer AUTHORIZER_REQUEST_FILTER= org.openmetadata.service.security.JwtFilter AUTHORIZER_ADMIN_PRINCIPALS=[admin] # john.doe from john.doe@example.com AUTHORIZER_PRINCIPAL_DOMAIN=open-metadata.org # Update with your Domain,The primary domain for the organization (example.com from john.doe@example.com). AUTHENTICATION_PROVIDER=custom-oidc # Use "custom-oidc" for Keycloak CUSTOM_OIDC_AUTHENTICATION_PROVIDER_NAME=KeyCloak # Name of the OIDC provider AUTHENTICATION_PUBLIC_KEYS=[http://localhost:8081/auth/realms/data-sec/protocol/openid-connect/certs, https://{your domain}/api/v1/system/config/jwks] # Update with your Domain and Make sure this "/api/v1/system/config/jwks" is always configured to enable JWT tokens AUTHENTICATION_AUTHORITY={http://localhost:8081/auth/realms/data-sec} AUTHENTICATION_CLIENT_ID=testsso AUTHENTICATION_CALLBACK_URL="https://{your domain}/callback" AUTHENTICATION_CLIENT_TYPE=confidential OIDC_CLIENT_ID=testsso # Replace with your Keycloak client ID OIDC_CLIENT_SECRET="{CLIENT_SECRET}" # Replace with your Keycloak client secret OIDC_TYPE="Keycloak" # Specify the OIDC provider (Keycloak) OIDC_DISCOVERY_URI="OIDC_DISCOVERY_URI=http://host.docker.internal:8081/realms/openmetadata/.well-known/openid-configuration" # Keycloak's discovery URI Update your Keycloak's Domain and Realm OIDC_CALLBACK="http://localhost:8585/callback" # Callback URL registered in Keycloak OIDC_SERVER_URL="http://localhost:8585" # OpenMetadata server URL ``` Altering the order of claims in `jwtPrincipalClaims` may lead to problems when matching a user from a token with an existing user in the system. The mapping process relies on the specific order of claims, so changing it can result in inconsistencies or authentication failures, as the system cannot ensure correct user mapping with a new claim order. ## 2. Start Docker ```commandline theme={null} docker compose --env-file ~/openmetadata_keycloak.env up -d ``` ## Configure Ingestion Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](/deployment/security/jwt-troubleshooting) to ensure seamless authentication. Go to KeyCloak Configuration # Keycloak SSO for Kubernetes Source: https://docs.open-metadata.org/v2.0.x/deployment/security/keycloak/kubernetes # Keycloak SSO for Kubernetes Check the Helm information [here](https://artifacthub.io/packages/search?repo=open-metadata). Here is an example for reference, showing where to place the values in the `values.yaml` file after setting up your Keycloak account and obtaining the application credentials. Check the more information about environment variable [here](/v2.0.x/deployment/security/configuration-parameters). ``` # Public Flow openmetadata: config: authorizer: className: "org.openmetadata.service.security.DefaultAuthorizer" containerRequestFilter: "org.openmetadata.service.security.JwtFilter" initialAdmins: # john.doe from john.doe@example.com - "admin" - "user1" - "user2" principalDomain: "open-metadata.org" # Update with your Domain,The primary domain for the organization (example.com from john.doe@example.com). authentication: clientType: public provider: "custom-oidc" publicKeys: - "{OMD-server-domain}/api/v1/system/config/jwks" # Update with your Domain and Make sure this "/api/v1/system/config/jwks" is always configured to enable JWT tokens - "{Keycloak-server-URL}/realms/{your-realm-name}/protocol/openid-connect/certs" authority: "{Keycloak-server-URL}/realms/{your-realm-name}/protocol/openid-connect/auth" clientId: "{Client ID}" # Update your Client ID callbackUrl: "http://localhost:8585/callback" ``` ``` # Auth Code Flow openmetadata: config: authorizer: className: "org.openmetadata.service.security.DefaultAuthorizer" containerRequestFilter: "org.openmetadata.service.security.JwtFilter" initialAdmins: # john.doe from john.doe@example.com - "admin" - "user1" - "user2" principalDomain: "open-metadata.org" # Update with your Domain,The primary domain for the organization (example.com from john.doe@example.com). authentication: clientType: confidential provider: "custom-oidc" publicKeys: - "{OMD-server-domain}/api/v1/system/config/jwks" # Update with your Domain and Make sure this "/api/v1/system/config/jwks" is always configured to enable JWT tokens - "{Keycloak-server-URL}/realms/{your-realm-name}/protocol/openid-connect/certs" authority: "{Keycloak-server-URL}/realms/{your-realm-name}/protocol/openid-connect/auth" clientId: "{Client ID}" # Update your Client ID callbackUrl: "http://localhost:8585/callback" oidcConfiguration: enabled: true oidcType: "Keycloak" clientId: secretRef: oidc-secrets secretKey: openmetadata-oidc-client-id clientSecret: secretRef: oidc-secrets secretKey: openmetadata-oidc-client-secret discoveryUri:"{Keycloak-server-URL}/realms/{your-realm-name}/.well-known/openid-configuration" # Keycloak's discovery URI Update your Keycloak's Domain and Realm callbackUrl: http://localhost:8585/callback serverUrl: http://localhost:8585 ``` Altering the order of claims in `jwtPrincipalClaims` may lead to problems when matching a user from a token with an existing user in the system. The mapping process relies on the specific order of claims, so changing it can result in inconsistencies or authentication failures, as the system cannot ensure correct user mapping with a new claim order. ## Configure Ingestion Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](/deployment/security/jwt-troubleshooting) to ensure seamless authentication. Go to KeyCloak Configuration # Fix PKI Not Found When Using Keycloak with Custom PKI Source: https://docs.open-metadata.org/v2.0.x/deployment/security/keycloak/troubleshooting Learn how to resolve PKI not found errors in OpenMetadata when using Keycloak behind Nginx with custom PKI by importing CA certificates into the truststore. # FAQ: Security with Keycloak ## How to resolve "PKI not found" error when connecting to Keycloak behind Nginx with a custom PKI? If you're using Keycloak behind an Nginx reverse proxy with a custom Public Key Infrastructure (PKI), OpenMetadata may fail to authenticate due to missing trusted certificates. This results in a **"PKI not found"** or TLS validation error. ### Resolution To allow OpenMetadata to trust your custom CA: 1. **Extend the OpenMetadata Docker image** and import your custom CA certificate into the Java truststore. 2. Use the following command (replace paths accordingly): ```bash theme={null} keytool -import -trustcacerts -keystore $JAVA_HOME/lib/security/cacerts \ -storepass changeit -noprompt -alias my-custom-ca \ -file /path/to/your/custom-ca.crt ``` 3. Alternatively, if you're using Helm, you can update your deployment by modifying the container image or using an initContainer to patch the truststore and setting: ```bash theme={null} OPENMETADATA_OPTS="-Djavax.net.ssl.trustStore=/path/to/keystore.jks \ -Djavax.net.ssl.trustStorePassword=changeit" ``` For guidance on extending the Docker image, refer to the official documentation: [Extending OpenMetadata Docker Image (GKE Example)](/v2.0.x/deployment/kubernetes/gke/airflow#extending-openmetadata-server-docker-image) This enables OpenMetadata to establish a secure connection with Keycloak behind your Nginx reverse proxy using a custom certificate authority. # LDAP Authentication | OpenMetadata Security Setup Source: https://docs.open-metadata.org/v2.0.x/deployment/security/ldap Configure LDAP integration to support centralized identity management using directory-based authentication systems. # Setting up Ldap Authentication Security requirements for your **production** environment: * **DELETE** the admin default account shipped by OM in case you had [Basic Authentication](/v2.0.x/deployment/security/basic-auth) enabled before configuring the authentication with Auth0 SSO. * **UPDATE** the Private / Public keys used for the [JWT Tokens](/v2.0.x/deployment/security/enable-jwt-tokens). The keys we provide by default are aimed only for quickstart and testing purposes. They should NEVER be used in a production installation. OpenMetadata allows using LDAP for validating email and password authentication. Once setup successfully, the user should be able to sign in to OpenMetadata using the Ldap credentials. Below are the configuration types to set up the LDAP Authentication: Configure LDAP Authentication for Docker deployment. Configure LDAP Authentication for Kubernetes deployment. Configure LDAP Authentication for Bare Metal deployment. ## Configure Ingestion Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](/deployment/security/jwt-troubleshooting) to ensure seamless authentication. # Ldap Authentication for Bare Metal Source: https://docs.open-metadata.org/v2.0.x/deployment/security/ldap/bare-metal # Ldap Authentication for Bare Metal ## Set up Configurations in openmetadata.yaml ### Authentication Configuration The following configuration controls the auth mechanism for OpenMetadata. Update the mentioned fields as required. ```yaml theme={null} authenticationConfiguration: provider: ${AUTHENTICATION_PROVIDER:-ldap} publicKeyUrls: ${AUTHENTICATION_PUBLIC_KEYS:-[{your domain}/api/v1/system/config/jwks]} # Update with your Domain and Make sure this "/api/v1/system/config/jwks" is always configured to enable JWT tokens authority: ${AUTHENTICATION_AUTHORITY:-https://accounts.google.com} enableSelfSignup : ${AUTHENTICATION_ENABLE_SELF_SIGNUP:-false} ldapConfiguration: host: ${AUTHENTICATION_LDAP_HOST:-localhost} port: ${AUTHENTICATION_LDAP_PORT:-10636} dnAdminPrincipal: ${AUTHENTICATION_LOOKUP_ADMIN_DN:-"cn=admin,dc=example,dc=com"} dnAdminPassword: ${AUTHENTICATION_LOOKUP_ADMIN_PWD:-"secret"} userBaseDN: ${AUTHENTICATION_USER_LOOKUP_BASEDN:-"ou=people,dc=example,dc=com"} mailAttributeName: ${AUTHENTICATION_USER_MAIL_ATTR:-email} # Optional maxPoolSize: ${AUTHENTICATION_LDAP_POOL_SIZE:-3} sslEnabled: ${AUTHENTICATION_LDAP_SSL_ENABLED:-true} truststoreConfigType: ${AUTHENTICATION_LDAP_TRUSTSTORE_TYPE:-TrustAll} # {CustomTrustStore, HostName, JVMDefault, TrustAll} trustStoreConfig: customTrustManagerConfig: trustStoreFilePath: ${AUTHENTICATION_LDAP_TRUSTSTORE_PATH:-} trustStoreFilePassword: ${AUTHENTICATION_LDAP_KEYSTORE_PASSWORD:-} trustStoreFileFormat: ${AUTHENTICATION_LDAP_SSL_KEY_FORMAT:-} verifyHostname: ${AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST:-} examineValidityDates: ${AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES:-} hostNameConfig: allowWildCards: ${AUTHENTICATION_LDAP_ALLOW_WILDCARDS:-} acceptableHostNames: ${AUTHENTICATION_LDAP_ALLOWED_HOSTNAMES:-[]} jvmDefaultConfig: verifyHostname: ${AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST:-} trustAllConfig: examineValidityDates: ${AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES:-true} ``` For the LDAP auth we need to set: OpenMetadata Specific Configuration : * `provider`: ldap * `publicKeyUrls`: `{http|https}://{your_domain}:{port}/api/v1/system/config/jwks` * `authority`: `{your_domain}` * `enableSelfSignup`: This has to be false for Ldap. Mandatory LDAP Specific Configuration: * `host`: hostName for the Ldap Server (Ex - localhost). * `port`: port of the Ldap Server to connect to (Ex - 10636). * `dnAdminPrincipal`: This is the DN Admin Principal(Complete path Example :- cn=admin,dc=example,dc=com ) with a lookup access in the Directory. * `dnAdminPassword`: Above Admin Principal Password. * `userBaseDN`: User Base DN(Complete path Example :- ou=people,dc=example,dc=com). Please see the below image for a sample LDAP Configuration in ApacheDS. apache-ldap Advanced LDAP Specific Configuration (Optional): * `maxPoolSize`: Connection Pool Size to use to connect to LDAP Server. * `sslEnabled`: Set to true if the SSL is enable to connect to LDAP Server. * `truststoreConfigType`: Truststore type. It is required. Can select from `CustomTrustStore`, `HostName`, `JVMDefault`, `TrustAll` * `trustStoreConfig`: Config for the selected truststore type. Please check below note for setting this up. Based on the different `truststoreConfigType`, we have following different `trustStoreConfig`. **1. TrustAll** Provides an SSL trust manager which will blindly trust any certificate that is presented to it, although it may optionally reject certificates that are expired or not yet valid. It can be convenient for testing purposes, but it is recommended that production environments use trust managers that perform stronger validation. ```yaml theme={null} truststoreConfigType: ${AUTHENTICATION_LDAP_TRUSTSTORE_TYPE:-TrustAll} trustStoreConfig: trustAllConfig: examineValidityDates: ${AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES:-true} ``` * `examineValidityDates`: Indicates whether to reject certificates if the current time is outside the validity window for the certificate. **2. JVMDefault** Provides an implementation of a trust manager that relies on the JVM's default set of trusted issuers. ```yaml theme={null} truststoreConfigType: ${AUTHENTICATION_LDAP_TRUSTSTORE_TYPE:-JVMDefault} trustStoreConfig: jvmDefaultConfig: verifyHostname: ${AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST:-true} ``` * `verifyHostname`: Controls using TrustAllSSLSocketVerifier vs HostNameSSLSocketVerifier. In case the certificate contains cn=hostname of the Ldap Server set it to true. **3. HostName** Provides an SSL trust manager that will only accept certificates whose hostname matches an expected value. ```yaml theme={null} truststoreConfigType: ${AUTHENTICATION_LDAP_TRUSTSTORE_TYPE:-HostName} trustStoreConfig: hostNameConfig: allowWildCards: ${AUTHENTICATION_LDAP_ALLOW_WILDCARDS:-false} acceptableHostNames: ${AUTHENTICATION_LDAP_ALLOWED_HOSTNAMES:-[localhost]} ``` * `allowWildCards`: Indicates whether to allow wildcard certificates which contain an asterisk as the first component of a CN subject attribute or dNSName subjectAltName extension. * `acceptableHostNames`: The set of hostnames and/or IP addresses that will be considered acceptable. Only certificates with a CN or subjectAltName value that exactly matches one of these names (ignoring differences in capitalization) will be considered acceptable. It must not be null or empty. **4. CustomTrustStore** Use the custom Truststore by providing the below details in the config. ```yaml theme={null} truststoreConfigType: ${AUTHENTICATION_LDAP_TRUSTSTORE_TYPE:-CustomTrustStore} trustStoreConfig: customTrustManagerConfig: trustStoreFilePath: ${AUTHENTICATION_LDAP_TRUSTSTORE_PATH:-/Users/parthpanchal/trusted.ks} trustStoreFilePassword: ${AUTHENTICATION_LDAP_KEYSTORE_PASSWORD:-secret} trustStoreFileFormat: ${AUTHENTICATION_LDAP_SSL_KEY_FORMAT:-JKS} verifyHostname: ${AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST:-true} examineValidityDates: ${AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES:-true} ``` * `trustStoreFilePath`: The path to the trust store file to use. It must not be null. * `trustStoreFilePassword`: The PIN to use to access the contents of the trust store. It may be null if no PIN is required. * `trustStoreFileFormat`: The format to use for the trust store. (Example :- JKS, PKCS12). * `verifyHostname`: Controls using TrustAllSSLSocketVerifier vs HostNameSSLSocketVerifier. In case the certificate contains cn=hostname of the Ldap Server set it to true. * `examineValidityDates`: Indicates whether to reject certificates if the current time is outside the validity window for the certificate. ### Authorizer Configuration This configuration controls the authorizer for OpenMetadata: ```yaml theme={null} authorizerConfiguration: adminPrincipals: ${AUTHORIZER_ADMIN_PRINCIPALS:-[admin]} principalDomain: ${AUTHORIZER_PRINCIPAL_DOMAIN:-"open-metadata.org"} ``` For the Ldap we need to set: * `adminPrincipals`: This is the list of admin Principal for the OpenMetadata , if mail in ldap is [example@open-metadata.org](mailto:example@open-metadata.org), then if we want this user to be admin in the OM, we should add 'example', in this list. * `principalDomain`: Company Domain. ## Configure Ingestion Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](/deployment/security/jwt-troubleshooting) to ensure seamless authentication. # Ldap Authentication for Docker | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/security/ldap/docker Integrate LDAP-based authentication in Docker for centralized credential management and user access across containerized applications. # Ldap Authentication for Docker To enable LDAP for docker deployment, there are a couple of files/certificates which are required to carry out the process. With the help of this documentation, we can provide those files/certificates to the docker container to use. To enable security for the Docker deployment, follow the next steps: ## Ways to configure LDAP using docker * #### [**Using Volumes**](#configure-using-volumes) * #### [**Extending docker image**](#extend-the-openmetadata-server-docker-image) ## Configure Using Volumes In `docker/docker-compose-quickstart/docker-compose.yml` file configure the volumes based on the `truststoreConfigType` **NO NEED TO ADD VOLUMES IF** `truststoreConfigType` **IS** `TrustAll` **OR** `HostName`. ### **Using JVMDefault** For docker container to access cacerts, copy the cacerts to `docker/ldap/config` and add the path in volumes. ```shell theme={null} volumes: - docker/ldap/config/cacerts:/usr/lib/jvm/java-17-openjdk/lib/security/cacerts ``` ### **Using CustomTrustStore** For docker container to access your truststore, copy the truststore to `docker/ldap/config` and add the path in volumes. ```shell theme={null} volumes: - docker/ldap/config/{YOUR_TRUSTSTORE}:/opt/openmetadata/ldap/truststore/{YOUR_TRUSTSTORE} ``` ## Extend the OpenMetadata server docker image Create a docker file and add the following details based on the `truststoreConfigType`. **NO NEED TO CREATE THIS FILE IF** `truststoreConfigType` **IS** `TrustAll` **OR** `HostName`. ### **Using JVMDefault** For docker container to access cacerts, copy the cacerts to `docker/ldap/config` as shown below. ```shell theme={null} FROM docker.open-metadata.org/openmetadata/server:2.0.1 COPY docker/ldap/config/cacerts /usr/lib/jvm/java-17-openjdk/lib/security/cacerts ``` ### **Using CustomTrustStore** For docker container to access your truststore, copy the truststore to `docker/ldap/config` as shown below. ```shell theme={null} FROM docker.open-metadata.org/openmetadata/server:2.0.1 COPY docker/ldap/config/{YOUR_TRUSTSTORE} /opt/openmetadata/ldap/truststore/{YOUR_TRUSTSTORE} ``` Run the following command from OpenMetadata root directory to create an image: ```text theme={null} docker build -f {DOCKER_FILE_PATH} -t {DOCKER_NAME}:{TAG} . ``` **NOTE:** After the image is created, in `docker/docker-compose-quickstart/docker-compose.yml` file, under openmetadata-server service replace the image name with the above created docker image. ```shell theme={null} image: {DOCKER_NAME}:{TAG} ``` ## Create an .env file Create an openmetadata\_ldap.env file and add the following contents as an example. Use the information generated when setting up the account. Based on the different `truststoreConfigType`, we have following different `trustStoreConfig`. ### Trust Store Config Type: TrustAll ```shell theme={null} AUTHENTICATION_PROVIDER=ldap AUTHENTICATION_LDAP_HOST={HOST} AUTHENTICATION_LDAP_PORT={PORT} AUTHENTICATION_LOOKUP_ADMIN_DN={ADMIN_DN} AUTHENTICATION_LOOKUP_ADMIN_PWD={ADMIN_DN_PASSWORD} AUTHENTICATION_USER_LOOKUP_BASEDN={USER_DN} AUTHENTICATION_USER_MAIL_ATTR={MAIL_ATTRIBUTE} AUTHENTICATION_LDAP_POOL_SIZE=3 AUTHENTICATION_LDAP_SSL_ENABLED=true AUTHENTICATION_LDAP_TRUSTSTORE_TYPE=TrustAll AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES=true ``` ### Trust Store Config Type: JVMDefault ```shell theme={null} AUTHENTICATION_PROVIDER=ldap AUTHENTICATION_LDAP_HOST={HOST} AUTHENTICATION_LDAP_PORT={PORT} AUTHENTICATION_LOOKUP_ADMIN_DN={ADMIN_DN} AUTHENTICATION_LOOKUP_ADMIN_PWD={ADMIN_DN_PASSWORD} AUTHENTICATION_USER_LOOKUP_BASEDN={USER_DN} AUTHENTICATION_USER_MAIL_ATTR={MAIL_ATTRIBUTE} AUTHENTICATION_LDAP_POOL_SIZE=3 AUTHENTICATION_LDAP_SSL_ENABLED=true AUTHENTICATION_LDAP_TRUSTSTORE_TYPE=TrustAll AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST=true ``` ### Trust Store Config Type: HostName ```shell theme={null} AUTHENTICATION_PROVIDER=ldap AUTHENTICATION_LDAP_HOST={HOST} AUTHENTICATION_LDAP_PORT={PORT} AUTHENTICATION_LOOKUP_ADMIN_DN={ADMIN_DN} AUTHENTICATION_LOOKUP_ADMIN_PWD={ADMIN_DN_PASSWORD} AUTHENTICATION_USER_LOOKUP_BASEDN={USER_DN} AUTHENTICATION_USER_MAIL_ATTR={MAIL_ATTRIBUTE} AUTHENTICATION_LDAP_POOL_SIZE=3 AUTHENTICATION_LDAP_SSL_ENABLED=true AUTHENTICATION_LDAP_TRUSTSTORE_TYPE=TrustAll AUTHENTICATION_LDAP_ALLOW_WILDCARDS=false AUTHENTICATION_LDAP_ALLOWED_HOSTNAMES={[ACCEPTABLE_HOSTNAMES]} ``` ### Trust Store Config Type: CustomTrustStore ```shell theme={null} AUTHENTICATION_PROVIDER=ldap AUTHENTICATION_LDAP_HOST={HOST} AUTHENTICATION_LDAP_PORT={PORT} AUTHENTICATION_LOOKUP_ADMIN_DN={ADMIN_DN} AUTHENTICATION_LOOKUP_ADMIN_PWD={ADMIN_DN_PASSWORD} AUTHENTICATION_USER_LOOKUP_BASEDN={USER_DN} AUTHENTICATION_USER_MAIL_ATTR={MAIL_ATTRIBUTE} AUTHENTICATION_LDAP_POOL_SIZE=3 AUTHENTICATION_LDAP_SSL_ENABLED=true AUTHENTICATION_LDAP_TRUSTSTORE_TYPE=TrustAll AUTHENTICATION_LDAP_TRUSTSTORE_PATH={TRUSTSTORE_FILEPATH} AUTHENTICATION_LDAP_KEYSTORE_PASSWORD={TRUSTSTORE_PASSWORD} AUTHENTICATION_LDAP_SSL_KEY_FORMAT={FORMAT} # JKS, PKCS12 AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST=true AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES=true ``` ## Start Docker ```commandline theme={null} docker compose --env-file ~/openmetadata_ldap.env up -d ``` ## Configure Ingestion Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](/deployment/security/jwt-troubleshooting) to ensure seamless authentication. # Ldap Authentication for Kubernetes Source: https://docs.open-metadata.org/v2.0.x/deployment/security/ldap/kubernetes # LDAP Authentication for Kubernetes This guide outlines how to configure LDAP authentication for Kubernetes deployments of OpenMetadata. It includes details on required configurations, optional settings, and best practices to ensure secure and efficient authentication. ## Authentication Configuration ```yaml theme={null} Update the `openmetadata.yaml` file with the following settings to enable LDAP authentication: openmetadata: config: authorizer: initialAdmins: ["admin"] # Add admin users here principalDomain: "example.com" # Organization domain for principal matching authentication: provider: ldap publicKeys: - "https:///api/v1/system/config/jwks" # Replace with your domain authority: "https://" # Replace with your domain enableSelfSignup: false ldapConfiguration: host: "ldap.example.com" # Replace with your LDAP server hostname port: 636 # Use 636 for secure LDAP (LDAPS) or 389 for standard LDAP dnAdminPrincipal: "cn=admin,dc=example,dc=com" dnAdminPassword: secretRef: ldap-admin-secret secretKey: openmetadata-ldap-secret userBaseDN: "ou=users,dc=example,dc=com" # Base DN for LDAP users mailAttributeName: "email" # Attribute for email in the LDAP schema sslEnabled: true # Enable SSL for secure LDAP truststoreConfigType: "TrustAll" # Trust store type (options: TrustAll, JVMDefault, HostName, CustomTrustStore) trustStoreConfig: trustAllConfig: examineValidityDates: true # Reject certificates outside the validity window jwtTokenConfiguration: enabled: true # Enable JWT tokens for secure communication # File Path on Airflow Container rsapublicKeyFilePath: "./conf/public_key.der" # File Path on Airflow Container rsaprivateKeyFilePath: "./conf/private_key.der" ``` ## Mandatory Fields for LDAP Configuration * **provider**: Set to `ldap` for enabling LDAP authentication. * **publicKeys**: Provide the public key URL in the format `{http|https}://{your_domain}:{port}/api/v1/system/config/jwks`. * **authority**: Specify your domain (e.g., `your_domain`). * **enableSelfSignup**: Set to `false` for LDAP. ## Key LDAP Fields * **host**: Hostname of the LDAP server (e.g., `localhost`). * **port**: Port of the LDAP server (e.g., `10636`). * **dnAdminPrincipal**: The Distinguished Name (DN) of the admin principal (e.g., `cn=admin,dc=example,dc=com`). * **dnAdminPassword**: Password for the admin principal. * **userBaseDN**: Base DN for user lookups (e.g., `ou=people,dc=example,dc=com`). ## Optional Advanced Configuration * **maxPoolSize**: Maximum connection pool size. * **sslEnabled**: Set to `true` to enable SSL connections to the LDAP server. * **truststoreConfigType**: Determines the type of trust store to use (`CustomTrustStore`, `HostName`, `JVMDefault`, or `TrustAll`). ## Example: TrustStore Configurations ### TrustAll Configuration ```yaml theme={null} openmetadata: config: ... authentication: ... ldapConfiguration: ... truststoreConfigType: TrustAll trustStoreConfig: examineValidityDates: true ... ``` ### JVMDefault Configuration ```yaml theme={null} openmetadata: config: ... authentication: ... ldapConfiguration: ... truststoreConfigType: JVMDefault trustStoreConfig: jvmDefaultConfig: verifyHostname: true ... ``` ### HostName Configuration ```yaml theme={null} openmetadata: config: ... authentication: ... ldapConfiguration: ... truststoreConfigType: HostName trustStoreConfig: hostNameConfig: allowWildCards: false acceptableHostNames: [localhost] ... ``` ### CustomTrustStore Configuration ```yaml theme={null} openmetadata: config: ... authentication: ... ldapConfiguration: ... trusttoreConfigType: CustomTrustStore trustStoreConfig: customTrustManagerConfig: trustStoreFilePath: /path/to/truststore.jks trustStoreFilePassword: secretRef: "" secretKey: "" trustStoreFileFormat: JKS verifyHostname: true examineValidityDates: true ... ``` ## Configure Ingestion Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](/deployment/security/jwt-troubleshooting) to ensure seamless authentication. # OIDC Based Authentication | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/security/oidc Enable OIDC-based authentication to unify identity providers using open standards, access tokens, and flexible integration patterns. # Setting up Any Oidc Provider Security requirements for your **production** environment: * **DELETE** the admin default account shipped by OM in case you had [Basic Authentication](/v2.0.x/deployment/security/basic-auth) enabled before configuring the authentication with Auth0 SSO. * **UPDATE** the Private / Public keys used for the [JWT Tokens](/v2.0.x/deployment/security/enable-jwt-tokens). The keys we provide by default are aimed only for quickstart and testing purposes. They should NEVER be used in a production installation. This guide provides instructions on setting up OpenID Connect (OIDC) configuration for your application. OpenID Connect is a simple identity layer built on top of the OAuth 2.0 protocol that allows clients to verify the identity of the end-user. Below configurations are universally applicable to all SSO provider like Google, Auth0, Okta, Keycloak, etc. OpenMetadata sessions are currently stored **in-memory**, which may cause issues when using **OIDC authentication** in a multi-replica setup. * If you are experiencing **authentication failures with "Missing state parameter" errors**, enabling **sticky sessions** can serve as a temporary workaround. Below are the configuration types to set up the OIDC Authentication with a Confidential Client type: ```yaml theme={null} authenticationConfiguration: clientType: ${AUTHENTICATION_CLIENT_TYPE:-confidential} publicKeyUrls: ${AUTHENTICATION_PUBLIC_KEYS:-[http://localhost:8585/api/v1/system/config/jwks]} oidcConfiguration: id: ${OIDC_CLIENT_ID:-""} type: ${OIDC_TYPE:-""} # google, azure etc. secret: ${OIDC_CLIENT_SECRET:-""} scope: ${OIDC_SCOPE:-"openid email profile"} discoveryUri: ${OIDC_DISCOVERY_URI:-""} useNonce: ${OIDC_USE_NONCE:-true} preferredJwsAlgorithm: ${OIDC_PREFERRED_JWS:-"RS256"} responseType: ${OIDC_RESPONSE_TYPE:-"code"} disablePkce: ${OIDC_DISABLE_PKCE:-true} callbackUrl: ${OIDC_CALLBACK:-"http://localhost:8585/callback"} serverUrl: ${OIDC_SERVER_URL:-"http://localhost:8585"} clientAuthenticationMethod: ${OIDC_CLIENT_AUTH_METHOD:-"client_secret_post"} tenant: ${OIDC_TENANT:-""} maxClockSkew: ${OIDC_MAX_CLOCK_SKEW:-""} customParams: ${OIDC_CUSTOM_PARAMS:-} ``` Check the more information about environment variable [here](/v2.0.x/deployment/security/configuration-parameters). # Okta SSO | OpenMetadata Authentication Integration Source: https://docs.open-metadata.org/v2.0.x/deployment/security/okta Configure Okta as your authentication provider to support secure login, identity federation, and enterprise-grade access control. # Okta SSO Follow the sections in this guide to set up Okta SSO. Security requirements for your **production** environment: * **DELETE** the admin default account shipped by OM in case you had [Basic Authentication](/v2.0.x/deployment/security/basic-auth) enabled before configuring the authentication with Okta SSO. * **UPDATE** the Private / Public keys used for the [JWT Tokens](/v2.0.x/deployment/security/enable-jwt-tokens). The keys we provide by default are aimed only for quickstart and testing purposes. They should NEVER be used in a production installation. ## Create Server Credentials This document will explain how to create an Okta app and configure it for OAuth. This will generate the information required for Single Sign On with Okta. ### Step 1: Create an Okta Account * Go to [Create Okta Account](https://developer.okta.com/signup/). * Provide the required input and click on Sign Up. * Else you can continue with Google or GitHub. ### Step 2: Create the OIDC App Integration. * Once done with **Signup/Sign** in, you will be redirected to the **Getting Started** page in Okta. create-oidc-app-integration * Click on **Applications -> Applications** in the left navigation panel. click-applications * Click on the **Create App Integration** button. create-app-integration ## Choose Your Authentication Flow After creating the account, choose the authentication flow you want to use: * [Implicit Flow](/v2.0.x/deployment/security/okta/implicit-flow) (Public) * [Auth Code Flow](/v2.0.x/deployment/security/okta/auth-code-flow) (Confidential) - **SPA (Single Page Application):** This type is designed for implicit flows. In this case, providing both the client ID and client secret will result in a failure because the implicit flow only requires the client ID for authentication. - **Web:** This type is intended for confidential clients. If you select this option, you must provide both the client ID and client secret. Simply passing the client ID will cause the authorization process to fail, as the Authorization Code flow requires both credentials for successful authentication. The [OIDC Authorization Code Flow](/v2.0.x/deployment/security/oidc) is used in this case, where the client secret is required to securely exchange the authorization code for tokens. ### Recommendation: * Use the **Web** type for confidential clients that require both a client ID and secret. * Use the **SPA** type for applications using implicit flows where only a client ID is needed. ## Configure Ingestion Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](/deployment/security/jwt-troubleshooting) to ensure seamless authentication. # OneLogin SSO | OpenMetadata Authentication Setup Source: https://docs.open-metadata.org/v2.0.x/deployment/security/one-login Configure OneLogin as your authentication source to manage user roles, sessions, and tokens across secure deployments. # OneLogin SSO Follow the sections in this guide to set up OneLogin SSO. Security requirements for your **production** environment: * **DELETE** the admin default account shipped by OM in case you had [Basic Authentication](/v2.0.x/deployment/security/basic-auth) enabled before configuring the authentication with OneLogin SSO. * **UPDATE** the Private / Public keys used for the [JWT Tokens](/v2.0.x/deployment/security/enable-jwt-tokens). The keys we provide by default are aimed only for quickstart and testing purposes. They should NEVER be used in a production installation. ## Create Server Credentials ### Step 1: Configure a new Application * Login to [OneLogin](https://www.onelogin.com/) as an administrator and click on Applications create-account * Click on the `Add App` button and search for `openid connect` * Select the `OpenId Connect (OIDC)` app create-account * Change the Display Name of the app to `Open Metadata` and click `Save` create-account * Configure the login Url (`http(s):///signin`) and redirect URI (`http(s):///callback`) as shown below create-account * Configure the users in the organization that can access OpenMetadata app by clicking on the `Users` create-account * Click on "SSO" and select `None (PKCE)` for Token Endpoint. create-account ### Step 2: Where to find the Credentials * Go to "SSO" and copy the Client ID create-account * Copy the Issuer URL After the applying these steps, you can update the configuration of your deployment: Configure OneLogin SSO for your Docker Deployment. Configure OneLogin SSO for your Bare Metal Deployment. Configure OneLogin SSO for your Kubernetes Deployment. ## Configure Ingestion Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](/deployment/security/jwt-troubleshooting) to ensure seamless authentication. # OneLogin SSO for Bare Metal | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/security/one-login/bare-metal Use OneLogin authentication on bare-metal deployments to ensure secure, token-driven access without relying on cloud-native services. # OneLogin SSO for Bare Metal ## Update conf/openmetadata.yaml Once the `Client Id` is generated, add the `Client Id` in `openmetadata.yaml` file in `client_id` field. Update the providerName config to the name you want to display in the `Sign In` button in the UI. For example, with the following configuration with `providerName` set to `OneLogin`, the users will see `Sign In with OneLogin SSO` in the `Sign In` page of the OpenMetadata UI. ```yaml theme={null} authenticationConfiguration: provider: "custom-oidc" providerName: "OneLogin" publicKeyUrls: - "{IssuerUrl}/certs" - "{your domain}/api/v1/system/config/jwks" # Update with your Domain and Make sure this "/api/v1/system/config/jwks" is always configured to enable JWT tokens authority: "{IssuerUrl}" clientId: "{client id}" callbackUrl: "http://localhost:8585/callback" ``` Then, * Update `authorizerConfiguration` to add login names of the admin users in `adminPrincipals` section as shown below. * Update the `principalDomain` to your company domain name. ```yaml theme={null} authorizerConfiguration: className: "org.openmetadata.service.security.DefaultAuthorizer" # JWT Filter containerRequestFilter: "org.openmetadata.service.security.JwtFilter" adminPrincipals: - "user1" - "user2" principalDomain: "open-metadata.org" ``` ## Configure Ingestion Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](/deployment/security/jwt-troubleshooting) to ensure seamless authentication. # One Login SSO for Docker | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/security/one-login/docker Deploy OneLogin authentication in Docker for fast, secure user validation and centralized identity management in container services. # One Login SSO for Docker To enable security for the Docker deployment, follow the next steps: ## 1. Create an .env file Create an `openmetadata_onelogin.env` file and add the following contents as an example. Use the information generated when setting up the account. ```shell theme={null} # OpenMetadata Server Authentication Configuration AUTHORIZER_CLASS_NAME=org.openmetadata.service.security.DefaultAuthorizer AUTHORIZER_REQUEST_FILTER=org.openmetadata.service.security.JwtFilter AUTHORIZER_ADMIN_PRINCIPALS=[admin] # Your `name` from name@domain.com AUTHORIZER_PRINCIPAL_DOMAIN=open-metadata.org # Update with your domain AUTHENTICATION_PROVIDER=custom-oidc AUTHENTICATION_PUBLIC_KEYS=[{public key url}, https://{your domain}/api/v1/system/config/jwks] # Update with your Domain and Make sure this "/api/v1/system/config/jwks" is always configured to enable JWT tokens AUTHENTICATION_AUTHORITY={issuer url} # Update with your Issuer URL AUTHENTICATION_CLIENT_ID={Client ID} # Update with your Client ID AUTHENTICATION_CALLBACK_URL=http://localhost:8585/callback ``` ## 2. Start Docker ```commandline theme={null} docker compose --env-file ~/openmetadata_onelogin.env up -d ``` ## Configure Ingestion Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](/deployment/security/jwt-troubleshooting) to ensure seamless authentication. # OneLogin SSO for Kubernetes | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/security/one-login/kubernetes Integrate OneLogin into Kubernetes clusters to securely manage authentication, access tokens, and identity roles at runtime. # One Login SSO for Kubernetes Check the Helm information [here](https://artifacthub.io/packages/search?repo=open-metadata). Once the `Client Id` is generated, see the snippet below for an example of where to place the client id value and update the authorizer configurations in the `values.yaml`. ```yaml theme={null} openmetadata: config: authorizer: className: "org.openmetadata.service.security.DefaultAuthorizer" # JWT Filter containerRequestFilter: "org.openmetadata.service.security.JwtFilter" initialAdmins: - "suresh" principalDomain: "open-metadata.org" authentication: provider: "custom-oidc" publicKeys: - "{your domain}/api/v1/system/config/jwks" # Update with your Domain and Make sure this "/api/v1/system/config/jwks" is always configured to enable JWT tokens - "{IssuerUrl}/certs" authority: "{IssuerUrl}" clientId: "{client id}" callbackUrl: "http://localhost:8585/callback" ``` ## Configure Ingestion Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](/deployment/security/jwt-troubleshooting) to ensure seamless authentication. # SAML SSO Source: https://docs.open-metadata.org/v2.0.x/deployment/security/saml # SAML SSO Security requirements for your **production** environment: * **DELETE** the admin default account shipped by OM. * **UPDATE** the Private / Public keys used for the [JWT Tokens](/v2.0.x/deployment/security/enable-jwt-tokens) in case it is enabled. ## Configuring Identity Provider and Service Provider ### Identity Provide (IDP) Configuration * Every IDP will have the following information 1. EntityId/Authority -> Same as IDP Openmetadata has an Entity Id 2. SignOn Url -> Service Provider SignOn Url 3. X509 Certificate -> In case the SP expects (wantAuthnRequestSigned) then provide certificate for validating. 4. Authority Url -> We just need to update the domain `localhost`. 5. NameID: This is sent as part of request and is provided by the IDP. Every IDP provides this information, we can download the XML Metadata and configure the OM taking the values from the XML. ### Service Provider (SP) Configuration * Openmetadata is the service provider, we just update the `localhost` to the hosted URI. 1. EntityId/Authority -> Normally a Url providing info about the provider. 2. SignOn Url -> Url to be used for signing purpose. 3. X509 Certificate -> In case the SP expects a signed response from IDP, the IDP can be configured with Signing Certificate given by SP. 4. Private Key -> In case SP expects a encrypted response from the IDP , the IDP can be configured with SPs public key for encryption and the Private Key can be used for SP for decrypting. When configuring the Private Key for the Service Provider, ensure you use the actual key content enclosed within the `-----BEGIN PRIVATE KEY-----` and `-----END PRIVATE KEY-----` block. Avoid using the Base64-encoded format of the key, as this is not the expected value. ```yaml theme={null} idp: entityId: ${SAML_IDP_ENTITY_ID:-""} ssoLoginUrl: ${SAML_IDP_SSO_LOGIN_URL:-"-----END PRIVATE KEY-----"} authorityUrl: ${SAML_AUTHORITY_URL:-"http://localhost:8585/api/v1/saml/login"} nameId: ${SAML_IDP_NAME_ID:-"urn:oasis:names:tc:SAML:2.0:nameid-format:emailAddress"} ``` To add a private key, you need to include it in the keystore and update the configuration details accordingly [here](https://github.com/open-metadata/OpenMetadata/blob/main/conf/openmetadata.yaml#L219). ```yaml theme={null} security: keyStoreFilePath: ${SAML_KEYSTORE_FILE_PATH:-"/path/to/keystore.jks"} keyStoreAlias: ${SAML_KEYSTORE_ALIAS:-"myKeystoreAlias"} keyStorePassword: ${SAML_KEYSTORE_PASSWORD:-"myKeystorePassword"} ``` SP Metadata XML is available at `http://localhost:8585/api/v1/saml/acs`; `localhost` needs to be updated with the correct URI. ### Security Configuration Security Configuration controls the SP requirement for the Security related aspects. The SP can be configured to send signed or encrypted or both request , and in return can also expect signed or encrypted or both responses from the IDP. ## Setup JWT Configuration Jwt Configuration is mandatory for Saml SSO. * Follow the guide here for JWT Configuration [Enable JWT Token](/v2.0.x/deployment/security/enable-jwt-tokens). Security requirements for your **production** environment: * **UPDATE** the Private / Public keys used for the [JWT Tokens](/v2.0.x/deployment/security/enable-jwt-tokens) the ones shipped with OM are for POC only. More specific details on different IDPs can be found below: Configure AWS as IDP. Configure AWS as IDP. ## Configure Ingestion Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](/deployment/security/jwt-troubleshooting) to ensure seamless authentication. # SAML AWS SSO Source: https://docs.open-metadata.org/v2.0.x/deployment/security/saml/aws # SAML AWS SSO Follow the sections in this guide to set up AWS SSO using SAML. Security requirements for your **production** environment: * **DELETE** the admin default account shipped by OM. * **UPDATE** the Private / Public keys used for the [JWT Tokens](/v2.0.x/deployment/security/enable-jwt-tokens) in case it is enabled. ## Create OpenMetadata application ### Step 1: Configure a new Application in AWS Console * Login to [AWS Console](https://aws.amazon.com/console/) as an administrator and search for IAM Identity Center. IAM-Identity-Center * Click on `Choose your identity source` and configure as per security requirements. identity-source * After identity source is set up successfully, goto step 2 and click on `Manage Access to application` and add all the required users who need access to application. manage-access * Click on `Set up Identity Center enabled applications`, and click `Add application`, and select `Add custom SAML 2.0 application`. saml-application * Set Display Name to `OpenMetadata` , and download the metadata xml file and save it someplace safe, it is needed to setup OM Server metadata-xml * Click on `Manage assignments to your cloud applications` and select `OpenMetadata` from list of applications. * Click on `Actions` and select `Edit Configurations` from list. Populate the shown values replacing `localhost:8585` with your `{domain}:{port}` and Submit. edit-configuration * Click on `Actions` again and select `Edit Attribute Mapping` from list. Populate the values as shown below and submit edit-attribute ### Step 2: Setup `OpenMetadata Server` * Open the downloaded metadata xml file, and populate the following properties in `openmetadata.yml` ```yaml theme={null} samlConfiguration: debugMode: ${SAML_DEBUG_MODE:-false} idp: entityId: ${SAML_IDP_ENTITY_ID:-"https://mocksaml.com/api/saml/sso"} ssoLoginUrl: ${SAML_IDP_SSO_LOGIN_URL:-"https://saml.example.com/entityid"} idpX509Certificate: ${SAML_IDP_CERTIFICATE:-""} authorityUrl: ${SAML_AUTHORITY_URL:-"http://localhost:8585/api/v1/saml/login"} nameId: ${SAML_IDP_NAME_ID:-"urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress"} sp: entityId: ${SAML_SP_ENTITY_ID:-"http://localhost:8585/api/v1/saml/acs"} acs: ${SAML_SP_ACS:-"http://localhost:8585/api/v1/saml/acs"} spX509Certificate: ${SAML_SP_CERTIFICATE:-""} callback: ${SAML_SP_CALLBACK:-"http://localhost:8585/saml/callback"} security: strictMode: ${SAML_STRICT_MODE:-false} tokenValidity: ${SAML_SP_TOKEN_VALIDITY:-"3600"} sendEncryptedNameId: ${SAML_SEND_ENCRYPTED_NAME_ID:-false} sendSignedAuthRequest: ${SAML_SEND_SIGNED_AUTH_REQUEST:-false} signSpMetadata: ${SAML_SIGNED_SP_METADATA:-false} wantMessagesSigned: ${SAML_WANT_MESSAGE_SIGNED:-false} wantAssertionsSigned: ${SAML_WANT_ASSERTION_SIGNED:-false} wantAssertionEncrypted: ${SAML_WANT_ASSERTION_ENCRYPTED:-false} wantNameIdEncrypted: ${SAML_WANT_NAME_ID_ENCRYPTED:-false} keyStoreFilePath: ${SAML_KEYSTORE_FILE_PATH:-""} keyStoreAlias: ${SAML_KEYSTORE_ALIAS:-""} keyStorePassword: ${SAML_KEYSTORE_PASSWORD:-""} ``` * Populate the above config from xml metadata populate-metadata * IDP Config `entityID` -> Populate it from Metadata XML Entity ID `HTTP-Redirect SSO Login URL` -> always select HTTP-Redirect Url for SSO Login Url `X509 Certificate` -> This is also available in the IDP XML. `NameIDFormat` -> from MetadataXML NameIDFormat `authorityUrl` -> set as /://:/api/v1/saml/login * SP Config `entityId` -> -> set as /://:/api/v1/saml/acs `acs` -> Assertion Consumer Url , set as /://:/api/v1/saml/acs `spX509Certificate` -> set to your X509 Signing Key `callback` -> set as /:///api/v1/saml/callback * Security Parameters can be configured in case we want to have signed or encrypted or both assertions. In any case we decided to use above config for security then it is mandatory to provide keystore config, from where the system can load the signing certificate or Private Key for encryption. ### Step 3: Setup JWT Configuration * Follow the guide here for JWT Configuration [Enable JWT Token](/v2.0.x/deployment/security/enable-jwt-tokens). Security requirements for your **production** environment: * **UPDATE** the Private / Public keys used for the [JWT Tokens](/v2.0.x/deployment/security/enable-jwt-tokens) the ones shipped with OM are for POC only. ### Step 4: Start the server * Set up for SAML is done, you should be routed to your IDP on trying to Sign-in. # SAML AZURE SSO Source: https://docs.open-metadata.org/v2.0.x/deployment/security/saml/azure # SAML AZURE SSO Follow the sections in this guide to set up Azure SSO using SAML. Security requirements for your **production** environment: * **DELETE** the admin default account shipped by OM. * **UPDATE** the Private / Public keys used for the [JWT Tokens](/v2.0.x/deployment/security/enable-jwt-tokens) in case it is enabled. ## Key Notes on SAML Configuration 1. **Set `AUTHENTICATION_PROVIDER` to `saml` (lowercase):** Ensure the `AUTHENTICATION_PROVIDER` field in your environment variables is explicitly set to `saml` for SAML authentication to function correctly. Without this, SAML integration will not work. 2. **Routing to IDP:** Users will only be routed to the IDP upon sign-in if `AUTHENTICATION_PROVIDER` is set to `saml`. ## Create OpenMetadata application ### Step 1: Configure a new Application in Microsoft Entra ID * Login to [Azure Portal](https://portal.azure.com) as an administrator and search for Microsoft Entra ID. EnterpriseApplications * Click on `Enterprise Applications` and then `+ New Application` . new-application * After that a new window will appear with different applications, click on `Create your own application`. create-own-application * Give your application a name and select `Integrate any other application you don't find in the gallery` and then click `Create`. name-application-create * Once you have the application created, open the app from list , and then click on `Single Sign-On` and then `SAML`. saml-create-single-sign-On * Edit `Basic SAML Configuration` and populate the values as shown below for `EntityId` and `Assertion Consumer Service Url`. These value should match the one configured with Openmetadata Server side for `samlConfiguration.sp.entityId` and `samlConfiguration.sp.acs` respectively. After this click `Save`. edit-basic-saml-configuration * Click on `Attributes and Claims` and click on the `Required Claim (NameId)`. edit-claims * You will see the values as below image, we need to set the value `Source Attribute` to a user mail value claim from the IDP. Click on `Edit` and then select the `Source Attribute` as `user.mail` or `user.userprincipalname` (in some cases this is also a mail) and then click `Save`. edit-claim-value * To Confirm the claim value we can navigate to user page and check the value of the user. In my case as you can see User Princpal Name is a my mail which i want to use for Openmetadata , so for me `user.userprincipalname` would be correct claim. user-claim-value Security requirements for your **production** environment: * You must always communicate via signed Request for both request from SP to IDP and response from IDP to SP. * To do so we need to add SP certificate to IDP , so that IDP can validate the signed Auth Request coming from SP. * Generate the certificate using below command and then upload the certificate to IDP. ```shell theme={null} openssl req -new -x509 -days 365 -nodes -sha256 -out saml.crt -keyout saml.pem openssl x509 -in saml.crt -out samlCER.cer -outform DER ``` * Under `Single Sign-On` you will see SAML Certificates, click on `Verification Certificates`. verification-certificate * You can then check the `Require Verification Certificates` and import the certification with .cer format we generated previously. ### Step 2: Setup `OpenMetadata Server` * Open the downloaded metadata xml file, and populate the following properties in `openmetadata.yml` ```yaml theme={null} authenticationConfiguration: provider: ${AUTHENTICATION_PROVIDER:-saml} samlConfiguration: debugMode: ${SAML_DEBUG_MODE:-false} idp: entityId: ${SAML_IDP_ENTITY_ID:-"https://mocksaml.com/api/saml/sso"} ssoLoginUrl: ${SAML_IDP_SSO_LOGIN_URL:-"https://saml.example.com/entityid"} idpX509Certificate: ${SAML_IDP_CERTIFICATE:-""} #Pass the certificate as a string authorityUrl: ${SAML_AUTHORITY_URL:-"http://localhost:8585/api/v1/saml/login"} nameId: ${SAML_IDP_NAME_ID:-"urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress"} sp: entityId: ${SAML_SP_ENTITY_ID:-"http://localhost:8585/api/v1/saml/acs"} acs: ${SAML_SP_ACS:-"http://localhost:8585/api/v1/saml/acs"} spX509Certificate: ${SAML_SP_CERTIFICATE:-""} callback: ${SAML_SP_CALLBACK:-"http://localhost:8585/saml/callback"} security: strictMode: ${SAML_STRICT_MODE:-false} tokenValidity: ${SAML_SP_TOKEN_VALIDITY:-"3600"} sendEncryptedNameId: ${SAML_SEND_ENCRYPTED_NAME_ID:-false} sendSignedAuthRequest: ${SAML_SEND_SIGNED_AUTH_REQUEST:-false} signSpMetadata: ${SAML_SIGNED_SP_METADATA:-false} wantMessagesSigned: ${SAML_WANT_MESSAGE_SIGNED:-false} wantAssertionsSigned: ${SAML_WANT_ASSERTION_SIGNED:-false} wantAssertionEncrypted: ${SAML_WANT_ASSERTION_ENCRYPTED:-false} wantNameIdEncrypted: ${SAML_WANT_NAME_ID_ENCRYPTED:-false} keyStoreFilePath: ${SAML_KEYSTORE_FILE_PATH:-""} keyStoreAlias: ${SAML_KEYSTORE_ALIAS:-""} keyStorePassword: ${SAML_KEYSTORE_PASSWORD:-""} ``` * Populate the above config from xml metadata populate-metadata * IDP Config `entityID` -> Populate it from Metadata XML Entity ID `HTTP-Redirect SSO Login URL` -> always select HTTP-Redirect Url for SSO Login Url `X509 Certificate` -> This is also available in the IDP XML. `NameIDFormat` -> from MetadataXML NameIDFormat `authorityUrl` -> set as /://:/api/v1/saml/login * SP Config `entityId` -> -> set as /://:/api/v1/saml/acs `acs` -> Assertion Consumer Url , set as /://:/api/v1/saml/acs `spX509Certificate` -> set to your X509 Signing Key `callback` -> set as /:///api/v1/saml/callback - Security Parameters can be configured in case we want to have signed or encrypted or both assertions. In any case we decided to use above config for security then it is mandatory to provide keystore config, from where the system can load the signing certificate or Private Key for encryption. - For **production** environment , it is always suggested to keep these true ```yaml theme={null} sendSignedAuthRequest: ${SAML_SEND_SIGNED_AUTH_REQUEST:-true} wantMessagesSigned: ${SAML_WANT_MESSAGE_SIGNED:-true} wantAssertionsSigned: ${SAML_WANT_ASSERTION_SIGNED:-true} ``` ### Step 3: Setup JWT Configuration * Follow the guide here for JWT Configuration [Enable JWT Token](/v2.0.x/deployment/security/enable-jwt-tokens). Security requirements for your **production** environment: * **UPDATE** the Private / Public keys used for the [JWT Tokens](/v2.0.x/deployment/security/enable-jwt-tokens) the ones shipped with OM are for POC only. ### Step 4: Start the server * Start the OpenMetadata server. With `AUTHENTICATION_PROVIDER` set to saml, you should be routed to the IDP upon sign-in. # Enable Semantic Search | OpenMetadata Deployment Guide Source: https://docs.open-metadata.org/v2.0.x/deployment/semantic-search Configure semantic search with vector embeddings in OpenMetadata to enable natural language queries against your metadata catalog using OpenSearch or Elasticsearch. # Enable Semantic Search ## Prerequisites * **OpenSearch or Elasticsearch** as your search backend * An external embedding provider: **OpenAI**, **AWS Bedrock**, **Google**, or **DJL** for HuggingFace models. * Network access from the OpenMetadata server to the embedding provider API (unless using DJL) ## Overview Semantic Search enhances OpenMetadata's search capabilities by using **vector embeddings** to understand the meaning behind queries, rather than relying solely on keyword matching. This means users and AI agents can search using natural language \-- for example, *"tables with customer demographics and purchase history"* -- and get meaningful results even if those exact words don't appear in the metadata. Semantic Search supports both **OpenSearch** and **Elasticsearch** as the search backend. The OpenSearch-specific hybrid search pipeline (which blends keyword and semantic scores server-side) is not available on Elasticsearch. Core vector/KNN search works the same on both backends. Semantic Search also powers the [Semantic Search MCP tool](/v2.0.x/how-to-guides/mcp/semantic-search), enabling AI assistants connected via the Model Context Protocol to perform natural language queries against your metadata catalog. ## How It Works For each entity, a structured text representation is constructed from its metadata -- including name, description, entity type, tags, glossary terms, owners, and other relevant fields. The text is sent to the configured embedding provider to generate a numerical vector (embedding), which is stored in a dedicated `dataAssetEmbeddings` index (`_dataAssetEmbeddings` if you've set a cluster alias) using the HNSW algorithm with cosine similarity. At query time, the search text is also embedded and a KNN (K-Nearest Neighbor) similarity search finds the most relevant results. Embeddings follow the same lifecycle as the entities themselves. When entities are created, updated, deleted, or restored, their embeddings are automatically kept in sync using the same indexing strategies the platform already uses for search. No manual intervention is required after initial setup. ### Supported Entity Types `table`, `glossary`, `glossaryTerm`, `chart`, `dashboard`, `dashboardDataModel`, `database`, `databaseSchema`, `dataProduct`, `pipeline`, `mlmodel`, `metric`, `apiEndpoint`, `apiCollection`, `page`, `storedProcedure`, `searchIndex`, `topic` ## Configuration Semantic Search settings are split across two sections of `openmetadata.yaml`. The master switch stays under `elasticsearch.naturalLanguageSearch`, while the embedding provider, model, and credentials live under `llmConfiguration` -- the same section used for platform-wide LLM completions. All settings can be overridden with environment variables. `LLM_ENABLED` and `LLM_PROVIDER` configure the platform's chat completion client. They do not enable or select Semantic Search's vector embeddings -- that's controlled by `SEMANTIC_SEARCH_ENABLED` and `llmConfiguration.embeddings.provider` instead. ### Enable Semantic Search | Environment Variable | Default | Description | | ----------------------------------- | --------- | ------------------------------------------------------------------ | | `SEMANTIC_SEARCH_ENABLED` | `false` | Master switch to enable semantic search | | `EMBEDDING_PROVIDER` | `bedrock` | Embedding provider to use: `openai`, `bedrock`, `google`, or `djl` | | `MAX_CONCURRENT_EMBEDDING_REQUESTS` | `10` | Maximum number of concurrent calls to the embedding provider | ```yaml theme={null} elasticsearch: naturalLanguageSearch: semanticSearchEnabled: ${SEMANTIC_SEARCH_ENABLED:-false} llmConfiguration: embeddings: provider: ${EMBEDDING_PROVIDER:-bedrock} maxConcurrentRequests: ${MAX_CONCURRENT_EMBEDDING_REQUESTS:-10} ``` ### Embedding Providers Choose one of the following embedding providers and configure it accordingly. Supports both OpenAI and Azure OpenAI endpoints. Credentials live under `llmConfiguration.openai`, shared with the platform's chat completion config; the embedding model and dimension live under `llmConfiguration.embeddings.openai`. | Environment Variable | Default | Description | | ---------------------------- | ------------------------ | --------------------------------------------------------------------- | | `LLM_OPENAI_API_KEY` | `""` | Your OpenAI API key | | `LLM_OPENAI_ENDPOINT` | `""` | API endpoint. For Azure, use `https://your-resource.openai.azure.com` | | `LLM_OPENAI_DEPLOYMENT` | `""` | Deployment name (required for Azure OpenAI) | | `LLM_OPENAI_API_VERSION` | `2024-02-01` | API version (Azure OpenAI) | | `OPENAI_EMBEDDING_MODEL_ID` | `text-embedding-3-small` | Embedding model to use | | `OPENAI_EMBEDDING_DIMENSION` | `1536` | Embedding vector dimension | ```yaml theme={null} elasticsearch: naturalLanguageSearch: semanticSearchEnabled: true llmConfiguration: openai: apiKey: ${LLM_OPENAI_API_KEY:-""} endpoint: ${LLM_OPENAI_ENDPOINT:-""} deploymentName: ${LLM_OPENAI_DEPLOYMENT:-""} apiVersion: ${LLM_OPENAI_API_VERSION:-"2024-02-01"} embeddings: provider: openai openai: embeddingModelId: ${OPENAI_EMBEDDING_MODEL_ID:-"text-embedding-3-small"} embeddingDimension: ${OPENAI_EMBEDDING_DIMENSION:-1536} ``` Uses AWS Bedrock for embedding generation. AWS credentials live under `llmConfiguration.bedrock.awsConfig`, shared with the platform's chat completion config; the embedding model and dimension live under `llmConfiguration.embeddings.bedrock`. Static credentials always take priority: if `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are both set, OpenMetadata uses them regardless of `BEDROCK_AWS_IAM_AUTH_ENABLED`. To actually authenticate with the server's IAM role, leave both of those empty. | Environment Variable | Default | Description | | --------------------------------- | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `BEDROCK_AWS_IAM_AUTH_ENABLED` | `true` | Use the server's IAM role instead of static credentials. Only takes effect when `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` are both empty — a static key pair always overrides it | | `AWS_DEFAULT_REGION` | `""` | AWS region | | `AWS_ACCESS_KEY_ID` | `""` | AWS access key. Must be left empty for IAM auth to be used | | `AWS_SECRET_ACCESS_KEY` | `""` | AWS secret access key. Must be left empty for IAM auth to be used | | `AWS_SESSION_TOKEN` | `""` | AWS session token (only used alongside static access key/secret) | | `AWS_BEDROCK_EMBED_MODEL_ID` | `amazon.titan-embed-text-v2:0` | Bedrock embedding model ID | | `AWS_BEDROCK_EMBEDDING_DIMENSION` | `512` | Embedding vector dimension | ```yaml theme={null} elasticsearch: naturalLanguageSearch: semanticSearchEnabled: true llmConfiguration: bedrock: awsConfig: enabled: ${BEDROCK_AWS_IAM_AUTH_ENABLED:-true} region: ${AWS_DEFAULT_REGION:-""} accessKeyId: ${AWS_ACCESS_KEY_ID:-""} secretAccessKey: ${AWS_SECRET_ACCESS_KEY:-""} sessionToken: ${AWS_SESSION_TOKEN:-""} embeddings: provider: bedrock bedrock: embeddingModelId: ${AWS_BEDROCK_EMBED_MODEL_ID:-"amazon.titan-embed-text-v2:0"} embeddingDimension: ${AWS_BEDROCK_EMBEDDING_DIMENSION:-512} ``` Uses Google's Gemini embedding models. Credentials live under `llmConfiguration.google`, shared with the platform's chat completion config; the embedding model and dimension live under `llmConfiguration.embeddings.google`. | Environment Variable | Default | Description | | ---------------------------- | ---------------------- | -------------------------- | | `LLM_GOOGLE_API_KEY` | `""` | Your Google API key | | `GOOGLE_EMBEDDING_MODEL_ID` | `gemini-embedding-001` | Embedding model to use | | `GOOGLE_EMBEDDING_DIMENSION` | `768` | Embedding vector dimension | ```yaml theme={null} elasticsearch: naturalLanguageSearch: semanticSearchEnabled: true llmConfiguration: google: apiKey: ${LLM_GOOGLE_API_KEY:-""} embeddings: provider: google google: embeddingModelId: ${GOOGLE_EMBEDDING_MODEL_ID:-"gemini-embedding-001"} embeddingDimension: ${GOOGLE_EMBEDDING_DIMENSION:-768} ``` Uses [Deep Java Library](https://djl.ai/) to run embedding models locally. No external API calls or credentials required. DJL downloads and runs the HuggingFace model in your server directly. This will have an impact on the necessary resources depending on the chosen model. If you are resource constrainted, use external providers. The example model we provide is a rather small one that fits development/testing use cases. In case of choosing DJL, choose a model that fits your use case. | Environment Variable | Default | Description | | --------------------- | ------------------------------------------------------------------- | ---------------------------- | | `DJL_EMBEDDING_MODEL` | `ai.djl.huggingface.pytorch/sentence-transformers/all-MiniLM-L6-v2` | HuggingFace model identifier | The embedding dimension is auto-detected from the model at startup. The default model `all-MiniLM-L6-v2` produces 384-dimensional vectors. ```yaml theme={null} elasticsearch: naturalLanguageSearch: semanticSearchEnabled: true llmConfiguration: embeddings: provider: djl djl: embeddingModel: ${DJL_EMBEDDING_MODEL:-"ai.djl.huggingface.pytorch/sentence-transformers/all-MiniLM-L6-v2"} ``` ## Docker Deployment To enable Semantic Search in a Docker deployment, set the required environment variables in your `docker-compose` override or `.env` file: ```yaml theme={null} environment: SEMANTIC_SEARCH_ENABLED: "true" EMBEDDING_PROVIDER: "openai" LLM_OPENAI_API_KEY: "sk-..." OPENAI_EMBEDDING_MODEL_ID: "text-embedding-3-small" OPENAI_EMBEDDING_DIMENSION: "1536" ``` ## Kubernetes Deployment For Kubernetes deployments using the OpenMetadata Helm chart, add the environment variables to your `values.yaml`: ```yaml theme={null} openmetadata: config: extraEnvs: - name: SEMANTIC_SEARCH_ENABLED value: "true" - name: EMBEDDING_PROVIDER value: "openai" - name: LLM_OPENAI_API_KEY valueFrom: secretKeyRef: name: openmetadata-secrets key: openai-api-key - name: OPENAI_EMBEDDING_MODEL_ID value: "text-embedding-3-small" - name: OPENAI_EMBEDDING_DIMENSION value: "1536" ``` Store sensitive values like API keys in Kubernetes Secrets and reference them with `secretKeyRef` rather than hardcoding them in `values.yaml`. ## Validating the Configuration After configuring your embedding provider, you can verify that everything is set up correctly by navigating to `Settings > Preferences > Health` in the OpenMetadata UI. This page shows the status of the embedding provider connection and will flag any misconfiguration. The **Semantic Search** health card only appears once `SEMANTIC_SEARCH_ENABLED` is set to `true`. It won't show up if semantic search is still disabled. ## Generating Embeddings Once Semantic Search is enabled, embeddings are generated and kept in sync automatically as entities are created or updated. To generate embeddings for all existing entities, run a **Reindex** from the OpenMetadata UI (`Settings > Applications > Search Indexing`). Every Reindex operation computes embeddings taking a fingerprint into account -- if the text representation of an entity has not changed since its last embedding, the embedding is not recomputed. This avoids unnecessary calls to the embedding provider and makes re-indexing efficient even for large catalogs. **OpenSearch only:** as a faster, dedicated alternative to a full Reindex, use the `reembed` CLI command. It initializes the vector service and processes entities in batches, applying the same fingerprint check described above -- entities whose content hasn't changed are skipped, and the existing vector index is not dropped or recreated. The speedup over a UI Reindex comes from running as a standalone, multithreaded batch job with configurable batch size and producer/consumer thread counts: ```bash theme={null} ./bin/openmetadata-ops.sh reembed --batch-size 100 --producer-threads 2 --consumer-threads 4 ``` This command is implemented against OpenSearch specifically and doesn't run on an Elasticsearch backend. On Elasticsearch, use the full Reindex from the UI described above instead. ## API Reference Semantic Search exposes a REST API endpoint for vector queries: ### POST `/api/v1/search/vector/query` Performs a semantic search against the vector index. **Request Body:** ```json theme={null} { "query": "customer demographics purchase history", "filters": { "entityType": ["table"], "owners": ["admin"], "tags": ["PII.Sensitive"], "domains": ["Marketing"], "tier": ["Tier.Tier1"], "serviceType": ["Postgres"] }, "size": 10, "k": 1000, "threshold": 0.0 } ``` | Parameter | Type | Default | Description | | ----------- | ------ | ------------ | --------------------------------------------------------------------------------------------------------- | | `query` | string | *(required)* | Natural language search text | | `filters` | map | `{}` | Filter map by entity type, owners, tags, domains, tier, service type, certification, or custom properties | | `size` | int | `10` | Number of distinct entities to return (max 100) | | `k` | int | `500` | KNN parameter -- number of nearest neighbors to consider (max 10,000) | | `threshold` | double | `0.0` | Minimum similarity score to include in results | Results are deduplicated by parent entity, so you will receive at most `size` distinct entities even if an entity has multiple text chunks. ## Troubleshooting ### Semantic Search returns no results * Verify that `SEMANTIC_SEARCH_ENABLED` is set to `true` and the server has been restarted. * Confirm your search backend (OpenSearch or Elasticsearch) is reachable and correctly configured. * Check that the `dataAssetEmbeddings` index (or `_dataAssetEmbeddings` if you've set a cluster alias) exists in your search backend. * Run a Reindex to generate embeddings for existing entities. ### Embedding generation fails * Verify network connectivity from the OpenMetadata server to your embedding provider. * Check that API keys and credentials are correct. * Review the OpenMetadata server logs for detailed error messages. # Upgrade OpenMetadata | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/upgrade Upgrade the platform version with step-by-step instructions on migration, compatibility, and new feature adoption. # Upgrade OpenMetadata In this guide, you will find all the necessary information to safely upgrade your OpenMetadata instance to 2.0.1. OpenMetadata 2.0 is a major release with breaking changes across the API, ingestion, configuration and UI. The highest-impact ones are listed below; the full, component-by-component breakdown lives in [Breaking Changes: 1.13 → 2.0](/v2.0.x/deployment/upgrade/breaking-changes/overview). * **Ingestion Framework:** all workflows have integrated `workflow.print_status()` inside the `workflow.execute()` call, to better handle logger lifecycles. If you drive workflows directly you can now remove the `print_status()` call — the only side effect is temporarily duplicated summary logs. ``` workflow_config = yaml.safe_load(CONFIG) workflow = MetadataWorkflow.create(workflow_config) workflow.execute() workflow.raise_from_status() workflow.print_status() # Not necessary anymore workflow.stop() ``` * **Collaboration:** `/v1/suggestions` is removed and suggestions are now Tasks; announcements move from `/v1/feed` to `/v1/announcements`. See [Collaboration](/v2.0.x/deployment/upgrade/breaking-changes/collaboration). * **Ingestion:** `ingestionPipeline.pipelineStatuses` is now an **array**, and the Databricks Pipeline connection requires `authType` instead of a top-level `token`. See [Ingestion & Connectors](/v2.0.x/deployment/upgrade/breaking-changes/ingestion-and-connectors). * **Great Expectations:** the validation action now requires **Great Expectations 1.3 or later**; the `great-expectations-1xx` extra no longer exists and test case results report row counts only. See [Ingestion & Connectors](/v2.0.x/deployment/upgrade/breaking-changes/ingestion-and-connectors). * **Configuration:** embedding and natural-language-query provider settings move out of `elasticsearch.naturalLanguageSearch` into a new top-level `llmConfiguration` block, and concurrent sessions are capped at 5 per user by default. See [Platform, Config & Security](/v2.0.x/deployment/upgrade/breaking-changes/platform-and-security). * **Explore:** the Explore page is redesigned. URL parameters change (`page`/`size` become `currentPage`/`pageSize`, plus a new `browsePath`), filters now stack with the browse location, and result ordering changes. See [Discovery & Search](/v2.0.x/deployment/upgrade/breaking-changes/discovery-and-search). * **Entity status:** field changed from **status** to **entityStatus** for **glossaryTerm** and **dataContract**, as it is introduced for different data assets. For Data Contracts the value also changed from **Active** to **Approved**. Component-by-component breakdown across API, Explore, collaboration, governance, data quality, lineage, ingestion, applications, platform and UI — plus an upgrade checklist. ## Prerequisites Everytime that you plan on upgrading OpenMetadata to a newer version, make sure to go over all these steps: ### Version Compatibility Matrix Before upgrading your OpenMetadata instance, verify that the versions of the external services used in your deployment meet the minimum supported requirements. Upgrading OpenMetadata without compatible versions of these services may cause migration failures or runtime issues. ## **Note:** If your current deployment uses versions lower than the minimum supported versions listed above, upgrade the respective services before proceeding with the OpenMetadata upgrade process. ### Backup your Metadata Before upgrading your OpenMetadata version we strongly recommend backing up the metadata. The source of truth is stored in the underlying database (MySQL and Postgres supported). During each version upgrade there is a database migration process that needs to run. It will directly attack your database and update the shape of the data to the newest OpenMetadata release. It is important that we backup the data because if we face any unexpected issues during the upgrade process, you will be able to get back to the previous version without any loss. You can learn more about how the migration process works [here](/deployment/upgrade/how-does-it-work). **During the upgrade, please note that the backup is only for safety and should not be used to restore data to a higher version**. Since version 1.4.0, **OpenMetadata encourages using the builtin-tools for creating logical backups of the metadata**: * [mysqldump](https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html) for MySQL * [pg\_dump](https://www.postgresql.org/docs/current/app-pgdump.html) for Postgres For PROD deployment we recommend users to rely on cloud services for their databases, be it [AWS RDS](https://docs.aws.amazon.com/rds/), [Azure SQL](https://azure.microsoft.com/en-in/products/azure-sql/database) or [GCP Cloud SQL](https://cloud.google.com/sql/). If you're a user of these services, you can leverage their backup capabilities directly: * [Creating a DB snapshot in AWS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateSnapshot.html) * [Backup and restore in Azure MySQL](https://learn.microsoft.com/en-us/azure/mysql/single-server/concepts-backup) * [About GCP Cloud SQL backup](https://cloud.google.com/sql/docs/mysql/backup-recovery/backups) You can refer to the following guide to get more details about the backup and restore: Learn how to back up MySQL or Postgres data. ### Understanding the "Running" State in OpenMetadata In OpenMetadata, the **"Running"** state indicates that the OpenMetadata server has received a response from Airflow confirming that a workflow is in progress. However, if Airflow unexpectedly stops or crashes before it can send a failure status update through the **Failure Callback**, OpenMetadata remains unaware of the workflow’s actual state. As a result, the workflow may appear to be stuck in **"Running"** even though it is no longer executing. This situation can also occur during an OpenMetadata upgrade. If an ingestion pipeline was running at the time of the upgrade and the process caused Airflow to shut down, OpenMetadata would not receive any further updates from Airflow. Consequently, the pipeline status remains **"Running"** indefinitely. Running State in OpenMetadata #### Expected Steps to Resolve To resolve this issue: * Ensure that Airflow is restarted properly after an unexpected shutdown. * Manually update the pipeline status if necessary. * Check Airflow logs to verify if the DAG execution was interrupted. #### Update `sort_buffer_size` (MySQL) or `work_mem` (Postgres) Before running the migrations, it is important to update these parameters to ensure there are no runtime errors. A safe value would be setting them to 20MB. **If using MySQL** You can update it via SQL (note that it will reset after the server restarts): ```sql theme={null} SET GLOBAL sort_buffer_size = 20971520 ``` To make the configuration persistent, you'd need to navigate to your MySQL Server install directory and update the `my.ini` or `my.cnf` [files](https://dev.mysql.com/doc/refman/8.0/en/option-files.html) with `sort_buffer_size = 20971520`. If using RDS, you will need to update your instance's [Parameter Group](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html) to include the above change. **If using Postgres** You can update it via SQL (not that it will reset after the server restarts): ```sql theme={null} SET work_mem = '20MB'; ``` To make the configuration persistent, you'll need to update the `postgresql.conf` [file](https://www.postgresql.org/docs/9.3/config-setting.html) with `work_mem = 20MB`. If using RDS, you will need to update your instance's [Parameter Group](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html) to include the above change. Note that this value would depend on the size of your `query_entity` table. If you still see `Out of Sort Memory Error`s during the migration after bumping this value, you can increase them further. After the migration is finished, you can revert this changes. #### Enable `pg_trgm` Extension (Azure PostgreSQL Flexible Server) If you are using **Azure Database for PostgreSQL (Flexible Server)**, the migration process requires the `pg_trgm` extension. By default, Azure restricts this extension and you may encounter: ``` ERROR: extension "pg_trgm" is not allow-listed for users in Azure Database for PostgreSQL ``` **Resolution Steps:** 1. **Allow the extension** - Go to **Azure Portal** → **PostgreSQL Flexible Server** → **Server Parameters** → Search for `azure.extensions` → Add `pg_trgm` (comma-separated if other extensions exist) 2. **Restart the PostgreSQL server** for changes to take effect 3. **Create the extension** by running: ```sql theme={null} CREATE EXTENSION IF NOT EXISTS pg_trgm; ``` 4. **Proceed with the migration** For detailed troubleshooting, see the [Kubernetes Upgrade Troubleshooting](/v2.0.x/deployment/upgrade/kubernetes-troubleshooting#azure-postgresql-pg_trgm-extension-requirement) section. ### MySQL Configuration Required for Airflow 3.x Migration If you are using MySQL as your Airflow metadata database and upgrading to Airflow 3.x (the new default in OpenMetadata 1.11), you must configure MySQL to allow temporary stored function creation during the migration process. #### Root Cause During the Airflow 3.x database migration on MySQL, Airflow needs to create a temporary stored function (`uuid_generate_v7`) to backfill UUIDs for the `task_instance` table. When MySQL runs with binary logging enabled (which is the default in most production setups), it blocks function creation unless `log_bin_trust_function_creators` is enabled or the user has SUPER privileges. Without this configuration, the migration fails with an error like: ``` FUNCTION airflow_db.uuid_generate_v7 does not exist ``` This is a known limitation when running Airflow 3.x migrations on MySQL with binary logging enabled. PostgreSQL users are not affected by this issue. For more details, see the Apache Airflow issues: * [https://github.com/apache/airflow/issues/49611](https://github.com/apache/airflow/issues/49611) * [https://github.com/apache/airflow/issues/54554](https://github.com/apache/airflow/issues/54554) #### Resolution **Option 1: Delete and Recreate the Airflow Database (Strongly Recommended)** The simplest and most reliable solution is to delete the existing Airflow database and let OpenMetadata recreate it fresh during startup. The Airflow database only stores workflow execution history and metadata—it does not contain any of your OpenMetadata configurations, connections, or ingestion pipeline definitions. This is the recommended approach because it avoids all migration complexities and ensures a clean state. Your ingestion pipelines and their configurations are stored in the OpenMetadata database, not in Airflow's database. ```bash theme={null} # Connect to your MySQL instance and drop the Airflow database docker exec -i openmetadata_mysql mysql -u USERNAME -pPASSWORD -e "DROP DATABASE IF EXISTS airflow_db;" ``` Then recreate the database with the proper character set and grant privileges: ```sql theme={null} CREATE DATABASE airflow_db CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; GRANT ALL PRIVILEGES ON airflow_db.* TO 'airflow_user'@'%' WITH GRANT OPTION; FLUSH PRIVILEGES; ``` Execute this via command line: ```bash theme={null} # Recreate the Airflow database docker exec -i openmetadata_mysql mysql -u USERNAME -pPASSWORD -e "CREATE DATABASE airflow_db CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; GRANT ALL PRIVILEGES ON airflow_db.* TO 'airflow_user'@'%' WITH GRANT OPTION; FLUSH PRIVILEGES;" # Restart the ingestion container to run migrations on the fresh database docker restart openmetadata_ingestion ``` Replace `USERNAME` and `PASSWORD` with your MySQL root credentials, and `airflow_user` with your actual Airflow database user if different. For Docker Quickstart deployments, the default root credentials are `root` / `password`. After the ingestion container restarts successfully, you must **redeploy all your ingestion pipelines** from the OpenMetadata UI. This registers the DAGs in the fresh Airflow database. *** **Option 2: Manual Migration Fix (If You Cannot Delete the Database)** If you have specific requirements to preserve the Airflow execution history and cannot delete the database, follow the manual steps below. **Step 1: Enable MySQL Configuration** First, enable `log_bin_trust_function_creators` in your MySQL instance to allow Airflow to create the necessary stored function: For Docker deployments, add this to your `docker-compose.yml` file under the MySQL service: ```yaml theme={null} services: mysql: command: "--log-bin-trust-function-creators=1" ``` For standalone MySQL instances, execute this query as a user with sufficient privileges: ```sql theme={null} SET GLOBAL log_bin_trust_function_creators = 1; ``` **Step 2: Clean Airflow Database** After enabling the MySQL configuration, choose one of the following options based on your situation: **Option 2a: Truncate Task Instance Table** If you want to avoid conflicting migration changes, you can truncate the `task_instance` table. This approach removes all task execution history but preserves your DAGs and connections. This will delete all historical task execution data. Only use this if you're okay with losing task run history. ```sql theme={null} -- Clean task_instance table to avoid migration conflicts USE airflow_db; -- Truncate task_instance table TRUNCATE TABLE task_instance; -- Verify the table is empty SELECT COUNT(*) FROM task_instance; ``` Execute this script: ```bash theme={null} # Run the cleanup script on your MySQL container docker exec -i openmetadata_mysql mysql -u USERNAME -pPASSWORD -e "USE airflow_db; TRUNCATE TABLE task_instance; SELECT COUNT(*) as remaining_rows FROM task_instance;" # Restart the ingestion container to apply migrations docker restart openmetadata_ingestion ``` **Option 2b: Fix Stuck Migrations (If Migration Already Failed)** If your migration is already stuck midway (the `task_instance` table was partially modified), you need to reset the migration state before restarting. Save the following SQL script as `fix_airflow_migration.sql`: ```sql theme={null} -- Fix Airflow 3.x migration issue -- This script fixes the partial migration of task_instance table USE airflow_db; -- Check if the migration was partially applied -- If 'id' column exists but isn't properly configured, we need to fix it -- First, check the current state SHOW COLUMNS FROM task_instance LIKE 'id'; -- Drop the problematic column if it exists SET @exist := (SELECT COUNT(*) FROM information_schema.COLUMNS WHERE TABLE_SCHEMA = 'airflow_db' AND TABLE_NAME = 'task_instance' AND COLUMN_NAME = 'id'); SET @sqlstmt := IF(@exist > 0, 'ALTER TABLE task_instance DROP COLUMN id', 'SELECT ''Column does not exist'' AS status'); PREPARE stmt FROM @sqlstmt; EXECUTE stmt; DEALLOCATE PREPARE stmt; -- Reset the alembic version to before this migration -- The migration that's failing is: d59cbbef95eb (Add UUID primary key to task_instance) -- We need to set it back to the previous version: 05234396c6fc UPDATE alembic_version SET version_num = '05234396c6fc' WHERE version_num = 'd59cbbef95eb'; -- Verify the changes SELECT * FROM alembic_version; SHOW COLUMNS FROM task_instance LIKE 'id'; ``` Then execute the script and restart the container: ```bash theme={null} # Run the fix script on your MySQL container docker exec -i openmetadata_mysql mysql -u USERNAME -pPASSWORD < fix_airflow_migration.sql # Restart the ingestion container docker restart openmetadata_ingestion ``` Replace `USERNAME` and `PASSWORD` with your actual MySQL credentials, and ensure the database name matches your configuration (default is `airflow_db`). ## Upgrade your installation Once your metadata is safe, follow the required upgrade instructions based on your environment: Upgrade your Kubernetes installation Upgrade your Docker installation Upgrade your Bare Metal installation ## Post-Upgrade Steps ### Reindex #### With UI Go to `Settings` -> `Applications` -> `Search Indexing` search-index-app Before initiating the process by clicking `Run Now`, ensure that the `Recreate Indexes` option is enabled to allow rebuilding the indexes as needed. In the configuration section, you can select the entities you want to reindex. create-project Since this is required after the upgrade, we want to reindex `All` the entities. ### (Optional) Update your OpenMetadata Ingestion Client If you are running the ingestion workflows **externally** or using a custom Airflow installation, you need to make sure that the Python Client you use is aligned with the OpenMetadata server version. For example, if you are upgrading the server to the version `x.y.z`, you will need to update your client with ```bash theme={null} pip install openmetadata-ingestion[]==x.y.z ``` #### With Kubernetes Follow these steps to reindex using the CLI: 1. List the CronJobs Use the following command to check the available CronJobs: ```bash theme={null} kubectl get cronjobs ``` Upon running this command you should see output similar to the following. ```CommandLine theme={null} kubectl get cronjobs NAME SCHEDULE TIMEZONE SUSPEND ACTIVE LAST SCHEDULE AGE cron-reindex 0/5 * * * * True 0 31m ``` 2. Create a Job from a CronJob Create a one-time job from an existing CronJob using the following command: ```bash theme={null} kubectl create job --from=cronjob/cron-deploy-pipelines ``` Replace `` with the actual name of the job. Upon running this command you should see output similar to the following. ```CommandLine theme={null} kubectl create job --from=cronjob/cron-reindex cron-reindex-one job.batch/cron-reindex-one created ``` 3. Check the Job Status Verify the status of the created job with: ```bash theme={null} kubectl get jobs ``` Upon running this command you should see output similar to the following. ```CommandLine theme={null} kubectl get jobs NAME STATUS COMPLETIONS DURATION AGE cron-reindex-one Complete 1/1 20s 109s ``` 4. view logs To view the logs use the below command. ```bash theme={null} kubectl logs job/ ``` Replace `` with the actual job name. The `plugin` parameter is a list of the sources that we want to ingest. An example would look like this `openmetadata-ingestion[mysql,snowflake,s3]==1.2.0`. You will find specific instructions for each connector in the Connectors section. Moreover, if working with your own Airflow deployment - not the `openmetadata-ingestion` image - you will need to upgrade as well the `openmetadata-managed-apis` version: ```bash theme={null} pip install openmetadata-managed-apis==x.y.z ``` ### Re Deploy Ingestion Pipelines #### With UI Go to `Settings` -> `{Services}` -> `{Databases}` -> `Pipelines` redeploy Select the pipelines you want to Re Deploy click `Re Deploy`. #### With Kubernetes Follow these steps to deploy pipelines using the CLI: 1. List the CronJobs Use the following command to check the available CronJobs: ```bash theme={null} kubectl get cronjobs ``` Upon running this command you should see output similar to the following. ```commandline theme={null} kubectl get cronjobs NAME SCHEDULE TIMEZONE SUSPEND ACTIVE LAST SCHEDULE AGE cron-deploy-pipelines 0/5 * * * * True 0 4m7s ``` 2. Create a Job from a CronJob Create a one-time job from an existing CronJob using the following command: ```bash theme={null} kubectl create job --from=cronjob/cron-reindex ``` Replace `` with the actual name of the job. Upon running this command you should see output similar to the following. ```commandline theme={null} kubectl create job --from=cronjob/cron-deploy-pipelines cron-deploy-pipeline-one job.batch/cron-deploy-pipeline-one created ``` 3. Check the Job Status Verify the status of the created job with: ```bash theme={null} kubectl get jobs ``` Upon running this command you should see output similar to the following. ```CommandLine theme={null} kubectl get jobs NAME STATUS COMPLETIONS DURATION AGE cron-deploy-pipeline-one Complete 1/1 13s 3m35s ``` 4. view logs To view the logs use the below command. ```bash theme={null} kubectl logs job/ ``` Replace `` with the actual job name. If you are seeing broken dags select all the pipelines from all the services and re deploy the pipelines. # Openmetadata-ops Script ## Overview The `openmetadata-ops` script is designed to manage and migrate databases and search indexes, reindex existing data into Elastic Search or OpenSearch, and redeploy service pipelines. ## Usage ```bash theme={null} sh openmetadata-ops.sh [-dhV] [COMMAND] ``` #### Commands * analyze-tables Migrates secrets from the database to the configured Secrets Manager. Note that this command does not support migrating between external Secrets Managers. * changelog Prints the change log of database migration. * check-connection Checks if a connection can be successfully obtained for the target database. * deploy-pipelines Deploys all the service pipelines. * drop-create Deletes any tables in the configured database and creates new tables based on the current version of OpenMetadata. This command also re-creates the search indexes. * info Shows the list of migrations applied and the pending migrations waiting to be applied on the target database. * migrate Migrates the OpenMetadata database schema and search index mappings. * migrate-secrets Migrates secrets from the database to the configured Secrets Manager. Note that this command does not support migrating between external Secrets Managers. * reindex Reindexes data into the search engine from the command line. * repair Repairs the DATABASE\_CHANGE\_LOG table, which is used to track all the migrations on the target database. This involves removing entries for the failed migrations and updating the checksum of migrations already applied on the target database. * validate Checks if all the migrations have been applied on the target database. ### Examples Display Help To display the help message: ```bash theme={null} sh openmetadata-ops.sh --help ``` ### Migrate Database Schema To migrate the database schema and search index mappings: ```bash theme={null} sh openmetadata-ops.sh migrate ``` ### Reindex Data To reindex data into the search engine: ```bash theme={null} sh openmetadata-ops.sh reindex ``` # Breaking Changes - API & Schema | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/upgrade/breaking-changes/api-and-schema Removed REST endpoints, changed request and response shapes, tightened validation and enum changes in OpenMetadata 2.0. # API & Schema Contracts Across the 1.13 → 2.0 jump the REST surface gains **208 endpoints** and loses **10**. The JSON Schema specification gains **86 files**, modifies **107** and removes **3**. ## Removed endpoints ### The Suggestions API is removed **Breaking.** Affects API clients, SDKs and automation bots. The entire `/v1/suggestions` namespace is gone: | Removed in 2.0 | | ------------------------------------------------------ | | `GET /v1/suggestions` | | `POST /v1/suggestions` | | `GET /v1/suggestions/{id}` | | `PUT /v1/suggestions/{id}` | | `PUT /v1/suggestions/{id}/accept` | | `PUT /v1/suggestions/{id}/reject` | | `PUT /v1/suggestions/accept-all` | | `PUT /v1/suggestions/reject-all` | | `DELETE /v1/suggestions/{suggestionId}` | | `DELETE /v1/suggestions/{entityType}/name/{entityFQN}` | Replace suggestion calls with `/v1/tasks` filtered by `type=Suggestion` (`category=MetadataUpdate`). Accept and reject become `POST /v1/tasks/{id}/resolve` and `PUT /v1/tasks/{id}/suggestion/apply`. Existing suggestions are migrated into `task_entity` by the 2.0.0 migration. See [Collaboration](/v2.0.x/deployment/upgrade/breaking-changes/collaboration). ## Changed request contracts ### `search_after` is now a repeated query parameter **Breaking.** Affects anyone paginating `/v1/search/query`. Each sort value is carried as its own parameter so values containing a comma — for example a glossary term fully qualified name — are safe. ```http theme={null} GET /api/v1/search/query?q=*&index=table&search_after=1712345678000,abc-123 ``` ```http theme={null} GET /api/v1/search/query?q=*&index=table&search_after=1712345678000&search_after=abc-123 ``` Split the cursor on the client and emit one `search_after` parameter per sort value. ### The `deleted` search parameter is deprecated **Deprecated.** `deleted` on `/v1/search/query` is annotated for removal in 2.0. Express deleted-entity filtering through `query_filter` instead. ### `testCaseStatus` accepts multiple values `GET /v1/dataQuality/testCases` binds `testCaseStatus` as a repeated parameter: ```http theme={null} GET /api/v1/dataQuality/testCases?testCaseStatus=Failed&testCaseStatus=Aborted ``` Values are validated against the `TestCaseStatus` enum, and an unknown value now returns `400 Bad Request` naming the allowed values. A single-value call is unchanged. ## Validation changes ### Entity name validation is tightened **Breaking.** Affects every write path, ingestion connectors and CSV import. `entityName` and `testCaseEntityName` change pattern: ```diff theme={null} - "pattern": "^((?!::).)*$" + "pattern": "^((?!::)[^>\"\\x00-\\x1f])*$" ``` In addition to the existing `::` restriction, names may no longer contain: * `>` (greater-than) * `"` (double quote) * any ASCII control character (`\x00`–`\x1f`) Entities already stored with these characters are not rewritten by the migration, but the next `PUT` or `PATCH` that revalidates the name will fail. Audit source systems whose object names can contain quotes or angle brackets before upgrading. ## Enum changes ### Removed enum values **Breaking.** | Schema | Enum | Removed values | | ------------------------------ | ----------- | ---------------------------------------------------------- | | `entity/applications/app.json` | `agentType` | `CollateAI`, `CollateAITierAgent`, `CollateAIQualityAgent` | Only `Metadata` remains. The corresponding application configuration schemas were deleted: * `entity/applications/configuration/external/collateAIQualityAgentAppConfig.json` * `entity/applications/configuration/external/collateAITierAgentAppConfig.json` * `entity/applications/configuration/private/internal/collateAITierAgentAppPrivateConfig.json` ### Added enum values **Additive** — but clients doing exhaustive `switch` or pattern matching on these enums need new branches. | Schema | Enum | Added values | | --------------------------------- | ---------------------- | -------------------------------------------------------------------------------------------------- | | `resourceDescriptor.json` | `operation` | `CreateTask`, `EditTask`, `ResolveTask`, `CloseTask`, `ReassignTask` | | `type/changeEventType.json` | — | `taskCreated`, `taskUpdated`, `entityLineageAdded`, `entityLineageDeleted`, `entityLineageUpdated` | | `type/entityRelationship.json` | `relationshipType` | `assignedTo` | | `type/workflowTriggerFields.json` | — | `entityStatus` | | `settings/settings.json` | `settingType` | `appConfiguration` | | `nodeSubType.json` | — | `policyAgentTask`, `createAndRunAIAutomationTask` | | `jobs/backgroundJob.json` | `jobType` | `CSV_IMPORT`, `CSV_EXPORT`, `AUDIT_EXPORT` | | `jobs/backgroundJob.json` | `status` | `CANCELLED` | | `ingestionPipeline.json` | `pipelineType` | `policyAgent` | | `entity/data/table.json` | `tableType` | `SemanticView` | | `entity/data/metric.json` | `metricType` | `SIMPLE`, `CUMULATIVE`, `DERIVED`, `CONVERSION` | | `databaseService.json` | `databaseServiceType` | `SapBw4Hana` | | `dashboardService.json` | `dashboardServiceType` | `Omni` | | `pipelineService.json` | `pipelineServiceType` | `Prefect`, `SapBw4HanaPipeline` | ## Default value changes **Behavioural.** Affects deployments that never set these explicitly. | Schema | Field | 1.13 | 2.0 | | ----------------------- | --------------------------------------------------- | ---------- | ----------- | | `rdfConfiguration.json` | `bulkEntityBatchSize` | `50` | `100` | | `rdfConfiguration.json` | `bulkRelationshipSourceBatchSize` | `25` | `100` | | `rdfConfiguration.json` | `inferenceEnabled` | `true` | **`false`** | | `workflowSettings.json` | `executorConfiguration.asyncJobAcquisitionInterval` | `10000` ms | `1000` ms | The 2.0.0 post-data migration also lowers `timerJobAcquisitionInterval` to `5000` ms and the `WorkflowEventConsumer` poll interval from 10 s to 1 s, so governance workflows fire near-real-time. RDF inference is off by default in 2.0. If you relied on inferred triples in SPARQL queries, set `RDF_INFERENCE_ENABLED=true` explicitly. ### Connector filter-pattern defaults **Behavioural.** Applies to new **and** existing ingestion runs. | Connector | Field | New default excludes | | --------- | -------------------- | -------------------------------------- | | Redshift | `tableFilterPattern` | `^(?:.*\.)?mv_tbl__.*__\d+$` | | Kafka | `topicFilterPattern` | `^__.*`, `^_schemas$`, `^_confluent.*` | | Redpanda | `topicFilterPattern` | `^__.*`, `^_schemas$`, `^_confluent.*` | If you deliberately ingest Redshift materialised-view backing tables or Kafka internal topics, override the filter pattern explicitly — the default now excludes them. ## Schema reference relocations **Behavioural.** Affects code generated from the specification and JSON Schema validators. Chart `function` and `kpiDetails` definitions moved out of `dataInsightCustomChart.json` into a new `dataInsight/custom/chartFunctions.json`: ```diff theme={null} - "$ref": "dataInsightCustomChart.json#/definitions/function" + "$ref": "chartFunctions.json#/definitions/function" ``` Affected files: `formulaHolder.json`, `lineChart.json`, `summaryCard.json` and `dataInsightCustomChartResultList.json`. Similarly, `entity/data/table.json` extracts the inline `columnProfile.cardinalityDistribution` object into a named definition. The wire shape is unchanged — only the pointer moved. Regenerate any client models built from the JSON Schema. If you resolve `$ref` pointers by hand, update the paths. ## Notable new endpoint groups **Additive.** These do not break anything, but they replace patterns you may currently implement client-side. | Namespace | Purpose | Replaces | | ------------------------------------------------------ | ------------------------------------------------------ | --------------------------------------- | | `/v1/tasks/**` | First-class Task entity (22 endpoints) | `/v1/feed/tasks/*`, `/v1/suggestions/*` | | `/v1/announcements/**` | Standalone Announcement entity | `/v1/feed?type=Announcement` | | `/v1/activity/**` | Ephemeral activity stream | System-generated `/v1/feed` threads | | `/v1/csvAsyncJobs/**` | CSV import/export job status and result download | WebSocket-only job tracking | | `/v1/csv/documentation/{entityType}` | Machine-readable CSV column documentation | Hard-coded column lists | | `/v1/services/overview` | One call for per-type and per-connector service counts | N per-service `GET` calls | | `/v1/lineage/hydrate` | Batch-hydrate up to 200 lineage nodes | N per-node entity `GET` calls | | `/v1/{entityType}/deleteStale` | Connector-driven stale-entity cleanup | Manual delete loops | | `/v1/search/export/async` | Queue a CSV export as a background job | Synchronous `/v1/search/export` | | `/v1/columns/name/{fqn}` | Fetch a single column by fully qualified name | Fetching the whole table | | `/v1/personas/{id}/context`, `/v1/personas/me/context` | Persona-scoped AI context | — | | `/v1/contextCenter/**`, `/v1/attachments/**` | Knowledge and Context Center, plus file assets | — | | `/v1/aiGovernance/**` and related | AI Governance Studio | — | `deleteStale` is available on 18 entity types: `tables`, `databases`, `databaseSchemas`, `storedProcedures`, `dashboards`, `charts`, `dashboard/datamodels`, `pipelines`, `topics`, `mlmodels`, `searchIndexes`, `containers`, `apiCollections`, `apiEndpoints`, `drives/directories`, `drives/files`, `drives/spreadsheets` and `drives/worksheets`. ## Deprecations to plan for | Item | Status in 2.0 | Replacement | | -------------------------------------------------------------- | ---------------------- | ------------------------------------------- | | `GET /v1/search/query?deleted=` | Deprecated for removal | `query_filter` | | `authenticationConfiguration.oidcConfiguration.sessionExpiry` | Deprecated fallback | `authenticationConfiguration.sessionExpiry` | | `EntityResource.patchInternal(uriInfo, ctx, id, patch)` (Java) | Deprecated | Overload taking `ChangeSource` | # Breaking Changes - Collaboration | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/upgrade/breaking-changes/collaboration The OpenMetadata 2.0 Task redesign, removal of the Suggestions API, standalone Announcements, the ephemeral Activity Stream and alert filter changes. # Collaboration: Tasks, Suggestions, Announcements & Feed 2.0 retires the thread-backed collaboration model. Tasks, suggestions, announcements and system activity each move out of `thread_entity` into purpose-built entities with their own tables, APIs and permissions. Human conversations remain on `/v1/feed`. | Concern | 1.13 storage | 2.0 storage | | --------------- | ------------------------------------- | ------------------------------------------------------ | | Conversations | `thread_entity` | `thread_entity` (unchanged) | | Tasks | `thread_entity` (`type=Task`) | **`task_entity`** | | Suggestions | `suggestions` table | **`task_entity`** (`type=Suggestion`) | | Announcements | `thread_entity` (`type=Announcement`) | **`announcement_entity`** | | System activity | `thread_entity` generated rows | **`activity_stream`** (partitioned, retention-bounded) | OpenMetadata 2.0 entity page showing the redesigned header and the Activity Feeds and Tasks tab ## The Task redesign **Breaking.** Affects API clients, bots and workflow integrations reading `/v1/feed/tasks/*`. Tasks are now a first-class entity backed by `task_entity`, with a full CRUD and versioning surface at `/v1/tasks` — 22 new endpoints: | Endpoint | Purpose | | -------------------------------------------------------------------- | -------------------------------- | | `GET` / `POST` / `PUT` `/v1/tasks` | List, create, upsert | | `GET /v1/tasks/{id}`, `GET /v1/tasks/name/{taskId}` | Fetch by UUID or human task id | | `PATCH /v1/tasks/{id}`, `DELETE /v1/tasks/{id}` | Update, delete | | `POST /v1/tasks/{id}/resolve`, `POST /v1/tasks/{id}/close` | Lifecycle transitions | | `PUT /v1/tasks/{id}/suggestion/apply` | Apply a suggestion payload | | `POST /v1/tasks/{id}/comments`, `PATCH` / `DELETE` `.../{commentId}` | Threaded comments | | `GET /v1/tasks/assigned`, `/created`, `/owned`, `/visible` | Scoped task lists | | `GET /v1/tasks/count`, `GET /v1/tasks/dataAccessRequests` | Counts and the data-access queue | | `POST /v1/tasks/bulk` | Bulk operations | | `GET /v1/tasks/{id}/versions[/{version}]` | Entity version history | ### The Task shape ```json theme={null} { "taskId": "TASK-1042", "category": "Approval", "type": "GlossaryApproval", "status": "Open", "priority": "Medium", "about": { "type": "glossaryTerm", "id": "..." }, "assignees": [], "reviewers": [], "watchers": [], "payload": { }, "resolution": { }, "dueDate": 1712345678000, "workflowInstanceId": "...", "workflowStageId": "...", "availableTransitions": [], "taskFormSchemaId": "...", "taskFormSchemaVersion": 0.1, "comments": [], "commentCount": 0, "domains": [], "tags": [], "externalReference": { } } ``` Required fields: `id`, `name`, `category`, `type`, `status`, `createdBy`. | Enum | Values | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `taskCategory` | `Approval`, `DataAccess`, `MetadataUpdate`, `Incident`, `Review`, `Custom` | | `taskType` | `GlossaryApproval`, `RequestApproval`, `DataAccessRequest`, `DescriptionUpdate`, `TagUpdate`, `OwnershipUpdate`, `TierUpdate`, `DomainUpdate`, `Suggestion`, `TestCaseResolution`, `IncidentResolution`, `PipelineReview`, `DataQualityReview`, `RecognizerFeedbackApproval`, `CustomTask` | | `taskStatus` | `Open`, `InProgress`, `Pending`, `Approved`, `Granted`, `ManualRevoke`, `Rejected`, `Completed`, `Cancelled`, `Failed`, `Revoked`, `Expired` | | `taskPriority` | `Critical`, `High`, `Medium`, `Low` | | `resolutionType` | `Approved`, `Rejected`, `Completed`, `Cancelled`, `TimedOut`, `AutoApproved`, `AutoRejected`, `Revoked`, `Expired` | Typed payload schemas ship for each task type: `glossaryApprovalPayload`, `descriptionUpdatePayload`, `tagUpdatePayload`, `ownershipUpdatePayload`, `tierUpdatePayload`, `domainUpdatePayload`, `suggestionPayload`, `reviewPayload`, `testCaseResolutionPayload`, `incidentResolutionPayload`, `dataAccessRequestPayload` and `genericTaskPayload`. ### Migration The migration converts every `thread_entity` row with `type='Task'` into a Task, computing `about` and `aboutFqnHash` from the entity link. A `task_migration_mapping` table records `old_thread_id → new_task_id` for traceability and redirects. ### `/v1/feed` task creation is now restricted **Breaking.** `/v1/feed` still exposes `GET /v1/feed/tasks/{id}`, `PUT /v1/feed/tasks/{id}/resolve` and `PUT /v1/feed/tasks/{id}/close`, but creating a task thread through `POST /v1/feed` now validates the task type and rejects anything outside the supported legacy set — description, tag, approval and test-case-failure-resolution tasks. Additional validations on `POST /v1/feed`: * `about` is required and must be non-blank. * `taskDetails` is required for `Task` threads and **forbidden** on non-task threads. * `RequestApproval` tasks must target an entity, not a field or column. * Tag-task `oldValue` and `suggestion` must be valid tag-label JSON. Move task creation to `POST /v1/tasks`. If you must stay on `/v1/feed`, restrict yourself to the four supported legacy task types and supply well-formed `taskDetails`. ### New task permissions **Behavioural.** Affects non-admin users and application bots. Five new policy operations exist in 2.0: `CreateTask`, `EditTask`, `ResolveTask`, `CloseTask` and `ReassignTask`. The migration backfills them so existing tenants keep working: | Migration step | Effect | | ---------------------------------------------- | ------------------------------------------------------------------- | | `addTaskAuthorPolicyToDataConsumerRole` | Seeds `TaskAuthorPolicy` and attaches it to the `DataConsumer` role | | `addCreateTaskRuleToDataConsumerPolicy` | Adds `DataConsumerPolicy-CreateTask-Rule` | | `addTaskRuleToDataConsumerPolicy` | Adds the per-entity `CreateTask`/`EditTask` grant | | `addCreateTaskOperationToApplicationBotPolicy` | Lets application bots file suggestions as tasks | Custom policies are **not** backfilled. Seed policies are create-if-not-exists, so if you replaced `DataConsumerPolicy` or `ApplicationBotPolicy` with your own definition, add the task operations yourself or non-admin users will get `403` when filing or patching tasks. Task authorization is also **self-approval guarded** — a task's creator cannot approve their own task. ## Suggestions become Tasks **Breaking.** Affects AI and automation bots and SDK users. | 1.13 | 2.0 | | ----------------------------------------------- | --------------------------------------------------------- | | `GET /v1/suggestions?entityFQN=…` | `GET /v1/tasks?…` filtered on `type=Suggestion` | | `POST /v1/suggestions` | `POST /v1/tasks` with `type: Suggestion` | | `PUT /v1/suggestions/{id}/accept` | `PUT /v1/tasks/{id}/suggestion/apply`, then resolve | | `PUT /v1/suggestions/{id}/reject` | `POST /v1/tasks/{id}/resolve` with a rejecting resolution | | `PUT /v1/suggestions/accept-all` / `reject-all` | `POST /v1/tasks/bulk` | Status mapping: | Task status | Suggestion status | | --------------------------------- | ----------------- | | `Open`, `InProgress`, `Pending` | `Open` | | `Completed`, `Approved` | `Accepted` | | `Rejected`, `Cancelled`, `Failed` | `Rejected` | The suggestion field path moves from an entity link to `payload.fieldPath` in dot notation (`columns.col_name.description`). ## Announcements are a standalone entity **Breaking.** Affects anything reading or writing announcements through the feed API. `/v1/feed` now rejects announcements outright with: ``` Announcements are no longer served from /v1/feed. Use /v1/announcements instead. ``` The guard fires on list (`threadType=Announcement`), get-by-id, patch, create, delete, posts and reactions — any request touching an announcement thread returns `400`. ``` GET/POST/PUT /v1/announcements GET /v1/announcements/{id} GET /v1/announcements/name/{fqn} PATCH /v1/announcements/{id} DELETE /v1/announcements/{id} PUT /v1/announcements/restore GET /v1/announcements/{id}/versions[/{version}] ``` ### Migration shape | Thread field | Announcement field | | --------------------------------------- | --------------------------------------------------- | | `id` | `id` | | — | `name` / `fullyQualifiedName` = `announcement-` | | `message` | `displayName` | | `announcement.description` ?? `message` | `description` | | `about` | `entityLink` | | `announcement.startTime` / `endTime` | `startTime` / `endTime` | | derived from times | `status` = `Active` \| `Scheduled` \| `Expired` | | `threadTs` | `createdAt` | | `reactions` | `reactions` | Announcements are full entities in 2.0 — versioned, soft-deletable and restorable — and the UI renders them in the entity header rather than only in the feed widget. ## The Activity Stream replaces system-generated feed threads **Breaking.** Affects anything treating the feed as an audit trail. System-generated activity (field changes, entity created/updated) no longer lives in `thread_entity`. It moves to a purpose-built, time-partitioned, retention-bounded `activity_stream` table with its own API: | Endpoint | Purpose | | ------------------------------------------------------------ | ---------------------------------- | | `GET /v1/activity` | Global stream | | `GET /v1/activity/my-feed` | Current user's feed | | `GET /v1/activity/following` | Followed entities | | `GET /v1/activity/user/{userId}` | A user's activity | | `GET /v1/activity/entity/{entityType}/{entityId}` | Per entity | | `GET /v1/activity/entity/{entityType}/name/{fqn}` | Per entity by fully qualified name | | `GET /v1/activity/about` | By entity link | | `GET /v1/activity/count` | Count | | `PUT` / `DELETE` `/v1/activity/{id}/reaction/{reactionType}` | Reactions | ### Activity is deleted after 30 days by default **Behavioural — data loss on old activity.** `activityStreamConfig` is configurable globally or per domain: | Field | Default | Meaning | | -------------------------- | -------- | ------------------------------------------------ | | `enabled` | `true` | Generate activity events for this scope | | `retentionDays` | **`30`** | Events older than this are deleted automatically | | `excludeEventTypes` | `[]` | Event types to skip | | `excludeEntityTypes` | `[]` | Entity types to skip | | `visibility` | — | Who can see events in this scope | | `scope` / `scopeReference` | — | Global or per domain | Events carry `domains` inherited from the source entity, enabling domain-scoped feed visibility. `oldValue` and `newValue` are explicitly documented as *"truncated for display, not for audit"*. Do not use the activity stream as an audit trail. For compliance history use **entity version history** (`/v1/{entityType}/{id}/versions`) and the **audit log** (`/v1/audit/logs`, which gains a searchable `search_text` column and an export endpoint in 2.0). Activity events are ephemeral by design. ## `thread_entity` is renamed **Behavioural.** Affects anyone querying the OpenMetadata database directly. ```sql theme={null} -- 2.0.0 post-data migration RENAME TABLE thread_entity TO thread_entity_legacy; -- MySQL ALTER TABLE IF EXISTS thread_entity RENAME TO thread_entity_legacy; -- Postgres ``` The feed repository resolves the legacy table dynamically, so migrated threads stay readable. Update any BI dashboards, retention jobs or support scripts that query `thread_entity` directly. ## Task Form Schemas **Additive.** `/v1/taskFormSchemas` stores per-task-type form definitions, referenced from a Task via `taskFormSchemaId` and `taskFormSchemaVersion`. This is what lets governance workflows render custom task forms. ## Change events for tasks and lineage **Additive.** Affects webhook and event-subscription consumers. `changeEventType` adds `taskCreated`, `taskUpdated`, `entityLineageAdded`, `entityLineageDeleted` and `entityLineageUpdated`. `changeEvent` adds a `recursive` flag marking cascade deletes — a single event is recorded for the deleted root, and cascaded descendants produce no individual events. Consumers that previously counted per-child delete events must read `recursive` instead. ## Alert and notification behaviour changes **Behavioural.** Affects existing alert subscriptions. ### Thread events are scoped by their parent entity In 1.13, the entity-FQN filter returned `true` unconditionally for thread change events — thread activity bypassed the filter entirely. In 2.0 a thread event is matched against the fully qualified name of the entity the thread is **about**. An alert scoped to `service.db.schema` that previously fired for *every* conversation and task in the system now fires only for threads about entities under that name. Alerts that looked noisy will go quiet; alerts you relied on for global thread coverage will stop firing. ### Filter matching is literal, not regular-expression Alert filter functions now match fully qualified names **literally**. Descendant matching is handled explicitly. An alert whose filter used regular-expression metacharacters (`.`, `*`, `|`) to match a family of names no longer matches — enumerate the names or rely on descendant matching. ### Other alert changes | Change | Effect | | ---------------------------------------------------------------------- | ----------------------------------------------- | | Observability status triggers no longer fire on thread events | Fewer spurious observability alerts | | Owner and user name filters match usernames containing a dot | Previously-missed recipients now match | | `testDestination` redacts destination config in the response | Secrets are no longer echoed back | | Filter expressions compiled once, combined condition validated at save | Invalid filters fail at save, not at fire time | | Recipients without contact info are skipped | Partial delivery instead of total batch failure | | Incident-task comment mentions and assignee alerts rewired | Mentions work again after the task migration | Audit every alert with an entity name filter after upgrading, and test with `POST /v1/events/subscriptions/testDestination`. ## Server-side feed and task time filters **Additive.** Both the feed list and task list APIs accept `startTs` and `endTs` for server-side time-range filtering, replacing client-side windowing. # Breaking Changes - Discovery & Search | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/upgrade/breaking-changes/discovery-and-search The redesigned OpenMetadata 2.0 Explore page, the new browse-and-filter query bar, changed URL parameters, facet scoping, staged ranking and async CSV export. # Discovery & Search (Explore) The Explore page is redesigned in 2.0. Filtering, browsing, pagination, ranking and export all behave differently. This is the most user-visible change in the release. The mental model changed from *"the tree replaces your filters"* to *"the tree sets a browse location that stacks with your filters"*. OpenMetadata 2.0 Explore page showing the Browse Estate panel, the query bar and result cards ## What changed at a glance | | 1.13 | 2.0 | | --------------------- | --------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | Left panel title | **Data Assets** | **Browse Estate** | | Left panel role | Tree selection **replaced** the quick filters | Tree sets a *browse location* that **ANDs with** the quick filters | | Active filter display | A single "Clear all" text link | A dedicated **query chip row** — one chip per browse level and per selected filter value, each removable | | Filter apply | Pick values, then click **Update** | **Immediate apply** on selection; the Update button is gone | | Pagination | Inline pagination inside the results card | A dedicated pagination card below the results, with a page-size selector | | Tab switching | Refetch with a spinner every time | Served from a short-lived client cache, then silently revalidated | | Result relevance | `searchFields` scoring | Staged ranking (`nameFirstLexicalThenSignals`) with an optional per-result explanation | | CSV export | Synchronous browser download | Background job surfaced in the **Background jobs** tray | ## Explore URL parameters changed **Breaking.** Affects bookmarks, saved links, embedded iframes, and anything that constructs Explore URLs. | Parameter | 1.13 | 2.0 | | --------------- | ------------------- | ------------------------------------------ | | Page number | `page` | **`currentPage`** | | Page size | `size` | **`pageSize`** | | Browse location | — | **`browsePath`** (JSON-encoded field list) | | Cursor paging | — | `cursorType`, `cursorValue` | | Free text | `search` | `search` (unchanged) | | Quick filters | `quickFilter` | `quickFilter` (unchanged) | | Sort | `sort`, `sortOrder` | unchanged | | Deleted | `showDeleted` | unchanged | ``` /explore/tables?search=orders&page=3&size=25&sort=_score&sortOrder=desc ``` ``` /explore/tables?search=orders¤tPage=3&pageSize=25&sort=_score&sortOrder=desc ``` A 1.13 link with `?page=3&size=25` still loads Explore, but silently lands on **page 1 at the default page size**. There is no error and no redirect. The route itself (`/explore/:tab`) is unchanged. Update deep links, embedded dashboards and internal documentation to the new parameter names. ### Page size is constrained to 15, 25 or 50 **Behavioural.** Explore accepts only 15, 25 and 50 rows per page. Any other `pageSize` — including a value inherited from the user's stored global page size — is coerced back to 15. Explore pagination card showing page controls and the records-per-page selector Records-per-page menu offering 15, 25 and 50 ## Browsing no longer clears your filters **Behavioural.** Affects every Explore user. In 1.13 the left tree drove the quick filters directly — clicking a service **overwrote** the filter state. In 2.0 the tree writes to its own `browsePath` parameter, which is compiled into a separate search filter and `AND`-ed with the dropdown filters. Explore with a Snowflake browse location and a Table type filter applied at the same time, with both shown as chips In the screenshot above, `Service Type: snowflake` (from the tree) and `Type: Table` (from the Data Assets dropdown) are active together, the tree counts are re-scoped to the filter, and both are rendered as removable chips. What this means in practice: * Selecting **Tier 1** and then browsing to a schema keeps the Tier 1 filter. * Clicking a **type leaf** (Tables, Columns) in the tree writes the parent levels into `browsePath` *and* sets the type in the Data Assets quick filter — both land in one navigation. * Removing a **browse chip truncates the path from that level down** — dropping the *Service* chip also drops the Database and Schema beneath it. * Category roots that cannot hold the selected asset type are greyed out. Selecting "Table" disables every non-Database service root. The resulting URL carries both parameters independently: ``` /explore/tables ?quickFilter={"query":{"bool":{"must":[{"bool":{"should":[{"term":{"entityType.keyword":"table"}}]}}]}}} &browsePath=[{"label":"serviceType","key":"serviceType","value":[{"key":"snowflake","label":"snowflake"}]}] ``` The Explore quick filter uses `entityType.keyword`, and values are stored **lowercased** (`tablecolumn`, not `tableColumn`). If you construct `quickFilter` URLs by hand, use the lowercased value. ## Facet options are scoped differently **Behavioural.** Changes which options appear in each dropdown. In 1.13 every dropdown's aggregation was computed against the *full* combined filter — including that dropdown's own selection. Selecting `Table` in **Data Assets** shrank the Data Assets dropdown to just `Table`. 2.0 excludes a facet's own field from its own aggregation, giving the conventional faceted-search model: * **Within one facet**, values are `OR`-ed and the option list keeps showing the alternatives. * **Across facets**, constraints are `AND`-ed. * The browse location is always applied, including to facet option lists. Data Assets quick filter dropdown showing option counts, entity icons and the helper text about the browse location The dropdown also gains entity icons, human-readable labels (`tableColumn` renders as **Column**), and the helper text *"Pick values to refine. Your browse location stays put."* UI tests that clicked **Update** to commit a dropdown selection must drop that step — selections now apply immediately. Expect option lists to be longer than in 1.13 and to change as you browse. ## The query chip row **New UI.** Replaces the "Clear all" text link. A **Query** row sits under the filter dropdowns and renders the whole active query as chips. Explore query chip row showing a Service Type chip, a Type chip and a Clear All action * Browse levels are labelled `In`, `Service Type`, `Service`, `Database` and `Schema`. * The Data Assets facet renders as `Type` with a human-readable label. * With nothing selected the row shows a placeholder: *"Browsing your whole data estate — pick a filter above or a location on the left and they stack here."* ### "Clear all" scope changed | Control | 1.13 | 2.0 | | ---------------------------------------------------------- | ---------------------------------------------- | ---------------------------------------------------------------------------------- | | "Clear all" link next to Tools | Reset all filters | Removed — replaced by **Clear All** on the chip row, which still resets everything | | Clear (×) on the **Advanced Search** applied-filter banner | Reset **all** filters, including quick filters | Clears **only** the advanced query; quick filters and browse location survive | Test suites keyed on `data-testid="clear-filters"` must move to the chip row (`data-testid="explore-query-filter-chips"`). Dismissing an advanced-search query no longer resets the rest of the filter state. ## Tools menu changes Explore Tools menu showing Export, Show Deleted, Advanced Search and the new Ranking Details toggle | Item | 1.13 | 2.0 | | ------------------- | -------------------- | ------------------------- | | Deleted toggle | Labelled **Deleted** | Labelled **Show Deleted** | | Export | Synchronous download | Queues a background job | | Advanced Search | unchanged | unchanged | | **Ranking Details** | — | **New toggle** | ### Ranking Details Turning on **Ranking Details** re-runs the search with explanation enabled and renders, per result card, the relevance score, the score explanation, and which ranking stages matched. It is part of the Explore fetch key, so toggling it forces a refetch. ## Result ordering changes **Behavioural.** Affects every search and Explore result list. Search settings gain a per-asset-type `ranking` block: ```json theme={null} { "algorithm": "nameFirstLexicalThenSignals", "enabled": true, "disMaxTieBreaker": 0.05, "stages": [ /* ordered lexical ranking stages */ ], "signals": { "boostMode": "sum", "scoreMode": "sum", "maxBoost": 2.0 }, "stopWords": [], "stopWordsByLanguage": { "en": ["a", "an", "and", "are", "as", "at", "by", "for", "from", "in", "into", "is", "of", "on", "or", "the", "to", "with"] } } ``` The model is **ordered lexical stages first** — name matches outrank description and context matches — then **bounded metadata signals** (Tier, usage) capped at `maxBoost: 2.0` so they act as tie-breakers rather than dominating relevance. Setting `ranking.enabled: false` falls back to the legacy `searchFields` scoring. The 2.0.0 migration writes the default ranking configuration into any **existing** stored search settings, merging in missing stages rather than overwriting operator customisations. Expect different result ordering after the upgrade: same query, same corpus, different order. If you have automated tests asserting the top result for a given query, re-baseline them. Tune the behaviour under **Settings → Search → Ranking**, which also gains a **Reset to Default** button in 2.0. ## CSV export is now a background job **Behavioural.** Affects Explore users exporting search results. Choosing **Tools → Export** no longer downloads a file directly. It queues a background job and shows *"Export started — track progress and download the CSV from Background jobs."* The job is tracked as `jobType: CSV_EXPORT` and downloaded from `GET /v1/csvAsyncJobs/{jobId}/result`. The export scope modal also changed: exporting "all" now covers the full tab result set with an accurate pre-count, capped at 200,000 rows. The synchronous endpoint still exists. `GET /v1/search/export` is unchanged and still streams CSV directly — only the **UI** switched to `GET /v1/search/export/async`. Scripted exporters do not need to change. ## Explore result caching **Behavioural.** Affects perceived freshness on tab switches. 2.0 adds a short-lived stale-while-revalidate cache keyed by the full search dependency string (filters, browse path, query, sort, page, page size, search index, ranking-details flag). * **Cache hit** — results render synchronously with **no spinner**, then a background refetch updates them. * **Cache miss** — normal loading path. * A stale-response guard drops in-flight responses whose key no longer matches the current search, so a slow response can no longer overwrite a newer result set. UI tests that wait for a loading spinner on tab switch need to key off content instead. ## Explore tree count semantics **Behavioural.** Tree counts now aggregate over the whole data-asset index at every level, so a node's count is the total matching objects in its **subtree** (parent ≥ child), and they respect the active quick filters, advanced query filter and browse path. In 1.13 counts came from the per-entity index for the immediate children only. A count refresh no longer rebuilds the tree from scratch: lazily-loaded expanded nodes keep their children, counts and selection. ## Search API changes The `search_after` parameter is now repeated once per sort value instead of comma-joined. See [API & Schema Contracts](/v2.0.x/deployment/upgrade/breaking-changes/api-and-schema) for the full search API contract changes. # Breaking Changes 1.13 to 2.0 | Official Documentation Source: https://docs.open-metadata.org/v2.0.x/deployment/upgrade/breaking-changes/overview Complete component-level breakdown of the API, schema, ingestion, configuration and UI breaking changes between OpenMetadata 1.13 and 2.0. # Breaking Changes: 1.13 → 2.0 OpenMetadata 2.0 is a major release. It retires the thread-backed collaboration model, replaces the Explore experience, moves LLM and embedding configuration out of the search block, adds a database-backed session store, and reshapes several API contracts. This section documents every change that can break an existing client, script, deployment or user workflow — broken out by component so you only have to read the parts that apply to you. Take a full database backup before upgrading. The 2.0.0 migration renames `thread_entity`, creates more than fifteen new tables, and rewrites application, service-connection and tag rows in place. There is no automated downgrade. ## How to read this section Every entry is classified so you can triage quickly. | Class | Meaning | | --------------- | -------------------------------------------------------------------------------------------------- | | **Breaking** | Existing callers or configurations fail outright. Action is required before or during the upgrade. | | **Behavioural** | Requests still succeed, but the result, ordering or side effect differs. | | **Deprecated** | Still works in 2.0, scheduled for removal. Migrate at your convenience. | | **Additive** | New surface only. Listed where it replaces something you may currently be using. | Each entry states **what changed**, **who is affected** and **what to do**. ## The changes most likely to break you | # | Change | Component | Class | | -- | ------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------- | ----------- | | 1 | `/v1/suggestions/*` removed — suggestions are now Tasks | [Collaboration](/v2.0.x/deployment/upgrade/breaking-changes/collaboration) | Breaking | | 2 | Announcements removed from `/v1/feed`, now `/v1/announcements` | [Collaboration](/v2.0.x/deployment/upgrade/breaking-changes/collaboration) | Breaking | | 3 | `ingestionPipeline.pipelineStatuses` is now an **array** | [Ingestion & Connectors](/v2.0.x/deployment/upgrade/breaking-changes/ingestion-and-connectors) | Breaking | | 4 | Embedding and NLQ provider config moved to top-level `llmConfiguration` | [Platform & Security](/v2.0.x/deployment/upgrade/breaking-changes/platform-and-security) | Breaking | | 5 | `searchIndexingAppConfig.recreateIndex` and `useDistributedIndexing` removed | [Applications & Automation](/v2.0.x/deployment/upgrade/breaking-changes/applications-and-automation) | Breaking | | 6 | Explore URL params `page`/`size` → `currentPage`/`pageSize`, plus a new `browsePath` | [Discovery & Search](/v2.0.x/deployment/upgrade/breaking-changes/discovery-and-search) | Breaking | | 7 | `search_after` is a repeated parameter, not a comma-joined string | [API & Schema](/v2.0.x/deployment/upgrade/breaking-changes/api-and-schema) | Breaking | | 8 | Databricks Pipeline connection requires `authType` instead of `token` | [Ingestion & Connectors](/v2.0.x/deployment/upgrade/breaking-changes/ingestion-and-connectors) | Breaking | | 9 | Great Expectations 1.3+ required; the `great-expectations-1xx` extra is gone | [Ingestion & Connectors](/v2.0.x/deployment/upgrade/breaking-changes/ingestion-and-connectors) | Breaking | | 10 | `maxActiveSessionsPerUser` defaults to 5; sessions are database-backed | [Platform & Security](/v2.0.x/deployment/upgrade/breaking-changes/platform-and-security) | Behavioural | | 11 | Ingestion images move from **Python 3.10 to 3.12** | [Ingestion & Connectors](/v2.0.x/deployment/upgrade/breaking-changes/ingestion-and-connectors) | Breaking | ## Deprecation notices These are not 2.0 breaking changes, but they change what you should build on now. **Airflow as the internal orchestrator is deprecated and will be removed in a future release.** The native Kubernetes Orchestrator released in 1.12 is its replacement. New deployments should standardise on the Kubernetes Orchestrator; existing Airflow-based deployments keep working in 2.0 but should plan a migration. **Great Expectations 0.x is no longer supported.** 2.0 requires Great Expectations 1.3 or later. See [Ingestion & Connectors](/v2.0.x/deployment/upgrade/breaking-changes/ingestion-and-connectors). ## Browse by component Removed endpoints, changed request and response shapes, tightened validation, enum and default changes. The redesigned Explore page, the browse-and-filter query bar, changed URL parameters, facet scoping and staged ranking. Task redesign, removal of the Suggestions API, standalone Announcements, the Activity Stream and alert filters. Policies and new task operations, glossary and ontology, classification, governance workflows and the Policy Agent. Incidents as Tasks, inlined incident status, auto-close, multi-status filters and the removed Data Insights DQ module. FQN-based edge APIs, batch hydration, time-window traversal semantics and new lineage change events. `pipelineStatuses`, Databricks Pipeline auth, Great Expectations 1.x, log-stream SSE and new connectors. Search Indexing and RDF app config, Data Insights modules, MCP tool contracts and CSV background jobs. `llmConfiguration`, session management, `openmetadata.yaml` changes and the 2.0.0 database migration. Design-system consolidation, App Mode, persona preferences and removed UI components. ## Upgrade checklist Search scripts, dashboards, Terraform and CI jobs for `/v1/suggestions`, `/v1/feed`, `search_after=`, `pipelineStatuses` and `recreateIndex`. The `elasticsearch.naturalLanguageSearch.*` provider blocks no longer exist. Port them to `llmConfiguration` before you upgrade. `SearchIndexingApplication`, `DataInsightsApplication`, `RdfIndexApp` and `McpApplication` configurations are all rewritten by the migration. `>`, `"` and ASCII control characters are rejected by `entityName` validation in 2.0. Expect these data migrations: suggestions → `task_entity`, thread tasks → `task_entity`, system feed threads → `activity_stream`, announcements → `announcement_entity`, and `thread_entity` renamed to `thread_entity_legacy`. Reindexing in 2.0 always recreates the index. There is no incremental mode, and the ranking configuration is backfilled into `searchSettings` during the migration. Check Explore result ordering, alert subscriptions that relied on partial FQN matching, Snowflake/Databricks/Unity Catalog `policyAgentConfig` defaults, and whether any user routinely exceeds five concurrent sessions. ## Version scope | | | | ---------------- | -------------------------------------------- | | **From** | OpenMetadata 1.13 line (1.13.0 – 1.13.4) | | **To** | OpenMetadata 2.0.0 | | **Schema spec** | 86 new schema files, 107 modified, 3 removed | | **REST surface** | 208 endpoints added, 10 removed | A few entries reference capabilities that ship in Collate rather than OpenMetadata OSS — AI Governance Studio, the Policy Agent, Context Center and AI Mode. They are documented here because their schemas and REST namespaces are part of the 2.0 specification and appear in the OpenAPI surface either way. # How do the Upgrade & Backup work? Source: https://docs.open-metadata.org/v2.0.x/deployment/upgrade/how-does-it-work Understand how the upgrade system works with a version-aware approach to safely migrate platform components and configurations. # How do the Upgrade & Backup work? If this is the first time you are trying to upgrade OpenMetadata, or you just have some doubts on how the backup and upgrade process work, this is the place to be. We will cover: * What is being backed up? * When should we restore? * What happens during the migration? ## Architecture Review Let's start with a simplified design review of OpenMetadata. You can find further details [here](/v2.0.x/main-concepts/high-level-design), but we'll now focus on the Server & the Database: simple architecture All the metadata is stored in a MySQL or Postgres instance. The shape of the data is managed using [Flyway](https://flywaydb.org/), and the migration scripts are handled [here](https://github.com/open-metadata/OpenMetadata/tree/main/bootstrap/sql). In a nutshell, we have a data model in which we store all of this information. The definition of this model (table names, schemas,...) is managed using Flyway migrations. **In every release, the structure of this data model can change**. This means that the shape of the database is tightly coupled to your OpenMetadata Server version. ## 1. What is being backed up? You can find all the necessary information on how to run the backups [here](/v2.0.x/deployment/backup-restore-metadata). When we backup the data, we are creating an SQL file that follows the shape of the database of a specific version. Thus, if we have some issues on our instance, and we ever need to restore that data, it will only fit to a database with that same version shape. backup ## 2. When should we restore? Now that we understand what is being backed up and how it looks like, when (and where) should we restore? * **When**: We restore the data if we need to get back in time. Restoring is never needed during the upgrade process. * **Where**: We will restore the data to a clean database with the Flyway migrations at the same version as the backed up data. The usual process will be: * We start with a clean database (no tables in it). * We run the migrations ([docs](/v2.0.x/deployment/bare-metal#4-prepare-the-openmetadata-database-and-indexes)) or we start the OpenMetadata server, which will automatically run the migrations. * With the server stopped, we [restore](/v2.0.x/deployment/backup-restore-metadata) the data. Note that the restore process will not work if we try to restore some data taken from version X to a database shaped with version Y. ## 3. What happens during the migration? We have been explaining how each OpenMetadata Server relies on a specific data model to store the metadata. What happens when we upgrade from version X to Y? backup The migration process will take care of getting the data shaped as X and transform it to the Y shape. After the migration is done, the server in version Y will be able to run properly. # OpenMetadata Documentation Source: https://docs.open-metadata.org/v2.0.x/index Unified platform for data discovery, lineage, and governance
# Unlock the Power of Metadata

Start with OpenMetadata and learn how to document, discover, and govern your data assets end-to-end.

⌘K
Deployment

Get Started with OpenMetadata

Choose how you want to deploy and use OpenMetadata based on your needs.

Best for: PoC, exploration
Quick Start

Quick Start

Try OpenMetadata quickly using a hosted sandbox or local Docker setup to explore features with minimal effort.

Read More Right Arrow
Best for: Teams, orgs
Production

Production

Deploy OpenMetadata securely at scale using Kubernetes, cloud-managed services, or bare metal for production workloads.

Read More Right Arrow
Best for: Existing users
Upgrade

Upgrade

Upgrade your OpenMetadata deployment to the latest supported minor version with compatibility checks and safe rollback options.

Read More Right Arrow
How-to Guides

Quick Setup Guides

Dive into our library of guides to master OpenMetadata's features.

Connectors

Seamless Data Integrations

Integrate diverse data sources with OpenMetadata.

Highlights

What’s New in OpenMetadata

Stay up to date with the latest features, enhancements, and platform updates in OpenMetadata.

Context Center
Context Center

Give your team a shared home for company knowledge. MCP assistants can search shared knowledge pills extracted from Documents alongside your metadata.

Read More Right Arrow
Connectors
New Connectors

Ingest metadata from two new sources: Prefect for pipeline orchestration, and Omni for dashboards and analytics.

Read More Right Arrow
Dimensional Validation
AI SDK

Build and embed custom AI agents with programmatic access to OpenMetadata through MCP tools across Python, TypeScript, Java, and CLI.

Read More Right Arrow
MCP Server
MCP Server

Enable seamless integration between OpenMetadata and external tools using the Model Context Protocol (MCP).

Read More Right Arrow
Dimensional Validation
Dimensional Validation

Validate data quality across key dimensions such as completeness, accuracy, consistency, and timeliness.

Read More Right Arrow
Data Quality as Code

Manage data quality checks using version-controlled configurations.

Read More Right Arrow
Quick Links

Trending Quick Links

Explore the most popular resources and links, handpicked by the OpenMetadata community.

©2026 OpenMetadata. All rights reserved.
# Quickstart | OpenMetadata Quickstart Guide Source: https://docs.open-metadata.org/v2.0.x/quick-start Get started with your platform using step-by-step onboarding instructions for setup, integration, and key feature exploration. # Quickstart ## Getting Started with OpenMetadata This section will guide you through the installation process and the initial steps for using OpenMetadata. Before you proceed with the installation, you have the option to explore OpenMetadata’s features using the sandbox environment. This allows you to try out its capabilities before setting it up. Get hands-on experience and discover how OpenMetadata can streamline your workflows by trying the sandbox environment today. Interact with a sample installation with 0 setup to explore our Discovery, Governance and Collaboration features. Additionally, you can explore the OpenMetadata by setting up one of the two approaches locally with your own customizations, allowing you to test and experiment with the features. Get OpenMetadata up and running with docker in under 5 minutes! Get OpenMetadata up and running with kubernetes in under 5 minutes! Set up and explore OpenMetadata's core features, from basic configuration to advanced functionalities, for a seamless onboarding experience. Unlock metadata insights for informed business decisions. # Getting Started with OpenMetadata for Data cataloging Source: https://docs.open-metadata.org/v2.0.x/quick-start/getting-started Follow this guide to get started quickly with core deployment, authentication, data access, and configuration for your platform setup. # Getting Started Welcome to OpenMetadata's unified platform for data discovery, observability, and governance. Our platform centralizes all data context to help you build high-quality data and AI assets. This guide provides the necessary information to set up your OpenMetadata environment in 30 minutes. ## How Does OpenMetadata Work? OpenMetadata is designed to support both technical and non-technical data practitioners across various use cases, including data discovery, lineage, observability, quality, collaboration, governance, and insights. The platform includes a library of 90+ turnkey connectors to easily ingest metadata from sources such as data warehouses, data lakes, streaming platforms, dashboards, and ML models. For custom data sources, APIs are available to streamline metadata ingestion. Metadata from these sources is organized into a Unified Metadata Graph, providing a single, comprehensive source of truth for your entire data estate. This centralized metadata is accessible through a unified user interface, eliminating the need for practitioners to switch between multiple catalogs, quality, or governance tools. OpenMetadata can also be extended with applications, such as AI-driven productivity tools like OpenMetadata AI, or through custom-built workflows that integrate the platform with existing systems. The platform’s native collaboration features support shared workflows, enabling different teams—data platform engineers, governance professionals, data scientists/analysts, and business users—to collaborate effectively in a single environment. ## Key Features of OpenMetadata Before we get started, here’s a quick summary of some of OpenMetadata’s main features: ### Discovery * Integrated catalog, data quality, and glossary * Natural language search, filtering, and faceting * 90+ turnkey data connectors ### Lineage * Table and column-level lineage * Automated data estate mapping with APIs * Lineage layers, search capabilities * Governance and PII automation, with manual customization ### Observability * Alerting and notifications * Incident management and third-party notifications * Pipeline monitoring, root cause analysis, and anomaly detection * Data profiler for performance insights ### Quality * Table and column-level test cases * No-code and SQL-based data quality tests * Test suites, reporting, and dashboards * Quality lineage maps and widgets for data insights ### Collaboration * Announcements, tasks, and team conversations * Slack/Teams integration for communication * Activity feed and team dashboards for tracking progress ### Governance * Business glossary and classification tags * Automated PII classification and description generation ### Insights * Data asset analytics and app usage metrics * Coverage KPIs and ownership dashboards * Data health and governance reports ## Onboarding Journey In this section, you will find a series of guides to help you get started with OpenMetadata. These guides cover both basic and advanced features of the platform, organized in a step-by-step format and divided into milestones that can be achieved over the course of your onboarding process. Discover the right data assets to make timely business decisions. # Day 1 Getting Started | OpenMetadata Day 1 Guide Source: https://docs.open-metadata.org/v2.0.x/quick-start/getting-started/day-1 Connect your first data sources, run ingestion, and invite users to start collaborating in OpenMetadata. # Getting Started: Day 1 Get started with your OpenMetadata service in a few simple steps: 1. **Set up a Data Connector**: Connect your data sources to begin collecting metadata. 2. **Ingest Metadata**: Run the metadata ingestion process to gather and push data insights. 3. **Invite Users**: Add team members to collaborate and manage metadata together. 4. **Explore the Features**: Dive into OpenMetadata's extensive feature set to unlock the full potential of your data. **Ready to begin? Let's get started!** ## Step 1: Set up a Data Connector Once you have logged into your OpenMetadata instance, set up a data connector to start ingesting metadata. OpenMetadata provides [90+ turnkey connectors](/v2.0.x/connectors) for a wide range of services, including: * Databases * Dashboards * Messaging services * Pipelines * ML models * Storage services * Other metadata services For [custom data sources](/v2.0.x/connectors/custom-connectors), metadata ingestion can also be set up via API. There's two options on how to set up a data connector: 1. **Run the connector in OpenMetadata**: In this scenario, an IP will be provided when you add the service. You must allow access to this IP in your data sources. Guide to start ingesting metadata seamlessly from your data sources. 2. **Run the connector in your infrastructure or on a local machine:**: This hybrid model allows organizations to run metadata ingestion components within their own infrastructure. This ensures that OpenMetadata's managed service doesn't need direct access to the underlying data. Only metadata is collected locally and securely transmitted to the platform, maintaining data privacy and security. ## Step 2: Ingest Metadata Once the connector is set up, configure a [metadata ingestion pipeline](/v2.0.x/how-to-guides/admin-guide/how-to-ingest-metadata) to import metadata into OpenMetadata on a regular schedule. * Navigate to **Settings > Services > Databases** and select the service you added. Ingestion Navigation * Go to the **Agents** tab and click **Add Metadata Agent**. Add Metadata Agent * Configure any required settings or filters for the ingestion process. Documentation is available in the side panel for reference. Configure Ingestion * Schedule the pipeline to ingest metadata at regular intervals. Schedule Ingestion * In addition to metadata ingestion, you can set up pipelines for lineage, profiler data, or dbt information. * Once the metadata ingestion is completed, the ingested data assets can be viewed under the **Explore** section in the main menu. Ingested Data * You can repeat these steps to ingest metadata from other data sources as needed. ## Step 3: Invite Users After Setting Up SMTP ### SMTP Configuration To invite users you will need to ensure that you have an SMTP server available. With the information for your SMTP server you can configure OpenMetadata to send email alerts by updating the details from the UI. To update the details from the UI, navigate to Settings > Preferences > Email Email Configuration If you encounter issues connecting to the SMTP server, ensure that the correct Certificate Authority (CA) is configured to trust the SMTP host. Additionally, use the DNS hostname instead of the IP address in the SMTP server endpoint configuration to avoid certificate validation errors. ### Inviting Users After metadata has been ingested into OpenMetadata, you can [invite users](/v2.0.x/how-to-guides/admin-guide/teams-and-users/invite-users) to collaborate on the data and assign different roles. * Go to **Settings > Team & User Management > Users**. Users Navigation * Click **Add User**, then enter their email and other required details to grant access to the platform. Adding New User * Organize users into different Teams, and assign them specific Roles. * Users inherit access permissions from their assigned teams and roles. * Admin access can be granted to users who need full access to all settings, including the ability to invite new users. Users Profile * New users will receive an email invitation to set up their accounts. ## Step 4: Explore Features of OpenMetadata OpenMetadata offers a comprehensive platform for data teams to: * Break down data silos * Securely share data assets across various sources * Foster collaboration around trusted data * Establish a documentation-first data culture within your organization Explore these features and unlock the full potential of your data using OpenMetadata. Discover the right data assets to make timely business decisions. Foster data team collaboration to enhance data understanding. Trust your data with quality tests & monitor the health of your data systems. Trace the path of data across tables, pipelines, and dashboards. Define KPIs and set goals to proactively hone the data culture of your company. Enhance your data platform governance using OpenMetadata. ## Deep Dive into OpenMetadata: Guides for Admins and Data Users Admin users can get started with OpenMetadata with just three quick and easy steps & know-it-all with the advanced guides. Get to know the basics of OpenMetadata and about the data assets that you can explore in the all-in-one platform. # Database service setup Source: https://docs.open-metadata.org/v2.0.x/quick-start/getting-started/day-1/database-service-setup Configure a database service connection in OpenMetadata and prepare it for metadata ingestion. # Setting Up a Database Service for Metadata Extraction You can quickly set up a database service for metadata extraction in OpenMetadata SaaS. Below is an example of how to configure a connection using the `Snowflake` Connector: 1. ## Log in to OpenMetadata SaaS * Navigate to **Settings > Services > Databases**. * Click on **Add New Service**. Adding Database Service 2. ## Select Database Type * Choose the database type for your service. In this example, select `Snowflake`. * Enter the required details, such as the **Name** and **Description**, to identify the database. Selecting Database Service 3. ## Enter Connection Details * Provide the necessary connection parameters, such as hostname, port, credentials, etc. * The side panel offers guidance with available documentation, and you can also refer to the specific `Snowflake` Connector [documentation](/v2.0.x/connectors)for more information. Configure Service Updating Connection Details 4. ## Test the Connection Click Test Connection to verify the setup. This will check if OpenMetadata can reach the Snowflake service. Verifying the Test Connection 5. ## Set Default Data Filters Configure default filters to control which databases, schemas, and tables are included or excluded during ingestion. #### Default Database Filter Pattern * **Includes / Excludes**:\ To add a filter pattern, simply type it in and press `Enter`. #### Default Schema Filter Pattern * **Includes / Excludes**:\ To add a filter pattern, simply type it in and press `Enter`. #### Default Table Filter Pattern * **Includes / Excludes**:\ To add a filter pattern, simply type it in and press `Enter`. These filters help streamline the ingestion process by targeting only the relevant data assets. # Try OpenMetadata in Docker Source: https://docs.open-metadata.org/v2.0.x/quick-start/local-docker-deployment Get OpenMetadata running locally in minutes with Docker. Step-by-step setup guide, configuration tips, and troubleshooting for your metadata platform. # Local Docker Deployment This installation doc will help you start a OpenMetadata standalone instance on your local machine. If you'd rather see the steps in a guided tutorial, we've got you covered! Otherwise, feel free to read the content below 👇