Azure Key Vault

The setup steps covers the use of the managed version of the Azure Key Vault as secrets manager but for the non-managed follow only the steps related to the Airflow server and CLI.

Setup

1. Create Principal

Service Principal

Go to Microsoft Entra ID and create an App Registration.
Inside the App Registration, go to Certificates & Secrets and create a Client secret. Note down the Value, it will be our clientSecret configuration.
From the App Registration overview page, note down the Application (client) ID and the Directory (tenant) ID.

Managed Identity (recommnded)

In your Azure subscription create Manged Identity
Use this created identity - for AKS users this means you need to use Pod Identity or Workload Identity (recommnded).

Note that the using Managed Identity require using default Authentication Credential.

2. Add RBAC roles

It if possible to use different Principals for OpenMetadata Server and the Ingestion. In that case the server needs higher privileges - Key Vault Secrets Officer - to be able to create/read/update secrets in the Vault. While the Airflow part only needs to read the secrets hence the role Key Vault Secrets Officer.

Open Metadata server

In your Key Vault overview page, note down the Vault URI.
Go to Access Control (IAM) and click on Add Role Assignment.
Give the permission Key Vault Secrets Officer to your Principal.

Airflow

In your Key Vault overview page, note down the Vault URI.
Go to Access Control (IAM) and click on Add Role Assignment.
Give the permission Key Vault Secrets User to your Principal.

3. Update configuration

We have to set up the secret manager provider we want to use, that in our case is azure-kv, and the credentials. The changes to be done in openmetadata.yaml file of the OpenMetadata server are:

Default Azure Credential

---
secretsManagerConfiguration:
  secretsManager: managed-azure-kv # or env var SECRET_MANAGER. For non-managed use 'azure-kv'.
  prefix: ${SECRET_MANAGER_PREFIX:-""} # Define the secret key ID as <prefix>-<clusterName>-<key>
  tags: ${SECRET_MANAGER_TAGS:-[]} # Add tags to the created resource. Format is `[key1:value1,key2:value2,...]`
  parameters:
    enabled: true
    vaultName: ${OM_SM_VAULT_NAME:-""}
pipelineServiceClientConfiguration:
  secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-airflow}

For Helm Values, you will need to add PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER as part of extraEnvs. This will look like below -

---
...
extraEnvs:
- name: PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER
  value: airflow
...

Client Secret Credential

---
secretsManagerConfiguration:
  secretsManager: managed-azure-kv # or env var SECRET_MANAGER. For non-managed use 'azure-kv'.
  prefix: ${SECRET_MANAGER_PREFIX:-""} # Define the secret key ID as <prefix>-<clusterName>-<key>
  tags: ${SECRET_MANAGER_TAGS:-[]} # Add tags to the created resource. Format is `[key1:value1,key2:value2,...]`
  parameters:
    enabled: true
    clientId: ${OM_SM_CLIENT_ID:-""}
    clientSecret: ${OM_SM_CLIENT_SECRET:-""}
    tenantId: ${OM_SM_TENANT_ID:-""}
    vaultName: ${OM_SM_VAULT_NAME:-""}
pipelineServiceClientConfiguration:
  secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-airflow}

For Helm Values, you will need to add PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER as part of extraEnvs. This will look like below -

---
...
extraEnvs:
- name: PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER
  value: airflow
...

The changes to be done in airflow.yaml file of the Airflow are:

Note that the Key Vault Name parameter is MANDATORY for the system to know where to store and retrieve the secrets.

And these are the changes required in airflow.cfg of our Airflow instance:

...
[openmetadata_secrets_manager]
azure_key_vault_name = <Key Vault Name>
azure_tenant_id = <Tenant / Directory ID>
azure_client_id = <App Registration ID>
azure_client_secret = <App Registration Secret>
...

As an alternative to editing the airflow.cfg file, we can also set the following environment variables:

AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_KEY_VAULT_NAME= <Key Vault Name>
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_TENANT_ID= <Tenant / Directory ID>
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_CLIENT_ID= <App Registration ID>
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_CLIENT_SECRET= <App Registration Secret>

If only the <Key Vault Name>, parameter is provided, we will use Azure’s default Authentication Credential.

Also if you are using Microsoft Entra Workload ID with Service Account Token Volume Projection then you need also to use projected service account instead one created by Airflow and OpenMetadata:airflow.yaml:

---
serviceAccount:
  create: false
  name: "name-of-your-service-account"

openmetadata.yaml:

---
serviceAccount:
  create: false
  name: "name-of-your-service-account"

3. Migrate Secrets & restart both servers

After updating the configuration files, we are ready to migrate the secrets and restart both services. In order to ensure that the current sensitive information is properly migrated to the Secrets Manager, you need to run the following command:

./bootstrap/openmetadata-ops.sh migrate-secrets

Make sure you are running it with the same environment variables required by the server. If everything goes as planned, all the data would be displayed using the parameters names which starts with openmetadata-... in your Key Vault console. Note: If we want to change the starting path for our secrets names from openmetadata to a different one, we have to change the property clusterName in our openmetadata.yaml. Also, if you inform the prefix value, it will be added before the clusterName, i.e., <prefix>-<clusterName>-<key> You can inform the tags as well as a list of strings [key1:value1,key2:value2,...]. These tags will be added to the created secret.

CLI

After enabling the Secret Manager, we also have to make a slight change in our workflows YAML files. In the workflowConfig we have to add the secret manager configuration:

workflowConfig:
  openMetadataServerConfig:
    secretsManagerProvider: azure-kv
    secretsManagerLoader: env
    hostPort: <OpenMetadata host and port>
    authProvider: <OpenMetadata auth provider>

Make sure to follow the steps here to allow the Python client to authenticate to Azure.

Note that the AZURE_KEY_VAULT_NAME variable is REQUIRED to know against which Key Vault service to point to.

You can specify as well the environment variables of your App Registration if you’re running the ingestion outside of Azure: docs.

Airflow

If you enabled the Secret Manager and you are using your own Airflow to run the ingestions, make sure to configure your YAML files as:

workflowConfig:
  openMetadataServerConfig:
    secretsManagerProvider: azure-kv
    secretsManagerLoader: airflow
    hostPort: <OpenMetadata host and port>
    authProvider: <OpenMetadata auth provider>

and follow the same environment variables to set up the Airflow configuration:

AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_KEY_VAULT_NAME= <Key Vault Name>
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_TENANT_ID= <Tenant / Directory ID>
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_CLIENT_ID= <App Registration ID>
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_CLIENT_SECRET= <App Registration Secret>

Note that the AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_KEY_VAULT_NAME variable is REQUIRED to know against which Key Vault service to point to.

Deployment

Azure Key Vault | OpenMetadata Secrets Manager Guide

Azure Key Vault

Setup

1. Create Principal

Service Principal

Managed Identity (recommnded)

2. Add RBAC roles

Open Metadata server

Airflow

3. Update configuration

Default Azure Credential

Client Secret Credential

3. Migrate Secrets & restart both servers

CLI

Airflow

​Azure Key Vault

​Setup

​1. Create Principal

​Service Principal

​Managed Identity (recommnded)

​2. Add RBAC roles

​Open Metadata server

​Airflow

​3. Update configuration

​Default Azure Credential

​Client Secret Credential

​3. Migrate Secrets & restart both servers

​CLI

​Airflow

Azure Key Vault

Setup

1. Create Principal

Service Principal

Managed Identity (recommnded)

2. Add RBAC roles

Open Metadata server

Airflow

3. Update configuration

Default Azure Credential

Client Secret Credential

3. Migrate Secrets & restart both servers

CLI

Airflow