Azure Key Vault

The setup steps covers the use of the managed version of the Azure Key Vault as secrets manager but for the non-managed follow only the steps related to the Airflow server and CLI.

Setup

1. Create Principal

Service Principal

1. Go to Microsoft Entra ID and create an App Registration.
2. Inside the App Registration, go to Certificates & Secrets and create a Client secret. Note down the Value, it will be our clientSecret configuration.
3. From the App Registration overview page, note down the Application (client) ID and the Directory (tenant) ID.

Managed Identity (recommnded)

1. In your Azure subscription create Manged Identity
2. Use this created identity - for AKS users this means you need to use Pod Identity or Workload Identity (recommnded).

2. Add RBAC roles

It if possible to use different Principals for OpenMetadata Server and the Ingestion. In that case the server needs higher privileges - Key Vault Secrets Officer - to be able to create/read/update secrets in the Vault.

While the Airflow part only needs to read the secrets hence the role Key Vault Secrets Officer.

Open Metadata server

1. In your Key Vault overview page, note down the Vault URI.
2. Go to Access Control (IAM) and click on Add Role Assignment.
3. Give the permission Key Vault Secrets Officer to your Principal.

Airflow

1. In your Key Vault overview page, note down the Vault URI.
2. Go to Access Control (IAM) and click on Add Role Assignment.
3. Give the permission Key Vault Secrets Users to your Principal.

3. Update configuration

We have to set up the secret manager provider we want to use, that in our case is azure-kv, and the credentials.

The changes to be done in openmetadata.yaml file of the OpenMetadata server are:

Default Azure Credential

For Helm Values, you will need to add PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER as part of extraEnvs. This will look like below -

Client Secret Credential

For Helm Values, you will need to add PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER as part of extraEnvs. This will look like below -

The changes to be done in airflow.yaml file of the Airflow are:

And these are the changes required in airflow.cfg of our Airflow instance:

As an alternative to editing the airflow.cfg file, we can also set the following environment variables:

If only the <Key Vault Name>, parameter is provided, we will use Azure's default Authentication Credential.

3. Migrate Secrets & restart both servers

After updating the configuration files, we are ready to migrate the secrets and restart both services.

In order to ensure that the current sensitive information is properly migrated to the Secrets Manager, you need to run the following command:

Make sure you are running it with the same environment variables required by the server.

If everything goes as planned, all the data would be displayed using the parameters names which starts with openmetadata-... in your Key Vault console.

Note: If we want to change the starting path for our secrets names from openmetadata to a different one, we have to change the property clusterName in our openmetadata.yaml. Also, if you inform the prefix value, it will be added before the clusterName, i.e., <prefix>-<clusterName>-<key>

You can inform the tags as well as a list of strings [key1:value1,key2:value2,...]. These tags will be added to the created secret.

CLI

After enabling the Secret Manager, we also have to make a slight change in our workflows YAML files. In the workflowConfig we have to add the secret manager configuration:

Make sure to follow the steps here to allow the Python client to authenticate to Azure.

You can specify as well the environment variables of your App Registration if you're running the ingestion outside of Azure: docs.

Airflow

If you enabled the Secret Manager and you are using your own Airflow to run the ingestions, make sure to configure your YAML files as:

and follow the same environment variables to set up the Airflow configuration: