deployment

No menu items for this category

AWS Secrets Manager

The setup steps covers the use of the managed version of the AWS Secrets Manager as secrets manager but for the non-managed follow only the steps related to the Airflow server and CLI.

These are the permissions required in the IAM policy to enable the AWS Secrets Manager in OpenMetadata.

We have to set up the secret manager provider we want to use, that in our case is aws, and the credentials for our AWS account.

The changes to be done in openmetadata.yaml file of the OpenMetadata server are:

And these are the changes required in airflow.cfg of our Airflow instance:

As an alternative to editing the airflow.cfg file, we can also set the following environment variables:

If no parameters are provided for the AWS account, or only <aws region>, it will use the default credentials. The default credential will look for credentials in:

  1. Environment variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
  2. Shared credential file - ~/.aws/credentials
  3. AWS config file - ~/.aws/config
  4. Assume Role provider
  5. Instance metadata service on an Amazon EC2 instance that has an IAM role configured

More info in AWS SDK for Java and Boto3 Docs

After updating the configuration files, we are ready to migrate the secrets and restart both services.

In order to ensure that the current sensitive information is properly migrated to the Secrets Manager, you need to run the following command:

Make sure you are running it with the same environment variables required by the server.

If everything goes as planned, all the data would be displayed using the secrets names which starts with /openmetadata/... in your AWS Secrets Manager console. The following image shows what it should look like:

secrets-manager-console

Note: If we want to change the starting path for our secrets names from openmetadata to a different one, we have to change the property clusterName in our openmetadata.yaml. Also, if you inform the prefix value, it will be added before the clusterName, i.e., /<prefix>/<clusterName>/<key>.

You can inform the tags as well as a list of strings [key1:value1,key2:value2,...]. These tags will be added to the resource created in AWS.

After enabling the Secret Manager, we also have to make a slight change in our workflows YAML files. In the workflowConfig we have to add the secret manager configuration:

Then, in the environment running the CLI make sure to have an environment variable AWS_DEFAULT_REGION with the rest of the required configurations from AWS.

If you enabled the Secret Manager and you are using your own Airflow to run the ingestions, make sure to configure your YAML files as:

and follow the same environment variables to set up the Airflow configuration: