Extract MWAA Metadata

There are two ways to extract metadata from an Amazon MWAA environment, depending on where the ingestion process runs:

REST API connection (recommended). The ingestion runs from OpenMetadata (UI-scheduled or via the CLI) and reads DAGs through the MWAA web server, using AWS credentials. No access to the underlying database is needed, and nothing has to be installed inside MWAA for metadata extraction (see below for the additional OpenLineage setup required for table-level lineage).
Running the ingestion inside MWAA. A DAG deployed in MWAA extracts metadata from the local Airflow backend and pushes it to OpenMetadata. Use this when your MWAA environment runs an Airflow version older than 2.4.3, or when the ingestion cannot be given AWS credentials.

Do not use the Backend connection type from outside MWAA. Backend reads the metadata database of the Airflow environment the ingestion workflow is running inside. OpenMetadata uses the hostPort field only to build clickable DAG links, never to fetch metadata. If you create an Airflow service pointing at your MWAA URL with a Backend connection and run it from OpenMetadata’s ingestion pipeline, you will silently ingest the DAGs of OpenMetadata’s own Airflow instead of MWAA’s. Backend is only valid when the workflow executes inside MWAA itself (option 2 below).Direct database connections (Postgres/MySQL) aren’t a general alternative either. MWAA’s metadata database lives in an AWS-managed private network, reachable only from inside that VPC or a peered network — the same constraint a private MWAA web server has for the REST API connection below. See the ECS approach below for a working example from inside the VPC. It isn’t an option for ingestion running entirely outside AWS.

1. REST API Connection (Recommended)

OpenMetadata calls the MWAA control plane’s InvokeRestApi action directly with your AWS credentials — no login token is created or exchanged. That action proxies the request to the Airflow REST API to read DAGs, runs, and task states. OpenMetadata always uses the Airflow v1 REST API for MWAA connections. MWAA handles the Airflow 2/3 version difference internally, so no client-side version detection happens.

Requirements

The MWAA environment must run Airflow 2.4.3 or newer (the minimum version supported by the MWAA InvokeRestApi API).
The IAM identity used in the connection needs airflow:InvokeRestApi, scoped to the environment’s Airflow role, not the environment itself:

{
  "Effect": "Allow",
  "Action": "airflow:InvokeRestApi",
  "Resource": "arn:aws:airflow:<region>:<account-id>:role/<environment-name>/<airflow-role>"
}

Replace <airflow-role> with the Apache Airflow role to grant: Admin, Op, User, Viewer, or Public.

If the MWAA environment uses a private web server, InvokeRestApi can’t be called from outside its VPC. The ingestion runtime must run inside, or have network connectivity to, that VPC.

Configuration

In the UI, create an Airflow pipeline service, select the REST API Connection type, and choose MWAA authentication. The equivalent YAML:

source:
  type: airflow
  serviceName: airflow_mwaa
  serviceConnection:
    config:
      type: Airflow
      hostPort: https://<environment-id>.c2.airflow.<region>.on.aws
      numberOfStatus: 10
      connection:
        type: RestAPI
        authConfig:
          mwaaConfig:
            mwaaEnvironmentName: <environment-name>
            awsConfig:
              awsRegion: <region>
              awsAccessKeyId: <access-key-id>
              awsSecretAccessKey: <secret-access-key>
              awsSessionToken: <session-token>  # only for temporary (STS) credentials
        apiVersion: auto
        verifySSL: true
  sourceConfig:
    config:
      type: PipelineMetadata
sink:
  type: metadata-rest
  config: {}
workflowConfig:
  loggerLevel: INFO
  openMetadataServerConfig:
    hostPort: <OpenMetadata host and port>
    authProvider: <OpenMetadata auth provider>

When the ingestion runs on AWS infrastructure with an attached IAM role (EC2, ECS, Lambda), omit the access key, secret, and session token. OpenMetadata picks up the role’s credentials automatically. Cross-account access is supported via assumeRoleArn. For the full parameter reference, see the REST API Connection documentation.

The REST API alone doesn’t capture table-level lineage. It requires the OpenLineage provider configured in MWAA. For the required setup, see the OpenLineage Setup Summary.

2. Running the Ingestion Inside MWAA

If the REST API connection isn’t an option, run the ingestion as a DAG inside MWAA using the Backend connection, which reads MWAA’s own metadata database directly. The OpenMetadata server must be reachable from MWAA. For more information about running connectors from MWAA, see Run the ingestion from AWS MWAA. There are three supported approaches:

Install the openmetadata-ingestion package as a requirement in the Airflow environment, then run the process using a PythonOperator.
Configure an ECS cluster and run the ingestion as an ECS Operator.
Install a plugin and run the ingestion with the PythonVirtualenvOperator.

2.1 Extracting MWAA Metadata with the PythonOperator

As the ingestion process will be happening locally in MWAA, we can prepare a DAG with the following YAML configuration:

source:
  type: airflow
  serviceName: airflow_mwaa
  serviceConnection:
    config:
      type: Airflow
      hostPort: http://localhost:8080
      numberOfStatus: 10
      connection:
        type: Backend
  sourceConfig:
    config:
      type: PipelineMetadata
sink:
  type: metadata-rest
  config: {}
workflowConfig:
  loggerLevel: INFO
  openMetadataServerConfig:
    hostPort: <OpenMetadata host and port>
    authProvider: <OpenMetadata auth provider>

2.2 Extracting MWAA Metadata with the ECS Operator

After setting up the ECS Cluster, you’ll need first to check the MWAA database connection

Getting MWAA Database Connection

To extract MWAA information we will need to take a couple of points in consideration:

How to get the underlying database connection info, and
How to make sure we can reach such database.

The happy path would be going to the Airflow UI > Admin > Configurations and finding the sql_alchemy_conn parameter. However, MWAA is not providing this information. Instead, we need to create a DAG to get the connection details once. The DAG can be deleted afterwards. We want to use a Python Operator that will retrieve the Airflow’s Session data:

import logging
import os
from datetime import timedelta

import yaml
from airflow import DAG

try:
    from airflow.operators.python import PythonOperator
except ModuleNotFoundError:
    from airflow.operators.python_operator import PythonOperator

from airflow.utils.dates import days_ago

from airflow.configuration import conf

default_args = {
    "owner": "user_name",
    "email": ["username@org.com"],
    "email_on_failure": False,
    "retries": 3,
    "retry_delay": timedelta(minutes=5),
    "execution_timeout": timedelta(minutes=60),
}

def get_data():
    from airflow.settings import Session

    logging.info("SQL ALCHEMY CONN")
    sqlalchemy_conn = conf.get("core", "sql_alchemy_conn", fallback=None)
    logging.info(sqlalchemy_conn)


with DAG(
    "airflow_database_connection",
    default_args=default_args,
    description="An example DAG which pushes Airflow data to OM",
    start_date=days_ago(1),
    is_paused_upon_creation=True,
    schedule_interval="@once",
    catchup=False,
) as dag:
    ingest_task = PythonOperator(
        task_id="db_connection",
        python_callable=get_data,
    )

After running the DAG, we can store the connection details and remove the dag file from S3. Note that trying to log the conf.get("core", "sql_alchemy_conn", fallback=None) details might either result in:

An empty string, depending on the Airflow version: If that’s the case, you can use update the line to be conf.get("database", "sql_alchemy_conn", fallback=None).
The password masked in ****. If that’s the case, you can use sqlalchemy_conn = list(conf.get("core", "sql_alchemy_conn", fallback=None)), which will return the results separated by commas.

Preparing the metadata extraction

Then, prepare the YAML config with the information you retrieved above. For example:

source:
  type: airflow
  serviceName: airflow_mwaa_ecs_op
  serviceConnection:
    config:
      type: Airflow
      hostPort: http://localhost:8080
      numberOfStatus: 10
      connection:
        type: Postgres
        username: adminuser
        password: ...
        hostPort: vpce-075b93a08ef7a8ffc-c88g16we.vpce-svc-0dfc8a341f816c134.us-east-2.vpce.amazonaws.com:5432
        database: AirflowMetadata
  sourceConfig:
    config:
      type: PipelineMetadata
sink:
  type: metadata-rest
  config: {}
workflowConfig:
  openMetadataServerConfig:
    enableVersionValidation: false
    hostPort: https://sandbox.open-metadata.org/api
    authProvider: openmetadata
    securityConfig:
      jwtToken: ...

2.3 Extracting MWAA Metadata with the Python Virtualenv Operator

This will be similar as the first step, where you just need the simple Backend connection YAML:

source:
  type: airflow
  serviceName: airflow_mwaa
  serviceConnection:
    config:
      type: Airflow
      hostPort: http://localhost:8080
      numberOfStatus: 10
      connection:
        type: Backend
  sourceConfig:
    config:
      type: PipelineMetadata
sink:
  type: metadata-rest
  config: {}
workflowConfig:
  loggerLevel: INFO
  openMetadataServerConfig:
    hostPort: <OpenMetadata host and port>
    authProvider: <OpenMetadata auth provider>

Connectors

Connector

Ingestion

Extract MWAA Metadata

Extract MWAA Metadata

1. REST API Connection (Recommended)

Requirements

Configuration

2. Running the Ingestion Inside MWAA

2.1 Extracting MWAA Metadata with the PythonOperator

2.2 Extracting MWAA Metadata with the ECS Operator

Getting MWAA Database Connection

Preparing the metadata extraction

2.3 Extracting MWAA Metadata with the Python Virtualenv Operator

​Extract MWAA Metadata

​1. REST API Connection (Recommended)

​Requirements

​Configuration

​2. Running the Ingestion Inside MWAA

​2.1 Extracting MWAA Metadata with the PythonOperator

​2.2 Extracting MWAA Metadata with the ECS Operator

​Getting MWAA Database Connection

​Preparing the metadata extraction

​2.3 Extracting MWAA Metadata with the Python Virtualenv Operator

Extract MWAA Metadata

1. REST API Connection (Recommended)

Requirements

Configuration

2. Running the Ingestion Inside MWAA

2.1 Extracting MWAA Metadata with the PythonOperator

2.2 Extracting MWAA Metadata with the ECS Operator

Getting MWAA Database Connection

Preparing the metadata extraction

2.3 Extracting MWAA Metadata with the Python Virtualenv Operator