Run the Glue Connector Externally - OpenMetadata Documentation

In this section, we provide guides and references to use the Glue connector. Configure and schedule Glue metadata and profiler workflows from the OpenMetadata UI:

Requirements
How Glue Maps to OpenMetadata
Metadata Ingestion
dbt Integration

How to Run the Connector Externally

To run the Ingestion via the UI you’ll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework anywhere.

External Schedulers

Get more information about running the Ingestion Framework Externally

Requirements

The connector reads metadata from the AWS Glue Data Catalog through the AWS Glue API. Grant the following AWS Identity and Access Management (IAM) permissions to the AWS identity that the connector uses:

glue:GetDatabases: Lists the Glue databases in the catalog.
glue:GetTables: Lists the tables and views in each Glue database, with their columns.

Iceberg tables need no additional permissions. For more information, see Iceberg Table Support. The following policy grants these permissions on every database and table in the catalog. Replace <AWS_REGION> and <ACCOUNT_ID> with your values.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "glue:GetDatabases",
        "glue:GetTables"
      ],
      "Resource": [
        "arn:aws:glue:<AWS_REGION>:<ACCOUNT_ID>:catalog",
        "arn:aws:glue:<AWS_REGION>:<ACCOUNT_ID>:database/*",
        "arn:aws:glue:<AWS_REGION>:<ACCOUNT_ID>:table/*/*"
      ]
    }
  ]
}

If AWS Lake Formation manages the catalog, also grant the identity the DESCRIBE permission on the databases and tables to ingest. Without that grant, Glue leaves those databases and tables out of its responses instead of returning an error.

Python Requirements

Use a Python version supported by the openmetadata-ingestion package that matches your OpenMetadata server. To find the supported range for your release, check the Requires-Python metadata of that release’s ingestion package. To run the Glue ingestion, you will need to install:

pip3 install "openmetadata-ingestion[glue]"

How Glue Maps to OpenMetadata

The connector doesn’t map a Glue database to an OpenMetadata database. The Glue Data Catalog becomes the OpenMetadata database, and each Glue database becomes a schema inside it.

AWS Glue	OpenMetadata	Name in OpenMetadata
Data Catalog	Database	The catalog ID (your AWS account ID, for example `123456789012`), unless you set `databaseName`.
Database	Schema	The Glue database name.
Table or view	Table	The Glue table name. Iceberg tables get the `Iceberg` table type, external tables `External`, views `View`, and all other tables `Regular`.

databaseName does more than rename the database. When you set it:

The connector creates one database with that name and stops using databaseFilterPattern to select catalogs, so leave that pattern empty.
Every Glue database the connector can see becomes a schema of that database, even when the Glue databases come from different catalogs. The ingestion run reports a warning when it sees more than one catalog.
If two Glue databases from different catalogs share a name, one overwrites the other and its tables go missing.

To keep one database per catalog, leave databaseName unset. To choose which Glue databases to ingest, use schemaFilterPattern.

Filter Patterns

Each filter pattern in sourceConfig matches the Glue name at its own level of this mapping. Patterns are case-insensitive regular expressions matched from the start of the name.

Field	Matches	Matching example	Non-matching example
`databaseFilterPattern`	Catalog ID, when `databaseName` is unset	`^123456789012$` includes the catalog `123456789012`.	The same pattern skips the catalog `210987654321`. An include pattern that names a Glue database, such as `^sales$`, matches no catalog ID. The connector then skips the whole catalog.
`schemaFilterPattern`	Glue database name	`^sales_` includes the Glue database `sales_raw`.	The same pattern skips the Glue database `marketing_raw`.
`tableFilterPattern`	Glue table name	`^orders` includes the table `orders_2024`.	The same pattern skips the table `customer_orders`.

The following sourceConfig applies these include patterns:

  sourceConfig:
    config:
      type: DatabaseMetadata
      databaseFilterPattern:
        includes:
          - "^123456789012$"
      schemaFilterPattern:
        includes:
          - "^sales_"
      tableFilterPattern:
        includes:
          - "^orders"

If a pattern contains only digits, such as 123456789012, wrap it in quotes. Otherwise, YAML reads it as a number, and the workflow rejects it because filter patterns must be strings. If you set useFqnForFiltering: true, patterns match the fully qualified name instead. For example, the Glue database sales_raw in the catalog 123456789012 of the service local_glue has the fully qualified name local_glue.123456789012.sales_raw.

Metadata Ingestion

All connectors are defined as JSON Schemas. Here you can find the structure to create a connection to Glue. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following JSON Schema

1. Define the YAML Config

2. Run with the CLI

First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run:

metadata ingest -c <path-to-yaml>

Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources.

dbt Integration

Learn more about how to ingest dbt models

​How to Run the Connector Externally

External Schedulers

​Requirements

​Python Requirements

​How Glue Maps to OpenMetadata

​Filter Patterns

​Metadata Ingestion

​1. Define the YAML Config

​2. Run with the CLI

​dbt Integration

dbt Integration

How to Run the Connector Externally

Requirements

Python Requirements

How Glue Maps to OpenMetadata

Filter Patterns

Metadata Ingestion

1. Define the YAML Config

2. Run with the CLI

dbt Integration