Run Databricks Connector with the CLI

Configure and schedule Databricks metadata and profiler workflows using your own Airflow instances.
​Requirements​
​Metadata Ingestion​
​Data Profiler and Quality Tests​
​DBT Integration​
Requirements
Follow this guide to learn how to set up Airflow to run the metadata ingestions.
Install the Python module for this connector
Once the virtual environment is set up and activated as described in Step 1, run the following command to install the Python module for this connector.
1
pip3 install 'openmetadata-ingestion[databricks]'
Copied!
Metadata Ingestion
All connectors are now defined as JSON Schemas. Here you can find the structure to create a connection to Databricks.
In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a JSON configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server.
The workflow is modeled around the following JSON Schema.
1. Create a configuration file using template JSON
Create a new file called databricks.json in the current directory. Note that the current directory should be the openmetadata directory.
Copy and paste the configuration template below into the databricks.json the file you created.
databrick.json
1
{
2
  "source": {
3
    "type": "databricks",
4
    "serviceName": "local_databricks",
5
    "serviceConnection": {
6
      "config": {
7
        "type":"Databricks",
8
        "token": "<databricks token>",
9
        "database":"<database>",
10
        "hostPort": "<databricks connection host & port>",
11
        "httpPath": "<http path of databricks cluster>"
12
      }
13
    },
14
    "sourceConfig": {
15
      "config": {
16
          "enableDataProfiler": true or false,
17
          "markDeletedTables": true or false,
18
          "includeTables": true or false,
19
          "includeViews": true or false,
20
          "generateSampleData": true or false,
21
          "sampleDataQuery": "<query to fetch table data>",
22
          "schemaFilterPattern": "<schema name regex list>",
23
          "tableFilterPattern": "<table name regex list>",
24
          "dbtConfigSource": "<configs for gcs, s3, local or file server to get the DBT files"
25
        }
26
      }
27
  },
28
  "sink": {
29
    "type": "metadata-rest",
30
    "config": {}
31
  },
32
  "workflowConfig": {
33
    "openMetadataServerConfig": {
34
      "hostPort": "<OpenMetadata host and port>",
35
      "authProvider": "<OpenMetadata auth provider>"
36
    }
37
  }
38
}
Copied!
Source Configuration - Service Connection
You can find all the definitions and types for the serviceConnection here.
token: Enter the generated token to connect to Databricks.
hostPort: Enter the fully qualified hostname and port number for your Hive deployment in the Host and Port field.
httpPath: Enter the http path of Databricks cluster.
database: If you want to limit metadata ingestion to a single database, enter the name of this database in the Database field. If no value is entered for this field, the connector will ingest metadata from all databases that the specified user is authorized to read.
connectionOptions (Optional): Enter the details for any additional connection options that can be sent to Databricks during the connection. These details must be added as Key-Value pairs.
connectionArguments: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Databricks during the connection. These details must be added as Key-Value pairs.
To specify LDAP Auth, use the following connectionArguments:
1
"connectionArguments": {
2
"auth": "LDAP"
3
}
Copied!
In this step, we will configure the Hive service settings required for this connector. Please follow the instructions below to ensure that you've configured the connector to read from your Hive service as desired.
Source Configuration - Source Config
The sourceConfig is defined here.
enableDataProfiler: **** true or false, to run the profiler (not the tests) during the metadata ingestion.
markDeletedTables: To flag tables as soft-deleted if they are not present anymore in the source system.
includeTables: true or false, to ingest table data. Default is true.
includeViews: true or false, to ingest views definitions.
generateSampleData: To ingest sample data based on sampleDataQuery.
sampleDataQuery: Defaults to select * from {}.{} limit 50.
schemaFilterPattern and tableFilternPattern: Note that the schemaFilterPattern and tableFilterPattern both support regex as include or exclude. E.g.,
1
"tableFilterPattern": {
2
  "includes": ["users", "type_test"]
3
}
Copied!
Sink Configuration
To send the metadata to OpenMetadata, it needs to be specified as "type": "metadata-rest".
Workflow Configuration
The main property here is the openMetadataServerConfig, where you can define the host and security provider of your OpenMetadata installation.
For a simple, local installation using our docker containers, this looks like:
1
"workflowConfig": {
2
  "openMetadataServerConfig": {
3
    "hostPort": "http://localhost:8585/api",
4
    "authProvider": "no-auth"
5
  }
6
}
Copied!
OpenMetadata Security Providers
We support different security providers. You can find their definitions here. An example of an Auth0 configuration would be the following:
1
"workflowConfig": {
2
    "openMetadataServerConfig": {
3
        "hostPort": "http://localhost:8585/api",
4
        "authProvider": "auth0",
5
        "securityConfig": {
6
            "clientId": "<client ID>",
7
            "secretKey": "<secret key>",
8
            "domain": "<domain>"
9
        }
10
    }
11
}
Copied!
2. Run with the CLI
First, we will need to save the JSON file. Afterward, and with all requirements installed, we can run:
1
metadata ingest -c <path-to-json>
Copied!
Note that from connector-to-connector, this recipe will always be the same. By updating the JSON configuration, you will be able to extract metadata from different sources.
Data Profiler and Quality Tests
The Data Profiler workflow will be using the orm-profiler processor. While the serviceConnection will still be the same to reach the source system, the sourceConfig will be updated from previous configurations.
1. Define the JSON configuration
This is a sample config for a Databricks profiler:
1
{
2
  "source": {
3
    "type": "databricks",
4
    "serviceName": "local_databricks",
5
    "serviceConnection": {
6
      "config": {
7
        "type":"Databricks",
8
        "token": "<databricks token>",
9
        "database":"<database>",
10
        "hostPort": "<databricks connection host & port>",
11
        "httpPath": "<http path of databricks cluster>"
12
      }
13
    },
14
    "sourceConfig": {
15
      "config": {
16
          "enableDataProfiler": true or false,
17
          "markDeletedTables": true or false,
18
          "includeTables": true or false,
19
          "includeViews": true or false,
20
          "generateSampleData": true or false,
21
          "sampleDataQuery": "<query to fetch table data>",
22
          "schemaFilterPattern": "<schema name regex list>",
23
          "tableFilterPattern": "<table name regex list>",
24
          "dbtConfigSource": "<configs for gcs, s3, local or file server to get the DBT files"
25
        }
26
      }
27
  },
28
  "processor": {
29
    "type": "orm-profiler",
30
    "config": {}
31
  },
32
  "sink": {
33
    "type": "metadata-rest",
34
    "config": {}
35
  },
36
  "workflowConfig": {
37
    "openMetadataServerConfig": {
38
      "hostPort": "<OpenMetadata host and port>",
39
      "authProvider": "<OpenMetadata auth provider>"
40
    }
41
  }
42
}
Copied!
Source Configuration
You can find all the definitions and types for the serviceConnection here.
The sourceConfig is defined here. If you don't need to add any fqnFilterPattern, the "type": "Profiler" is still required to be present.
Note that the fqnFilterPattern supports regex as include or exclude. E.g.,
1
"fqnFilterPattern": {
2
  "includes": ["service.database.schema.*"]
3
}
Copied!
Processor
To choose the orm-profiler. It can also be updated to define tests from the JSON itself instead of the UI:
1
 "processor": {
2
    "type": "orm-profiler",
3
    "config": {
4
        "test_suite": {
5
            "name": "<Test Suite name>",
6
            "tests": [
7
                {
8
                    "table": "<Table FQN>",
9
                    "table_tests": [
10
                        {
11
                            "testCase": {
12
                                "config": {
13
                                    "value": 100
14
                                },
15
                                "tableTestType": "tableRowCountToEqual"
16
                            }
17
                        }
18
                    ],
19
                    "column_tests": [
20
                        {
21
                            "columnName": "<Column Name>",
22
                            "testCase": {
23
                                "config": {
24
                                    "minValue": 0,
25
                                    "maxValue": 99
26
                                },
27
                                "columnTestType": "columnValuesToBeBetween"
28
                            }
29
                        }
30
                    ]
31
                }
32
            ]
33
        }
34
     }
35
  },
Copied!
tests is a list of test definitions that will be applied to table, informed by its FQN. For each table, one can then define a list of table_tests and column_tests. Review the supported tests and their definitions to learn how to configure the different cases here.
Workflow Configuration
The same as the metadata ingestion.
2. Run with the CLI
Again, we will start by saving the JSON file.
Then, we can run the workflow as:
1
metadata profile -c <path-to-json>
Copied!
Note how instead of running ingest, we are using the profile command to select the Profiler workflow.
DBT Integration
You can learn more about how to ingest DBT models' definitions and their lineage here.

Run Databricks Connector using Airflow SDK

Delta Lake

Last modified 17h ago

Copy link

Edit on GitHub

Contents

Requirements

Install the Python module for this connector

Metadata Ingestion

1. Create a configuration file using template JSON

2. Run with the CLI

Data Profiler and Quality Tests

1. Define the JSON configuration

2. Run with the CLI

DBT Integration