Kafka

Prerequisites
OpenMetadata is built using Java, DropWizard, Jetty, and MySQL.
1.Python 3.8 or above
Install from PyPI
Install Using PyPI
1
pip3 install 'openmetadata-ingestion[kafka]'
Copied!
Run Manually
1
metadata ingest -c ./examples/workflows/confluent_kafka.json
Copied!
Configuration
confluent_kafka.json
1
{
2
  "source": {
3
    "type": "kafka",
4
    "serviceName": "local_kafka",
5
    "serviceConnection": {
6
      "config": {
7
        "type": "Kafka",
8
        "bootstrapServers": "localhost:9092",
9
        "schemaRegistryURL": "http://localhost:8081",
10
        "consumerConfig": {},
11
        "schemaRegistryConfig": {}
12
      }
13
    },
14
    "sourceConfig": {
15
      "config": {
16
        "topicFilterPattern": {
17
          "excludes": ["_confluent.*"]
18
        }
19
      }
20
    }
21
 ...
Copied!
1.service_name - Service Name for this Kafka cluster. If you added Kafka cluster through OpenMetadata UI, make sure the service name matches the same.
2.filter_pattern - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata
Publish to OpenMetadata
Below is the configuration to publish Kafka data into the OpenMetadata service.
Add metadata-rest sink along with metadata-server config
confluent_kafka.json
1
{
2
  "source": {
3
    "type": "kafka",
4
    "serviceName": "local_kafka",
5
    "serviceConnection": {
6
      "config": {
7
        "type": "Kafka",
8
        "bootstrapServers": "localhost:9092",
9
        "schemaRegistryURL": "http://localhost:8081",
10
        "consumerConfig": {},
11
        "schemaRegistryConfig": {}
12
      }
13
    },
14
    "sourceConfig": {
15
      "config": {
16
        "topicFilterPattern": {
17
          "excludes": ["_confluent.*"]
18
        }
19
      }
20
    }
21
  },
22
  "sink": {
23
    "type": "metadata-rest",
24
    "config": {
25
    }
26
  },
27
  "workflowConfig": {
28
    "openMetadataServerConfig": {
29
      "hostPort": "http://localhost:8585/api",
30
      "authProvider": "no-auth"
31
    }
32
  }
33
}
Copied!

Run Looker Connector with the CLI

MariaDB

Last modified 6d ago

Copy link

Edit on GitHub

Contents

Install from PyPI

Run Manually

Configuration

Publish to OpenMetadata