OpenMetadata
Search…
Kafka
This guide will help install Kafka connector and run manually
Prerequisites
OpenMetadata is built using Java, DropWizard, Jetty, and MySQL.
  1. 1.
    Python 3.8 or above

Install from PyPI

Install Using PyPI
1
pip3 install 'openmetadata-ingestion[kafka]'
Copied!

Run Manually

1
metadata ingest -c ./examples/workflows/confluent_kafka.json
Copied!

Configuration

confluent_kafka.json
1
{
2
"source": {
3
"type": "kafka",
4
"serviceName": "local_kafka",
5
"serviceConnection": {
6
"config": {
7
"type": "Kafka",
8
"bootstrapServers": "localhost:9092",
9
"schemaRegistryURL": "http://localhost:8081",
10
"consumerConfig": {},
11
"schemaRegistryConfig": {}
12
}
13
},
14
"sourceConfig": {
15
"config": {
16
"topicFilterPattern": {
17
"excludes": ["_confluent.*"]
18
}
19
}
20
}
21
...
Copied!
  1. 1.
    service_name - Service Name for this Kafka cluster. If you added Kafka cluster through OpenMetadata UI, make sure the service name matches the same.
  2. 2.
    filter_pattern - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata

Publish to OpenMetadata

Below is the configuration to publish Kafka data into the OpenMetadata service.
Add metadata-rest sink along with metadata-server config
confluent_kafka.json
1
{
2
"source": {
3
"type": "kafka",
4
"serviceName": "local_kafka",
5
"serviceConnection": {
6
"config": {
7
"type": "Kafka",
8
"bootstrapServers": "localhost:9092",
9
"schemaRegistryURL": "http://localhost:8081",
10
"consumerConfig": {},
11
"schemaRegistryConfig": {}
12
}
13
},
14
"sourceConfig": {
15
"config": {
16
"topicFilterPattern": {
17
"excludes": ["_confluent.*"]
18
}
19
}
20
}
21
},
22
"sink": {
23
"type": "metadata-rest",
24
"config": {
25
}
26
},
27
"workflowConfig": {
28
"openMetadataServerConfig": {
29
"hostPort": "http://localhost:8585/api",
30
"authProvider": "no-auth"
31
}
32
}
33
}
Copied!