Develop the Ingestion Code

We recommend you to take some time to understand how the Ingestion Framework works by reading this small article.

The main takes for developing a new connector are:

To understand that each of our Source Types (Databases, Dashboards, etc) have a Topology attached.
To understand that the process flow is implemented as a generator chain, going through each step.

Service Topology

The Topology defines a series of Nodes and Stages that get executed in a hierarchical way and describe how we extract the needed data from the sources.

Starting from the Root node we process the entities in a depth first approach, following the topology tree through the node's children.

From the Service Topology you can understand what methods you need to implement:

producer: Methods that will fetch the entities we need to process
processor: Methods that will yield a given Entity
post_process: Methods that will yield a given Entity but are ran after all entities from that node were processed.

Example - DatabaseServiceTopology

Can be found in ingestion/src/metadata/ingestion/source/database/database_service.py

Service Source

Now that you understand how the Ingestion Process works, you need to understand the Service Source.

A Service Source is an abstract class that is the base for any Connector from that Source Type. They tend to have a lot of methods and are pretty overwhelming at first glance but you don't need to worry. You'll need to check which abstract methods you need to implement in your connector.

Hint

You can start slow, yielding nothing with yield from [] to see the whole flow running and then slowly implement the features you want.

On this note, also remember that you don't need to implement everything. You could contribute by start and implement just the Metadata Extraction features but without extracting Owner/Tags or deal with Lineage.

Example - DatabaseServiceSource

Can be found in ingestion/src/metadata/ingestion/source/database/database_service.py

Next Step

With the Code ready to go, we can now proceed to make a small change in the UI to be able to configure the Connector properly from there.

Apply the UI Changes

Learn what you need to do to be able see the Connector properly in the UI