Learn how to create a Custom Drive Connector to ingest metadata from file system-like services into OpenMetadata using the ingestion framework.
Custom Drive Connector
Just like other services in OpenMetadata (Database, Pipelines, Dashboards, Messaging, etc.), it’s possible to create a Custom Drive Connector to bring metadata from a storage or file system-like service into OpenMetadata.In this guide, we’ll walk through how to implement your own Custom Drive Connector by extending the ingestion framework. The implementation pattern follows other service types closely, making the transition smooth for anyone familiar with the ecosystem.
You need to yield CreateEntityRequest objects wrapped in an Either type. This pattern handles both successful entity creation and errors.Import necessary types:
Extend the same pattern with CreateSpreadsheetRequest and CreateWorksheetRequest if your Drive service exposes those concepts.
Step 3 - Package Your Custom Drive Connector
To use the connector, package it as a Python module. A minimal setup.py may look like:
from setuptools import setup, find_packagessetup( name="custom_drive_connector", version="0.1", packages=find_packages(), install_requires=[],)
Build the package:
python setup.py sdist
Step 4 - Update the Ingestion Image
To run your Custom Drive Connector inside Docker (e.g., with Airflow or directly from the UI), the ingestion image must include your module.Dockerfile example:
This ensures that your code is baked into the ingestion container. Always align the ingestion image tag with the OpenMetadata version that is running your server/UI.
Step 5 - Run OpenMetadata with Custom Image
If you’re using Docker Compose, update your setup to use the new image.
run: docker compose -f docker-compose.yml up --build
Step 6 - Configure and Run the Connector
The Custom Drive service form does not expose a Source Python Class field. Add the fully qualified class name to your ingestion YAML, then run it with the ingestion CLI.