Skip to main content

​
Metadata Ingestion - Incremental Extraction

The default Metadata Ingestion roughly follows these steps:
  1. Fetch all the information from the Source.
  2. Compare the information with the OpenMetadata information to update it properly.
  3. Compare the information with the OpenMetadata information to delete entities that were deleted.
While on one hand this is a great simple way of doing things that works for most use cases since at every ingestion pipeline run we get the whole Source state, on other hand this is fetching and comparing a lot of data without need since if there were no structural changes we already know there is nothing to update on OpenMetadata. We implemented the Incremental Extraction feature to improve the performance by diminishing the extraction and comparison of uneeded data. How this is done depends a lot on the Source itself, but the general idea is to follow these steps:
  1. Fetch the last successful pipeline run.
  2. Add a small safety margin.
  3. Get all the structural changes since then.
  4. Flag deleted entities.
  5. Fetch/Compare only the entities with structural changes.
  6. Delete entities flagged for deletion.

​
Validate Incremental Runs

Validate incremental extraction after enabling it and after changing source permissions or filters:
  1. Make a known structural change in the source, such as creating a test table or adding a column.
  2. Run the incremental metadata ingestion pipeline.
  3. Open the pipeline run details and confirm that the run completed successfully. If the run failed or completed with partial success, review its logs before validating the catalog.
  4. Open the affected asset in OpenMetadata and confirm that the structural change appears. For a deletion, confirm that the asset is marked as deleted when the pipeline has the corresponding Mark Deleted option enabled.
  5. Review the run logs for warnings about source change-history access, skipped objects, or filters. A successful run confirms that the workflow completed, but you should still verify representative source changes in OpenMetadata.
Incremental extraction uses the most recent successful run for that ingestion pipeline as its starting point. If no successful run is found within Successful Pipeline Run Lookback Days, the pipeline performs a full extraction instead.

​
Schedule a Periodic Full Run

Use two metadata ingestion pipelines for the same database service when you want frequent incremental updates and a periodic full reconciliation:
  1. Keep Incremental Extraction enabled on the frequently scheduled pipeline.
  2. Add a second metadata ingestion pipeline for the same service and leave Incremental Extraction disabled.
  3. Give the pipelines distinct names so their run histories are easy to identify.
  4. Schedule the full pipeline less frequently, such as weekly, and avoid overlapping the two schedules.
  5. Use the full run to reconcile the complete source state. Review its logs and spot-check any metadata that was missing after incremental runs.
Configure both pipelines with the same database, schema, and table filters. Different scopes can produce inconsistent results when any Mark Deleted option is enabled.

​
External Ingestion

When using the Incremental Extraction feature with External Ingestions (ingesting using YAML files instead of setting it up from the UI), you must pass the ingestion pipeline fully qualified name to the configuration. This should be {service_name}{pipeline_name} Example:

​
Feature available for

​
Databases

BigQuery

BETA | OpenMetadata

Redshift

BETA | OpenMetadata

Snowflake

BETA | OpenMetadata

Unity Catalog

BETA | OpenMetadata