No menu items for this category

This page is about running the Ingestion Framework externally!

There are mainly 2 ways of running the ingestion:

  1. Internally, by managing the workflows from OpenMetadata.
  2. Externally, by using any other tool capable of running Python code.

If you are looking for how to manage the ingestion process from OpenMetadata, you can follow this doc.

Run the ingestion from GitHub Actions

You can find a fully working demo of this setup here.

The process to run the ingestion from GitHub Actions is the same as running it from anywhere else.

  1. Get the YAML configuration,
  2. Prepare the Python Script
  3. Schedule the Ingestion

For any connector and workflow, you can pick it up from its doc page.

In the GitHub Action we will just be triggering a custom Python script. This script will:

  • Load the secrets from environment variables (we don't want any security risks!),

  • Prepare the Workflow class from the Ingestion Framework that contains all the logic on how to run the metadata ingestion,

  • Execute the workflow and log the results.

  • A simplified version of such script looks like follows:

Note how we are securing the credentials using environment variables. You will need to create these env vars in your GitHub repository. Follow the GitHub docs for more information on how to create and use Secrets.

In the end, we'll map these secrets to environment variables in the process, that we can pick up with os.getenv.

Now that we have all the ingredients, we just need to build a simple GitHub Actions with the following steps:

  • Install Python

  • Prepare virtual environment with the openmetadata-ingestion package

  • Run the script!

  • It is as simple as this. Internally the function run we created will be sending the results to the OpenMetadata server, so there's nothing else we need to do here.

A first version of the action could be:

A very interesting option that GitHub Actions provide is the ability to get alerts in Slack after our action fails.

This can become specially useful if we want to be notified when our metadata ingestion is not working as expected. We can use the same setup as above with a couple of slight changes:

We have:

  • Marked the Run Ingestion step with a specific id and with continue-on-error: true. If anything happens, we don't want the action to stop.
  • We added a step with slackapi/slack-github-action@v1.23.0. By passing a Slack Webhook link via a secret, we can send any payload to a
  • specific Slack channel. You can find more info on how to set up a Slack Webhook here.
  • If our ingestion step fails, we still want to mark the action as failed, so we are forcing the failure we skipped before.