connectors

No menu items for this category

Usage Workflow

Learn how to configure the Usage workflow from the UI to ingest Query history data from your data sources.

This workflow is available ONLY for the following connectors:

If your database service is not yet supported, you can use this same workflow by providing a Query Log file!

Learn how to do so πŸ‘‡

Once the metadata ingestion runs correctly and we are able to explore the service Entities, we can add Query Usage information.

This will populate the Queries tab from the Table Entity Page.

table-entity-page

Table Entity Page

We can create a workflow that will obtain the query log and table creation information from the underlying database and feed it to OpenMetadata. The Usage Ingestion will be in charge of obtaining this data.

From the Service Page, go to the Ingestions tab to add a new ingestion and click on Add Usage Ingestion.

add-ingestion

Add Ingestion

Here you can enter the Usage Ingestion details:

configure-usage-ingestion

Configure the Usage Ingestion

Query Log Duration

Specify the duration in days for which the profiler should capture usage data from the query logs. For example, if you specify 2 as the value for the duration, the data profiler will capture usage information for 48 hours prior to when the ingestion workflow is run.

Stage File Location

Mention the absolute file path of the temporary file name to store the query logs before processing.

Result Limit

Set the limit for the query log results to be run at a time.

After clicking Next, you will be redirected to the Scheduling form. This will be the same as the Metadata Ingestion. Select your desired schedule and click on Deploy to find the usage pipeline being added to the Service Ingestions.

schedule-and-deploy

View Service Ingestion pipelines