Upgrade from 0.12 to 0.13

Upgrading from 0.12 to 0.13 can be done directly on your instances. This page will list few general details you should take into consideration when running the upgrade.

0.13.2 Highlights

OpenMetadata 0.13.2 is a stable release. Please check the release notes

If you are upgrading production this is the recommended version to upgrade.

Service Connection Updates

• Oracle:

  • Oracle connector now supports thick mode enabled by default.
    • Added: instantClientDirectory with the path to the Oracle libraries. They are provided by default for release 19 in the ingestion container.

• Superset:

  • The Superset connector now supports the ability to extract metadata directly from the database underlying Superset, making it useful for users who have SSO enabled.
    • Added: connection field the connection can be of type ApiConnection, postgresConnection or mysqlConnection
    • Removed: username, password & provider fields as now these fields will be part of ApiConnection

Breaking Changes for 0.13.2 Stable Release

EntityName

To better manage and harmonize entityName value and allow users to form better expectations around these values the team introduced an enforcement of the entityName format using regex pattern.

All the OpenMetadata entities entityName fields will enforce by the default the following regex pattern:

• ^[\w'\- .&]+$: match any word characters (equivalent to [a-zA-Z0-9_]) or the characters '- .&. For example when creating a pipeline service the following name will not be allowed MyPipelineIngestion?! while the following will be My-Pipeline .

Some entities are enforcing specific patterns:

• Users: ([\w\-.]|[^@])+$ match any word characters (equivalent to [a-zA-Z0-9_]) or the characters -. and don't match the character @
• Webhook: ^[\w'\-.]+$ match any word characters (equivalent to [a-zA-Z0-9_]) or the characters -.
• Table: ^[\w'\- ./]+$ match any word characters (equivalent to [a-zA-Z0-9_]) or the characters '-. /
• Location: ^[\w'\-./]+$ match any word characters (equivalent to [a-zA-Z0-9_]) or the characters -.'/
• Type: ^[a-z][\w]+$ match any word characters (equivalent to [a-zA-Z0-9_]) starting with a lowercase letter (e.g. tHisChar will match, ThisChar will not)

If an entity name does not follow the pattern an error will be returned by the OpenMetadata platform.

The change should be transparent for the end user and no action should be required.

EntityLink

Similar to the implementation done for entityName, entityLink will now enforce a specific pattern. The structure of entityLink is in the form <#E::{entities}::{entityType}::{field}::{fieldName}::{fieldValue}>

All the OpenMetadata entities entityLink fields will enforce by the default the following regex pattern:

• ^<#E::\w+::[\w'\- .&/\:+\"\\]+>$: this means that {entities} value needs to match any word characters (equivalent to [a-zA-Z0-9_]) and the part after {entities} can match any word characters (equivalent to [a-zA-Z0-9_]) or the characters '- .&/\:+\"\

If an entity name does not follow the pattern an error will be returned by the OpenMetadata platform.

The change should be transparent for the end user and no action should be required.

Tags API

Tags APIs were coded long before other entities' APIs were added. Due to this, tag API does not follow the API convention that all the other entities are following. This issue makes backward incompatible changes to follow the same convention as glossaries.

You can find the full list of API paths changes in the following issue.

Metabase and Domo Dashboards name

With the new restrictions on the EntityName and to ensure unicity of the assets, the Metabase and Domo Dashboard sources now ingest the name of the charts and dashboards with their internal ID value. Their name value will be used as the display name, but not as the OpenMetadata name anymore.

The recommended approach here is to create a new service and ingest the metadata again. If you ingest from the same service, the assets in there will end up being duplicated, as the name in OpenMetadata is used to identify each asset.

Let us know if you have any questions around this topic.

Ingestion Framework sources directory structure

We have converted source modules (e.g., redshift.py) into packages containing all the necessary information (e.g., redshift/metadata.py). This has helped us reorganise functionalities and easily focus on each connector independently.

If you're extending any of the sources, you'll need to update your imports. You can take a look at the new structure here.

MySQL Helm Chart Version Reverted to 8.8.23

OpenMetadata Helm Chart Release with Application Version 0.13.2 updates the Bitnami MySQL Helm Chart version from 9.2.1 to 8.8.23. This is a breaking change and users will face an issue as mentioned in the documentation here. Please note that OpenMetadata Dependencies Helm Chart is not recommended for production use cases. The steps mentioned in the section will help you fix the issue.

0.13.1 Highlights

OpenMetadata 0.13.1 is a stable release. Please check the release notes.

Deprecating botPrincipals from OpenMetadata Configuration

Starting with 0.13.0, we have deprecated the initial configurations for Authorizer Bot Principals. This means that all the configs will be taken care from UI once it's up and running under settings >> bots section. The default once available and enabled will be ingestion-bot which will have JWT Token configured. You can edit the same for configuring service principals instead of using the default JWT Auth mechanism. For more information on configuring Bots, you can visit Enable Security.

Service Connection Updates

• Databricks

  • Databricks connector now supports usage and lineage extraction.
    • Added: supportsUsageExtraction, supportsLineageExtraction

• Dagster

  • Dagster connector now support fetching metadata from dagster cloud.
    • Added: configSource
    • Removed: hostPort, numberOfStatus

• Airbyte

  • Airbyte connector now support Basic Authentication.
    • Added: username, password

Lineage Backend

• The import for the Airflow Lineage Backend has been updated from airflow_provider_openmetadata.lineage.openmetadata.OpenMetadataLineageBackend to airflow_provider_openmetadata.lineage.backend.OpenMetadataLineageBackend.
• We removed support from Airflow v1.
• The failure callback now only updates the pipeline status if the Pipeline already exists in OpenMetadata.

Webhooks

OpenMetadata will be deprecating the existing webhooks for Slack, MSTeams.

Before upgrading to 0.13.1 it is recommended to save the existing Webhook configs(like webhook url) to use them later.

We have added Alerts/Notifications , which can be configured to receive customised alerts on updates in OM using Triggers, Filtering Information to different destinations like Slack, MsTeams or even Emails. Please use the same webhook config that you had saved from previous version to configure the Alerts Destination after upgrading.

dbt Workflow

• Starting with 0.13.1 dbt ingestion has been separated from the metadata ingestion. It can now be configured as a separate workflow after completing the metadata ingestion workflow
• We will remove the dbt configuration from your existing metadata ingestion pipelines and they will keep working as expected.
• After upgrading you will have to create the dbt workflow for the dbt ingestion to start working again.
• dbt workflow can be configured via going to services->selecting your service->ingestion tab->Add dbt Ingestion

Breaking Changes for 0.13.1 Stable Release

OpenMetadata Release 0.13.1 introduces below breaking changes -

Webhooks

Starting from 0.13.1 , OpenMetadata will be deprecating the existing webhooks for Slack, MSTeams.

Before upgrading to 0.13.1 it is recommended to save the existing Webhook configs(like webhook url) to use them later.

We have added Alerts/Notifications , which can be configured to receive customised alerts on updates in OM using Triggers, Filtering Information to different destinations like Slack, MsTeams or even Emails. Please use the same webhook config that you had saved from previous version to configure the Alerts Destination after upgrading.

OpenMetadata Release 0.13.x introduces below breaking changes:

Docker Volumes

OpenMetadata Release 0.13.x introduces Default Docker Volumes for Database (MySQL, PostgreSQL) and ElasticSearch with Docker deployment.

• If you are looking for the fresh deployment of 0.13.x - here
• If you are looking for upgrading of the new version i.e 0.13.x - here

MySQL Helm Chart Version Updated to 9.2.1

OpenMetadata Helm Chart Release with Application Version 0.13.1 updates the Bitnami MySQL Helm Chart version to 9.2.1 from 8.8.23. This is not a breaking change but existing user's trying to upgrade will experience a slight delay in OpenMetadata Dependencies Helm Chart Upgrades as it pulls new docker image for MySQL. Please note that OpenMetadata Dependencies Helm Chart is not recommended for production use cases. Please follow the kubernetes deployment for new installation or upgrade kubernetes for upgrading OpenMetadata in Kubernetes.

dbt Workflow

dbt ingestion has been separated from the metadata ingestion. It can now be configured as a separate workflow after completing the metadata ingestion workflow.

We will remove the dbt configuration from your existing metadata ingestion pipelines and they will keep working as expected.

After upgrading you will have to create the dbt workflow for the dbt ingestion to start working again.

Airflow Lineage Backend

• The import for the Airflow Lineage Backend has been updated from airflow_provider_openmetadata.lineage.openmetadata.OpenMetadataLineageBackend to airflow_provider_openmetadata.lineage.backend.OpenMetadataLineageBackend.
• We removed support from Airflow v1.
• The failure callback now only updates the pipeline status if the Pipeline already exists in OpenMetadata.