Upgrade from 0.12 to 0.13
Upgrading from 0.12 to 0.13 can be done directly on your instances. This page will list few general details you should take into consideration when running the upgrade.
0.13.2 Highlights
OpenMetadata 0.13.2 is a stable release. Please check the release notes
If you are upgrading production this is the recommended version to upgrade.
Service Connection Updates
Oracle:
- Oracle connector now supports thick mode enabled by default.
- Added:
instantClientDirectory
with the path to the Oracle libraries. They are provided by default for release 19 in the ingestion container.
- Added:
- Oracle connector now supports thick mode enabled by default.
Superset:
- The Superset connector now supports the ability to extract metadata directly from the database underlying Superset, making it useful for users who have SSO enabled.
- Added:
connection
field the connection can be of typeApiConnection
,postgresConnection
ormysqlConnection
- Removed:
username
,password
&provider
fields as now these fields will be part ofApiConnection
- Added:
- The Superset connector now supports the ability to extract metadata directly from the database underlying Superset, making it useful for users who have SSO enabled.
Breaking Changes for 0.13.2 Stable Release
EntityName
To better manage and harmonize entityName
value and allow users to form better expectations around these values the team introduced an enforcement of the entityName
format using regex pattern.
All the OpenMetadata entities entityName
fields will enforce by the default the following regex pattern:
^[\w'\- .&]+$
: match any word characters (equivalent to[a-zA-Z0-9_]
) or the characters'- .&
. For example when creating a pipeline service the following name will not be allowedMyPipelineIngestion?!
while the following will beMy-Pipeline
.
Some entities are enforcing specific patterns:
- Users:
([\w\-.]|[^@])+$
match any word characters (equivalent to[a-zA-Z0-9_]
) or the characters-.
and don't match the character@
- Webhook:
^[\w'\-.]+$
match any word characters (equivalent to[a-zA-Z0-9_]
) or the characters-.
- Table:
^[\w'\- ./]+$
match any word characters (equivalent to[a-zA-Z0-9_]
) or the characters'-. /
- Location:
^[\w'\-./]+$
match any word characters (equivalent to[a-zA-Z0-9_]
) or the characters-.'/
- Type:
^[a-z][\w]+$
match any word characters (equivalent to[a-zA-Z0-9_]
) starting with a lowercase letter (e.g.tHisChar
will match,ThisChar
will not)
If an entity name does not follow the pattern an error will be returned by the OpenMetadata platform.
The change should be transparent for the end user and no action should be required.
EntityLink
Similar to the implementation done for entityName
, entityLink
will now enforce a specific pattern. The structure of entityLink
is in the form <#E::{entities}::{entityType}::{field}::{fieldName}::{fieldValue}>
All the OpenMetadata entities entityLink
fields will enforce by the default the following regex pattern:
^<#E::\w+::[\w'\- .&/\:+\"\\]+>$
: this means that{entities}
value needs to match any word characters (equivalent to[a-zA-Z0-9_]
) and the part after{entities}
can match any word characters (equivalent to[a-zA-Z0-9_]
) or the characters'- .&/\:+\"\
If an entity name does not follow the pattern an error will be returned by the OpenMetadata platform.
The change should be transparent for the end user and no action should be required.
Tags API
Tags APIs were coded long before other entities' APIs were added. Due to this, tag API does not follow the API convention that all the other entities are following. This issue makes backward incompatible changes to follow the same convention as glossaries.
You can find the full list of API paths changes in the following issue.
Metabase and Domo Dashboards name
With the new restrictions on the EntityName
and to ensure unicity of the assets, the Metabase and Domo Dashboard sources now ingest the name
of the charts and dashboards with their internal ID value. Their name
value will be used as the display name, but not as the OpenMetadata name
anymore.
The recommended approach here is to create a new service and ingest the metadata again. If you ingest from the same service, the assets in there will end up being duplicated, as the name
in OpenMetadata is used to identify each asset.
Let us know if you have any questions around this topic.
Ingestion Framework sources directory structure
We have converted source modules (e.g., redshift.py
) into packages containing all the necessary information (e.g., redshift/metadata.py
). This has helped us reorganise functionalities and easily focus on each connector independently.
If you're extending any of the sources, you'll need to update your imports. You can take a look at the new structure here.
MySQL Helm Chart Version Reverted to 8.8.23
OpenMetadata Helm Chart Release with Application Version 0.13.2
updates the Bitnami MySQL Helm Chart version from 9.2.1
to 8.8.23
. This is a breaking change and users will face an issue as mentioned in the documentation here. Please note that OpenMetadata Dependencies Helm Chart is not recommended for production use cases. The steps mentioned in the section will help you fix the issue.
0.13.1 Highlights
OpenMetadata 0.13.1 is a stable release. Please check the release notes.
Deprecating botPrincipals from OpenMetadata Configuration
Starting with 0.13.0
, we have deprecated the initial configurations for Authorizer Bot Principals. This means that all the configs will be taken care from UI once it's up and running under settings >> bots section. The default once available and enabled will be ingestion-bot
which will have JWT Token configured. You can edit the same for configuring service principals instead of using the default JWT Auth mechanism. For more information on configuring Bots, you can visit Enable Security.
Service Connection Updates
Databricks
- Databricks connector now supports usage and lineage extraction.
- Added:
supportsUsageExtraction
,supportsLineageExtraction
- Added:
- Databricks connector now supports usage and lineage extraction.
Dagster
- Dagster connector now support fetching metadata from dagster cloud.
- Added:
configSource
- Removed:
hostPort
,numberOfStatus
- Added:
- Dagster connector now support fetching metadata from dagster cloud.
Airbyte
- Airbyte connector now support Basic Authentication.
- Added:
username
,password
- Added:
- Airbyte connector now support Basic Authentication.
Lineage Backend
- The import for the Airflow Lineage Backend has been updated from
airflow_provider_openmetadata.lineage.openmetadata.OpenMetadataLineageBackend
toairflow_provider_openmetadata.lineage.backend.OpenMetadataLineageBackend
. - We removed support from Airflow v1.
- The failure callback now only updates the pipeline status if the Pipeline already exists in OpenMetadata.
Webhooks
OpenMetadata will be deprecating the existing webhooks for Slack, MSTeams.
Before upgrading to 0.13.1 it is recommended to save the existing Webhook configs(like webhook url) to use them later.
We have added Alerts/Notifications , which can be configured to receive customised alerts on updates in OM using Triggers, Filtering Information to different destinations like Slack, MsTeams or even Emails. Please use the same webhook config that you had saved from previous version to configure the Alerts Destination after upgrading.
dbt Workflow
- Starting with
0.13.1
dbt ingestion has been separated from the metadata ingestion. It can now be configured as a separate workflow after completing the metadata ingestion workflow - We will remove the dbt configuration from your existing metadata ingestion pipelines and they will keep working as expected.
- After upgrading you will have to create the dbt workflow for the dbt ingestion to start working again.
- dbt workflow can be configured via going to
services->selecting your service->ingestion tab->Add dbt Ingestion
Breaking Changes for 0.13.1 Stable Release
OpenMetadata Release 0.13.1 introduces below breaking changes -
Webhooks
Starting from 0.13.1 , OpenMetadata will be deprecating the existing webhooks for Slack, MSTeams.
Before upgrading to 0.13.1 it is recommended to save the existing Webhook configs(like webhook url) to use them later.
We have added Alerts/Notifications , which can be configured to receive customised alerts on updates in OM using Triggers, Filtering Information to different destinations like Slack, MsTeams or even Emails. Please use the same webhook config that you had saved from previous version to configure the Alerts Destination after upgrading.
OpenMetadata Release 0.13.x introduces below breaking changes:
Docker Volumes
OpenMetadata Release 0.13.x introduces Default Docker Volumes for Database (MySQL, PostgreSQL) and ElasticSearch with Docker deployment.
- If you are looking for the fresh deployment of 0.13.x - here
- If you are looking for upgrading of the new version i.e 0.13.x - here
MySQL Helm Chart Version Updated to 9.2.1
OpenMetadata Helm Chart Release with Application Version 0.13.1
updates the Bitnami MySQL Helm Chart version to 9.2.1
from 8.8.23
. This is not a breaking change but existing user's trying to upgrade will experience a slight delay in OpenMetadata Dependencies Helm Chart Upgrades as it pulls new docker image for MySQL. Please note that OpenMetadata Dependencies Helm Chart is not recommended for production use cases. Please follow the kubernetes deployment for new installation or upgrade kubernetes for upgrading OpenMetadata in Kubernetes.
dbt Workflow
dbt ingestion has been separated from the metadata ingestion. It can now be configured as a separate workflow after completing the metadata ingestion workflow.
We will remove the dbt configuration from your existing metadata ingestion pipelines and they will keep working as expected.
After upgrading you will have to create the dbt workflow for the dbt ingestion to start working again.
Airflow Lineage Backend
- The import for the Airflow Lineage Backend has been updated from
airflow_provider_openmetadata.lineage.openmetadata.OpenMetadataLineageBackend
toairflow_provider_openmetadata.lineage.backend.OpenMetadataLineageBackend
. - We removed support from Airflow v1.
- The failure callback now only updates the pipeline status if the Pipeline already exists in OpenMetadata.