Upgrade from 0.9 to 0.10
The 0.10 Release consists of backward-incompatible changes. We do not support database migration from the 0.9.0 release. Please follow the steps carefully and backup your database before proceeding. 0.10.0 installations require brand new installation, and we have a migration tool to transfer all your entity descriptions, tags, owners, etc. to the 0.10.0 release
Please reach out to us on Slack, we can schedule a zoom session to help you upgrade your production instance.
openmetadata-ingestion client to the latest available version.
Make sure that you backed up your metadata.
To migrate the metadata from the 0.9 to the 0.10 version you need to ingest all the connectors in the 0.10 version with
the same service name as done in the 0.9 version. For example, if you have ingested snowflake connector in 0.9 version
with service name
Snowflake_Prod then in the 0.10 version, you need to ingest the snowflake connector again with the
All connectors are now defined as JSON Schemas. Here you can find the structure to create a connection to Snowflake. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following JSON Schema.
1. Setup your 0.10.1 installation
- OpenMetadata 0.9, keep it running as it is with its own MySQL and ES
- Install OpenMetadata 0.10.1, make it completely isolated from your 0.9 install new MySQL DB and other ES indexes
- Install 0.10.1 connectors
- Run ingestion of all your entities using the new connector. Keep the service names the same as your 0.9
installations. Once all the entities are ingested into OpenMetadata 0.10.1. For example, if you have ingested
snowflake connector in 0.9 version with service name Snowflake_Prod then in the 0.10 version, you need to ingest the
snowflake connector again with the name
- Run migration using the newly installed connector
2. Define the YAML Config
This is a sample config:
source: type: migrate serviceName: local_metadata serviceConnection: config: type: OpenMetadata hostPort: http://<hostport of 0.9.0 Openmetadata Server>/api authProvider: no-auth includeTables: true includeUsers: true includeTopics: true includePipelines: true includeTags: true includeGlossaryTerms: true includeMessagingServices: true includeDatabaseServices: true includePipelineServices: true enableVersionValidation: false limitRecords: 100000 sourceConfig: config: enableDataProfiler: false stage: type: migrate config: dirPath: <Directory Path to store data> bulkSink: type: migrate config: dirPath: <Directory Path to store data> workflowConfig: openMetadataServerConfig: hostPort: http://<hostport of 0.10.x Openmetadata Server>/api authProvider: no-auth
Source Configuration - Service Connection
You can find all the definitions and types for the
hostPort: Enter the OpenMetadata 0.9 Server Config. Must include API end point ex:
includeTables: Include tables for migration
includeUsers: Include users for migration
includeTopics: Include topics for migration
includePipelines: Include pipelines for migration
includeGlossaryTerms: include glossary terms for migration
includeTags: Include tags for migration.
includeMessagingServices: Include messaging services for migration.
includeDatabaseServices: Include database services for migration.
enableVersionValidation: Enable the server OpenMetadata and client OpenMetadata version validation. For migrating the metadata this has to be set to
limitRecords: Limit the number records that is fetched by 0.9 OpenMetadata API (1 to 1000000, default = 10).
stage: type: migrate config: dirPath: <Directory Path to store data>
The data while migrating from the 0.9 to 0.10 version needs to be stored in a file directory. in the dir Path field enter a valid file directory path to store the metadata.
bulkSink: type: migrate config: dirPath: <Directory Path to store data>
dirPath needs to be set as the same passed in stage config, which contains the file directory path where the
metadata is stored.
The main property here is the
openMetadataServerConfig, where you can define the host and security provider of your
For a simple, local installation using our docker containers, this looks like:
workflowConfig: openMetadataServerConfig: hostPort: http://<hostport of 0.10.x Openmetadata Server>/api authProvider: no-auth
openMetadataServerConfig you need to pass details about the 0.10 OpenMetadata Server.
3. Run with the CLI
First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run:
metadata ingest -c <path-to-yaml>