OpenMetadata
Search…
Upgrade from 0.9 to 0.10
The 0.10.x release contains backward incompatible changes this guide will help you to migrate your metadata from the 0.9 version to the 0.10 version. Please ensure that you've taken a backup of metadata of your OpenMetadata instance prior to upgrading to the 0.10 release.

Requirements

  • Update the openmetadata-ingestion client to the latest available version.
1
pip install --upgrade openmetadata-ingestion
Copied!
  • Back up your metadata. The steps to backup metadata depends on the method of deployment are defined below:
  • To migrate the metadata from the 0.9 to the 0.10 version you need to ingest all the connectors in the 0.10 version with the same service name as done in the 0.9 version. For example, if you have ingested snowflake connector in 0.9 version with service name Snowflake_Prod then in the 0.10 version, you need to ingest the snowflake connector again with the name Snowflake_Prod.

Metadata Migration

All connectors are now defined as JSON Schemas. Here you can find the structure to create a connection to Snowflake.
In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server.
The workflow is modeled around the following JSON Schema.

1. Setup your 0.10.1 installation

1. OpenMetadata 0.9, keep it running as it is with its own MySQL and ES
2. Install OpenMetadata 0.10.1, make it completely isolated from your 0.9 install new MySQL DB and other ES indexes
3. Install 0.10.1 connectors
4. Run ingestion of all your entities using the new connector. Keep the service names the same as your 0.9 installations. Once all the entities are ingested into OpenMetadata 0.10.1. For example, if you have ingested snowflake connector in 0.9 version with service name Snowflake_Prod then in the 0.10 version, you need to ingest the snowflake connector again with the name Snowflake_Prod.
5. Run migration using the newly installed connector

2. Define the YAML Config

This is a sample config:
1
source:
2
type: migrate
3
serviceName: local_metadata
4
serviceConnection:
5
config:
6
type: OpenMetadata
7
hostPort: http://<hostport of 0.9.0 Openmetadata Server>/api
8
authProvider: no-auth
9
includeTables: true
10
includeUsers: true
11
includeTopics: true
12
includePipelines: true
13
includeTags: true
14
includeGlossaryTerms: true
15
includeMessagingServices: true
16
includeDatabaseServices: true
17
includePipelineServices: true
18
enableVersionValidation: false
19
limitRecords: 100000
20
sourceConfig:
21
config:
22
enableDataProfiler: false
23
stage:
24
type: migrate
25
config:
26
dirPath: <Directory Path to store data>
27
bulkSink:
28
type: migrate
29
config:
30
dirPath: <Directory Path to store data>
31
workflowConfig:
32
openMetadataServerConfig:
33
hostPort: http://<hostport of 0.10.x Openmetadata Server>/api
34
authProvider: no-auth
Copied!

Source Configuration - Service Connection

You can find all the definitions and types for the serviceConnection here.
  • hostPort: Enter the OpenMetadata 0.9 Server Config. Must include API end point ex: http://localhost:8585/api
  • includeTables: Include tables for migration
  • includeUsers: Include users for migration
  • includeTopics: Include topics for migration
  • includePipelines: Include pipelines for migration
  • includeGlossaryTerms: include glossary terms for migration
  • includeTags: Include tags for migration.
  • includeMessagingServices: Include messaging services for migration.
  • includeDatabaseServices: Include database services for migration.
  • enableVersionValidation: Enable the server OpenMetadata and client OpenMetadata version validation. For migrating the metadata this has to be set to false
  • limitRecords: Limit the number records that is fetched by 0.9 OpenMetadata API (1 to 1000000, default = 10).

Stage Configuration

1
stage:
2
type: migrate
3
config:
4
dirPath: <Directory Path to store data>
Copied!
The data while migrating from the 0.9 to 0.10 version needs to be stored in a file directory. in the dir Path field enter a valid file directory path to store the metadata.

BulkSink Configuration

1
bulkSink:
2
type: migrate
3
config:
4
dirPath: <Directory Path to store data>
Copied!
The dirPath needs to be set as the same passed in stage config, which contains the file directory path where the metadata is stored.

Workflow Configuration

The main property here is the openMetadataServerConfig, where you can define the host and security provider of your OpenMetadata installation.
For a simple, local installation using our docker containers, this looks like:
1
workflowConfig:
2
openMetadataServerConfig:
3
hostPort: http://<hostport of 0.10.x Openmetadata Server>/api
4
authProvider: no-auth
Copied!
Note: In openMetadataServerConfig you need to pass details about the 0.10 OpenMetadata Server.

2. Run with the CLI

First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run:
1
metadata ingest -c <path-to-yaml>
Copied!