Upgrade from 0.9 to 0.10

The 0.10.x release contains backward incompatible changes this guide will help you to migrate your metadata from the 0.9 version to the 0.10 version. Please ensure that you've taken a backup of metadata of your OpenMetadata instance prior to upgrading to the 0.10 release.
Requirements
Update the openmetadata-ingestion client to the latest available version.
1
pip install --upgrade openmetadata-ingestion
Copied!
Back up your metadata. The steps to backup metadata depends on the method of deployment are defined below:
Docker
Kubernetes
Bare Metal
Backup Metadata
Backup Metadata
Backup Metadata
To migrate the metadata from the 0.9 to the 0.10 version you need to ingest all the connectors in the 0.10 version with the same service name as done in the 0.9 version. For example, if you have ingested snowflake connector in 0.9 version with service name Snowflake_Prod then in the 0.10 version, you need to ingest the snowflake connector again with the name Snowflake_Prod.
Metadata Migration
All connectors are now defined as JSON Schemas. Here you can find the structure to create a connection to Snowflake.
In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server.
The workflow is modeled around the following JSON Schema.
1. Setup your 0.10.1 installation
1. OpenMetadata 0.9, keep it running as it is with its own MySQL and ES
2. Install OpenMetadata 0.10.1, make it completely isolated from your 0.9 install new MySQL DB and other ES indexes
3. Install 0.10.1 connectors
4. Run ingestion of all your entities using the new connector. Keep the service names the same as your 0.9 installations. Once all the entities are ingested into OpenMetadata 0.10.1. For example, if you have ingested snowflake connector in 0.9 version with service name Snowflake_Prod then in the 0.10 version, you need to ingest the snowflake connector again with the name Snowflake_Prod.
5. Run migration using the newly installed connector
2. Define the YAML Config
This is a sample config:
1
source:
2
  type: migrate
3
  serviceName: local_metadata
4
  serviceConnection:
5
    config:
6
      type: OpenMetadata
7
      hostPort: http://<hostport of 0.9.0 Openmetadata Server>/api
8
      authProvider: no-auth
9
      includeTables: true
10
      includeUsers: true
11
      includeTopics: true
12
      includePipelines: true
13
      includeTags: true
14
      includeGlossaryTerms: true
15
      includeMessagingServices: true
16
      includeDatabaseServices: true
17
      includePipelineServices: true
18
      enableVersionValidation: false
19
      limitRecords: 100000
20
  sourceConfig:
21
    config:
22
      enableDataProfiler: false
23
stage:
24
  type: migrate
25
  config:
26
    dirPath: <Directory Path to store data>
27
bulkSink:
28
  type: migrate
29
  config:
30
    dirPath: <Directory Path to store data>
31
workflowConfig:
32
  openMetadataServerConfig:
33
    hostPort: http://<hostport of 0.10.x Openmetadata Server>/api
34
    authProvider: no-auth
Copied!
Source Configuration - Service Connection
You can find all the definitions and types for the serviceConnection here.
hostPort: Enter the OpenMetadata 0.9 Server Config. Must include API end point ex: http://localhost:8585/api
includeTables: Include tables for migration
includeUsers: Include users for migration
includeTopics: Include topics for migration
includePipelines: Include pipelines for migration
includeGlossaryTerms: include glossary terms for migration
includeTags: Include tags for migration.
includeMessagingServices: Include messaging services for migration.
includeDatabaseServices: Include database services for migration.
enableVersionValidation: Enable the server OpenMetadata and client OpenMetadata version validation. For migrating the metadata this has to be set to false
limitRecords: Limit the number records that is fetched by 0.9 OpenMetadata API (1 to 1000000, default = 10).
Stage Configuration
1
stage:
2
  type: migrate
3
  config:
4
    dirPath: <Directory Path to store data>
Copied!
The data while migrating from the 0.9 to 0.10 version needs to be stored in a file directory. in the dir Path field enter a valid file directory path to store the metadata.
BulkSink Configuration
1
bulkSink:
2
  type: migrate
3
  config:
4
    dirPath: <Directory Path to store data>
Copied!
The dirPath needs to be set as the same passed in stage config, which contains the file directory path where the metadata is stored.
Workflow Configuration
The main property here is the openMetadataServerConfig, where you can define the host and security provider of your OpenMetadata installation.
For a simple, local installation using our docker containers, this looks like:
1
workflowConfig:
2
  openMetadataServerConfig:
3
    hostPort: http://<hostport of 0.10.x Openmetadata Server>/api
4
    authProvider: no-auth
Copied!
Note: In openMetadataServerConfig you need to pass details about the 0.10 OpenMetadata Server.
2. Run with the CLI
First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run:
1
metadata ingest -c <path-to-yaml>
Copied!

Upgrade OpenMetadata on Kubernetes

Next - Data Discovery

Keyword Search

Last modified 16h ago

Copy link

Edit on GitHub

Contents

Requirements

Metadata Migration

1. Setup your 0.10.1 installation

2. Define the YAML Config

2. Run with the CLI