OpenMetadata
Search…
Roadmap
Here is the OpenMetadata Roadmap for the next 3 releases.
We are doing a monthly release and we are going to evolve fast and adapt to community needs. Below roadmap is subject to change based on community needs and feedback. our roadmap yet, please file an Issue Github or ping us on Slack
If you would like to prioritize any feature or would like to add a new feature that is not in our roadmap yet, please file an Issue on Github or ping us on Slack.
Please check our Releases page to see all the features we shipped so far.

Roadmap

0.10.0 Release - Apr 27th, 2022

Support for Database Schema

OpenMetadata supports databases, service name databases, and tables. We’ve added Database Schema as part of the FQN. For each external data source, we ingest the database, as well as the tables that are contained underneath the schemas.

Support for Hard Delete

OpenMetadata supported soft deletions. Now, we also support the hard deletion of entities through the UI, APIs, and ingestion. Hard deleting an entity removes the entity and all of its relationships. This will also generate a change event.

Deploy Ingestion from UI

OpenMetadata has refactored the service connections to simplify the ingestion jobs from both the ingestion framework as well as the UI. We now use the pydantic models automatically generated from the JSON schemas for the connection definition. The ‘Add Service’ form is automatically generated in the UI based on the JSON schema specifications for the various connectors that are supported in OpenMetadata.

Download DBT Manifest Files from Amazon S3 or Google Cloud Storage

Previously, when ingesting the models and lineage from DBT, we passed the path of the DBT manifest and catalog files directly into the workflow. We’ve worked on improving the quality of life of DBT. Now, we can dynamically download these files from Amazon S3 or Google Cloud Storage. This way we can have any other process to connect to the DBT, extract the catalog, and put it into any cloud service. We just need the path name and workflow job details from the metadata extraction to be able to ingest metadata.

JSON Schema based Connection Definition

Each service (database, dashboard, messaging, or pipeline service) has its own configuration specifications, with some unique requirements for some services. Instead of the ad hoc definitions of the source module in Python for each connector, we’ve worked on the full refactoring of the ingestion framework. We now use the pydantic models automatically generated from the JSON schemas for the connection definition.

Airflow Rest APIs

The Airflow REST APIs have been refactored. With our API centric model, we are creating a custom airflow rest API directly on top of Airflow using plugins. This passes the connection information to automatically generate all the dags and prepares handy methods to help us test the connection to the source before creating the service.

UI Changes

  • The UI improvements are directed toward providing a consistent user experience.
  • Hard Deletion of Entities: With the support for the hard deletion of entities, we can permanently delete tables, topics, or services. When the entity is hard deleted, the entity and all its relationships are removed. This generates an ‘EntityDeleted’ change event.
  • Dynamic “Add Service” Forms: The ‘Add Service’ form is automatically generated in the UI based on the JSON schema specifications for the various connectors that are supported in OpenMetadata.
  • UI Support for Database Schema as part of FQN: The database schema has been introduced in the 0.10 release. All the entity pages now support Database Schema in the UI.
  • Lineage Editor: Improvements have been made to the lineage editor.
  • Teams: While signing up in OpenMetadata, the teams with restricted access are hidden and only the joinable teams are displayed.
  • Team Owner: An Owner field has been added to the Team entity. Only team owners can update the teams.
  • Activity Feeds: The Activity Feeds UI supports infinite scrolling.
  • Add User: A user can be added from the Users page.

Security Changes

Support Refresh Tokens for Auth0 and Okta SSO
The JWT tokens generated by the SSO providers expire by default in about an hour, making the user re-login often. In this release, we’ve added support for refresh tokens for Auth0 and Okta SSO. The tokens are refreshed silently behind the scenes to provide an uninterrupted user experience. In future releases, we’ll continue to stabilize authentication and add refresh tokens for the other SSO providers.
Custom OIDC SSO
OpenMetadata now supports integration with your custom built OIDC SSO for authentication. This is supported both on the front end for user authentication as well as on the ingestion side.
Azure SSO
Support has been added for Azure SSO on Airflow.

0.11 Release - Jun 9th, 2022

Data Insights

  • Show users/teams how their data doing
  • Cost analysis
  • How to improve Data culture

Data Quality

  • Data SLAs
  • Freshness and completeness
  • Ability to deploy GE tests from OpenMetadata UI

Lineage

  • Column Level Lineage support
  • Incident detection through Lineage

Collaboration

  • Tiering Report on how to improve data assets and data culture
  • Email notifications
  • Webbrowser notifications

ML Features

  • support ML feature UI
  • Sagemake connector