Skip to main content
GCS

GCS

PROD
Feature List
✓ Metadata
This page contains the setup guide and reference information for the GCS connector. Configure and schedule GCS metadata workflows from the CLI:

​
Requirements

To run the GCS ingestion, you will need to install: 
pip3 install "openmetadata-ingestion[datalake-gcs]"

OpenMetadata 1.0 or later

To deploy OpenMetadata, check the Deployment guides.
 We need the following permissions in GCP:

​
GCS Permissions

For all the buckets that we want to ingest, we need to provide the following:
  • storage.buckets.get
  • storage.buckets.list
  • storage.objects.get
  • storage.objects.list

​
OpenMetadata Manifest

In any other connector, extracting metadata happens automatically. In this case, we will be able to extract high-level metadata from buckets, but in order to understand their internal structure we need users to provide an openmetadata.json file at the bucket root. Supported File Formats: [ "csv", "tsv", "avro", "parquet", "json", "json.gz", "json.zip" ] You can learn more about this here. Keep reading for an example on the shape of the manifest file.

​
OpenMetadata Manifest

Our manifest file is defined as a JSON Schema, and can look like this:

​
Global Manifest

You can also manage a single manifest file to centralize the ingestion process for any container, named openmetadata_storage_manifest.json. You can also keep local manifests openmetadata.json in each container, but if possible, we will always try to pick up the global manifest during the ingestion.

​
Metadata Ingestion

All connectors are defined as JSON Schemas. Here you can find the structure to create a connection to Athena. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following JSON Schema

​
1. Define the YAML Config

This is a sample config for Athena: