Kubernetes Native Orchestrator

Starting with OpenMetadata 1.12, you can run ingestion pipelines directly using native Kubernetes, eliminating the need for Apache Airflow. This is ideal for organizations that:

Already run workloads on Kubernetes and prefer native solutions
Don’t need the full feature set of Apache Airflow

Orchestration Modes

The Kubernetes orchestrator supports two modes for running ingestion pipelines:

Option 1: OMJob Operator (Recommended)

Uses custom Kubernetes CRDs (OMJob and CronOMJob) managed by the OpenMetadata operator.

Resource	Description
CronOMJob	Scheduled pipelines - runs on a cron schedule
OMJob	On-demand pipelines - one-off execution when triggered

Recommended for production. The OMJob Operator provides guaranteed exit handler execution and failure diagnostics.

Advantages:

Exit Handler Guarantee: Even if the ingestion pod crashes (OOMKilled, node failure, etc.), the operator ensures pipeline status is always reported back to OpenMetadata
Failure Diagnostics: Automatically collects detailed error context from pod logs and events when pipelines fail
Pod Lifecycle Monitoring: The operator watches pod events and updates pipeline status in real-time

Requirements:

Elevated permissions to install Custom Resource Definitions (CRDs)
The OMJob Operator deployment running in your cluster

Option 2: Native Kubernetes Jobs

Uses standard Kubernetes resources (Job and CronJob) without any custom CRDs.

Resource	Description
CronJob	Scheduled pipelines - runs on a cron schedule
Job	On-demand pipelines - one-off execution when triggered

Advantages:

No CRD installation required - uses only built-in Kubernetes resources
Works in environments with restricted permissions
Simpler setup

Limitations:

No guaranteed exit handler - if a pod is killed unexpectedly, status updates may not reach OpenMetadata
No automatic failure diagnostics

Features

Native K8s Integration

Pipelines run as standard Kubernetes Jobs, making them easy to monitor with existing K8s tooling.

Automatic Status Updates

Pipeline status is automatically reported back to OpenMetadata, including success/failure details.

Failure Diagnostics

When pipelines fail, detailed diagnostics are collected from pod logs and events. (OMJob Operator only)

Resource Control

Configure CPU, memory, node selectors, and security contexts for ingestion pods.

Setup Option 1: OMJob Operator (Recommended)

This setup uses custom CRDs for guaranteed exit handler execution and failure diagnostics.

Prerequisites

OpenMetadata deployed on Kubernetes (Helm chart recommended)
Permissions to install CRDs in your cluster
Ingestion image accessible from your cluster (docker.getcollate.io/openmetadata/ingestion-base)

Helm Values Configuration

Always pin ingestionImage to the same version as your OpenMetadata server — never use :latest. The :latest tag moves whenever a new release is published; any Kubernetes node without the previous image cached will silently pull the newer version on the next scheduled job, causing a server/client version mismatch. The ingestion major version must match the server major version (for example, a 2.0.x server requires a 2.0.x ingestion image).

# Enable the OMJob Operator
omjobOperator:
  enabled: true
  image:
    repository: docker.getcollate.io/openmetadata/omjob-operator
    tag: "2.0.5"
    pullPolicy: IfNotPresent
  resources:
    requests:
      cpu: "100m"
      memory: "128Mi"
    limits:
      cpu: "500m"
      memory: "256Mi"

openmetadata:
  config:
    pipelineServiceClientConfig:
      enabled: true
      type: "k8s"
      metadataApiEndpoint: http://openmetadata:8585/api

      k8s:
        # Use the OMJob Operator
        useOMJobOperator: true
        
        # Container image for ingestion jobs
        ingestionImage: "docker.getcollate.io/openmetadata/ingestion-base:2.0.5"
        imagePullPolicy: "IfNotPresent"
        imagePullSecrets: ""
        
        # Service account for ingestion jobs
        serviceAccountName: "openmetadata-ingestion"
        
        # Job lifecycle settings
        ttlSecondsAfterFinished: 86400  # Keep completed jobs for 24 hours
        activeDeadlineSeconds: 7200      # Max 2 hour runtime
        backoffLimit: 3                  # Retry up to 3 times
        
        # Job history
        successfulJobsHistoryLimit: 3
        failedJobsHistoryLimit: 3
        
        # Pod security context
        securityContext:
          runAsUser: 1000
          runAsGroup: 1000
          fsGroup: 1000
          runAsNonRoot: true
        
        # Resource limits
        resources:
          limits:
            cpu: "2"
            memory: "4Gi"
          requests:
            cpu: "500m"
            memory: "1Gi"
        
        # Enable failure diagnostics (only works with OMJob Operator)
        enableFailureDiagnostics: true
        
        # RBAC - set to false if managed externally
        rbac:
          enabled: true

Required RBAC Permissions

When using the OMJob Operator, additional permissions are needed for the custom resources:

rules:
  # Pod management for pipeline jobs and diagnostics
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "create", "update", "patch", "delete"]
  # ConfigMaps for pipeline configuration
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list", "create", "update", "patch", "delete"]
  # Secrets for pipeline credentials
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "create", "update", "patch", "delete"]
  # Events for diagnostics
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list"]
  # Jobs and CronJobs management
  - apiGroups: ["batch"]
    resources: ["jobs", "cronjobs"]
    verbs: ["get", "list", "create", "update", "patch", "delete"]
  # OMJob CRDs
  - apiGroups: ["pipelines.openmetadata.org"]
    resources: ["omjobs"]
    verbs: ["get", "list", "create", "update", "patch", "delete"]
  - apiGroups: ["pipelines.openmetadata.org"]
    resources: ["omjobs/status"]
    verbs: ["get", "patch"]
  - apiGroups: ["pipelines.openmetadata.org"]
    resources: ["cronomjobs"]
    verbs: ["get", "list", "create", "update", "patch", "delete"]
  - apiGroups: ["pipelines.openmetadata.org"]
    resources: ["cronomjobs/status"]
    verbs: ["get", "patch"]

Setup Option 2: Native Kubernetes Jobs

This setup uses standard Kubernetes Jobs and CronJobs without any custom CRDs.

Prerequisites

OpenMetadata deployed on Kubernetes (Helm chart recommended)
RBAC permissions for the OpenMetadata service account to manage Jobs, CronJobs, ConfigMaps, and Secrets
Ingestion image accessible from your cluster (docker.getcollate.io/openmetadata/ingestion-base)

Helm Values Configuration

openmetadata:
  config:
    pipelineServiceClientConfig:
      enabled: true
      type: "k8s"
      metadataApiEndpoint: http://openmetadata:8585/api

      k8s:
        # Do NOT use the OMJob Operator (default)
        useOMJobOperator: false
        
        # Container image for ingestion jobs
        ingestionImage: "docker.getcollate.io/openmetadata/ingestion-base:2.0.5"
        imagePullPolicy: "IfNotPresent"
        imagePullSecrets: ""
        
        # Service account for ingestion jobs
        serviceAccountName: "openmetadata-ingestion"
        
        # Job lifecycle settings
        ttlSecondsAfterFinished: 86400
        activeDeadlineSeconds: 7200
        backoffLimit: 3
        
        # Job history
        successfulJobsHistoryLimit: 3
        failedJobsHistoryLimit: 3
        
        # Pod security context
        securityContext:
          runAsUser: 1000
          runAsGroup: 1000
          fsGroup: 1000
          runAsNonRoot: true
        
        # Resource limits
        resources:
          limits:
            cpu: "2"
            memory: "4Gi"
          requests:
            cpu: "500m"
            memory: "1Gi"
        
        # RBAC - set to false if managed externally
        rbac:
          enabled: true

Required RBAC Permissions

rules:
  # Pod management for pipeline jobs
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "create", "update", "patch", "delete"]
  # ConfigMaps for pipeline configuration
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list", "create", "update", "patch", "delete"]
  # Secrets for pipeline credentials
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "create", "update", "patch", "delete"]
  # Events for diagnostics
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list"]
  # Jobs and CronJobs management
  - apiGroups: ["batch"]
    resources: ["jobs", "cronjobs"]
    verbs: ["get", "list", "create", "update", "patch", "delete"]

Storing Pipeline Logs in S3

By default, OpenMetadata reads the logs of a run from its Kubernetes pod, so the logs are lost when the pod is deleted. This happens, for example, when ttlSecondsAfterFinished expires or when a node is scaled down or replaced. The UI then shows No pods found for this pipeline or unable to retrieve container logs. To keep the logs, store them in an S3 bucket or in S3-compatible storage such as MinIO. While a pipeline runs, the ingestion sends its logs to the OpenMetadata server. The server writes them to the bucket, and the UI reads them from there.

1. Create a Bucket

Create a bucket used only for pipeline logs. Only the OpenMetadata server needs access to it, because the ingestion pods send their logs through the server.

When the server starts, it replaces the bucket’s lifecycle configuration with one rule that deletes logs after 30 days. Any lifecycle rules the bucket already has are removed.

2. Configure the Server

Add the log storage settings to your Helm values with extraEnvs, then run helm upgrade:

extraEnvs:
  - name: PIPELINE_SERVICE_CLIENT_LOG_TYPE
    value: "s3"
  - name: PIPELINE_SERVICE_CLIENT_LOG_BUCKET_NAME
    value: "<bucket-name>"
  - name: PIPELINE_SERVICE_CLIENT_LOG_PREFIX   # optional folder inside the bucket
    value: "ingestion-logs"
  - name: PIPELINE_SERVICE_CLIENT_LOG_REGION
    value: "<aws-region>"
  # Credentials, read from a Kubernetes secret
  - name: PIPELINE_SERVICE_CLIENT_LOG_AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: <secret-name>
        key: <access-key-id-key>
  - name: PIPELINE_SERVICE_CLIENT_LOG_AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: <secret-name>
        key: <secret-access-key-key>
  # To use the IAM role of the server pod instead, remove the two keys above and set:
  # - name: PIPELINE_SERVICE_CLIENT_AWS_IAM_AUTH_ENABLED
  #   value: "true"
  # For S3-compatible storage such as MinIO, also set its URL:
  # - name: PIPELINE_SERVICE_CLIENT_LOG_AWS_ENDPOINT_URL
  #   value: "http://minio:9000"

After the upgrade, the server log shows Log storage initialized successfully: type=s3. If it doesn’t, look for an error about the bucket in the server log.

3. Enable Log Storage for Each Pipeline

A pipeline sends its logs to the bucket only when its enableStreamableLogs property is true. The property is false by default and the UI has no setting for it, so set it with the API. Then redeploy the pipeline so that the change reaches Kubernetes:

OM=https://<openmetadata-host>/api/v1
TOKEN=<jwt-token>   # for example, an admin's personal access token

# Find the pipeline id from its fully qualified name, such as "mysql_prod.mysql_prod_metadata"
ID=$(curl -s "$OM/services/ingestionPipelines/name/<pipeline-fqn>" \
  -H "Authorization: Bearer $TOKEN" | jq -r .id)

# Enable log storage for the pipeline
curl -X PATCH "$OM/services/ingestionPipelines/$ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json-patch+json" \
  -d '[{"op": "add", "path": "/enableStreamableLogs", "value": true}]'

# Redeploy the pipeline
curl -X POST "$OM/services/ingestionPipelines/deploy/$ID" \
  -H "Authorization: Bearer $TOKEN"

Repeat this for every pipeline whose logs you want to keep. Editing the pipeline later in the UI keeps the setting.

The PIPELINE_SERVICE_CLIENT_LOG_ENABLED server setting doesn’t enable log storage for your pipelines. Set enableStreamableLogs on each pipeline as shown above.

From then on, the logs of every new run of these pipelines stay visible in the UI after the pods are deleted. The logs of each run are stored in their own logs.txt file, in a folder per pipeline and run, so the logs of earlier runs stay in the bucket too. The UI shows the logs of the latest run. Runs that finished before the change are not copied to the bucket.

For validating your setup, viewing pipeline logs, troubleshooting, and migrating from Airflow, see the Operations & Troubleshooting guide.

​Kubernetes Native Orchestrator

​Orchestration Modes

​Option 1: OMJob Operator (Recommended)

​Option 2: Native Kubernetes Jobs

​Features

Native K8s Integration

Automatic Status Updates

Failure Diagnostics

Resource Control

​Setup Option 1: OMJob Operator (Recommended)

​Prerequisites

​Helm Values Configuration

​Required RBAC Permissions

​Setup Option 2: Native Kubernetes Jobs

​Prerequisites

​Helm Values Configuration

​Required RBAC Permissions

​Storing Pipeline Logs in S3

​1. Create a Bucket

​2. Configure the Server

​3. Enable Log Storage for Each Pipeline

Kubernetes Native Orchestrator

Orchestration Modes

Option 1: OMJob Operator (Recommended)

Option 2: Native Kubernetes Jobs

Features

Setup Option 1: OMJob Operator (Recommended)

Prerequisites

Helm Values Configuration

Required RBAC Permissions

Setup Option 2: Native Kubernetes Jobs

Prerequisites

Helm Values Configuration

Required RBAC Permissions

Storing Pipeline Logs in S3

1. Create a Bucket

2. Configure the Server

3. Enable Log Storage for Each Pipeline