Kubernetes Orchestrator Operations & Troubleshooting

This guide covers validating your Kubernetes orchestrator setup, viewing pipeline logs, troubleshooting common issues, and migrating from Airflow. For initial setup, see the Kubernetes Native Orchestrator guide.

Validating the Setup

1. Check Service Health

Navigate to Settings → Preferences → Health in the OpenMetadata UI to verify the Kubernetes pipeline client is properly configured and can connect to the Kubernetes API.

2. Deploy a Test Pipeline

Create a simple metadata ingestion pipeline from the OpenMetadata UI. The pipeline should:

Show “Deployed” status
Display the Kubernetes Job/CronJob name

3. Check Kubernetes Resources

# List ingestion ConfigMaps
kubectl get configmaps -l app.kubernetes.io/managed-by=openmetadata

# List ingestion Jobs
kubectl get jobs -l app.kubernetes.io/managed-by=openmetadata

# List ingestion CronJobs (native mode)
kubectl get cronjobs -l app.kubernetes.io/managed-by=openmetadata

# List CronOMJobs (operator mode)
kubectl get cronomjobs -l app.kubernetes.io/managed-by=openmetadata

# View pod logs
kubectl logs -l app.kubernetes.io/component=ingestion -f

Pipeline Logs

By default, pipeline logs are retrieved directly from Kubernetes pod logs, so the logs of a run are lost when its pod is deleted. To keep them, see Storing Pipeline Logs in S3. OpenMetadata implements log pagination for large log files, splitting them into ~1MB chunks for efficient retrieval. To view logs:

Navigate to Settings → Services → Agents
Select your pipeline
Click on Logs to view them directly on OpenMetadata UI

Alternatively, read the logs directly with kubectl:

List the ingestion pods, newest last, with the pipeline each one belongs to:

kubectl get pods -n <namespace> -l app.kubernetes.io/component=ingestion \
  --sort-by=.metadata.creationTimestamp -L app.kubernetes.io/pipeline

Read the main container of the pod you want:

kubectl logs -n <namespace> <pod-name> -c main

With the OMJob operator, each run has two pods: one ending in -main (the ingestion itself) and one ending in -exit (it reports the final status). Read the -main pod. Kubernetes only keeps these logs while the pod exists and its container is still on the node.

Pods of Finished Runs Are Not Cleaned Up

Known issue in 2.0.x with the OMJob operator: pods from finished runs are not deleted when ttlSecondsAfterFinished expires, so they accumulate in the namespace. The fix ships in 2.1 (OpenMetadata#33319).

Until you upgrade, delete the finished pods yourself.

Deleting a finished pod also deletes its logs. Copy any logs you still need before you run these commands. Running pods are not touched.

Run these two commands periodically, or schedule them as a job:

kubectl delete pods -n <namespace> -l app.kubernetes.io/managed-by=openmetadata \
  --field-selector=status.phase=Succeeded

kubectl delete pods -n <namespace> -l app.kubernetes.io/managed-by=openmetadata \
  --field-selector=status.phase=Failed

Troubleshooting

Server/Client Version Mismatch

If you see an error like server version X does not match client version Y, the ingestion job is running a different version of the ingestion library than the OpenMetadata server expects. Most common cause: ingestionImage is set to :latest in your Helm values. When a new OpenMetadata release is published, :latest moves to that version. Any Kubernetes node without the old image cached will pull the new one on the next job run, silently upgrading the ingestion client while your server stays on the previous version. Fix: Pin ingestionImage to the exact version of your server:

k8s:
  ingestionImage: "docker.getcollate.io/openmetadata/ingestion-base:2.0.5"

Replace 2.0.5 with your actual server version. The ingestion image must match the server’s full x.y.z version: for example, a 2.0.5 server requires a 2.0.5 ingestion image. After updating, redeploy OpenMetadata:

helm upgrade openmetadata open-metadata/openmetadata -f values.yaml -n openmetadata

Pipeline stuck in “Queued” state

If the pipeline cannot start and remains in “Queued” state, check if the pod can be scheduled:

kubectl get pods -l app.kubernetes.io/pipeline=<pipeline-name>
kubectl describe pod <pod-name>

Common causes:

Image pull errors (check imagePullSecrets)
Insufficient cluster resources (increase CPU/memory limits or add nodes)
Node selector constraints

Permission Denied Errors

If you see RBAC-related errors:

kubectl auth can-i create jobs --as=system:serviceaccount:<namespace>:openmetadata

Ensure the service account has the required permissions.

Ingestion Pod Crashes (OOMKilled)

Increase memory limits in the Helm values:

k8s:
  resources:
    limits:
      memory: "8Gi"
    requests:
      memory: "2Gi"

CronJob Not Triggering

Check CronJob status and events:

kubectl get cronjob <cronjob-name> -o yaml
kubectl describe cronjob <cronjob-name>

Common issues:

Invalid cron expression
startingDeadlineSeconds too short
Concurrency policy blocking execution

Migrating from Airflow

If you’re migrating from Airflow to the Kubernetes orchestrator:

Stop existing Airflow-managed pipelines - Disable or delete pipelines managed by Airflow
Update Helm values - Switch type: "airflow" to type: "k8s"
Redeploy OpenMetadata - Apply the new Helm configuration
Re-deploy pipelines - Navigate to each pipeline and click “Deploy” to create the Kubernetes resources

The migration does not automatically transfer pipeline schedules. You’ll need to re-configure and deploy each pipeline after switching to the Kubernetes orchestrator.

Comparison: Airflow vs Kubernetes Orchestrator

Feature	Airflow	K8s Native	K8s with OMJob Operator
Infrastructure	Requires Airflow deployment	Uses existing K8s cluster	Uses existing K8s cluster
CRD Installation	N/A	Not required	Required
Exit Handler Guarantee	✅ Airflow handles	❌ Best effort	✅ Guaranteed
Failure Diagnostics	❌	❌	✅
UI for DAGs	✅ Airflow UI	OpenMetadata UI	OpenMetadata UI
Resource efficiency	Always running	Jobs on-demand	Jobs on-demand
K8s-native monitoring	Extra setup	✅ Native	✅ Native

​Kubernetes Orchestrator Operations & Troubleshooting

​Validating the Setup

​1. Check Service Health

​2. Deploy a Test Pipeline

​3. Check Kubernetes Resources

​Pipeline Logs

​Pods of Finished Runs Are Not Cleaned Up

​Troubleshooting

​Server/Client Version Mismatch

​Pipeline stuck in “Queued” state

​Permission Denied Errors

​Ingestion Pod Crashes (OOMKilled)

​CronJob Not Triggering

​Migrating from Airflow

​Comparison: Airflow vs Kubernetes Orchestrator

Kubernetes Orchestrator Operations & Troubleshooting

Validating the Setup

1. Check Service Health

2. Deploy a Test Pipeline

3. Check Kubernetes Resources

Pipeline Logs

Pods of Finished Runs Are Not Cleaned Up

Troubleshooting

Server/Client Version Mismatch

Pipeline stuck in “Queued” state

Permission Denied Errors

Ingestion Pod Crashes (OOMKilled)

CronJob Not Triggering

Migrating from Airflow

Comparison: Airflow vs Kubernetes Orchestrator