Skip to main content

Publishing Results & Best Practices

Publishing Results to OpenMetadata

Results can be published back to OpenMetadata for tracking, alerting, and visualization:

DataFrame Validation Results

Benefits of Publishing Results

  • Historical tracking: View trends over time
  • Alerting: Trigger notifications on failures
  • Dashboards: Centralized data quality monitoring
  • Collaboration: Share results across teams
  • Compliance: Maintain audit trails

Error Handling and Retries

Implement robust error handling:

Dynamic Test Generation

Generate tests programmatically based on metadata:

Multi-Table Validation

Validate multiple tables in a workflow:

Best Practices Summary

  1. Version control test configurations: Store YAML configs in git
  2. Use environment variables: Never hardcode credentials
  3. Implement retries: Handle transient failures gracefully
  4. Publish results: Enable tracking and alerting in OpenMetadata
  5. Monitor execution: Track metrics for test runs
  6. Handle errors explicitly: Don’t silently swallow failures
  7. Document tests: Use descriptive names and descriptions
  8. Validate incrementally: Test early and often in pipelines
  9. Separate concerns: Let data stewards define tests, engineers execute them
  10. Test your tests: Ensure test definitions are correct

Next Steps