Publishing Results & Best Practices
Publishing Results to OpenMetadata
Results can be published back to OpenMetadata for tracking, alerting, and visualization:DataFrame Validation Results
Benefits of Publishing Results
- Historical tracking: View trends over time
- Alerting: Trigger notifications on failures
- Dashboards: Centralized data quality monitoring
- Collaboration: Share results across teams
- Compliance: Maintain audit trails
Error Handling and Retries
Implement robust error handling:Dynamic Test Generation
Generate tests programmatically based on metadata:Multi-Table Validation
Validate multiple tables in a workflow:Best Practices Summary
- Version control test configurations: Store YAML configs in git
- Use environment variables: Never hardcode credentials
- Implement retries: Handle transient failures gracefully
- Publish results: Enable tracking and alerting in OpenMetadata
- Monitor execution: Track metrics for test runs
- Handle errors explicitly: Don’t silently swallow failures
- Document tests: Use descriptive names and descriptions
- Validate incrementally: Test early and often in pipelines
- Separate concerns: Let data stewards define tests, engineers execute them
- Test your tests: Ensure test definitions are correct
Next Steps
- Review the Test Definitions Reference
- Learn about TestRunner
- Explore DataFrame Validation
- Return to Data Quality as Code Overview
- Check our Examples and Tutorials out