Column-Level Test Definitions

Column-level tests validate properties of specific columns. For table-level tests and importing instructions, see the Test Definitions Reference.

Column-Level Tests

ColumnValuesToBeNotNull

Validates that a column contains no null or missing values. Parameters:

column (str, required): Name of the column to validate

Example:

from metadata.sdk.data_quality import ColumnValuesToBeNotNull

# Email column must not contain nulls
test = ColumnValuesToBeNotNull(column="email")

Use Cases:

Ensure required fields are populated
Validate data completeness
Enforce NOT NULL constraints

ColumnValuesToBeUnique

Validates that all values in a column are unique with no duplicates. Parameters:

column (str, required): Name of the column to validate

Example:

from metadata.sdk.data_quality import ColumnValuesToBeUnique

# Customer ID must be unique
test = ColumnValuesToBeUnique(column="customer_id")

Use Cases:

Validate primary keys
Ensure unique identifiers
Detect duplicate records

ColumnValuesToBeInSet

Validates that all values in a column belong to a specified set. Parameters:

column (str, required): Name of the column to validate
allowed_values (list[str], required): List of acceptable values

Example:

from metadata.sdk.data_quality import ColumnValuesToBeInSet

# Status must be one of these values
test = ColumnValuesToBeInSet(
    column="status",
    allowed_values=["active", "inactive", "pending"]
)

# Country codes must be valid
test = ColumnValuesToBeInSet(
    column="country_code",
    allowed_values=["US", "UK", "CA", "AU", "DE"]
)

Use Cases:

Validate enum values
Enforce categorical constraints
Validate lookup values

ColumnValuesToBeNotInSet

Validates that column values do not contain any forbidden values. Parameters:

column (str, required): Name of the column to validate
forbidden_values (list[str], required): List of values that must not appear

Example:

from metadata.sdk.data_quality import ColumnValuesToBeNotInSet

# Prevent test email addresses
test = ColumnValuesToBeNotInSet(
    column="email",
    forbidden_values=["test@test.com", "admin@admin.com", "noreply@example.com"]
)

Use Cases:

Detect test data in production
Blacklist invalid values
Filter out placeholder values

ColumnValuesToMatchRegex

Validates that column values match a specified regular expression pattern. Parameters:

column (str, required): Name of the column to validate
regex (str, required): Regular expression pattern

Example:

from metadata.sdk.data_quality import ColumnValuesToMatchRegex

# Validate email format
test = ColumnValuesToMatchRegex(
    column="email",
    regex=r"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$"
)

# Validate phone number format (US)
test = ColumnValuesToMatchRegex(
    column="phone",
    regex=r"^\+?1?\d{10}$"
)

# Validate date format (YYYY-MM-DD)
test = ColumnValuesToMatchRegex(
    column="date_string",
    regex=r"^\d{4}-\d{2}-\d{2}$"
)

Use Cases:

Validate data format consistency
Ensure pattern compliance
Detect malformed data

ColumnValuesToNotMatchRegex

Validates that column values do not match a forbidden regular expression pattern. Parameters:

column (str, required): Name of the column to validate
regex (str, required): Regular expression pattern that values must NOT match

Example:

from metadata.sdk.data_quality import ColumnValuesToNotMatchRegex

# Prevent test emails
test = ColumnValuesToNotMatchRegex(
    column="email",
    regex=r".*@test\.com$"
)

# Detect test data markers
test = ColumnValuesToNotMatchRegex(
    column="name",
    regex=r"^test.*"
)

Use Cases:

Detect test data patterns
Prevent specific formats
Identify security risks

ColumnValuesToBeBetween

Validates that all values in a column fall within a specified numeric range. Parameters:

column (str, required): Name of the column to validate
min_value (float, optional): Minimum acceptable value
max_value (float, optional): Maximum acceptable value

Example:

from metadata.sdk.data_quality import ColumnValuesToBeBetween

# Age should be reasonable
test = ColumnValuesToBeBetween(
    column="age",
    min_value=0,
    max_value=120
)

# Price must be positive
test = ColumnValuesToBeBetween(
    column="price",
    min_value=0.01,
    max_value=999999.99
)

# Percentage values
test = ColumnValuesToBeBetween(
    column="completion_rate",
    min_value=0,
    max_value=100
)

Use Cases:

Validate numeric constraints
Detect outliers
Ensure value ranges

ColumnValueMaxToBeBetween

Validates that the maximum value in a column falls within a specified range. Parameters:

column (str, required): Name of the column to validate
min_value (float, optional): Minimum acceptable maximum value
max_value (float, optional): Maximum acceptable maximum value

Example:

from metadata.sdk.data_quality import ColumnValueMaxToBeBetween

# Max temperature should be reasonable
test = ColumnValueMaxToBeBetween(
    column="temperature",
    min_value=-50,
    max_value=50
)

# Highest score should not exceed 100
test = ColumnValueMaxToBeBetween(
    column="score",
    min_value=90,
    max_value=100
)

Use Cases:

Monitor data ranges
Detect upper outliers
Validate maximum constraints

ColumnValueMinToBeBetween

Validates that the minimum value in a column falls within a specified range. Parameters:

column (str, required): Name of the column to validate
min_value (float, optional): Minimum acceptable minimum value
max_value (float, optional): Maximum acceptable minimum value

Example:

from metadata.sdk.data_quality import ColumnValueMinToBeBetween

# Min temperature validation
test = ColumnValueMinToBeBetween(
    column="temperature",
    min_value=-50,
    max_value=0
)

# Minimum age should be at least 18
test = ColumnValueMinToBeBetween(
    column="age",
    min_value=18,
    max_value=25
)

Use Cases:

Monitor lower bounds
Detect lower outliers
Validate minimum constraints

ColumnValueMeanToBeBetween

Validates that the mean (average) value falls within a specified range. Parameters:

column (str, required): Name of the column to validate
min_value (float, optional): Minimum acceptable mean value
max_value (float, optional): Maximum acceptable mean value

Example:

from metadata.sdk.data_quality import ColumnValueMeanToBeBetween

# Average rating should be between 3 and 4.5
test = ColumnValueMeanToBeBetween(
    column="rating",
    min_value=3.0,
    max_value=4.5
)

# Average response time monitoring
test = ColumnValueMeanToBeBetween(
    column="response_time_ms",
    min_value=100,
    max_value=500
)

Use Cases:

Statistical validation
Detect data drift
Monitor averages

ColumnValueMedianToBeBetween

Validates that the median value falls within a specified range. Parameters:

column (str, required): Name of the column to validate
min_value (float, optional): Minimum acceptable median value
max_value (float, optional): Maximum acceptable median value

Example:

from metadata.sdk.data_quality import ColumnValueMedianToBeBetween

# Median salary range
test = ColumnValueMedianToBeBetween(
    column="salary",
    min_value=50000,
    max_value=75000
)

Use Cases:

Robust central tendency checks
Detect skewed distributions
Monitor typical values

ColumnValueStdDevToBeBetween

Validates that the standard deviation falls within a specified range. Parameters:

column (str, required): Name of the column to validate
min_value (float, optional): Minimum acceptable standard deviation
max_value (float, optional): Maximum acceptable standard deviation

Example:

from metadata.sdk.data_quality import ColumnValueStdDevToBeBetween

# Response time variability
test = ColumnValueStdDevToBeBetween(
    column="response_time",
    min_value=0,
    max_value=100
)

Use Cases:

Detect unexpected variability
Monitor data consistency
Validate distribution stability

ColumnValuesSumToBeBetween

Validates that the sum of all values falls within a specified range. Parameters:

column (str, required): Name of the column to validate
min_value (float, optional): Minimum acceptable sum
max_value (float, optional): Maximum acceptable sum

Example:

from metadata.sdk.data_quality import ColumnValuesSumToBeBetween

# Total revenue validation
test = ColumnValuesSumToBeBetween(
    column="revenue",
    min_value=1000000,
    max_value=5000000
)

Use Cases:

Validate totals
Monitor aggregates
Detect unexpected volumes

ColumnValuesMissingCount

Validates the count of missing or null values. Parameters:

column (str, required): Name of the column to validate
missing_count_value (int, optional): Expected number of missing values
missing_value_match (list[str], optional): Additional strings to treat as missing

Example:

from metadata.sdk.data_quality import ColumnValuesMissingCount

# Expect exactly 100 missing values
test = ColumnValuesMissingCount(
    column="optional_field",
    missing_count_value=100
)

# Treat custom strings as missing
test = ColumnValuesMissingCount(
    column="status",
    missing_value_match=["N/A", "Unknown", ""]
)

Use Cases:

Monitor data completeness
Track missing data patterns
Validate optional fields

ColumnValueLengthsToBeBetween

Validates that string lengths fall within a specified range. Parameters:

column (str, required): Name of the column to validate
min_length (int, optional): Minimum acceptable string length
max_length (int, optional): Maximum acceptable string length

Example:

from metadata.sdk.data_quality import ColumnValueLengthsToBeBetween

# Username length validation
test = ColumnValueLengthsToBeBetween(
    column="username",
    min_length=3,
    max_length=20
)

# Description length constraints
test = ColumnValueLengthsToBeBetween(
    column="description",
    min_length=10,
    max_length=500
)

Use Cases:

Validate string constraints
Prevent truncation
Ensure format compliance

ColumnValuesToBeAtExpectedLocation

Validates that latitude and longitude values are within a configured radius of an expected location from a reference column, such as a city or postal code. Parameters:

column (str, required): Reference location column, such as a city or postal code column
location_reference_type (str, required): Type of location reference. Supported values are CITY and POSTAL_CODE
longitude_column_name (str, required): Name of the longitude column in the table
latitude_column_name (str, required): Name of the latitude column in the table
radius (float, required): Allowed distance from the expected location in meters

Example:

from metadata.sdk.data_quality import ColumnValuesToBeAtExpectedLocation

test = ColumnValuesToBeAtExpectedLocation(
    column="city",
    location_reference_type="CITY",
    longitude_column_name="longitude",
    latitude_column_name="latitude",
    radius=5000.0,
)

Use Cases:

Validate latitude and longitude coordinates against a city or postal code column
Detect records with coordinates outside an expected delivery or service area
Check geospatial data quality before downstream mapping or distance calculations

Customizing Tests

All tests support customization through fluent methods:

from metadata.sdk.data_quality import ColumnValuesToBeNotNull

test = ColumnValuesToBeNotNull(column="email") \
    .with_name("email_completeness_check") \
    .with_display_name("Email Completeness Validation") \
    .with_description("Ensures all customer records have email addresses") \
    .with_compute_row_count(True)

Or pass values directly to the constructor:

test = ColumnValuesToBeNotNull(
    column="email",
    name="email_completeness_check",
    display_name="Email Completeness Validation",
    description="Ensures all customer records have email addresses"
)

Next Steps

Learn how to use these tests with TestRunner
Apply tests to DataFrame Validation
Explore Advanced Usage patterns

Data Quality and Observability

Column-Level Test Definitions Reference

Column-Level Test Definitions

Column-Level Tests

ColumnValuesToBeNotNull

ColumnValuesToBeUnique

ColumnValuesToBeInSet

ColumnValuesToBeNotInSet

ColumnValuesToMatchRegex

ColumnValuesToNotMatchRegex

ColumnValuesToBeBetween

ColumnValueMaxToBeBetween

ColumnValueMinToBeBetween

ColumnValueMeanToBeBetween

ColumnValueMedianToBeBetween

ColumnValueStdDevToBeBetween

ColumnValuesSumToBeBetween

ColumnValuesMissingCount

ColumnValueLengthsToBeBetween

ColumnValuesToBeAtExpectedLocation

Customizing Tests

Next Steps

​Column-Level Test Definitions

​Column-Level Tests

​ColumnValuesToBeNotNull

​ColumnValuesToBeUnique

​ColumnValuesToBeInSet

​ColumnValuesToBeNotInSet

​ColumnValuesToMatchRegex

​ColumnValuesToNotMatchRegex

​ColumnValuesToBeBetween

​ColumnValueMaxToBeBetween

​ColumnValueMinToBeBetween

​ColumnValueMeanToBeBetween

​ColumnValueMedianToBeBetween

​ColumnValueStdDevToBeBetween

​ColumnValuesSumToBeBetween

​ColumnValuesMissingCount

​ColumnValueLengthsToBeBetween

​ColumnValuesToBeAtExpectedLocation

​Customizing Tests

​Next Steps

Column-Level Test Definitions

Column-Level Tests

ColumnValuesToBeNotNull

ColumnValuesToBeUnique

ColumnValuesToBeInSet

ColumnValuesToBeNotInSet

ColumnValuesToMatchRegex

ColumnValuesToNotMatchRegex

ColumnValuesToBeBetween

ColumnValueMaxToBeBetween

ColumnValueMinToBeBetween

ColumnValueMeanToBeBetween

ColumnValueMedianToBeBetween

ColumnValueStdDevToBeBetween

ColumnValuesSumToBeBetween

ColumnValuesMissingCount

ColumnValueLengthsToBeBetween

ColumnValuesToBeAtExpectedLocation

Customizing Tests

Next Steps