​

Profiler Workflow

​

UI configuration

​

1. Add a Profiler Agent

1. Navigate to Settings > Services > Databases.
2. Select the service.
3. Click the Agents tab.
4. Click Add Agent.
5. Select Add Profiler Agent.

​

2. Configure the Profiler Agent

​

Filter Patterns

• Database filter pattern (Optional): Regex to include or exclude databases by name.
• Schema filter pattern (Optional): Regex to include or exclude schemas by name.
• Table filter pattern (Optional): Regex to include or exclude tables by name.
• Classification filter pattern (Optional): Regex to target tables by classification, tag, tier, or glossary term. The profiler computes metrics only for tables whose classification matches the pattern.
• Use FQN For Filtering: Regex patterns apply to the fully qualified name (FQN) instead of the raw asset name when enabled. For example, service_name.db_name.schema_name.table_name. Enable this when different schemas share the same table name across multiple databases.

​

Scope & Behaviour

​

Advanced Config

Sampling Settings

• Profile Sample: Set a fixed sample size for each profiler run.
• Profile Sample Type: Select Percentage for a fraction of the table or Rows for an absolute number of rows.
• Sampling Method Type: For percentage samples on PostgreSQL and Snowflake, select BERNOULLI to sample rows independently or SYSTEM to sample disk pages. Row sampling is more statistically uniform but slower. Page sampling is faster but less statistically uniform.

Execution Settings

• Thread Count: Set the number of threads for parallel metric computation, from 1 to 20. For Snowflake, set this to 1 because of a known issue with snowflake-connector-python. If omitted, the profiler calculates it from the number of metrics.
• Timeout (in sec.): Set how long a profiling job waits before moving to the next table. The default is 43,200 seconds (12 hours). The profiler waits for the current query to finish before timing out. Set a query or connection timeout on the database engine to limit hanging queries.
• Include Views: Enable profiling for views. Including views might reduce profiler performance.

Metric Settings

• Metrics: Add or deselect metrics to control what the profiler computes and reduce profiling time or cost.
• Compute Table Metrics: Compute table-level metrics, such as row count and column count. Enabled by default.
• Compute Column Metrics: Compute column-level metrics, such as null count, distinct count, and mean. Enabled by default.
• Use System Table Statistics: Read available table metrics from system catalogs instead of scanning data. Keep database statistics up to date, for example by running ANALYZE on PostgreSQL. Metrics unavailable in system tables use the default scan method.

​

3. Schedule and Deploy

1. Click Next to open the scheduling step.
2. Select how the profiler runs:
   • Schedule: Set a recurring schedule to run the profiler automatically at a fixed interval.
   • On Demand: Run the profiler manually whenever needed.
3. Click Deploy.

​

4. Updating Profiler setting at the table level

​

Profiler Options

• Percentage: Value must be between 1 and 100. This will sample the table based on a percentage
• Row Count: The table will be sampled based on a number of rows (i.e. 1,000, 2,000), etc.

• Column Name: this is the name of the column that will be used as the partition field
• Interval Type:
  • TIME-UNIT: a business logic timestamp/date/datetime (e.g. order date, sign up datetime, etc.)
  • INGESTION-TIME: a process logic timestamp/date/datetime (i.e. when was my data ingested in my table)
  • COLUMN-VALUE: a value representing a chunk of data (e.g. Product Type A, B, C, etc.)
  • INTEGER-RANGE: a range of integer that will be used as the partition (e.g. Customer ID between 1 and 10)

• Interval: the interval value (e.g. 1, 2, etc.)
• Interval Unit:
  • HOUR
  • DAY
  • MONTH
  • YEAR

• Value: a list of value to use for the partitioning logic

• Start Range: the start of the range (inclusive)
• End Range: the end of the range (inclusive)

​

5. Updating Profiler setting at the platform level

​

YAML Configuration

source:
  type: <connectorType (e.g. bigquery, redshift, etc.)>
  serviceName: <serviceName (same as metadata ingestion service name)>
  sourceConfig:
    config:
      type: Profiler
      computeTableMetrics: true
      computeColumnMetrics: true
      # profileSampleConfig:
      #   sampleConfigType: DYNAMIC
      #   config:
      #     smartSampling: true
      # threadCount: 5
      # timeoutSeconds: 43200
      # databaseFilterPattern:
      #   includes:
      #     - database1
      #     - database2
      #   excludes:
      #     - database3
      #     - database4
      # schemaFilterPattern:
      #   includes:
      #     - schema1
      #     - schema2
      #   excludes:
      #     - schema3
      #     - schema4
      # tableFilterPattern:
      #   includes:
      #     - table1
      #     - table2
      #   excludes:
      #     - table3
      #     - table4
processor:
  type: orm-profiler
  config: {}  # Remove braces if adding properties
    # tableConfig:
    #   - fullyQualifiedName: <table fqn>
    #     profileSampleConfig:
    #       sampleConfigType: STATIC  # omit entirely to inherit dynamic sampling
    #       config:
    #         profileSample: <number between 0 and 99>
    #     columnConfig:
    #       excludeColumns:
    #         - <column name>
    #       includeColumns:
    #         - columnName: <column name>
    #         - metrics:
    #           - MEAN
    #           - MEDIAN
    #           - ...
    #     partitionConfig:
    #       enablePartitioning: <set to true to use partitioning>
    #       partitionColumnName: <partition column name>
    #       partitionIntervalType: <TIME-UNIT, INTEGER-RANGE, INGESTION-TIME, COLUMN-VALUE>
    #       Pick one of the variation shown below
    #       ----'TIME-UNIT' or 'INGESTION-TIME'-------
    #       partitionInterval: <partition interval>
    #       partitionIntervalUnit: <YEAR, MONTH, DAY, HOUR>
    #       ------------'INTEGER-RANGE'---------------
    #       partitionIntegerRangeStart: <integer>
    #       partitionIntegerRangeEnd: <integer>
    #       -----------'COLUMN-VALUE'----------------
    #       partitionValues:
    #         - <value>
    #         - <value>
sink:
  type: metadata-rest
  config: {}
workflowConfig:
  # loggerLevel: DEBUG  # DEBUG, INFO, WARN or ERROR
  openMetadataServerConfig:
    hostPort: <OpenMetadata host and port>
    authProvider: <OpenMetadata auth provider>

​

2. Run with the CLI

metadata profile -c <path-to-yaml>

​

Profiler Best Practices

​

1. Profile what you Need

​

2. Sampling and Partitioning your Tables

​

3. Excluding/Including Specific Columns/Metrics

​

4. Set Up Multiple Workflow