Skip to main content

​
Auto PII Tagging

Auto PII tagging for Sensitive/NonSensitive at the column level is performed based on the two approaches described below.

​
Tagging logic

  1. Column Name Scanner: We validate the column names of the table against a set of regex rules that help us identify common English patterns to identify email addresses, SSN, bank accounts, etc.
  2. Entity Recognition: The workflow samples rows directly from the source and validates them against an Entity Recognition engine that identifies sensitive information from a list of supported entities. The confidence parameter lets you tune the minimum score required to tag a column as PII.Sensitive. Enabling Store Sample Data saves those rows in OpenMetadata, but is not required for classification to run.
Note that if a column is already tagged as PII, we will ignore its execution.

​
Troubleshooting

​
Recognizer ran but tag was not applied

If the auto-classification logs show a recognizer attempted to classify a column but no tag appears:
  • Sampled values are masked or anonymized — the content recognizer found no match above the confidence threshold. Switch the recognizer Target to Column Name to tag based on the column name alone, regardless of data content.
  • Confidence score too low — the recognizer ran but scored below the threshold. Lower the confidence threshold or add context to boost the score.
  • Column already tagged as PII — columns with an existing PII tag are skipped. Check the column’s existing tags.

​
SSL: CERTIFICATE_VERIFY_FAILED

If you see an error similar to:
This is a scenario that we identified on some corporate Windows laptops. The bottom-line here is that the profiler is trying to download the Entity Recognition model but having certificate issues when trying the request. A solution here is to manually download the model on the ingestion container / Airflow host by running:
If using Docker, you might want to customize the OpenMetadata-ingestion image to have this command run there by default.