DQ-Data Quality APP

DQ - Data Quality APP
Implementing Data Quality checks are essential part of designing Data Pipelines. Data Quality ensures dataset meets criteria for accuracy, completeness, validity, consistency, uniqueness, timeliness, and fitness for purpose, and it is critical to all data governance initiatives within an organization. This APP uses Python’s library for Data Quality Validation and Snowflake’s SnowPark API for data frames. Once APP gets onboarded into Snowflake account, Data Quality Checks can be implemented by calling stored procedures with relevant parameters. Below is the list of objects used in this framework. Installing APP on Snowflake account creates Tables, Sequences & Stored Procs required for this framework to function.

This App provides you the capability to use inbuilt data quality checks and add your own rule as per requirement without much efforts.
Data Quality app provides the following capabilities:
- Add data quality rules
- Data quality results with failed records in details

***** SEQUENCE *****

- GE_FUNCTION_RULE_MAP_SEQ
- DQ_VIOLATIONS_SEQ
- DQ_RULES_SEQ
- PROCESS_METADATA_SEQ

***** TABLES ******

- DATASETS - Stores Table level information.
- ATTRIBUTE - Stores Column Level information.
- DQ_RULES - Stores Data Quality Rules Information.
- GE_FUNCTION_RULE_MAP - Functions & Data Quality Rule Mapping information.
- DQ_VIOLATIONS - Stores Failed Records in JSON Format
- SENSITIVE_ATTRIBUTES - Stores Attributes to be identified as PI & PII information.
- PROCESS_METADATA - Stores Process Metadata information

***** STORED PROCEDURE ******

- CREATE_METADATA(DB VARCHAR, SCHEMA VARCHAR, TBL_NAME VARCHAR);

- CREATE_DQ_RULE(TABLE_NAME VARCHAR,COLUMN_NAME VARCHAR,RULE_ID VARCHAR,EXPRESSION VARCHAR);

- CREATE PROCESS_METADATA(DB VARCHAR,SCHEMA VARCHAR,TBL_NAME VARCHAR);

For full steps, refer sample worksheet.

搜集汇总

背景与挑战

背景概述

DQ-Data Quality APP是一个基于Python和Snowflake SnowPark API的数据质量验证工具，提供内置检查规则和自定义规则功能，能记录详细的质量检查结果和失败记录。该应用在Snowflake账户中创建必要的表、序列和存储过程来支持数据质量框架的运行。