DATA VALIDATOR

Data Validator - Feature Summary

The 'Data Validator' is a solution designed to ensure high data quality across source and destination systems through a flexible, rule-driven approach. It empowers teams to define, execute, monitor, and audit various types of data validation checks seamlessly.

**Key Features**

Test Types Supported:

- Row Count Check: Validates if row counts between source and target tables are within a ±5% tolerance.
- Grain Check: Ensures the uniqueness of primary keys across both source and destination tables.
- KPI Check: Compares the results of custom KPI SQL logic or aggregated column values between the source and destination. Passes if the difference is within ±5%.

Scheduling Support:

- Test cases are scheduled using Snowflake tasks for automatic execution.
- Offers full CRON-style configuration with time zone support.
- Automatically log results, including comparison values, pass/fail status, and metadata for analysis in dashboards or the "Show Test Cases" view.

User Interface Capabilities:

- Add Test Case: Intuitive interface to define test parameters (e.g., databases, tables, columns, SQL queries).
- Show Test Cases: Displays existing test cases based on selected filters like table type and test type.
- Execute Test Case: On-demand Execution of test cases.
- View Test Results: Offers detailed visibility into execution status, timestamps, result IDs, and pass/fail outcomes, enabling historical trend analysis.
- Schedule Test Cases: Allows to setup a customized schedule to run the checks automatically.

搜集汇总

背景与挑战

背景概述

该数据集提供了一套灵活的数据验证解决方案，支持行数核对、主键唯一性检查和KPI比对等多种验证类型，可通过自动化调度执行并实时监控结果。系统配备直观的用户界面，支持测试案例的创建、执行、结果查看及定时任务设置。