遇见数据集

A Systematic Review of Tools for AI-Augmented Data Quality Management in Data Warehouses

收藏
Zenodo2025-07-14 更新2026-05-26 收录
官方服务:

资源简介:

As part of the “From Data Quality for AI to AI for Data Quality: A Systematic Review of Tools for AI-Augmented Data Quality Management in Data Warehouses” (Tamm & Nikifovora, 2025), a systematic review of DQ tools was conducted to evaluate their automation capabilities, particularly in detecting and recommending DQ rules in data warehouse - a key component of data ecosystems. To attain this objective, five key research questions were established. Q1. What is the current landscape of DQ tools? Q2. What functionalities do DQ tools offer? Q3. Which data storage systems DQ tools support? and where does the processing of the organization’s data occur? Q4. What methods do DQ tools use for rule detection? Q5. What are the advantages and disadvantages of existing solutions? Candidate DQ tools were identified through a combination of rankings from technology reviewers and academic sources. A Google search was conducted using keyword (“the best data quality tools” OR “the best data quality software” OR “top data quality tools” OR “top data quality software”) AND "2023" (search conducted in December 2023). Additionally, this list was complemented by DQ tools found in academic articles, identified with two queries in Scopus, namely "data quality tool" OR "data quality software" and ("information quality" OR "data quality") AND ("software" OR "tool" OR "application") AND "data quality rule". For selecting DQ tools for further systematic analysis, several exclusion criteria were applied. Tools from sponsored, outdated (pre-2023), non-English, or non-technical sources were excluded. Academic papers were restricted to those published within the last ten years, focusing on the computer science field. This resulted in 151 DQ tools, which are provided in the file "DQ Tools Selection". To structure the review process and facilitate answering the established questions (Q1-Q3), a review protocol was developed, consisting of three sections. The initial tool assessment was based on availability, functionality, and trialability (e.g., open-source, demo version, or free trial). Tools that were discontinued or lacked sufficient information were excluded. The second phase (and protocol section) focused on evaluating the functionalities of the identified tools. Initially, the core DQM functionalities were assessed, such as data profiling, custom DQ rule creation, anomaly detection, data cleansing, report generation, rule detection, data enrichment. Subsequently, additional data management functionalities such as master data management, data lineage, data cataloging, semantic discovery, and integration were considered. The final stage of the review examined the tools' compatibility with data warehouses and General Data Protection Regulation (GDPR) compliance. Tools that did not meet these criteria were excluded. As such, the 3rd section of the protocol evaluated the tool's environment and connectivity features, such as whether it operates in the cloud, hybrid, or on-premises, its API support, input data types (.txt, .csv, .xlsx, .json), and its ability to connect to data sources including relational and non-relational databases, data warehouses, cloud data storages, data lakes. Additionally, it assessed whether the tool processes data on-premises or in the vendor’s cloud environment. Tools were excluded based on criteria such as not supporting data warehouses or processing data externally. These protocols (filled) are available in file "DQ Tools Analysis"

本研究作为《面向AI的数据质量到以AI赋能数据质量:数据仓库中AI增强型数据质量管理工具系统综述》(Tamm & Nikifovora, 2025)的组成部分,针对数据质量(Data Quality, DQ)工具开展了系统综述,旨在评估其自动化能力,尤其是在数据仓库——数据生态系统的核心组件——中的数据质量规则检测与推荐能力。 为达成该研究目标,设定了五项核心研究问题: Q1. 当前数据质量工具的发展格局如何? Q2. 数据质量工具具备哪些功能? Q3. 数据质量工具支持哪些数据存储系统?企业数据的处理场所在何处? Q4. 数据质量工具采用何种方法进行规则检测? Q5. 现有解决方案的优势与劣势分别是什么? 研究人员通过技术评审机构与学术来源的排名组合识别候选数据质量工具。2023年12月通过谷歌(Google)搜索,使用关键词组合:("最佳数据质量工具" OR "最佳数据质量软件" OR "顶级数据质量工具" OR "顶级数据质量软件") AND "2023"。此外,通过Scopus数据库的两条查询补充工具列表:分别为"data quality tool" OR "data quality software",以及("information quality" OR "data quality") AND ("software" OR "tool" OR "application") AND "data quality rule"。 为遴选用于后续系统分析的数据质量工具,研究人员设置了多项排除标准:排除来自赞助、过时(2023年之前)、非英语或非技术来源的工具;学术论文限定为近十年内发表的计算机科学领域文献。最终共遴选出151款数据质量工具,相关信息收录于文件"DQ Tools Selection"(DQ工具遴选清单)。 为梳理综述流程并辅助回答前述研究问题(Q1-Q3),研究人员制定了一份综述协议,包含三个部分。第一阶段为工具初步评估,基于可用性、功能与可试用性(例如开源、演示版或免费试用)进行筛选,停用或信息不足的工具将被排除。第二阶段聚焦于已识别工具的功能评估:首先评估核心数据质量管理(Data Quality Management, DQM)功能,包括数据概要分析、自定义数据质量规则创建、异常检测、数据清洗、报告生成、规则检测与数据增强;随后考量额外的数据管理功能,例如主数据管理、数据血缘、数据编目、语义发现与集成。第三阶段审查工具对数据仓库的兼容性与通用数据保护条例(General Data Protection Regulation, GDPR)合规性,不符合该标准的工具将被排除。该部分协议还评估工具的运行环境与连接特性,例如是否支持云、混合或本地部署,其应用程序编程接口(Application Programming Interface, API)支持情况、输入数据类型(.txt、.csv、.xlsx、.json),以及能否连接至关系型与非关系型数据库、数据仓库、云存储、数据湖等数据源;同时评估工具是在本地还是厂商云环境中处理数据。不符合标准(如不支持数据仓库或在外部处理数据)的工具将被排除。 已完成填写的该综述协议收录于文件"DQ Tools Analysis"(DQ工具分析清单)。

提供机构:
Zenodo
创建时间:
2025-07-14
二维码
社区交流群
二维码
科研交流群
商业服务