DCA-Bench

该数据集名为DCA-Bench，旨在评估大型语言模型代理在识别隐藏数据集质量问题方面的能力。它包含了来自八个开放数据集平台的多样化真实世界数据集质量问题。数据集根据涉及的文件数量和问题数量进行分类，包含18个标签以便归类，如“数据问题”、“文档问题”、“基础设施问题”以及“伦理/法律风险”。该数据集规模为收集到的91个数据集问题，其任务是检测隐藏的数据集质量问题。The dataset named DCA-Bench is designed to evaluate the capability of Large Language Model (LLM) Agents in identifying hidden dataset quality issues. It encompasses a diverse range of real-world dataset quality problems sourced from eight open dataset platforms. The dataset is categorized based on the number of involved files and the count of issues, with 18 labels for classification, including "Data Issues", "Documentation Issues", "Infrastructure Issues", and "Ethical/Legal Risks". Comprising a total of 91 collected dataset issues, the core task of this benchmark is to detect hidden dataset quality issues.

搜集汇总

背景与挑战

背景概述

DCA-Bench是一个用于评估LLM代理在在线数据集平台上检测数据质量问题的基准测试数据集，包含221个真实案例，覆盖8个平台，并分为4种类型和18个标签。该数据集提供多级提示和基于GPT-4的自动评估方案，旨在测试代理在复杂环境中的问题发现能力，支持用户贡献新数据点。