CentificAIResearch/BA-Agent-Bench

--- license: cc-by-4.0 language: - en size_categories: - n<1K task_categories: - text-generation - question-answering tags: - business-analysis - requirements-engineering - user-stories - benchmark - llm-evaluation - agentic-rl configs: - config_name: default data_files: train.parquet --- # BA-Agent-Bench (Subset) **BA Agent Bench** — purpose-built benchmark for evaluating **LLMs and multi-agent systems on enterprise requirements generation capability**. Measures whether a model or agent can take real production work-item specs and produce BA-grade user stories with acceptance criteria, matching the decomposition and rigor of certified Business Analysts. The full benchmark evaluates 8 systems (7 frontier LLMs + Centific BA Toolkit pipeline) on 7 enterprise features against 119 ground-truth user stories authored by certified Business Analysts. Composite score across **Alignment (35%) · Coherence (24%) · Completeness (18%) · Compliance (10%) · Testability (9%) · Spec Quality (4%)**. This dataset card is an **8-feature stratified subset** of the benchmark — sampled for public release so practitioners can inspect inputs and gold standard. Full benchmark, leaderboard, and harness → **[centific.com/benchmark/agentic-rl/1#start](https://www.centific.com/benchmark/agentic-rl/1#start)**. --- ## Why BA-Agent-Bench? Existing BA / requirements-engineering benchmarks rely on toy specs or single-shot story generation. Real BA work has: - Multi-document source specs (main requirement + KB / reference docs) - Domain-rich decomposition: per-entity, per-workflow, per-stage stories - BA-authored gold standard with full acceptance criteria - 7 quality dimensions (alignment, coherence, completeness, testability, specification quality, trust, compliance) BA-Agent-Bench targets all of the above on **production work items from an enterprise import/export logistics platform**. --- ## Subset Composition This subset contains **8 features** covering a range of domains, document counts, and decomposition complexity: | task_id | docs | stories | domain | |---|---|---|---| | T-001 | 4 | 27 | Customs Client ID processing | | T-002 | 4 | 13 | Commencement configuration | | T-003 | 4 | 12 | Brokerage Cargo Reports | | T-004 | 4 | 17 | Cargo Reporting & Declaration | | T-005 | 4 | 18 | TIN / OEM Client ID re-issue | | T-006 | 4 | 12 | Reference File Processing | | T-007 | 4 | 20 | OEM Client ID app improvement | | T-008 | 3 | 8 | Shipment-level Underbond | **Total:** 31 source documents · 127 golden stories · 8 distinct domains --- ## Schema Each row has the following columns: | Column | Type | Description | |---|---|---| | `task_id` | str | Original work-item identifier | | `title` | str | Feature title | | `description` | str | Feature description, scope, and business value | | `input_documents` | list[dict] | Source requirement docs: `{filename, content}` | | `golden_stories` | list[dict] | BA-authored user stories with acceptance criteria | ### `golden_stories` row | Field | Type | Description | |---|---|---| | `story_id` | str | Story identifier within feature | | `title` | str | Short story title | | `description` | str | Full "As a / I want / So that" narrative | | `acceptance_criteria` | str \| list[str] | Given/When/Then or free-text AC | | `story_points` | int | Effort estimate | | `state` | str | Workflow state when story was authored | --- ## How to Load ```python from datasets import load_dataset ds = load_dataset("CentificAIResearch/BA-Agent-Bench") print(ds["train"][0]) ``` Or read parquet directly: ```python import pandas as pd df = pd.read_parquet("hf://datasets/CentificAIResearch/BA-Agent-Bench/train.parquet") print(df.columns.tolist()) for _, row in df.iterrows(): print(row["task_id"], len(row["golden_stories"])) ``` --- ## Full Dataset & Related Resources This subset is drawn from a larger benchmark suite covering 26+ enterprise features with full evaluation infrastructure (LLM-judge metrics, multi-model leaderboard, automated scoring harness). - **Benchmark portal:** [centific.com/benchmark/agentic-rl/1#start](https://www.centific.com/benchmark/agentic-rl/1#start) --- ## License CC-BY-4.0 — free to use with attribution. ## Citation ```bibtex @dataset{ba_agent_bench_2026, title = {{BA-Agent-Bench}: Benchmark for {BA}-grade Story Generation and Multi-agent {BA} Pipelines}, author = {Centific AI Research}, year = {2026}, publisher = {HuggingFace}, url = {https://huggingface.co/datasets/CentificAIResearch/BA-Agent-Bench} } ```BA-Agent-Bench (Subset) is a purpose-built benchmark for evaluating LLMs and multi-agent systems on enterprise requirements generation capability. It measures whether a model or agent can take real production work-item specs and produce BA-grade user stories with acceptance criteria, matching the decomposition and rigor of certified Business Analysts. The subset contains 8 features covering domains such as Customs Client ID processing, Cargo Reporting & Declaration, and Reference File Processing, with a total of 31 source documents and 127 golden stories authored by certified Business Analysts. Each row includes task_id, title, description, input_documents (a list of source requirement documents), and golden_stories (a list of user stories with fields like story_id, title, description, acceptance_criteria, story_points, and state). This subset is sampled from a larger benchmark suite for public release, allowing practitioners to inspect inputs and gold standards. It is designed for NLP tasks like text-generation and question-answering, supporting enterprise business analysis, requirements engineering, and agent evaluation.

搜集汇总

数据集介绍

构建方式

BA-Agent-Bench数据集专为评估大型语言模型及多智能体系统在业务需求生成方面的能力而构建。从企业级进出口物流平台的实际生产工作项中提取了8个特征领域，涵盖海关客户ID处理、货物报告与申报、参考文件处理等多样化的业务场景。每个特征领域均配有多份源需求文档，共计31份文档，并经由认证业务分析师手工编写了127条黄金标准用户故事及验收标准。构建过程严格遵循业务分析的专业分解逻辑，确保每个特征都能映射到实体、工作流及阶段级别的细粒度故事，形成了高质量、高复杂度的基准测试数据集。

特点

该数据集的核心特点在于其专业性与真实性。不同于依赖简单规格或单次故事生成的现有基准，BA-Agent-Bench基于真实的多文档源规格，要求模型或代理从主需求与参考文档中提取信息并进行领域丰富的分解。评估体系涵盖7个质量维度，包括对齐度、连贯性、完整性、可测试性、规范质量、可信度与合规性，并分配了差异化的权重。此外，数据集中包含了认证业务分析师编写的黄金标准作为参考，使得评估不仅关注生成结果的数量，更注重其与专业分析师的匹配程度，从而全面衡量模型在复杂企业需求工程任务上的表现。

使用方法

用户可以通过HuggingFace datasets库直接加载数据集，示例如下：`from datasets import load_dataset; ds = load_dataset("CentificAIResearch/BA-Agent-Bench")`，即可访问训练集数据。每条记录包含任务标识符、特征标题与描述、输入文档列表以及黄金用户故事列表。此外，用户也可以直接读取Parquet文件，使用Pandas库进行处理，便于进一步的数据探索与分析。对于寻求完整评估方案的实践者，官方提供了包含所有评分指标、排行榜及自动化测试框架的全量基准测试平台，可通过数据集卡片中的链接访问，从而实现模型在真实企业环境下的全面验证与对比。

背景与挑战

背景概述

BA-Agent-Bench是由Centific AI Research于2026年创建的基准测试数据集，专注于评估大型语言模型（LLM）及多智能体系统在企业需求生成中的能力。其核心研究问题在于，现有需求工程基准通常基于简化的模拟场景或单次故事生成，无法反映真实业务分析师（BA）工作的复杂性。该数据集以企业进出口物流平台的生产级工作项为蓝本，涵盖了7个质量维度，包括对齐度、连贯性、完整性、可测试性、规范质量、信任度和合规性，并提供了经认证BA撰写的119条黄金标准用户故事。通过引入多文档源规范、领域丰富的分解任务以及8个不同领域的特征子集，BA-Agent-Bench填补了从学术玩具样例到工业级质量评估的空白，为LLM在需求工程领域的实际应用确立了新的评价标杆。

当前挑战

BA-Agent-Bench主要应对两大挑战。在领域问题层面，传统需求生成评估依赖简易规范或单轮故事编写，忽略了多文档交叉引用、跨实体与跨工作流的细致分解、以及涉及合规性、测试性等深层质量维度的真实BA工作流程，导致模型难以胜任企业级任务。在构建过程中，挑战包括从企业物流平台收集31份原始生产文档、协同认证BA撰写出127条高保真黄金故事，并在8个异构领域内保持一致的分解粒度与质量标注标准。此外，设计覆盖7个维度的复合评分机制（如对齐度占35%、连贯性占24%等）需要精细的度量校准，以平衡不同维度对整体性能的真实反映。

常用场景

经典使用场景

在需求工程与业务分析的交叉领域，BA-Agent-Bench数据集专为评估大语言模型及多智能体系统在企业级需求生成任务中的表现而设计。其经典使用场景聚焦于衡量模型能否基于真实生产环境中的工作项规格说明，产出堪比认证业务分析师水准的用户故事与验收标准。该数据集包含来自企业进出口物流平台的8个功能特性、31份源文档及127条由专业业务分析师撰写的黄金标准用户故事，覆盖多文档源规格分解、领域密集型拆分及七维质量评估框架，为自动化需求生成提供了严谨的标准化测试基准。

实际应用

在实际企业应用场景中，BA-Agent-Bench为需求工程自动化工具的选型与优化提供了可量化的评估手段。产品团队可借助该基准测试不同大语言模型或多智能体系统在生成符合业务分析师标准的用户故事方面的表现，从而筛选出最适合集成到企业需求管理流程中的AI方案。该数据集也服务于企业数字化转型中需求阶段的质量控制，通过自动评估生成的用户故事与黄金标准之间的对齐程度，降低因需求理解偏差导致的开发返工成本，提升从业务意图到技术实现的转化效率。

衍生相关工作

BA-Agent-Bench的问世催生了一系列富有启发性的衍生研究工作。围绕该基准，研究者已开发出基于LLM评判的自动化评分框架和跨模型排行榜，支持对8个前沿大语言模型及Centific BA Toolkit多智能体管线的系统化比较。这些工作进一步推动了面向需求工程的多智能体强化学习研究，探索了将业务分析师专业知识编码为奖励信号、训练智能体自主生成高质量用户故事的技术路线。此外，该基准也为用户故事自动生成、验收标准规范化以及需求文档质量评估等子任务提供了可复现的对比基线，促进了需求工程领域实验评估的标准化进程。