zarnite/zarn-workspace-rag-qa

--- language: - en license: apache-2.0 annotations_creators: - expert-generated - machine-generated language_creators: - expert-generated - machine-generated source_datasets: - original task_categories: - question-answering - text-generation tags: - zarnite - benchmark - rag - grounded-qa - retrieval - gold-track - benchmark-starter pretty_name: Zarn Workspace RAG QA size_categories: - 10K<n<100K configs: - config_name: default data_files: - split: train path: data/train.jsonl - split: validation path: data/validation.jsonl - split: test path: data/test.jsonl --- # Zarn Workspace RAG QA ## Dataset Description Small document bundles paired with grounded answers, evidence, and explicit refusals when context is missing. ## Team Attribution This dataset was created and reviewed by the Zarnite team through internal benchmark design, generation, and quality-control workflows. It should be presented as a Zarnite-authored benchmark starter pack, not as a purely human-collected field corpus. ## Ecosystem Need Tier High Ecosystem Need ## Why This Category Is Attractive RAG systems fail most often on groundedness and abstention, so richer retrieval benchmarks with traps and known gaps are highly useful. ## Benchmark Goal Evaluate grounded answering, citation precision, answerability judgment, and refusal quality under partial workspace context. ## Included In This Folder - `data/train.jsonl`, `data/validation.jsonl`, `data/test.jsonl`: starter benchmark splits with 1200 total rows. - `schema.json`: JSON Schema for row validation. - `benchmark_spec.json`: metrics, quality gates, and target release scale. - `LICENSE.md`: folder-local license notice for self-contained publishing. - `PUBLISHING.md`: repo-specific publish instructions for Hugging Face. - `hf_repo_template.json`: machine-readable repo template used by the uploader script. ## Target Public Scale - Train: 24,000 - Validation: 3,000 - Test: 3,000 - Total target rows: 30,000 ## Recommended Metrics - `answer_and_citation_f1` - `answerability_accuracy` - `unsupported_claim_rate` - `citation_precision` - `groundedness` ## Gold-Track Benchmark Assets - `ANNOTATION_GUIDELINES.md`: how to expand rows without drifting from the benchmark purpose. - `REVIEW_PROTOCOL.md`: how to audit validation and test rows with dual review and adjudication. - `BASELINE_EVAL_SPEC.json`: expected output contract, slice reporting, and release thresholds. - `RELEASE_CHECKLIST.md`: final pre-publish checks for the public Hugging Face release. - `SCORING_PROFILE.json`: prediction keys, scoring expectations, and slice reporting requirements. - `prediction_template.jsonl`: starter template for benchmark submissions or baseline runs. ## Expanded Row Anatomy - `knowledge_bundle`: multiple documents with sections, partial authority, and known gaps. - `query_context`: who is asking and what type of retrieval task this is. - `unsupported_claim_traps`: tempting details the model must not invent. - `answerability`: whether the question should be answered or refused. - `difficulty_rationale`: why the row belongs in its difficulty bucket instead of a weaker slice. - `benchmark_slices`: named reporting slices such as approval friction, proof preservation, or citation traps. - `adversarial_features`, `expected_failure_modes`, and `review_readiness`: what the row is testing and how a gold-track reviewer should treat it. - `evidence_manifest`, `reference_variants`, and `negative_examples`: the source evidence boundary, acceptable alternate answers, and concrete failure cases. ## Hugging Face Deployment This folder is self-contained and can be uploaded as its own Hugging Face dataset repository. - Suggested repo id: `zarnite/zarn-workspace-rag-qa` - Example upload command: `python upload_to_huggingface.py --dataset-folder "push/high-ecosystem-need/Zarn-Workspace-RAG-QA" --repo-id "zarnite/zarn-workspace-rag-qa"` - You can swap the namespace by passing `--namespace YOUR_USERNAME` to the uploader. ## Local Evaluation - Example eval command: `python run_priority_eval.py --dataset-folder "push/high-ecosystem-need/Zarn-Workspace-RAG-QA" --splits validation test` - `prediction_template.jsonl` gives the required output shape for local or leaderboard-style submissions. ## License This package is marked `apache-2.0`. The rows in this folder are original starter examples for benchmark packaging.### 数据集元数据 - 语言：英语 - 许可证：Apache-2.0 - 注释创建者：专家生成、机器生成 - 语言创建者：专家生成、机器生成 - 源数据集：原创 - 任务类别：问答、文本生成 - 标签：zarnite、基准测试（benchmark）、检索增强生成（Retrieval-Augmented Generation，RAG）、基于上下文的问答（grounded QA）、检索（retrieval）、金标准赛道（gold-track）、基准测试入门包（benchmark-starter） - 友好名称：Zarn工作区检索增强生成问答（Zarn Workspace RAG QA） - 规模类别：10000 < 数据量 < 100000 - 配置： - 配置名称：default（默认配置） 数据文件： - 训练集（train）：data/train.jsonl - 验证集（validation）：data/validation.jsonl - 测试集（test）：data/test.jsonl # Zarn工作区检索增强生成问答（Zarn Workspace RAG QA） ## 数据集说明 本数据集包含小型文档集，搭配基于上下文的答案（grounded answers）、佐证依据，以及当上下文缺失时的明确拒答示例。 ## 团队归属 本数据集由Zarnite团队通过内部基准测试设计、数据生成与质量管控流程完成创建与审核，应作为Zarnite出品的基准测试入门套件进行展示，而非单纯的人工采集的真实语料库。 ## 生态需求层级 高生态需求层级 ## 该类数据集的价值所在 检索增强生成（Retrieval-Augmented Generation，RAG）系统最常于上下文一致性与拒答准确性环节出现失误，因此包含陷阱与已知缺失上下文的高质量检索基准测试集具备极高应用价值。 ## 基准测试目标 评估在部分工作区上下文条件下的基于上下文作答能力、引用精度、可回答性判断以及拒答质量。 ## 本文件夹包含内容 - `data/train.jsonl`、`data/validation.jsonl`、`data/test.jsonl`：基准测试入门拆分集，总计1200条数据。 - `schema.json`：用于数据行校验的JSON Schema文件。 - `benchmark_spec.json`：包含评估指标、质量门槛与目标发布规模的配置文件。 - `LICENSE.md`：本文件夹独立发布所需的本地许可证声明。 - `PUBLISHING.md`：针对Hugging Face平台的专属仓库发布指南。 - `hf_repo_template.json`：上传脚本所用的机器可读格式仓库模板。 ## 公开发布目标规模 - 训练集：24000条 - 验证集：3000条 - 测试集：3000条 - 总目标数据行数：30000条 ## 推荐评估指标 - `答案与引用F1值（answer_and_citation_f1）` - `可回答性准确率（answerability_accuracy）` - `无依据声明率（unsupported_claim_rate）` - `引用精度（citation_precision）` - `上下文一致性（groundedness）` ## 金标准赛道基准测试资源 - `ANNOTATION_GUIDELINES.md`：如何在不偏离基准测试目标的前提下扩展数据行的指南。 - `REVIEW_PROTOCOL.md`：如何通过双审与仲裁机制对验证集与测试集数据行进行审核的规程。 - `BASELINE_EVAL_SPEC.json`：包含预期输出规范、分维度报告与发布阈值的基线评估配置文件。 - `RELEASE_CHECKLIST.md`：面向Hugging Face公开发布的最终预发布检查清单。 - `SCORING_PROFILE.json`：包含预测字段规范、评分预期与分维度报告要求的评分配置文件。 - `prediction_template.jsonl`：基准测试提交或基线运行所需的入门模板文件。 ## 扩展数据行结构说明 - `knowledge_bundle`：包含多个带章节、部分权限标记与已知上下文缺失的文档集合。 - `query_context`：查询发起方信息与该任务所属的检索类型。 - `unsupported_claim_traps`：模型不得编造的误导性细节陷阱。 - `answerability`：判断问题是否应当被作答或拒答。 - `difficulty_rationale`：该数据行所属难度层级的判定依据，以及为何不属于更低难度维度的理由。 - `benchmark_slices`：命名化的报告维度，例如审批摩擦、佐证留存或引用陷阱等。 - `adversarial_features`、`expected_failure_modes`与`review_readiness`：该数据行的测试目标，以及金标准赛道审核人员的处理方式。 - `evidence_manifest`、`reference_variants`与`negative_examples`：分别对应佐证依据边界、可接受的替代答案以及具体的失效案例。 ## Hugging Face平台部署 本文件夹为独立完整的包，可直接作为专属Hugging Face数据集仓库上传。 - 推荐仓库ID：`zarnite/zarn-workspace-rag-qa` - 示例上传命令：`python upload_to_huggingface.py --dataset-folder "push/high-ecosystem-need/Zarn-Workspace-RAG-QA" --repo-id "zarnite/zarn-workspace-rag-qa"` - 可通过向上传脚本传入`--namespace YOUR_USERNAME`参数替换命名空间。 ## 本地评估 - 示例评估命令：`python run_priority_eval.py --dataset-folder "push/high-ecosystem-need/Zarn-Workspace-RAG-QA" --splits validation test` - `prediction_template.jsonl`提供了本地评估或排行榜式提交所需的输出格式规范。 ## 许可证 本套件采用`Apache-2.0`许可证。本文件夹内的数据行均为基准测试打包所用的原创入门示例。