evgenypal/k8s-docs-rag-bench

---
language:
- en
license: cc-by-4.0
size_categories:
- 1K<n<10K
task_categories:
- question-answering
- text-retrieval
task_ids:
- extractive-qa
- open-domain-qa
- document-retrieval
pretty_name: k8s-docs-rag-bench
tags:
- kubernetes
- rag
- retrieval-augmented-generation
- benchmark
- technical-documentation
- lora
- llm-as-a-judge
configs:
- config_name: qa
data_files:
- split: train
path: qa_train.jsonl
- split: eval
path: qa_eval.jsonl
- split: test
path: qa_test.jsonl
- config_name: corpus
data_files:
- split: train
path: corpus.jsonl
- config_name: judge_labels
data_files:
- split: train
path: judge_labels.jsonl
---

# k8s-docs-rag-bench

- **Paper:** [Analyzing Quality--Latency--Resource Trade-offs in a Technical Documentation RAG Assistant Using LoRA Adaptation](https://huggingface.co/papers/2605.28222) (arXiv:[2605.28222](https://arxiv.org/abs/2605.28222))
- **Code:** [github.com/EugPal/rag-lora-tradeoffs](https://github.com/EugPal/rag-lora-tradeoffs)

A small, fully-grounded benchmark for retrieval-augmented question answering
(RAG) over the official **Kubernetes documentation**, together with the full
set of LLM-judge labels used in the accompanying preprint
*"Analyzing Quality-Latency-Resource Trade-offs in a Technical Documentation
RAG Assistant Using LoRA Adaptation"*.

The benchmark is intended for:

- testing dense / sparse / hybrid retrieval over real, multi-page technical
documentation;
- training small open-weight LLMs (e.g., Llama-3.x) on doc-grounded QA via
LoRA;
- reproducing the paper's pipeline-ablation and LoRA-rank studies.

---

## Configurations

| Config | Rows | Description |
|----------------|---------|----------------------------------------------------------|
| `qa` | 5,144 | Question / extractive-answer pairs (train + eval + test).|
| `corpus` | 7,908 | Semantically chunked corpus over the K8s `/docs/` tree. |
| `judge_labels` | 172,700 | LLM-judge labels (10 regimes x 22 systems x 785 test Q). |

Total on-disk size: ~128 MB (uncompressed JSONL).

### Splits inside `qa`

| Split | Rows | Pages covered |
|-------|-------|---------------|
| train | 3,614 | 578 |
| eval | 745 | 131 |
| test | 785 | 153 |

The split is **page-level**: every Kubernetes documentation page lives in
exactly one of the three splits, so a question in `test` is grounded in a
page that has never been seen in `train` or `eval`. This gives a realistic
estimate of out-of-page generalization.

---

## Data fields

### `qa_{train,eval,test}.jsonl`

```json
{
"id": "kubernetes-test-manualv2-0001",
"question": "What does Kubernetes facilitate?",
"answer": "It facilitates both declarative configuration and automation.",
"answer_mode": "normal",
"context_policy": "retriever_only",
"source_chunk": "concepts-0",
"source_page": "concepts",
"page_kind": "concept",
"provenance": "manual"
}
```

| Field | Type | Notes |
|------------------|--------|------------------------------------------------------------------------------|
| `id` | string | Stable per-row identifier. |
| `question` | string | Natural-language question; English. |
| `answer` | string | Short extractive reference answer. |
| `answer_mode` | string | Always `normal` (single short answer). |
| `context_policy` | string | Always `retriever_only` -- the question is answerable from the corpus alone. |
| `source_chunk` | string | ID of the corpus chunk that supports the answer. |
| `source_page` | string | ID of the K8s docs page the chunk came from. |
| `page_kind` | string | `concept` / `task` / `tutorial` / `reference` / etc. |
| `provenance` | string | Always `manual` -- every QA pair was hand-checked. |

### `corpus.jsonl`

```json
{
"id": "concepts-0",
"page_id": "concepts",
"url": "https://kubernetes.io/docs/concepts/",
"title": "Concepts",
"section": "concepts",
"text": "Kubernetes is an open source ...",
"chunk_index": 0,
"page_kind": "concept",
"section_anchor": "Introduction"
}
```

7,908 semantic chunks built with the recipe described in the paper (semantic
splitter, target ~256 tokens, headed by the nearest section anchor). The
chunk text is the only thing fed to the retriever.

### `judge_labels.jsonl`

Each row records the LLM-judge verdict for a single (regime, system,
test-question) triple. There are **10 retrieval regimes** x **22 generator
configurations** x **785 test questions** = **172,700** rows.

```json
{
"regime": "01_base__neutral",
"example_id": "kubernetes-test-manualv2-0947",
"config": "8B r32 qv_only",
"base_model_name_or_path": "meta-llama/Llama-3.1-8B-Instruct",
"rank": 32,
"target_mode": "qv_only",
"prediction": "References to a set of network endpoints.",
"correctness": 4,
"groundedness": 5,
"evidence": "an EndpointSlice contains references to a set of network endpoints",
"rationale": "The answer matches the core statement ...",
"prompt_version": "groundedness_v1_raw",
"judge_model": "gpt-5.4-mini"
}
```

- `regime` enumerates the 10 pipeline ablations (base / no reranker / dense
only / sparse only / hybrid with classical BM25), each in `neutral` and
`explicit_grounded` prompt variants.
- `config` enumerates the 22 generators -- two baselines (`3B baseline`, `8B
baseline`) plus 5 LoRA ranks `r in {4, 8, 16, 32, 64}` x 2 target-module
sets `target_mode in {qv_only, full_attention}` on each base model.
- `correctness` and `groundedness` are integer judge scores on a 1-5 scale.
- `evidence` quotes the supporting passage the judge anchored on.

---

## How to load

```python
from datasets import load_dataset

qa = load_dataset("evgenypal/k8s-docs-rag-bench", "qa")
corpus = load_dataset("evgenypal/k8s-docs-rag-bench", "corpus", split="train")
labels = load_dataset("evgenypal/k8s-docs-rag-bench", "judge_labels", split="train")

print(qa)
# DatasetDict({
# train: Dataset(num_rows=3614, ...),
# eval: Dataset(num_rows=745, ...),
# test: Dataset(num_rows=785, ...),
# })
```

A minimal RAG pipeline that reproduces the paper's main regime:

```python
# pseudo-code
chunks = list(corpus)
dense = bge_m3_dense(chunks)
sparse = bge_m3_sparse(chunks)

for row in qa["test"]:
hits = rrf(dense.search(row["question"]), sparse.search(row["question"]))
top_k = bge_reranker_v2_m3(row["question"], hits)[:5]
answer = llm.answer(row["question"], context=top_k)
```

---

## Source and version

- **Source**: <https://kubernetes.io/docs/> (official Kubernetes
documentation), full crawl of the `/docs/` tree.
- **Snapshot date**: 2026-02-04.
- **Pages crawled**: 956 HTML pages.
- **Software version**: at the time of the snapshot the docs reflect
Kubernetes v1.34 / v1.35 (the documentation site shows the latest release).
- The corpus contains *prose only* (headings, paragraphs, lists, inline code,
fenced code blocks). No images.

The benchmark questions were authored manually by the paper authors against
this exact snapshot. Every QA pair is grounded in a single chunk
(`source_chunk`) of a single page (`source_page`) of the corpus.

---

## License

Released under **CC-BY-4.0**. See `LICENSE`.

The original Kubernetes documentation is itself licensed under
**CC-BY-4.0** (c) The Kubernetes Authors. This dataset is a derivative work
of that documentation: the `corpus` configuration contains chunked extracts
of K8s docs prose; the `qa` configuration contains questions and answers
authored by us about that prose.

When you use this dataset, please attribute both:

- *The Kubernetes Authors* -- for the underlying documentation.
- The accompanying paper (citation below) -- for the QA pairs and judge
labels.

---

## Citation

If you use this dataset, please cite the accompanying preprint:

```bibtex
@misc{palnikov2026rag,
title = {Analyzing Quality--Latency--Resource Trade-offs in a Technical
Documentation RAG Assistant Using LoRA Adaptation},
author = {Palnikov, Evgenii and Gavrilova, Elizaveta},
year = {2026},
eprint = {2605.28222},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2605.28222}
}
```

---

## Limitations and known biases

- **Single domain.** All questions are about Kubernetes. Conclusions about
retrieval and LoRA effects may not transfer to other technical-doc
domains.
- **Single language.** English only.
- **LLM-judge labels.** `judge_labels.jsonl` is the output of `gpt-5.4-mini`
acting as judge. It correlates with, but is not identical to, human
judgment. We release the raw quoted `evidence` and `rationale` per row so
downstream consumers can re-judge with a different model if they want.
- **Sparse channel.** The main pipeline uses BGE-M3's native sparse channel
(not classical BM25); BM25 is included only as an ablation regime
(`09_hybrid_bm25__neutral`, `10_hybrid_bm25__explicit_grounded`).

---

## Contact

For questions, issues, or corrections, please open an issue on the
HuggingFace dataset repository or contact the paper authors.

搜集汇总

数据集介绍

构建方式

k8s-docs-rag-bench数据集基于Kubernetes官方文档构建，旨在评估检索增强生成（RAG）管道的性能。数据集的构建过程始于对Kubernetes文档站点/docs/目录下956个HTML页面的完整爬取，抓取时间点为2026年2月4日，对应Kubernetes v1.34/v1.35版本。原始文档中的标题、段落、列表、内联代码和围栏代码块等散文内容被保留，而图像则被排除。随后，这些文档内容经过语义分块处理，生成约7,908个语义块，每个块的目标长度为256个token，并附带了最近的章节锚点作为标题。在此基础上，研究者手工编写了5,144个问答对，每个问答对都严格对应到一个特定块和一个特定页面，并通过手工检查确保答案的准确性。数据集采用页面级分割，将每个文档页面完整地分配到训练集、验证集或测试集中，从而避免了跨页面的信息泄露，确保了对未见页面泛化能力的真实评估。此外，数据集还包含172,700行由GPT-5.4-mini作为评判模型生成的标签数据，涵盖了10种检索策略与22种生成器配置的组合。

特点

该数据集具备多项鲜明的特性。首先，它聚焦于单一技术领域——Kubernetes，这虽然限制了结论的普适性，但确保了评估环境的专业性和一致性。其次，数据集的构建高度精良：问答对均源自手工编写，确保了语义的准确性与引用链条的完整性；页面级分割策略在同类基准中较少采用，能够更真实地反映检索系统对全新文档的泛化能力。第三，数据集支持多维度实验设计：包含训练集、验证集和测试集，适用于检索器微调与LoRA参数高效微调等任务。此外，数据集内嵌了丰富的评估元数据——judge_labels配置中记录了每个系统在每道测试题上的正确性与扎根性评分、支持性证据及其推理过程，这为可解释性分析与评判模型的复现提供了便利。数据集整体采用CC-BY-4.0许可协议，兼容开放科学精神，且其配套论文公布了详细的分层消融实验与LoRA秩搜索范式，为后续研究者提供了可靠的参考基准。

使用方法

使用k8s-docs-rag-bench数据集的过程极为便捷。用户可通过HuggingFace Datasets库中的load_dataset函数直接加载三个配置：qa配置包含问答对（3,614训练、745验证、785测试），corpus配置包含7,908个语义块，judge_labels配置包含172,700行评判标签。典型的使用流程是：首先从corpus中构建嵌入索引，可以使用稠密检索（如BGE-M3的稠密向量）与稀疏检索（如BGE-M3的稀疏向量）的混合策略，再通过重排序器（如BGE-Reranker-v2-M3）对检索结果排序，最后将检索到的Top-K文档片段输入到大语言模型中生成答案。研究者也可直接复用论文中的10种检索消融设置与22种生成器配置，以复现质量-延迟-资源权衡分析。judge_labels配置支持直接对比模型输出与评判分数，或使用其中的证据和推理字段重新评判。数据集提供稳定的ID与显式的字段结构，便于进行广泛的检索与生成联合优化实验。

背景与挑战

背景概述

k8s-docs-rag-bench是一个面向检索增强生成（RAG）技术的小型、全标注基准数据集，由Evgenii Palnikov和Elizaveta Gavrilova于2026年创建，旨在评估基于Kubernetes官方技术文档的问答系统性能。该数据集涵盖5144个手动编写的问题-答案对，以及7908个语义切分的文档块和172,700个由大语言模型作为评判者生成的标签，支持稠密/稀疏/混合检索策略与LoRA微调技术的系统研究。其核心研究问题在于平衡RAG系统中的质量、延迟与计算资源之间的权衡，为技术文档领域提供了首个专门化的评估平台，对推动云原生技术文档的智能问答系统研究具有重要影响力。

当前挑战

该数据集所解决的领域挑战在于：技术文档（如Kubernetes官方文档）具有多层级、多页面且内容高度专业化的特点，传统检索方法难以准确捕获用户意图，而RAG系统在质量、延迟与资源消耗之间往往存在矛盾，缺乏统一的标准化评估基准。构建过程中面临的主要挑战包括：如何设计页面级分割策略以确保泛化性能的真实评估，如何手动撰写与特定文档块精确锚定、同时覆盖概念、任务、教程等多种页面类型的问题-答案对，以及如何系统性地标注不同检索链路（如纯稠密、稀疏、混合BM25）与不同LoRA配置（如秩参数与目标模块变化）下的系统表现，以形成可靠的大语言模型评判标签集。

常用场景

经典使用场景

k8s-docs-rag-bench作为一款面向Kubernetes官方文档的检索增强生成基准测试数据集，其经典使用场景集中在评估和优化技术文档领域的问答系统。研究者可基于该数据集测试稠密、稀疏及混合检索方法在真实多页技术文档上的表现，同时借助其精心设计的页面级数据划分（训练集、验证集、测试集分别覆盖不同文档页面），模拟检索器在未见页面上的泛化能力。该数据集还特别支持通过LoRA微调小型开放权重语言模型（如Llama-3.x系列），在文档锚定的问答任务中探索质量-延迟-资源三者间的权衡关系，为构建高效、轻量的技术文档RAG助手提供可复现的实验平台。

实际应用

在实际应用层面，k8s-docs-rag-bench所模拟的正是企业级技术文档智能问答助手的典型部署场景。Kubernetes文档作为全球最复杂的开源技术文档之一，其层级化、多类型（概念、任务、教程、参考）的文档结构对检索和生成系统提出了严峻挑战。该数据集可用于开发面向DevOps团队的即时问答工具，帮助运维人员在处理集群配置、故障排查等任务时，从海量文档中快速定位精确的解决方案。此外，其研究成果可直接指导云原生企业构建私有化、低延迟的文档辅助系统，在资源受限环境中通过LoRA适配实现轻量化部署，提升技术支持和知识管理效率。

衍生相关工作

该数据集衍生的经典工作体现在其配套论文《Analyzing Quality–Latency–Resource Trade-offs in a Technical Documentation RAG Assistant Using LoRA Adaptation》中，该研究基于此数据集系统评估了10种检索管线变体（包括基础混合检索、无重排序器、仅稠密、仅稀疏及经典BM25消融）与22种生成器配置（涵盖不同基数模型、LoRA秩及目标模块）的协同效应。这项工作直接推动了后续关于RAG管线中检索粒度与生成质量关系的研究，并启发了一系列针对技术文档的知识密集型任务评估标准。同时，数据集中开放的LLM评判标签（含原始证据引用和推理过程）为检验自动化评估方法的可靠性提供了基准，促进了LLM-as-a-Judge范式在特定领域内的验证与改进。