KRLabsOrg/tool-output-extraction-swebench-gliner

--- license: apache-2.0 task_categories: - token-classification language: - en tags: - information-extraction - ner - gliner - extractive-qa - coding-agents - tool-output - context-pruning size_categories: - 10K<n<100K --- # Tool Output Extraction (extractive / GLiNER2 format) Extractive variant of [KRLabsOrg/tool-output-extraction-swebench](https://huggingface.co/datasets/KRLabsOrg/tool-output-extraction-swebench), formatted for fine-tuning span-extraction models ([GLiNER2](https://github.com/fastino-ai/GLiNER2), BERT-for-QA, etc.). Each tool observation from the parent dataset is chunked into ~400-token windows (preserving line boundaries) so it fits into encoder-style models with a 512-token context. The query is concatenated in front of each chunk, extractive-QA style, and gold evidence is mapped to verbatim spans within the tool-output portion. Chunks with no evidence become natural negative examples. ## Format Each row is a GLiNER2 training record with the query concatenated into the input: ```json { "input": "Query: Find the code block...\n\nTool output:\n193: ...\n194: ...\n...\n233: columns = []\n...", "output": { "entities": {"RELEVANT": ["233: columns = []\n234: for col in data.columns:\n..."]} }, "meta": { "instance_id": "astropy__astropy-12544", "source": "swe", "tool_type": "read_file", "query": "Find the code block in read_table_fits...", "chunk_index": 7, "total_chunks": 15, "has_evidence": true, "chunk_start_line": 193, "chunk_end_line": 233 } } ``` Design choices: - **Query concatenation.** Query is prepended as `Query: ...\n\nTool output:\n<chunk>` so the model conditions on it directly, like an extractive-QA model. This avoids relying on GLiNER2's per-type `entity_descriptions` for per-example queries. - **Single entity type `RELEVANT`.** All examples share the same type; the task-specific signal comes from the query in the input. - **Verbatim spans.** Every entity mention is a verbatim substring of `input`, validated with GLiNER2's `InputExample.validate()`. ## Splits | Split | Chunks | Positive | Negative | Source examples | |-------|-------:|---------:|---------:|----------------:| | train | 51,917 | 17,450 | 34,467 | 10,508 | | dev | 2,579 | 422 | 2,157 | 240 | | test | 9,595 | 1,090 | 8,505 | 618 | Negatives in the train split are subsampled (30% kept) to limit class imbalance. Dev and test preserve the natural distribution. ## Usage with GLiNER2 ```python from gliner2.training.data import InputExample, TrainingDataset import json def load_split(path): examples = [] with open(path) as f: for line in f: d = json.loads(line) examples.append(InputExample( text=d["input"], entities=d["output"]["entities"], )) return TrainingDataset(examples=examples) train_ds = load_split("gliner_train.jsonl") dev_ds = load_split("gliner_dev.jsonl") ``` At inference, format new inputs the same way: ```python query = "Find the failing test block" chunk = open("pytest_output.txt").read() text = f"Query: {query}\n\nTool output:\n{chunk}" # model.extract_entities(text, entity_types=["RELEVANT"]) -> list of verbatim spans ``` ## Source Generated from [KRLabsOrg/tool-output-extraction-swebench](https://huggingface.co/datasets/KRLabsOrg/tool-output-extraction-swebench) (11,477 examples, 27 tool types, derived from SWE-bench repositories and synthetic multi-ecosystem observations). See the [paper](https://arxiv.org/abs/2604.04979) for construction details. ## Citation ```bibtex @misc{kovács2026squeeztaskconditionedtooloutputpruning, title={Squeez: Task-Conditioned Tool-Output Pruning for Coding Agents}, author={Ádám Kovács}, year={2026}, eprint={2604.04979}, archivePrefix={arXiv}, primaryClass={cs.SE}, url={https://arxiv.org/abs/2604.04979}, } ``` ## License Apache 2.0license: apache-2.0 任务类别： - 令牌分类（Token Classification） 语言： - 英语 标签： - 信息抽取 - 命名实体识别（NER） - GLiNER2 - 抽取式问答（Extractive QA） - 代码智能体（coding-agents） - 工具输出 - 上下文剪枝 样本规模： - 10K<n<100K --- # 工具输出抽取（抽取式/GLiNER2格式） 本数据集为[KRLabsOrg/tool-output-extraction-swebench](https://huggingface.co/datasets/KRLabsOrg/tool-output-extraction-swebench)的抽取式变体，专为跨度抽取模型（如GLiNER2、用于问答的BERT等）的微调进行格式化。 原父数据集中的每一条工具观测结果均被切分为约400个Token的窗口（保留行边界），以适配拥有512个Token上下文的编码器风格模型。将查询以抽取式问答的风格拼接至每个窗口的前端，且将标注的证据映射至工具输出部分中的逐字跨度。不含证据的切块将作为自然负样本。 ## 格式 每一行均为一条GLiNER2训练记录，查询已拼接至输入中： json { "input": "Query: Find the code block... Tool output: 193: ... 194: ... ... 233: columns = [] ...", "output": { "entities": {"RELEVANT": ["233: columns = [] 234: for col in data.columns: ..."]} }, "meta": { "instance_id": "astropy__astropy-12544", "source": "swe", "tool_type": "read_file", "query": "Find the code block in read_table_fits...", "chunk_index": 7, "total_chunks": 15, "has_evidence": true, "chunk_start_line": 193, "chunk_end_line": 233 } } ## 设计考量 - **查询拼接**：将查询以`Query: ... Tool output: <chunk>`的格式前置，使模型直接以抽取式问答的方式进行条件建模，无需依赖GLiNER2针对每个示例查询的逐类型`entity_descriptions`。 - **单一实体类型`RELEVANT`**：所有示例共享同一实体类型，任务特定的信号由输入中的查询提供。 - **逐字跨度**：每个实体提及均为`input`中的逐字子字符串，已通过GLiNER2的`InputExample.validate()`方法完成验证。 ## 拆分集 | 拆分集 | 切块数 | 正样本数 | 负样本数 | 源示例数 | |-------|-------:|---------:|---------:|----------------:| | 训练集 | 51,917 | 17,450 | 34,467 | 10,508 | | 开发集 | 2,579 | 422 | 2,157 | 240 | | 测试集 | 9,595 | 1,090 | 8,505 | 618 | 训练拆分集中的负样本进行了下采样（保留30%）以缓解类别不平衡问题。开发集与测试集则保留其自然分布。 ## GLiNER2使用方法 python from gliner2.training.data import InputExample, TrainingDataset import json def load_split(path): examples = [] with open(path) as f: for line in f: d = json.loads(line) examples.append(InputExample( text=d["input"], entities=d["output"]["entities"], )) return TrainingDataset(examples=examples) train_ds = load_split("gliner_train.jsonl") dev_ds = load_split("gliner_dev.jsonl") 在推理阶段，需以相同格式格式化新输入： python query = "查找失败的测试代码块" chunk = open("pytest_output.txt").read() text = f"Query: {query} Tool output: {chunk}" # model.extract_entities(text, entity_types=["RELEVANT"]) -> 逐字跨度列表 ## 数据集来源 本数据集源自[KRLabsOrg/tool-output-extraction-swebench](https://huggingface.co/datasets/KRLabsOrg/tool-output-extraction-swebench)（包含11,477个示例、27种工具类型，源自SWE-bench仓库与合成的多生态系统观测结果）。有关构建细节，请参阅[论文](https://arxiv.org/abs/2604.04979)。 ## 引用格式 bibtex @misc{kovács2026squeeztaskconditionedtooloutputpruning, title={Squeez: Task-Conditioned Tool-Output Pruning for Coding Agents}, author={Ádám Kovács}, year={2026}, eprint={2604.04979}, archivePrefix={arXiv}, primaryClass={cs.SE}, url={https://arxiv.org/abs/2604.04979}, } ## 许可证 Apache 2.0