gray311/AgenticOPD

---
license: other
language:
- en
task_categories:
- question-answering
- text-generation
tags:
- function-calling
- knowledge-injection
- on-policy-distillation
- catastrophic-forgetting
- tool-use
pretty_name: AgenticOPD — Unified Knowledge-Injection Benchmark
configs:
- config_name: bfcl_api
data_files: bfcl_api.jsonl
- config_name: bfcl_single
data_files: bfcl_single.jsonl
- config_name: squad
data_files:
- split: train
path: squad_train.jsonl
- split: validation
path: squad_validation.jsonl
- config_name: ms_marco_mqa
data_files:
- split: train
path: ms_marco_mqa_train.jsonl
- split: val
path: ms_marco_mqa_val.jsonl
- split: test
path: ms_marco_mqa_test.jsonl
source_datasets:
- gorilla-llm/Berkeley-Function-Calling-Leaderboard
- rajpurkar/squad
- Yewei-Liu/ms_marco_mqa
---

# AgenticOPD — Unified Knowledge-Injection Benchmark

Three source datasets (BFCL, SQuAD, MS MARCO MQA) normalized to one schema so the same
On-Policy Distillation (OPD) training/eval code can consume them interchangeably. Built to
study parametric knowledge injection (tool docs + text passages) into 7B LLMs while
avoiding catastrophic forgetting.

Licenses of original data apply to each subset: BFCL (Apache-2.0),
SQuAD (CC-BY-SA-4.0), MS MARCO MQA (MIT, derived from MS MARCO under its own terms).

## Unified schema

One JSONL row = **one knowledge unit + all its probes**.

```json
{
"unit_id": "bfcl_api:GorillaFileSystem",
"source": "bfcl_api" | "bfcl_single" | "squad" | "ms_marco_mqa",
"domain": "GorillaFileSystem" | "<Wikipedia title>" | null,
"split": "train" | "validation" | "test" | null,

"knowledge": {
"text": "<LM-friendly text — always a string>",
"format": "tool_api_markdown" | "tool_single_markdown" | "passage",
"structured": [ ...function specs... ] | null
},

"probes": [
{
"probe_id": "multi_turn_base_0",
"question": "<single turn or first turn>",
"question_turns": [[msg,...], [msg,...]] | null,
"answer": {
"type": "ast_call" | "exec_call" | "call_trajectory" | "span" | "free_text",
"value": <polymorphic>,
"evaluator_hint": "bfcl_ast" | "bfcl_exec" | "bfcl_multi_turn" | "squad_em_f1" | "string_match_loose"
},
"probe_meta": { ... source-specific extras ... }
}
]
}
```

## Files

| File | Unit granularity | Knowledge | Probe count |
|---|---|---|---|
| `bfcl_api.jsonl` | per API class (8) | full API markdown + structured specs | all MT samples involving the API |
| `bfcl_single.jsonl` | per ST sample | the sample's own func doc(s) | 1 per unit |
| `squad_{train,validation}.jsonl` | per (title, context) | passage | ~5 Q per unit |
| `ms_marco_mqa_{train,val,test}.jsonl` | per context | passage | 15 Q per unit |

## How `answer.value` looks per `answer.type`

- **ast_call**: `[{"fn_name": {"arg": [allowed_values...]}}]` — BFCL AST matcher
- **exec_call**: `"fn_name(arg=val, ...)"` — executable string
- **call_trajectory**: `[[step1, step2], [step1], ...]` — nested list per turn for BFCL MT
- **span**: `["accepted text 1", "accepted text 2", ...]` — SQuAD; `answer_start` in `probe_meta`
- **free_text**: `["single answer string"]` — MS MARCO (list for consistency)

## Evaluator routing

Use `probes[*].answer.evaluator_hint` to dispatch to the right scorer:
- `bfcl_ast` → BFCL AST checker (`github.com/ShishirPatil/gorilla/...`)
- `bfcl_exec` → execute + compare
- `bfcl_multi_turn` → execute trajectory on seeded backend state (`probe_meta.initial_config`)
- `squad_em_f1` → SQuAD official EM/F1 (allow any string in `value` list)
- `string_match_loose` → contains / LLM-judge fallback

许可证：其他
语言：英语
任务类别：
- 问答
- 文本生成
标签：
- 函数调用
- 知识注入
- 策略内蒸馏（On-Policy Distillation，OPD）
- 灾难性遗忘
- 工具使用
展示名称：AgenticOPD——统一知识注入基准
配置项：
- 配置名称：bfcl_api，数据文件：bfcl_api.jsonl
- 配置名称：bfcl_single，数据文件：bfcl_single.jsonl
- 配置名称：squad，数据文件：
- 划分集：训练集，路径：squad_train.jsonl
- 划分集：验证集，路径：squad_validation.jsonl
- 配置名称：ms_marco_mqa，数据文件：
- 划分集：训练集，路径：ms_marco_mqa_train.jsonl
- 划分集：验证集，路径：ms_marco_mqa_val.jsonl
- 划分集：测试集，路径：ms_marco_mqa_test.jsonl
源数据集：
- gorilla-llm/伯克利函数调用排行榜（Berkeley-Function-Calling-Leaderboard）
- rajpurkar/SQuAD
- Yewei-Liu/ms_marco_mqa

# AgenticOPD——统一知识注入基准

将三个源数据集（BFCL、SQuAD、MS MARCO MQA）归一化至统一数据模式，使得同一套策略内蒸馏（On-Policy Distillation，OPD）训练与评估代码可无缝适配不同数据集。本基准旨在研究向70亿参数大语言模型（Large Language Model，LLM）注入参数化知识（工具文档与文本段落）的同时避免灾难性遗忘。

各子集沿用原始数据集的许可证：BFCL采用Apache-2.0协议，SQuAD采用CC-BY-SA-4.0协议，MS MARCO MQA采用MIT协议（其衍生自MS MARCO，需遵循MS MARCO的原有条款）。

## 统一数据模式

每条JSONL行对应**一个知识单元及其所有测试探针**。

json
{
"unit_id": "bfcl_api:GorillaFileSystem",
"source": "bfcl_api" | "bfcl_single" | "squad" | "ms_marco_mqa",
"domain": "GorillaFileSystem" | "<维基百科标题>" | null,
"split": "train" | "validation" | "test" | null,

"knowledge": {
"text": "<适配大语言模型的文本 — 始终为字符串>",
"format": "tool_api_markdown" | "tool_single_markdown" | "passage",
"structured": [ ...函数规格列表... ] | null
},

"probes": [
{
"probe_id": "multi_turn_base_0",
"question": "<单轮问题或首轮问题>",
"question_turns": [[消息,...], [消息,...]] | null,
"answer": {
"type": "ast_call" | "exec_call" | "call_trajectory" | "span" | "free_text",
"value": <多态取值>,
"evaluator_hint": "bfcl_ast" | "bfcl_exec" | "bfcl_multi_turn" | "squad_em_f1" | "string_match_loose"
},
"probe_meta": { ... 源数据集特定的额外信息 ... }
}
]
}

## 文件说明

| 文件名 | 单元粒度 | 知识内容 | 探针数量 |
|---|---|---|---|
| `bfcl_api.jsonl` | 按API类（共8类） | 完整API Markdown文档与结构化规格 | 该API相关的所有多轮样本 |
| `bfcl_single.jsonl` | 按单轮样本 | 样本对应的函数文档 | 每个单元1个探针 |
| `squad_{train,validation}.jsonl` | 按（标题，上下文）对 | 文本段落 | 每个单元约5个探针 |
| `ms_marco_mqa_{train,val,test}.jsonl` | 按上下文 | 文本段落 | 每个单元15个探针 |

## 不同`answer.type`对应的`answer.value`格式

- **ast_call**：`[{"fn_name": {"arg": [允许取值...]}}]` — 适配BFCL的抽象语法树（Abstract Syntax Tree，AST）匹配器
- **exec_call**：`"fn_name(arg=val, ...)"` — 可执行字符串
- **call_trajectory**：`[[步骤1, 步骤2], [步骤1], ...]` — 嵌套列表，用于表示BFCL多轮任务的调用轨迹
- **span**：`["匹配文本1", "匹配文本2", ...]` — 适配SQuAD数据集；`answer_start`字段存储于`probe_meta`中
- **free_text**：`["单答案字符串"]` — 适配MS MARCO数据集（采用列表格式以统一规范）

## 评估器路由规则

通过`probes[*].answer.evaluator_hint`字段可调度至对应的评估器：
- `bfcl_ast` → BFCL抽象语法树检查器（`github.com/ShishirPatil/gorilla/...`）
- `bfcl_exec` → 执行代码并比对结果
- `bfcl_multi_turn` → 在初始化的后端状态（`probe_meta.initial_config`）上执行调用轨迹
- `squad_em_f1` → SQuAD官方精确匹配（Exact Match，EM）/ F1评分（支持`value`列表中的任意字符串）
- `string_match_loose` → 宽松包含匹配/大语言模型评估兜底方案