hyunseoki/memory-reasoning-split-eval-sets

--- license: odc-by task_categories: - question-answering - text-generation language: - en tags: - reasoning - factual-recall - ablation - qwen3 - memory-offloading pretty_name: Memory-Reasoning-Split Stage D Eval Sets size_categories: - 1K<n<10K --- # Memory-Reasoning-Split Stage D Eval Sets Curated + subsetted closed-book evaluation sets used to measure the per-domain factual degradation × reasoning retention trade-off in the [`memory_reasoning_split`](https://github.com/hyunseoklee-ai/memory_split) Stage D/E forget-corpus ablation. All factual rows share a unified schema so a single evaluator can score them: ``` { "question": str, "aliases": list[str], # any normalized alias match counts as a hit "relation": str, "topic": str, "source_dataset": str (popqa_general / sciq subset only; custom splits omit) } ``` ## Files | File | Rows | Kind | Source | |---|---:|---|---| | `popqa_general.jsonl` | 1232 | general-domain closed-book QA | [`akariasai/PopQA`](https://huggingface.co/datasets/akariasai/PopQA) test subset, reshaped to unified schema with aliases + relation + topic | | `math_facts.jsonl` | 100 | hand-curated closed-book | constants, theorems, formulas (calculus, linear algebra, probability, geometry, trigonometry, number theory, ...) | | `code_api_facts.jsonl` | 101 | hand-curated closed-book | Python/NumPy/PyTorch/Pandas/JS/C++/Rust/Go/SQL API signatures + shell/git/HTTP trivia | | `sciq.jsonl` | 500 | subset of [`allenai/sciq`](https://huggingface.co/datasets/allenai/sciq) `test` split, reshaped | converted to the unified QA schema | | `humaneval_prompts.jsonl` | 164 | full [`openai_humaneval`](https://huggingface.co/datasets/openai_humaneval) `test` split | prompt + canonical_solution + test + entry_point | **Why `popqa_general`?** Stage D's forget corpus is Wikipedia-derived, so the most natural "Did you just break general-domain factual recall?" probe is general-topic PopQA (1232 rows across relations like `occupation`, `place_of_birth`, `capital`, etc.). Pairs with the NER-masked training corpus at [`hyunseoki/popqa-mini-ner-knowledge-masks`](https://huggingface.co/datasets/hyunseoki/popqa-mini-ner-knowledge-masks). ## Usage Closed-book factual eval (vLLM-backed): ```bash python scripts/eval_stage_d_factual.py \ --config configs/stage_d/d0_no_forget.yaml \ --adapter_checkpoint outputs/stage_d/d1_wikipedia/checkpoint-1338 \ --eval_jsonl popqa_general.jsonl \ --tag popqa_general --output_dir outputs/stage_d_eval/d1 ``` HumanEval pass@1 (sandboxed subprocess scorer): ```bash python scripts/eval_stage_d_humaneval.py \ --config configs/stage_d/d0_no_forget.yaml \ --adapter_checkpoint outputs/stage_d/d4_all_domains/checkpoint-1338 \ --eval_jsonl humaneval_prompts.jsonl \ --output_dir outputs/stage_d_eval/d4 ``` The `scripts/run_stage_d_eval.sh` orchestrator runs all five splits (`popqa_general`, `math_facts`, `code_api_facts`, `sciq`, `humaneval`) plus MATH-500 and AMC23 reasoning tasks across every trained Stage D/E model in parallel. ## Reproducibility - `math_facts` / `code_api_facts` were hand-authored for this project; all 201 items have ≥ 3 accepted aliases to absorb surface-form variation. - `sciq.jsonl` was produced by `scripts/prepare_stage_d_eval_sets.py` with a deterministic first-500 slice of the SciQ test split. - `humaneval_prompts.jsonl` was produced by the same script; it keeps the upstream prompt/test/entry_point unchanged for standard pass@1 scoring. - `popqa_general.jsonl` was produced by the same script from the `popqa_sharded_test` subset; each row preserves PopQA's `relation`, canonical `topic` (subject), and the full alias list for normalized match. ## Intended use Drop-in eval suite for measuring **per-domain factual retention** and **code usability** of adapters trained with selective forgetting over different forget corpora. Pairs with the retain corpus [`hyunseoki/qwen3-0p6b-openthoughts-self-distill-10k`](https://huggingface.co/datasets/hyunseoki/qwen3-0p6b-openthoughts-self-distill-10k), the NER-masked forget corpus [`hyunseoki/popqa-mini-ner-knowledge-masks`](https://huggingface.co/datasets/hyunseoki/popqa-mini-ner-knowledge-masks), and the dedup index [`hyunseoki/openthoughts3-dedup-index`](https://huggingface.co/datasets/hyunseoki/openthoughts3-dedup-index). All assets are grouped under the [Qwen3 Lambda Gates collection](https://huggingface.co/collections/hyunseoki/qwen3-lambda-gates-knowledge-reasoning-disentanglement-69e20c8e64960042ed4c3159). ## License / Attribution SciQ rows © Allen AI (CC BY-NC 3.0). HumanEval rows © OpenAI (released under the MIT license in the `openai_humaneval` dataset card; please follow its terms). PopQA rows © Asai et al. (MIT license). The hand-curated `math_facts` and `code_api_facts` are CC BY-SA 4.0.> 许可证：odc-by > 任务类别： > - 问答 > - 文本生成 > 语言： > - 英语 > 标签： > - 推理 > - 事实性召回 > - 消融实验 > - Qwen3 > - 内存卸载 > 易读名称：内存-推理拆分阶段D评测集（Memory-Reasoning-Split Stage D Eval Sets） > 样本规模：1000 < 样本数 < 10000 # 内存-推理拆分阶段D评测集（Memory-Reasoning-Split Stage D Eval Sets） 本数据集为经过精选与子集筛选的闭卷评测集（closed-book evaluation sets），用于衡量[`memory_reasoning_split`](https://github.com/hyunseoklee-ai/memory_split) 阶段D/E遗忘语料消融实验中，各领域事实性退化与推理能力保留之间的权衡关系。 所有事实型样本均采用统一的数据格式（schema），因此可通过单一评测器完成评分： { "question": str, "aliases": list[str], # 任意归一化别名匹配均视为命中 "relation": str, "topic": str, "source_dataset": str # 仅popqa_general / sciq子集包含该字段；自定义拆分无需包含 } ## 文件列表 | 文件名 | 样本数量 | 类型 | 来源 | |---|---:|---|---| | `popqa_general.jsonl` | 1232 | 通用领域闭卷问答数据集 | 源自[`akariasai/PopQA`](https://huggingface.co/datasets/akariasai/PopQA) 测试子集，已重构为包含别名、关系与主题的统一格式 | | `math_facts.jsonl` | 100 | 手工精选闭卷评测集 | 涵盖常数、定理、公式（微积分、线性代数、概率论、几何学、三角学、数论等领域） | | `code_api_facts.jsonl` | 101 | 手工精选闭卷评测集 | 涵盖Python/NumPy/PyTorch/Pandas/JS/C++/Rust/Go/SQL 的API签名，以及shell/git/HTTP 技术常识 | | `sciq.jsonl` | 500 | 问答数据集 | 源自[`allenai/sciq`](https://huggingface.co/datasets/allenai/sciq) 测试拆分的500条子集，已重构为统一问答格式 | | `humaneval_prompts.jsonl` | 164 | 评测数据集 | 完整的[`openai_humaneval`](https://huggingface.co/datasets/openai_humaneval) 测试拆分，包含提示词（prompt）、标准解法（canonical_solution）、测试用例（test）与入口点（entry_point） | ## 为何选用`popqa_general`？ 阶段D的遗忘语料源自维基百科，因此最自然的“是否破坏了通用领域事实性召回”探测任务即为涵盖职业、出生地、首都等关系的1232条通用主题PopQA样本。该评测集可与[`hyunseoki/popqa-mini-ner-knowledge-masks`](https://huggingface.co/datasets/hyunseoki/popqa-mini-ner-knowledge-masks) 中的命名实体识别（Named Entity Recognition, NER）掩码训练语料搭配使用。 ## 使用方法 ### 闭卷事实性评测（基于vLLM） bash python scripts/eval_stage_d_factual.py --config configs/stage_d/d0_no_forget.yaml --adapter_checkpoint outputs/stage_d/d1_wikipedia/checkpoint-1338 --eval_jsonl popqa_general.jsonl --tag popqa_general --output_dir outputs/stage_d_eval/d1 ### HumanEval pass@1 评测（沙箱子进程评分） bash python scripts/eval_stage_d_humaneval.py --config configs/stage_d/d0_no_forget.yaml --adapter_checkpoint outputs/stage_d/d4_all_domains/checkpoint-1338 --eval_jsonl humaneval_prompts.jsonl --output_dir outputs/stage_d_eval/d4 `scripts/run_stage_d_eval.sh` 调度脚本可并行运行全部5个拆分子集（`popqa_general`、`math_facts`、`code_api_facts`、`sciq`、`humaneval`）以及MATH-500与AMC23推理任务，覆盖所有已训练的阶段D/E模型。 ## 可复现性说明 1. `math_facts`与`code_api_facts`为本项目手工编写，全部201条样本均包含至少3个可接受的别名，以适配表面形式的变体。 2. `sciq.jsonl`由`scripts/prepare_stage_d_eval_sets.py`基于SciQ测试拆分的前500条确定性切片生成。 3. `humaneval_prompts.jsonl`由同一脚本生成，保留了上游数据集的原始提示词、测试用例与入口点，以支持标准pass@1评测。 4. `popqa_general.jsonl`由同一脚本基于`popqa_sharded_test`子集生成，每条样本均保留了PopQA的`relation`（关系）、规范`topic`（主题/实体）以及用于归一化匹配的完整别名列表。 ## 预期用途 本评测套件可直接用于衡量针对不同遗忘语料训练的选择性遗忘适配器的**各领域事实性保留能力**与**代码可用性**。本套件可搭配以下资源使用：保留语料[`hyunseoki/qwen3-0p6b-openthoughts-self-distill-10k`](https://huggingface.co/datasets/hyunseoki/qwen3-0p6b-openthoughts-self-distill-10k)、命名实体识别掩码遗忘语料[`hyunseoki/popqa-mini-ner-knowledge-masks`](https://huggingface.co/datasets/hyunseoki/popqa-mini-ner-knowledge-masks) 以及去重索引[`hyunseoki/openthoughts3-dedup-index`](https://huggingface.co/datasets/hyunseoki/openthoughts3-dedup-index)。所有相关资源均归类于[Qwen3 Lambda Gates 知识与推理解耦合集](https://huggingface.co/collections/hyunseoki/qwen3-lambda-gates-knowledge-reasoning-disentanglement-69e20c8e64960042ed4c3159) 下。 ## 许可证与归属声明 SciQ样本 © Allen AI（遵循CC BY-NC 3.0协议）。HumanEval样本 © OpenAI（在`openai_humaneval`数据集卡片中以MIT许可证发布，请遵循其使用条款）。PopQA样本 © Asai等人（遵循MIT许可证）。手工精选的`math_facts`与`code_api_facts`遵循CC BY-SA 4.0协议。