bertybaums/marc

--- language: - en license: cc-by-4.0 size_categories: - 10K<n<100K task_categories: - visual-question-answering - text-classification tags: - arc - metaphor - figurative-language - mechanistic-interpretability - grid-puzzles - abstraction-and-reasoning - multimodal-integration pretty_name: "MARC: Metaphor Abstraction and Reasoning Corpus" configs: - config_name: tasks data_files: "tasks/train.parquet" - config_name: task_subsets data_files: "task_subsets/train.parquet" - config_name: descriptions data_files: "descriptions/train.parquet" - config_name: baseline data_files: "baseline/train.parquet" - config_name: figurative data_files: "figurative/train.parquet" dataset_info: - config_name: tasks features: - name: task_id dtype: int32 - name: arc_name dtype: string - name: source dtype: string - name: num_train dtype: int32 - name: see_description dtype: string - name: do_description dtype: string - name: grid_description dtype: string splits: - name: train num_examples: 577 - config_name: task_subsets features: - name: task_id dtype: int32 - name: model_name dtype: string - name: subset dtype: string splits: - name: train num_examples: 1188 - config_name: descriptions features: - name: fig_id dtype: int32 - name: task_id dtype: int32 - name: generator_model dtype: string - name: variant dtype: string - name: source_domain dtype: string - name: metaphor dtype: string - name: figurative_see dtype: string - name: figurative_do dtype: string - name: figurative_grid dtype: string splits: - name: train num_examples: 1500 - config_name: baseline features: - name: trial_id dtype: int32 - name: task_id dtype: int32 - name: model_name dtype: string - name: condition dtype: string - name: num_examples dtype: int32 - name: correct dtype: int32 - name: cell_accuracy dtype: float32 splits: - name: train num_examples: 3952 - config_name: figurative features: - name: trial_id dtype: int32 - name: fig_id dtype: int32 - name: task_id dtype: int32 - name: model_name dtype: string - name: num_examples dtype: int32 - name: correct dtype: int32 - name: cell_accuracy dtype: float32 - name: variant dtype: string - name: source_domain dtype: string splits: - name: train num_examples: 8225 --- # MARC: Metaphor Abstraction and Reasoning Corpus ## What This Is MARC identifies puzzles where figurative language and visual examples are *genuinely complementary*: the model fails given examples alone, fails given the metaphor alone, but succeeds when both are presented together. We call this the **MARC property**. The corpus provides 78 MARC-verified puzzles with 1,230 domain-diverse figurative descriptions and complete behavioral trial data for three language models. Suppose you are staring at a grid puzzle — coloured cells in rows and columns, some pattern lurking beneath the surface. A handful of training examples show input grids paired with their correct outputs, but the transformation rule eludes you. Now someone offers a hint: "Think of it as a garden, where the green cells are plants spreading to fill empty soil." If that helps — and if neither the examples nor the metaphor would have sufficed on its own — then this puzzle exhibits the MARC property. ## The MARC Property A puzzle satisfies the MARC property for a given model when three conditions hold simultaneously: 1. **Examples alone fail.** The model cannot solve the puzzle from training input-output pairs alone. 2. **Figurative description alone fails.** The model cannot solve the puzzle from the metaphorical clue alone. 3. **Figurative + examples succeeds.** The model solves the puzzle when given both the metaphor and some number of training examples. This isolates cases where language and perception are genuinely complementary — neither channel suffices on its own, but their combination does. ## Data Sources The corpus draws on two sources: - **LARC tasks** (task IDs 0–399): 400 ARC-AGI puzzles (Chollet, 2019) extended with crowdsourced literal descriptions from the LARC dataset (Acquaviva et al., 2022). Figurative descriptions were generated by Claude and verified behaviorally. - **MARC submissions** (task IDs 1000–1176): 177 hand-crafted puzzles with human-authored figurative clues. Grid puzzles themselves (the actual input-output grid pairs) are not included in this dataset. They are available from the original ARC-AGI repository and the LARC dataset. Task IDs and `arc_name` fields provide the link. ## Dataset Configs ### `tasks` (577 rows) Task metadata. For LARC tasks, `see_description`, `do_description`, and `grid_description` contain literal descriptions of the puzzle. For MARC submissions, these fields contain the figurative clue (no literal descriptions exist). | Column | Description | |--------|-------------| | `task_id` | Unique task identifier (0–399: LARC; 1000–1176: submissions) | | `arc_name` | Original ARC filename (8-character hex ID) | | `source` | `'larc'` or `'marc-submission'` | | `num_train` | Number of training examples (1–10) | | `see_description` | What structures are visible in the input | | `do_description` | The transformation rule | | `grid_description` | How output dimensions relate to input | ### `task_subsets` (1,188 rows) Classification of each task by each model into one of four categories based on baseline performance. | Column | Description | |--------|-------------| | `task_id` | Links to `tasks` | | `model_name` | Which model this classification applies to | | `subset` | `'examples_sufficient'`, `'language_sufficient'`, `'both_required'`, or `'unsolvable'` | ### `descriptions` (1,500 rows) Figurative descriptions — the metaphorical clues. This is the core contribution. Each MARC-verified puzzle has an original clue plus up to 24 domain-diverse alternatives. | Column | Description | |--------|-------------| | `fig_id` | Unique description identifier | | `task_id` | Links to `tasks` | | `generator_model` | `'claude-agent'` (generated) or `'human'` (hand-crafted) | | `variant` | `'original'`, `'alt-1'`, `'alt-2'`, ... | | `source_domain` | Metaphor domain: `'biology'`, `'warfare'`, `'cooking'`, etc. (NULL for originals) | | `metaphor` | One-line metaphor concept | | `figurative_see` | Figurative "what you see" (empty for submissions) | | `figurative_do` | Figurative "what to do" (empty for submissions) | | `figurative_grid` | Figurative grid description (empty for submissions) | **Domain coverage:** 12 core domains (warfare, biology, cooking, music, gardening, navigation, dance, theater, architecture, astronomy, chemistry, weather) each cover all 78 MARC-verified tasks. 51 additional domains appear in smaller numbers. ### `baseline` (3,952 rows) Baseline trial results under three conditions (no figurative language involved). | Column | Description | |--------|-------------| | `trial_id` | Unique trial identifier | | `task_id` | Links to `tasks` | | `model_name` | Subject model | | `condition` | `'examples_only'`, `'language_only'`, or `'both'` | | `num_examples` | Training examples shown | | `correct` | 1 = exact match, 0 = incorrect | | `cell_accuracy` | Fraction of cells matching (0.0–1.0) | ### `figurative` (8,225 rows) Figurative trial results. Each row is one (metaphor variant, model, number of examples) combination. | Column | Description | |--------|-------------| | `trial_id` | Unique trial identifier | | `fig_id` | Links to `descriptions` | | `task_id` | Links to `tasks` | | `model_name` | Subject model | | `num_examples` | 0 = figurative only, 1–N = figurative + k examples | | `correct` | 1 = exact match, 0 = incorrect | | `cell_accuracy` | Fraction of cells matching | | `variant` | Denormalized from `descriptions` for convenience | | `source_domain` | Denormalized from `descriptions` for convenience | ## Models Tested | Model | Parameters | Architecture | Notes | |-------|-----------|--------------|-------| | gpt-oss-120b | 120B | MoE, open-weight | Primary verification model | | gpt-oss-20b | 21B (3.6B active) | MoE, open-weight | Mechanistic interpretability target | | qwen3.5-400b | 400B | Dense | Baseline only (no figurative trials) | All trials used temperature 0.0. Reasoning models (gpt-oss-*) use a two-pass protocol: Pass 1 for reasoning, Pass 2 for structured output extraction. ## Key Statistics | | Count | |---|---:| | Total tasks | 577 | | MARC-verified puzzles (120b) | 78 | | Figurative descriptions | 1,500 | | — original clues | 270 | | — domain-diverse alternatives | 1,230 | | Distinct source domains | 63 | | Core domains (full 78-task coverage) | 12 | | Baseline trials | 3,952 | | Figurative trials | 8,225 | | MARC-valid alternatives (120b) | 725/1,230 (59%) | ## Usage ```python from datasets import load_dataset # Load a specific config tasks = load_dataset("bertybaums/marc", "tasks") descriptions = load_dataset("bertybaums/marc", "descriptions") figurative = load_dataset("bertybaums/marc", "figurative") # Get all MARC-valid alternatives for biology domain bio = [d for d in descriptions["train"] if d["source_domain"] == "biology"] # Check MARC property: find tasks where figurative+examples succeeds # but figurative-alone and examples-alone both fail import pandas as pd fig_df = figurative["train"].to_pandas() marc_valid = fig_df.groupby("fig_id").apply( lambda g: (g[g.num_examples == 0].correct == 0).all() and (g[g.num_examples > 0].correct == 1).any() ) ``` ## Intended Uses - **Mechanistic interpretability:** How do LLMs internally integrate figurative language with visual-spatial pattern recognition? The domain-diverse alternatives enable controlled comparisons: same puzzle, different metaphor. - **Figurative language understanding:** Which source domains produce more effective metaphors for abstract reasoning tasks? The 12-domain factorial supports systematic comparison. - **Abstraction and reasoning:** The MARC property identifies a specific failure mode (examples alone insufficient) and a specific remedy (figurative scaffolding). What makes some puzzles amenable to this scaffolding and others not? - **Scaling analysis:** Comparing MARC validity rates across 20B vs. 120B models reveals how figurative reasoning capacity scales with model size. ## Limitations - Figurative descriptions were generated by Claude, not humans (except the 177 MARC submissions). The metaphors may reflect Claude's biases in how it maps grid operations to conceptual domains. - Behavioral trials use temperature 0.0, but reasoning models may still exhibit minor non-determinism across identical prompts. - The dataset does not include the grid puzzles themselves (input-output pairs). Researchers need the original ARC/LARC data to see what the metaphors describe. - MARC verification was performed against gpt-oss-120b. A metaphor that is MARC-valid for 120b may not be for smaller models (and vice versa). - Prompt text is not included in this release to keep the dataset compact. Researchers needing full prompts can reconstruct them from the task data + descriptions, or contact the authors. ## Citation If you use this dataset, please cite: ```bibtex @misc{baumgaertner2026marc, title={MARC: Metaphor Abstraction and Reasoning Corpus}, author={Baumgaertner, Bert}, year={2026}, url={https://huggingface.co/datasets/bertybaums/marc} } ``` ## Acknowledgments Grid puzzles are drawn from ARC-AGI (Chollet, 2019) and LARC (Acquaviva et al., 2022). Figurative descriptions were generated using Claude (Anthropic). Behavioral experiments were conducted on the MindRouter infrastructure at the University of Idaho. ## License This dataset is released under [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/). The underlying ARC puzzles are licensed under Apache 2.0; LARC descriptions are licensed under CC-BY-4.0.语言：英语 许可协议：CC-BY-4.0 数据规模：10000 < 样本数 < 100000 任务类别：视觉问答(visual question answering)、文本分类(text classification) 标签：ARC、隐喻(metaphor)、比喻性语言(figurative language)、机械可解释性(mechanistic interpretability)、网格谜题(grid puzzles)、抽象与推理(abstraction and reasoning)、多模态融合(multimodal integration) 美观名称："MARC：隐喻抽象与推理语料库(Metaphor Abstraction and Reasoning Corpus)" 配置项： - 配置名称：tasks，数据文件：tasks/train.parquet - 配置名称：task_subsets，数据文件：task_subsets/train.parquet - 配置名称：descriptions，数据文件：descriptions/train.parquet - 配置名称：baseline，数据文件：baseline/train.parquet - 配置名称：figurative，数据文件：figurative/train.parquet 数据集信息： - 配置名称：tasks，字段： - task_id：32位整数，唯一任务标识符 - arc_name：字符串，原始ARC文件名（8位十六进制ID） - source：字符串，取值为'larc'或'marc-submission' - num_train：32位整数，训练示例数量 - see_description：字符串，输入可见结构说明 - do_description：字符串，转换规则说明 - grid_description：字符串，输出与输入维度关联说明 划分：训练集，样本数577 - 配置名称：task_subsets，字段： - task_id：32位整数，关联至tasks配置 - model_name：字符串，模型名称 - subset：字符串，分类类别 划分：训练集，样本数1188 - 配置名称：descriptions，字段： - fig_id：32位整数，唯一描述标识符 - task_id：32位整数，关联至tasks配置 - generator_model：字符串，生成模型或人工 - variant：字符串，提示变体 - source_domain：字符串，隐喻来源领域 - metaphor：字符串，隐喻概念 - figurative_see：字符串，比喻性所见说明 - figurative_do：字符串，比喻性操作说明 - figurative_grid：字符串，比喻性网格描述 划分：训练集，样本数1500 - 配置名称：baseline，字段： - trial_id：32位整数，唯一测试标识符 - task_id：32位整数，关联至tasks配置 - model_name：字符串，测试模型 - condition：字符串，测试条件 - num_examples：32位整数，训练示例数量 - correct：32位整数，是否正确 - cell_accuracy：32位浮点数，单元格匹配率 划分：训练集，样本数3952 - 配置名称：figurative，字段： - trial_id：32位整数，唯一测试标识符 - fig_id：32位整数，关联至descriptions配置 - task_id：32位整数，关联至tasks配置 - model_name：字符串，测试模型 - num_examples：32位整数，示例数量 - correct：32位整数，是否正确 - cell_accuracy：32位浮点数，单元格匹配率 - variant：字符串，提示变体 - source_domain：字符串，隐喻来源领域 划分：训练集，样本数8225 # MARC：隐喻抽象与推理语料库(Metaphor Abstraction and Reasoning Corpus) ## 数据集简介 MARC数据集旨在筛选出**比喻性语言与视觉示例真正互补**的谜题：仅依靠视觉示例时模型无法求解，仅依靠隐喻提示时模型同样无法求解，但当二者同时提供时模型即可成功求解。我们将这一特性称为**MARC特性**。本语料库包含78个经MARC验证的谜题、1230个跨领域多样化的比喻性描述，以及针对3个大语言模型(Large Language Model, LLM)的完整行为测试数据。 假设你正盯着一道网格谜题——由行与列中的彩色单元格构成，背后隐藏着某种规律。少量训练示例展示了输入网格与其对应正确输出的配对，但你始终无法洞悉其中的转换规则。此时有人给出一条提示："不妨将其视作一座花园，绿色单元格就像是正在蔓延、填满空白土壤的植物。"如果这条提示确实起到了作用，且单独依靠示例或隐喻均不足以让你解出谜题，那么这道谜题便符合MARC特性。 ## MARC特性 当同时满足以下三个条件时，某道谜题对于给定模型而言便符合MARC特性： 1. **仅依靠示例失败**：模型无法仅通过训练输入-输出配对求解该谜题。 2. **仅依靠比喻性描述失败**：模型无法仅通过隐喻提示求解该谜题。 3. **比喻性描述+示例成功**：当同时获得隐喻提示与若干训练示例时，模型可成功求解该谜题。 这一特性精准隔离出了语言与感知真正互补的场景——单一模态无法独立完成任务，但二者结合即可达成目标。 ## 数据来源 本语料库的数据来源于两个渠道： - **LARC任务**（任务ID 0–399）：400道ARC-AGI（Abstraction and Reasoning Corpus for Artificial General Intelligence）谜题（Chollet, 2019），搭配了LARC数据集（Acquaviva等人, 2022）中众包得到的字面描述。比喻性描述由Claude生成，并经过行为测试验证。 - **MARC提交任务**（任务ID 1000–1176）：177道手工制作的谜题，搭配人工撰写的比喻性提示。 本数据集未包含网格谜题本身（即实际的输入-输出网格配对），这些内容可从原始ARC-AGI仓库与LARC数据集中获取。任务ID与`arc_name`字段提供了对应的关联信息。 ## 数据集配置 ### `tasks`（577条数据） 任务元数据。对于LARC任务，`see_description`、`do_description`与`grid_description`字段包含该谜题的字面描述；对于MARC提交任务，这些字段则存储比喻性提示（无字面描述）。 | 列名 | 描述 | |------|------| | `task_id` | 唯一任务标识符（0–399为LARC任务；1000–1176为提交任务） | | `arc_name` | 原始ARC文件名（8位十六进制ID） | | `source` | 取值为`'larc'`或`'marc-submission'` | | `num_train` | 训练示例的数量（1–10） | | `see_description` | 输入中可见的结构说明 | | `do_description` | 转换规则说明 | | `grid_description` | 输出维度与输入维度的关联说明 | ### `task_subsets`（1188条数据） 基于基线测试表现，将每个任务按不同模型划分为四类。 | 列名 | 描述 | |------|------| | `task_id` | 关联至`tasks`配置 | | `model_name` | 该分类对应的模型名称 | | `subset` | 取值为`'examples_sufficient'`（仅示例足够）、`'language_sufficient'`（仅语言足够）、`'both_required'`（二者均需）或`'unsolvable'`（无法求解） | ### `descriptions`（1500条数据） 比喻性描述——即隐喻提示，为本数据集的核心贡献。每道经MARC验证的谜题均包含一条原始提示，以及最多24个跨领域多样化的替代提示。 | 列名 | 描述 | |------|------| | `fig_id` | 唯一描述标识符 | | `task_id` | 关联至`tasks`配置 | | `generator_model` | 取值为`'claude-agent'`（AI生成）或`'human'`（人工撰写） | | `variant` | 取值为`'original'`（原始提示）、`'alt-1'`、`'alt-2'`等（替代提示） | | `source_domain` | 隐喻来源领域，如`'biology'`（生物学）、`'warfare'`（战争）、`'cooking'`（烹饪）等（原始提示无此字段） | | `metaphor` | 单行隐喻概念 | | `figurative_see` | 比喻性的“所见之物”说明（提交任务无此字段） | | `figurative_do` | 比喻性的“操作指引”说明（提交任务无此字段） | | `figurative_grid` | 比喻性的网格描述（提交任务无此字段） | **领域覆盖范围**：12个核心领域（战争、生物学、烹饪、音乐、园艺、导航、舞蹈、戏剧、建筑、天文学、化学、气象），每个领域均覆盖全部78道经MARC验证的谜题；另有51个额外的小众领域，覆盖数量较少。 ### `baseline`（3952条数据） 三种无比喻性语言参与的基线测试结果。 | 列名 | 描述 | |------|------| | `trial_id` | 唯一测试标识符 | | `task_id` | 关联至`tasks`配置 | | `model_name` | 测试所用模型 | | `condition` | 测试条件：`'examples_only'`（仅示例）、`'language_only'`（仅语言）或`'both'`（二者兼具） | | `num_examples` | 提供的训练示例数量 | | `correct` | 1表示完全匹配，0表示错误 | | `cell_accuracy` | 单元格匹配率（取值范围0.0–1.0） | ### `figurative`（8225条数据） 带比喻性语言的测试结果。每一行对应一组（隐喻变体、模型、示例数量）的组合。 | 列名 | 描述 | |------|------| | `trial_id` | 唯一测试标识符 | | `fig_id` | 关联至`descriptions`配置 | | `task_id` | 关联至`tasks`配置 | | `model_name` | 测试所用模型 | | `num_examples` | 0表示仅使用比喻性语言，1–N表示搭配N个训练示例与比喻性语言 | | `correct` | 1表示完全匹配，0表示错误 | | `cell_accuracy` | 单元格匹配率 | | `variant` | 为方便使用，从`descriptions`配置中反规范化的字段 | | `source_domain` | 为方便使用，从`descriptions`配置中反规范化的字段 | ## 测试模型 | 模型 | 参数规模 | 架构 | 备注 | |------|----------|------|------| | gpt-oss-120b | 1200亿 | 混合专家模型(Mixture of Experts, MoE)，开源权重 | 主要验证模型 | | gpt-oss-20b | 210亿（活跃参数36亿） | 混合专家模型，开源权重 | 机械可解释性研究目标模型 | | qwen3.5-400b | 4000亿 | 稠密模型 | 仅参与基线测试（无比喻性语言测试） | 所有测试均使用温度系数（temperature）为0.0的设置。推理类模型（gpt-oss系列）采用两步流程：第一步进行推理，第二步提取结构化输出结果。 ## 关键统计数据 | | 数量 | |---|---:| | 总任务数 | 577 | | 经120b模型验证的MARC谜题 | 78 | | 比喻性描述总数 | 1500 | | — 原始提示 | 270 | | — 跨领域多样化替代提示 | 1230 | | 不同来源领域数 | 63 | | 覆盖全部78道谜题的核心领域数 | 12 | | 基线测试总次数 | 3952 | | 带比喻性语言的测试总次数 | 8225 | | 经120b模型验证的有效替代提示数 | 725/1230（占比59%） | ## 使用示例 python from datasets import load_dataset # 加载指定配置的数据集 tasks = load_dataset("bertybaums/marc", "tasks") descriptions = load_dataset("bertybaums/marc", "descriptions") figurative = load_dataset("bertybaums/marc", "figurative") # 获取所有生物学领域的有效替代提示 bio = [d for d in descriptions["train"] if d["source_domain"] == "biology"] # 筛选符合MARC特性的任务：即仅使用比喻性语言或仅使用示例均无法求解，但二者结合即可求解 import pandas as pd fig_df = figurative["train"].to_pandas() marc_valid = fig_df.groupby("fig_id").apply( lambda g: (g[g.num_examples == 0].correct == 0).all() and (g[g.num_examples > 0].correct == 1).any() ) ## 预期用途 - **机械可解释性研究**：大语言模型如何在内部将比喻性语言与视觉空间模式识别相结合？跨领域多样化的替代提示支持可控对比实验：同一谜题搭配不同隐喻。 - **比喻性语言理解研究**：哪些来源领域能够生成针对抽象推理任务的更有效隐喻？12个核心领域的因子设计支持系统性对比分析。 - **抽象与推理研究**：MARC特性精准识别出一种特定的失败模式（仅依靠示例无法求解）与一种特定的补救方案（比喻性提示辅助）。探究为何部分谜题可通过此类提示得到优化，而其余则不行。 - **缩放规律分析**：对比20亿参数与120亿参数模型的MARC特性有效率，可揭示比喻性推理能力如何随模型规模增长而提升。 ## 局限性 - 除177道MARC提交任务外，其余比喻性描述均由Claude生成，因此隐喻可能反映出Claude在将网格操作映射至概念领域时存在的固有偏差。 - 所有测试均使用温度系数为0.0的设置，但推理类模型仍可能在相同提示下出现微小的非确定性输出差异。 - 本数据集未包含网格谜题本身（输入-输出配对），研究人员需获取原始ARC/LARC数据集才能查看隐喻所描述的具体内容。 - MARC特性验证仅针对gpt-oss-120b模型完成。对于某一模型而言符合MARC特性的隐喻，未必适用于其他模型（反之亦然）。 - 为保持数据集紧凑性，本版本未包含完整提示文本。需要完整提示的研究人员可通过任务数据与描述字段自行重构，或联系作者获取。 ## 引用 若您使用本数据集，请引用以下文献： bibtex @misc{baumgaertner2026marc, title={MARC: Metaphor Abstraction and Reasoning Corpus}, author={Baumgaertner, Bert}, year={2026}, url={https://huggingface.co/datasets/bertybaums/marc} } ## 致谢 网格谜题来源于ARC-AGI（Chollet, 2019）与LARC（Acquaviva等人, 2022）数据集。比喻性描述通过Claude（Anthropic公司）生成。行为测试在爱达荷大学MindRouter基础设施上完成。 ## 许可协议 本数据集采用[CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)协议发布。底层ARC谜题采用Apache 2.0协议许可；LARC描述采用CC-BY-4.0协议许可。