omega-transformative

# Transformative Math Problems

This dataset contains transformative mathematical problem settings in paper "OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization" that test the most challenging form of generalization: the ability to abandon familiar but ineffective strategies in favor of qualitatively different and more efficient approaches.

## Overview

Transformative generalization presents the greatest challenge in mathematical reasoning evaluation. These tasks go beyond simple extension or composition, requiring a "jump out of the box"—a creative reframing or redescription that bypasses the limitations of standard reasoning tactics. Models must discard familiar yet ineffective strategies and develop qualitatively different approaches to solve problems.

## Quick Start

```python
from datasets import load_dataset

# Load all transformative settings
dataset = load_dataset("sunyiyou/omega-transformative")

# Load a specific transformative setting with train/test splits
matrix_data = load_dataset("sunyiyou/omega-transformative", "trans_matrix_rank")
train_data = matrix_data["train"] # Training problems from basic domain
test_data = matrix_data["test"] # Transformative problems requiring advanced techniques

# Load just the training or test split
train_only = load_dataset("sunyiyou/omega-transformative", "trans_matrix_rank", split="train")
test_only = load_dataset("sunyiyou/omega-transformative", "trans_matrix_rank", split="test")
```

## Dataset Description

Each transformative setting combines training data from a specific mathematical domain with test problems that require advanced techniques beyond the standard approaches taught in the training data. The training problems provide foundational knowledge, while test problems challenge models to develop novel solution strategies.

## Citation

If you use this dataset, please cite the original work:

```bibtex
@article{sun2024omega,
title = {OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization},
author = {Yiyou Sun and Shawn Hu and Georgia Zhou and Ken Zheng and Hannaneh Hajishirzi and Nouha Dziri and Dawn Song},
journal = {arXiv preprint arXiv:2506.18880},
year = {2024},
}
```

## Related Resources

- **Explorative Dataset**: See [omega-explorative](https://huggingface.co/datasets/sunyiyou/omega-explorative) for explorative reasoning challenges
- **Compositional Dataset**: See [omega-compositional](https://huggingface.co/datasets/sunyiyou/omega-compositional) for compositional reasoning challenges
- **Paper**: See the full details in [paper](https://arxiv.org/pdf/2506.18880)
- **Code Repository**: See generation code on [github](https://github.com/sunblaze-ucb/math_ood)

# 变革性数学问题集

本数据集源自论文《OMEGA：大语言模型（Large Language Model）能否在数学领域跳出固有思维？评估探索性、组合性与变革性泛化能力》中的变革性数学问题设定，用于检验难度最高的泛化能力：即摒弃熟悉却无效的策略，转而采用本质不同且更高效的方法的能力。

## 概述

变革性泛化是数学推理评估中最具挑战性的任务。此类任务超越了简单的扩展或组合，需要“跳出固有框架”——即通过创造性的重构或重新描述，规避标准推理策略的局限。模型必须舍弃熟悉却无效的思路，转而构建本质迥异的解决方案。

## 快速上手

python
from datasets import load_dataset

# 加载全部变革性问题设定
dataset = load_dataset("sunyiyou/omega-transformative")

# 加载包含训练/测试划分的特定变革性问题设定
matrix_data = load_dataset("sunyiyou/omega-transformative", "trans_matrix_rank")
train_data = matrix_data["train"] # 基础领域的训练问题
test_data = matrix_data["test"] # 需采用进阶技巧的变革性测试问题

# 仅加载训练或测试划分
train_only = load_dataset("sunyiyou/omega-transformative", "trans_matrix_rank", split="train")
test_only = load_dataset("sunyiyou/omega-transformative", "trans_matrix_rank", split="test")

## 数据集说明

每个变革性问题设定均结合了特定数学领域的训练数据，以及训练数据中未讲授、需采用进阶技巧才能解决的测试问题。训练问题用于提供基础知识，而测试问题则要求模型开发全新的解题策略。

## 引用

若使用本数据集，请引用如下原始文献：

bibtex
@article{sun2024omega,
title = {OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization},
author = {Yiyou Sun and Shawn Hu and Georgia Zhou and Ken Zheng and Hannaneh Hajishirzi and Nouha Dziri and Dawn Song},
journal = {arXiv preprint arXiv:2506.18880},
year = {2024},
}

## 相关资源

- **探索性数据集**：如需探索性推理挑战数据集，请访问 [omega-explorative](https://huggingface.co/datasets/sunyiyou/omega-explorative)
- **组合性数据集**：如需组合性推理挑战数据集，请访问 [omega-compositional](https://huggingface.co/datasets/sunyiyou/omega-compositional)
- **论文原文**：详细内容请查阅 [论文](https://arxiv.org/pdf/2506.18880)
- **代码仓库**：数据集生成代码可访问 [GitHub](https://github.com/sunblaze-ucb/math_ood)