Hugging-zheng/SciJudgeBench

---
language:
- en
license: apache-2.0
task_categories:
- text-classification
tags:
- scientific-evaluation
- citation-prediction
- preference-learning
- arxiv
configs:
- config_name: default
data_files:
- split: train
path: train.jsonl
- split: test
path: test.jsonl
- split: test_ood_iclr
path: test_ood_iclr.jsonl
- split: test_ood_year
path: test_ood_year.jsonl
size_categories:
- 100K<n<1M
---

# SciJudge Dataset

Training and evaluation data for scientific paper citation prediction, from the paper **[AI Can Learn Scientific Taste](https://arxiv.org/abs/2603.14473)**.

Given two academic papers (title, abstract, publication date), the task is to predict which paper has a higher citation count.

## Dataset Splits

| Split | Examples | Description |
|-------|----------|-------------|
| `train` | 720,341 | Training preference pairs from arXiv papers |
| `test` | 8,830 | Main evaluation set (880 valid pairs × ~10 prompt variations) |
| `test_ood_iclr` | 611 | Out-of-distribution evaluation on ICLR papers |
| `test_ood_year` | 514 | Out-of-distribution evaluation on recent (2025) papers |

## Data Format

Each example is a JSON object with the following fields:

### Core Fields (all splits)

| Field | Type | Description |
|-------|------|-------------|
| `messages` | list | Conversation in chat format (`system`, `user` roles) |
| `correct_answer` | str | Ground truth: `"A"` or `"B"` |
| `paper_a_citations` | int | Citation count of Paper A |
| `paper_b_citations` | int | Citation count of Paper B |
| `paper_a_title` | str | Title of Paper A |
| `paper_b_title` | str | Title of Paper B |
| `paper_a_date` | str | Publication date of Paper A (YYYY-MM-DD) |
| `paper_b_date` | str | Publication date of Paper B (YYYY-MM-DD) |
| `paper_a_category` | str | Primary category (e.g., "Physics", "Computer Science") |
| `paper_b_category` | str | Primary category |
| `paper_a_subcategory` | str | arXiv subcategory (e.g., "cs.CL cs.LG") |
| `paper_b_subcategory` | str | arXiv subcategory |
| `paper_a_arxiv_id` | str | arXiv paper ID |
| `paper_b_arxiv_id` | str | arXiv paper ID |
| `paper_a_abstract` | str | Abstract of Paper A |
| `paper_b_abstract` | str | Abstract of Paper B |

### Additional Fields in `test_ood_iclr`

| Field | Type | Description |
|-------|------|-------------|
| `paper_a_rating` | float | ICLR review rating of Paper A |
| `paper_b_rating` | float | ICLR review rating of Paper B |
| `year` | int | ICLR submission year |

## Usage

```python
from datasets import load_dataset

dataset = load_dataset("OpenMOSS-Team/SciJudgeBench")

# Access splits
train = dataset["train"]
test = dataset["test"]
test_iclr = dataset["test_ood_iclr"]
test_year = dataset["test_ood_year"]
```

## Citation

```bibtex
@article{scijudge2025,
title={AI Can Learn Scientific Taste},
year={2025}
}
```