RaR-Science

## Dataset Summary **RaR-Science** is a dataset curated for training and evaluating language models on science domain using structured rubric-based supervision. Each example includes a science related question, a reference answer, and checklist-style rubric annotations generated via OpenAI's o3-mini. This dataset is introduced in ****[*Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains*](https://arxiv.org/abs/2507.17746)****. ## Dataset Structure ### Data Fields Each example contains: - `question`: the open-ended medical question - `reference_answer`: high-quality expert reference response - `question_source`: source of the original question - `rubric_list`: list of rubric criteria used to evaluate the model response - `rubric`: dictionary mapping each rubric criterion to a score - `rubric_count`: number of rubric criteria used ## Usage ```python from datasets import load_dataset ds = load_dataset("ScaleAI/RaR-Science") ``` ## Citation If you use this dataset, please cite: ``` @article{gunjal2025rubrics, title={Rubrics as rewards: Reinforcement learning beyond verifiable domains}, author={Gunjal, Anisha and Wang, Anthony and Lau, Elaine and Nath, Vaskar and He, Yunzhong and Liu, Bing and Hendryx, Sean}, journal={arXiv preprint arXiv:2507.17746}, year={2025} } ```