Rose-STL-Lab/Zephyrus

---
pretty_name: ZephyrusBench
license: mit
language:
- en
task_categories:
- question-answering
- text-generation
tags:
- weather
- climate
- earth-science
- benchmark
- agent
- reasoning
size_categories:
- 1K<n<10K
---

# ZephyrusBench

[](https://arxiv.org/abs/2510.04017)
[](https://iclr.cc/virtual/2026/poster/10008692)
[](https://github.com/Rose-STL-Lab/Zephyrus)
[](https://huggingface.co/datasets/Rose-STL-Lab/Zephyrus)

ZephyrusBench is a weather-science benchmark released with the paper *Zephyrus: An Agentic Framework for Weather Science*. It contains **2,230** question-answer pairs across **49** tasks spanning geospatial reasoning, temporal reasoning, forecasting, simulation, climatology, and scientific question answering.

Accepted at the **International Conference on Learning Representations, 2026**.

**Paper and Resources**

- Paper: [arXiv](https://arxiv.org/abs/2510.04017)
- Poster: [ICLR 2026 Poster](https://iclr.cc/virtual/2026/poster/10008692)
- Code: [Rose-STL-Lab/Zephyrus](https://github.com/Rose-STL-Lab/Zephyrus)
- Dataset: [Rose-STL-Lab/Zephyrus on Hugging Face](https://huggingface.co/datasets/Rose-STL-Lab/Zephyrus)

## Dataset Summary

- Domain: weather science and climate-related reasoning
- Language: English
- Format: one or more JSON files
- Primary use: benchmarking inference and evaluation pipelines in the Zephyrus codebase

The full Zephyrus stack pairs the benchmark with:

- WeatherBench 2 data access
- Natural Earth geolocation utilities
- forecasting tools
- simulation tools
- climatology tools

## Usage

### Use With the Zephyrus Repository

1. Setup the Code Execution Server.
2. Download the dataset files from this Hugging Face repo.
3. Place them in a local directory.
4. Point `dataset_path` in `configs/paths/default.yaml` to that directory.
5. Run inference or evaluation from the Zephyrus repository.

For full setup instructions, including WeatherBench 2, Natural Earth, caches, and model/server configuration, see the project [README](https://github.com/Rose-STL-Lab/Zephyrus).

## Citation

If you use ZephyrusBench, please cite:

```bibtex
@inproceedings{
varambally2026zephyrus,
title={Zephyrus: An Agentic Framework for Weather Science},
author={Sumanth Varambally and Marshall Fisher and Jas Thakker and Yiwei Chen and Zhirui Xia and Yasaman Jafari and Ruijia Niu and Manas Jain and Veeramakali Vignesh Manivannan and Zachary Novack and Luyu Han and Srikar Eranky and Salva R{\"u}hling Cachay and Taylor Berg-Kirkpatrick and Duncan Watson-Parris and Yian Ma and Rose Yu},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=aVeaNahsID}
}
```