opencompass/ATLAS

--- annotations_creators: - expert-generated - found language_creators: - expert-generated - found language: - en - zh license: cc-by-sa-4.0 multilinguality: - multilingual size_categories: - n<1K source_datasets: - original pretty_name: ATLAS configs: - config_name: default data_files: - split: test path: atlas_test_pub.jsonl - split: val path: atlas_val.jsonl --- <div align="center"> [](https://creativecommons.org/licenses/by-nc-sa/4.0/) [](https://arxiv.org/abs/2511.14366) [](https://huggingface.co/datasets/opencompass/ATLAS) </div> --- # ATLAS: A High-Difficulty, Multidisciplinary Benchmark for Frontier Scientific Reasoning ## 📊 Overview **ATLAS (AGI-Oriented Testbed for Logical Application in Science)** is a high-difficulty, multidisciplinary benchmark designed to evaluate frontier scientific reasoning capabilities of Large Language Models (LLMs). As existing benchmarks show saturated performance, ATLAS provides a reliable measuring stick for progress towards Artificial General Intelligence. ### 🌟 Key Features - **🎯 800+ Original High-Quality Questions**: All questions are newly created or significantly adapted to prevent data contamination - **🔬 7 Core Scientific Domains**: Mathematics, Physics, Chemistry, Biology, Computer Science, Earth Science, and Materials Science - **🏛️ 25+ Leading Institutions**: Contributed by PhD-level experts from top universities and research institutions - **💎 High-Fidelity Answers**: Complex, open-ended answers involving multi-step reasoning and LaTeX expressions - **🛡️ Contamination-Resistant**: Rigorous quality control with multi-round expert peer review and adversarial testing ### 📈 Leaderboard Highlights Latest results evaluated with **OpenAI-o4-mini** as judge (Public Validation Set): | Rank | Model | Organization | Accuracy (Avg) | |------|-------|--------------|----------------| | 1 | OpenAI GPT-5-High | OpenAI | 42.9% | | 2 | Gemini-2.5-Pro | Google | 35.3% | | 3 | Grok-4 | xAI | 34.1% | | 4 | OpenAI o3-High | OpenAI | 33.8% | | 5 | DeepSeek-R1-0528 | DeepSeek AI | 26.4% | > 📝 **Note**: Results show that even the most advanced models struggle with ATLAS, demonstrating its effectiveness as a frontier benchmark. For Test set complete submission: [ATLAS Test Submission](https://huggingface.co/spaces/opencompass/ATLAS) --- ## 🚀 Quick Start ### Installation ```bash pip install opencompass ``` ### Load Dataset ```python from datasets import load_dataset # Load ATLAS dataset dataset = load_dataset("opencompass/ATLAS") # Access validation split val_data = dataset['val'] print(f"Validation samples: {len(val_data)}") # Access test split (for inference only) test_data = dataset['test'] ``` ### Basic Evaluation Setup ```python from mmengine.config import read_base with read_base(): from opencompass.configs.datasets.atlas.atlas_gen import atlas_datasets # Update your judge model information atlas_datasets[0]["eval_cfg"]["evaluator"]["judge_cfg"]["judgers"][0].update(dict( abbr="YOUR_MODEL_ABBR", openai_api_base="YOUR_API_URL", path="YOUR_MODEL_PATH", key="YOUR_API_KEY", # tokenizer_path="o3", # Optional: update if using a different model )) ``` ### Evaluate on Test Split ```python from mmengine.config import read_base with read_base(): from opencompass.configs.datasets.atlas.atlas_gen import atlas_datasets # Configure for test split atlas_datasets[0]["abbr"] = "atlas-test" atlas_datasets[0]["split"] = "test" atlas_datasets[0]["eval_cfg"]["evaluator"]["dataset_cfg"]["abbr"] = "atlas-test" atlas_datasets[0]["eval_cfg"]["evaluator"]["dataset_cfg"]["split"] = "test" ``` > ⚠️ **Important**: The test split is only supported for inference mode. Use `-m infer` flag when running OpenCompass. ### Run Evaluation ```bash # Evaluate on validation set python run.py configs/eval_atlas.py # Evaluate on test set (inference only) python run.py configs/eval_atlas.py -m infer ``` --- ## 📚 Dataset Structure ### Data Fields - `subject_name`: Subject name in English (e.g., "Biology", "Physics") - `question`: The scientific question/problem statement - `answer_ideas`: Reasoning ideas and approaches for solving the problem - `refined_standard_answer`: List of standard answers (may contain multiple sub-answers) - `sub_subject_name`: Specific sub-discipline (e.g., "Molecular Biology", "Quantum Mechanics") ### Data Splits | Split | Count | Purpose | |-------|-------|---------| | **Validation** | ~300 | Public evaluation, reproducible results | | **Test** | ~500 | Hidden evaluation, contamination-resistant | ### Example Data Point ```json { "subject_name": "Biology", "question": "Explain how CRISPR-Cas9 gene editing works at the molecular level...", "answer_ideas": "[\"Cas9 protein binds to guide RNA...\"]", "refined_standard_answer": [ "1. Guide RNA (gRNA) directs Cas9 to target DNA sequence...", "2. Cas9 creates double-strand break...", "3. Cell repairs through NHEJ or HDR pathways..." ], "sub_subject_name": "Molecular Biology and Biotechnology" } ``` --- ## 🎯 Evaluation Protocol ATLAS uses an **LLM-as-Judge** evaluation framework with the following characteristics: ### Judge Model - Default: **OpenAI-o4-mini** (for leaderboard consistency) - Customizable: You can use your own judge model ### Evaluation Process 1. **Model Inference**: Generate answers in structured JSON format 2. **Answer Extraction**: Parse final answers from model outputs 3. **LLM Judging**: Compare candidate answers with standard answers 4. **Scoring**: Calculate accuracy and pass@k metrics ### Answer Format Models should output answers in the following JSON format: ```json { "answers": [ "answer to sub-question 1", "answer to sub-question 2", ... ] } ``` ### Evaluation Metrics - **Accuracy (Avg)**: Average correctness across all questions - **mG-Pass@2**: Majority voting accuracy with 2 samples - **mG-Pass@4**: Majority voting accuracy with 4 samples --- ## 📜 Citation If you use ATLAS in your research, please cite: ```bibtex @misc{liu2025atlashighdifficultymultidisciplinarybenchmark, title={ATLAS: A High-Difficulty, Multidisciplinary Benchmark for Frontier Scientific Reasoning}, author={Hongwei Liu and Junnan Liu and Shudong Liu and Haodong Duan and Yuqiang Li and Mao Su and Xiaohong Liu and Guangtao Zhai and Xinyu Fang and Qianhong Ma and Taolin Zhang and Zihan Ma and Yufeng Zhao and Peiheng Zhou and Linchen Xiao and Wenlong Zhang and Shijie Zhou and Xingjian Ma and Siqi Sun and Jiaye Ge and Meng Li and Yuhong Liu and Jianxin Dong and Jiaying Li and Hui Wu and Hanwen Liang and Jintai Lin and Yanting Wang and Jie Dong and Tong Zhu and Tianfan Fu and Conghui He and Qi Zhang and Songyang Zhang and Lei Bai and Kai Chen}, year={2025}, eprint={2511.14366}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2511.14366}, } ```

搜集汇总

数据集介绍

构建方式

ATLAS（AGI-Oriented Testbed for Logical Application in Science）是一个面向大语言模型前沿科学推理能力的高难度、多学科基准数据集。其构建过程由来自全球超过25所顶尖高校及科研机构的博士级专家主导，所有题目均为全新创作或经过显著改编，以杜绝数据污染问题。数据集包含约800道高质量题目，覆盖数学、物理、化学、生物学、计算机科学、地球科学与材料科学七大核心科学领域。每道题目均配有复杂的开放性答案，涉及多步推理与LaTeX表达式表达，并经过多轮专家同行评审与对抗性测试，确保答案的高保真度与可靠性。数据集分为公开验证集（约300题）与隐藏测试集（约500题），分别用于可重复评估与防污染评测。

使用方法

使用ATLAS数据集时，用户可通过Hugging Face Datasets库直接加载，命令为`load_dataset("opencompass/ATLAS")`，获取验证集与测试集。评估需借助OpenCompass框架，安装后通过配置文件导入`atlas_datasets`，并设置裁判模型（如OpenAI API）的参数。对于测试集，需配置为推理模式并使用`-m infer`标志运行。模型需按指定JSON格式输出答案，包含多个子答案的列表。评估过程涵盖答案提取、LLM判分与指标计算，支持自定义裁判模型，但排行榜默认使用OpenAI-o4-mini以保证一致性。详细配置示例与运行命令在官方文档中已完整提供。

背景与挑战

背景概述

在大型语言模型（LLM）能力日益逼近饱和的当下，如何精准度量其在科学推理这一高阶智力活动上的真正进展，成为了人工智能领域亟待突破的瓶颈。为此，由来自全球超过25所顶尖高校与科研机构的博士级专家联合构建的ATLAS（AGI-Oriented Testbed for Logical Application in Science）基准测试应运而生。该数据集于2025年正式发布，其核心研究问题聚焦于评估LLM在多学科、高难度前沿科学问题上的逻辑推演与知识整合能力。ATLAS横跨数学、物理、化学、生物学、计算机科学、地球科学与材料科学七大核心领域，涵盖800余道原创或深度改编的高保真题目，旨在为迈向通用人工智能（AGI）提供一把可靠且极具挑战性的标尺，其发布迅速引起了领域内对现有模型极限能力的重新审视。

当前挑战

ATLAS所面临的挑战首先源于其要解决的领域根本问题：现有基准测试的性能已趋饱和，亟需一个能区分顶尖模型细微能力差异的“试金石”，而设计这样一套既避免数据污染又具备高区分度的跨学科难题本身便是一项艰巨任务。在构建过程中，团队遭遇了多重困难：其一，确保800多道题目均为原创或显著改编，以根除预训练数据中的记忆效应，这对问题来源与创新性提出了极高要求。其二，实现跨学科知识的深度融合与答案的高保真度，题目往往需要多步推理并嵌入复杂的LaTeX表达式，对专家出题与审核的严谨性构成严峻考验。其三，通过多轮同行评审与对抗性测试来实施严格的质量控制，以确保证据链的完整性与评分标准的客观性，这一过程耗费了大量的人力与时间成本。

常用场景

经典使用场景

ATLAS数据集的核心应用在于作为评估前沿科学推理能力的标杆。该基准覆盖数学、物理、化学、生物学、计算机科学、地球科学与材料科学七大核心领域，包含800余道由顶尖博士级专家原创或深度改编的高难度题目。研究者常通过ATLAS验证大规模语言模型在多学科、多步骤推理任务中的真实表现，例如在跨领域科学问题求解中检验模型对复杂逻辑链条的驾驭能力。其高保真答案与抗污染设计确保了评测结果的可靠性与区分度，成为衡量模型向通用人工智能迈进的关键试金石。

解决学术问题

ATLAS致力于解决现有科学推理基准性能饱和、易受数据污染等核心学术难题。传统评测集如MMLU或GSM8K已难以区分顶尖模型间的能力差异，而ATLAS通过引入原创性高难度问题与严格的专家同行评审机制，为学术界提供了一个更具挑战性与鲁棒性的评测工具。该数据集揭示了大模型在深层科学推理中的局限性，推动了关于模型泛化能力、逻辑严谨性及跨学科知识整合的深入研究，为理解当前人工智能的认知边界与演进方向提供了至关重要的实证基础。

实际应用

在实际应用中，ATLAS为人工智能研发机构与科研社群提供了模型迭代优化的明确指南。研发团队可依据模型在ATLAS上的细分学科表现，精准定位其在数学推导、物理建模或生物机制理解等维度的薄弱环节，从而有针对性地改进训练策略、数据配比或推理架构。此外，该基准也被高等教育机构用于评估AI辅助科研工具的可靠性，例如在自动化文献分析、实验方案设计或复杂科学假说生成等场景中筛选具备真实推理能力的系统，推动人工智能从辅助工具向科学发现协作者的范式跃迁。

数据集最近研究

最新研究方向

在当前大规模语言模型（LLM）能力评测趋于饱和的背景下，ATLAS基准数据集应运而生，聚焦于高阶科学推理能力的评估，填补了现有基准在难度与学科广度上的空白。该数据集涵盖数学、物理学、化学、生物学等七大核心科学领域，由全球25余家顶尖机构的博士级专家精心设计，包含800余道原创高难度题目，并通过多轮专家评审与对抗测试确保抗污染性。前沿研究方向紧密围绕LLM在跨学科复杂推理、多步逻辑推导及开放性科学问题解答上的真实能力极限，尤其关注如OpenAI GPT-5-High和Gemini-2.5-Pro等顶级模型在ATLAS上仅达42.9%和35.3%准确率的显著瓶颈。这一成果不仅揭示了当前AI系统在科学推理上的根本局限，也为迈向通用人工智能（AGI）提供了更具挑战性的评估标尺，推动了科学智能评测范式的革新。