Evaluating Large Language Models on Multimodal Chemistry Olympiad Exams

Overview

This repository contains resources for our paper "Evaluating large language models on multimodal chemistry olympiad exams" published at Communications Chemistry.

Paper link: https://www.nature.com/articles/s42004-025-01782-x

Abstract

Multimodal scientific reasoning remains a significant challenge for large language models (LLMs), particularly in chemistry, where problem-solving relies on symbolic diagrams, molecular structures, and structured visual data. Here, we systematically evaluate 40 proprietary and open-source multimodal LLMs, including GPT-5, o3, Gemini-2.5-Pro, and Qwen2.5-VL, on a curated benchmark of Olympiad-style chemistry questions drawn from over two decades of U.S. National Chemistry Olympiad (USNCO) exams. These questions require integrated visual and textual reasoning across diverse modalities. We find that many models struggle with modality fusion, where in some cases, removing the image even improves accuracy, indicating misalignment in vision-language integration. Chain-of-Thought prompting consistently enhances both accuracy and visual grounding, as demonstrated through ablation studies and occlusion-based interpretability. Our results reveal critical limitations in the scientific reasoning abilities of current MLLMs, providing actionable strategies for developing more robust and interpretable multimodal systems in chemistry. This work provides a timely benchmark for measuring progress in domain-specific multimodal AI and underscores the need for further advances at the intersection of artificial intelligence and scientific reasoning.

README

This repository contains the metadata and evaluation script for our USNCO‑V multimodal chemistry benchmark.

No copyrighted text or images are distributed—readers manually crop diagrams from publicly available USNCO PDFs.

Repository structure
.

├── usnco-v_local.json # metadata for all 204 local questions (including 2025 exam)

├── usnco-v_national.json # metadata for all 269 national questions (including 2025 exam)

├── image-local/ # folder for all local question images (not included)

├── image-national/ # folder for all national question images (not included)

├── sample_output/ # sample model output files for evaluation

│ ├── predictions-local.json

│ └── predictions-national.json

├── evaluate.py # evaluation script

├── generate_qwen25vl.py # sample script to generate predictions using Qwen2.5-VL

└── README.md # this file

Metadata format

usnco_v_meta.json is a JSON array; each entry has:

- qid (string): unique question ID (e.g. "2025-12")

- pdf (string): local PDF filename

- pdf_url (string): URL to USNCO PDF

- year (int): year of the exam

- type (string): "local" or "national"

- original_qid (int): question number printed in the margin

- image_id (string): corresponding image name (without .png) in `image-local` or `image-national` folder

- image_type (string): one of five categories ("table", "chart", …)

- answer (string): correct option letter ("A", "B", "C", "D")

- question (string): empty (placeholder)

- choices (string): empty (placeholder)

- summary (string): brief summary of the question and choices

- category (dict): knowledge categories in three hierarchies, including "main_category", "subcategory", and "skill_or_focus"

- human_rate (float): percentage of students who answered correctly (0-100), only available for national questions in certain years

Prerequisites

1. **Python 3.10+**

2. **Dependencies**

pip install -U transformers

pip install huggingface_hub hf-transfer

pip install accelerate bitsandbytes peft datasets

pip install flash-attn --no-build-isolation

pip install prettytable

How to obtain each diagram image

For each metadata entry:

1. Download the PDF

wget "<pdf_url>" -O "<pdf>"

2. Open <pdf> in your PDF viewer.

3. Locate the question block labeled `original_qid` in the left margin.

4. Select the diagram (and its caption, if any) using the viewer’s rectangular‐selection tool.

5. Copy or export the selection as a PNG image, and save it as `{image_id}.png` in the corresponding folder (`image-local` or `image-national`)

6. Repeat for all questions. (Approximately 1-2 hours total for 473 questions.)

Running thep prediction script

We provide a sample script `generate_qwenvl25.py` to generate predictions using the Qwen2.5-VL model.

To run the script, use:

python generate_qwenvl25.py --model_dir qwen2.5-vl-3b-inst --data_dir . --mode local

where,

- `--model_dir`: directory of the Qwen2.5-VL model

- `--data_dir`: directory containing either `usnco-v_local.json` and `image-local/`, or `usnco-v_national.json` and `image-national/`. Note that you must curate complete dataset before running the script (cannot run with metadata only).

- `--mode`: either `local` or `national`, indicating which set of questions to use.

Running the evaluation script

Your model should output a JSON file mapping each qid to its predicted option letters.

Note that to allow calculate pass@5, consistency@5, multiple predictions should be stored as a list in `prediction`.

For example:

[

{

"qid": "2025-12",

"answer": "C",

"prediction": ["C", "A", "C", "C", "C"]

},

{

"qid": "2025-20",

"answer": "A",

"prediction": ["A", "A", "B", "A", "A"]

},

...

]

A sample output file is provided in `sample_output/predictions-*.json`.

To run the evaluation, use:

python evaluate.py sample_output/predictions-local.json