meituan/DiningBench

---
license: cc-by-nc-nd-4.0
task_categories:
- image-classification
- image-text-to-text
language:
- en
- zh
tags:
- food
- nutrition
- vqa
---

# DiningBench

[**Paper**](https://huggingface.co/papers/2604.10425) | [**Github**](https://github.com/meituan/DiningBench)

This directory contains **DiningBench** benchmark assets: **JSON Lines** annotations (six `*.jsonl` files for three tasks, Chinese and English-translated pairs), and an image archive **`images.tar.gz`** (after extraction, a **`images/`** tree matching paths referenced in the JSONL). DiningBench targets fine-grained food classification, nutrition estimation, and visual question answering (VQA). Official evaluation scripts and instructions are in the GitHub repository [meituan/DiningBench](https://github.com/meituan/DiningBench) (ACL 2026 Main, paper companion code).

<table style="border: none; width: 75%;">
<tr>
<td style="border: none; width: 33%; padding: 5px;">
<img src="https://cdn-uploads.huggingface.co/production/uploads/6911df7ac770065d9869b7f5/g5rT8fngDqDwtvv_zMsSA.png" width="100%" />
</td>
<td style="border: none; width: 33%; padding: 5px;">
<img src="https://cdn-uploads.huggingface.co/production/uploads/6911df7ac770065d9869b7f5/gpZpSvpO-VZ-4W6a-VIzX.png" width="100%" />
</td>
<td style="border: none; width: 33%; padding: 5px;">
<img src="https://cdn-uploads.huggingface.co/production/uploads/6911df7ac770065d9869b7f5/9Dg1ybZt_SVQaFle3G05-.png" width="100%" />
</td>
</tr>
</table>

## Files

| File | Lines (samples) | Task |
|------|-----------------|------|
| `classification.jsonl` | 2884 | Fine-grained classification (multiple choice) |
| `classification_en.jsonl` | 2884 | Same task (English translation) |
| `nutrition.jsonl` | 1650 | Nutrition estimation |
| `nutrition_en.jsonl` | 1650 | Same task (English translation) |
| `vqa.jsonl` | 804 | Visual question answering |
| `vqa_en.jsonl` | 804 | Same task (English translation) |
| `images.tar.gz` | — | Images (extract to `images/`; see below) |

`*_en.jsonl` is the English translation of the matching file without `_en`.

## Images

Images are shipped in **`images.tar.gz`**. **Extract** it before use; you should obtain an **`images/`** directory consistent with JSONL paths (e.g. `images/02180.jpg`). Run evaluation from a working directory where those relative paths resolve, or place the `images/` tree accordingly.

## JSON field overview by task

### Classification (`classification*.jsonl`)

- `id`: sample id
- `standard_image`: standard / reference image path
- `user_images`: list of user-side multi-view paths (may be empty)
- `options`: candidate dish names
- `prompt`: model prompt (expects a choice letter)
- `ground_truth`: correct option letter (e.g. `"E"`)

### Nutrition estimation (`nutrition*.jsonl`)

- `id`, `standard_image`, `user_images`, `prompt`: same as above
- `ground_truth`: object; typically includes dish name and nutrients.(aligned with the JSON schema requested in `prompt`)

### VQA (`vqa*.jsonl`)

- `id`: sample id
- `images`: image paths for this item (single or multiple)
- `prompt`: question and answer-format instructions
- `ground_truth`: reference short answer
- `reasoning`: reference reasoning (scoring usually uses `ground_truth`)
- `vqa_detail_info`: extra metadata (category, question, answer, visual cues, etc.; structure varies by item)

## Evaluation

Inference and metric scripts live in [https://github.com/meituan/DiningBench](https://github.com/meituan/DiningBench), mainly `eval_classification.py`, `eval_nutrition.py`, and `eval_vqa.py`. Pass JSONL files here as `--test_jsonl_path` (and related flags), and configure your model API or vLLM. Environment setup and examples are in that repo’s `README.md`.

## Citation

If you use DiningBench data or results in a publication, cite the following paper:

```bibtex
@misc{jin2024diningbench,
title={DiningBench: A Hierarchical Multi-view Benchmark for Perception and Reasoning in the Dietary Domain},
author={Song Jin and Juntian Zhang and Xun Zhang and Zeying Tian and Fei Jiang and Guojun Yin and Wei Lin and Yong Liu and Rui Yan},
year={2024},
eprint={2604.10425},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
```