PEACE

# PEACE: Empowering Geologic Map Holistic Understanding with MLLMs

[[`Code`](https://github.com/microsoft/PEACE)] [[`Paper`](https://arxiv.org/pdf/2501.06184)] [[`Data`](https://huggingface.co/datasets/microsoft/PEACE)]

<p align="left">
<img src="./images/GeoMap-Bench_Pipeline.png" width="800">
</p>

## Introduction
We construct a geologic map benchmark, GeoMap-Bench, to evaluate the performance of MLLMs on geologic map understanding across different abilities, the overview of it is as shown in below Table.

<table>
<thead>
<tr>
<th style="text-align:left;" >Property</th>
<th style="text-align:left;" >Description</th>
</tr>
</thead>
<tbody>
<tr>
<th style="text-align:left;" rowspan="2"><strong>Source</strong></th>
<th style="text-align:left;" ><strong>USGS(English)</strong></th>
</tr>
<tr>
<th style="text-align:left;" ><strong>CGS(Chinese)</strong></th>
</tr>
<tr>
<th style="text-align:left;" ><strong>Content</strong></th>
<th style="text-align:left;" ><strong>Image-question pair with annotated answer</strong></th>
</tr>
<tr>
<th style="text-align:left;" ><strong>Scale</strong></th>
<th style="text-align:left;" ><strong>124 images and 3,864 questions</strong></th>
</tr>
<tr>
<th style="text-align:left;" ><strong>Resolution</strong></th>
<th style="text-align:left;" ><strong>6,146² pixels on average</strong></th>
</tr>
<tr>
<th style="text-align:left;" rowspan="3"><strong>Question Type</strong></th>
<th style="text-align:left;" ><strong>1.Multiple-choicequestion</strong></th>
</tr>
<tr>
<th style="text-align:left;" ><strong>2.Fill-in-the-blankquestion</strong></th>
</tr>
<tr>
<th style="text-align:left;" ><strong>3.Essayquestion</strong></th>
</tr>
<tr>
<td style="text-align:left;" rowspan="5"><strong>Covering Ability</strong></td>
<td style="text-align:left;" ><strong>1.Extracting</strong></td>
</tr>
<tr>
<td style="text-align:left;" ><strong>2.Grounding</strong></td>
</tr>
<tr>
<td style="text-align:left;" ><strong>3.Referring</strong></td>
</tr>
<tr>
<td style="text-align:left;" ><strong>4.Reasoning</strong></td>
</tr>
<tr>
<td style="text-align:left;" ><strong>5.Analyzing</strong></td>
</tr>
<tr>
<td style="text-align:left;" ><strong>Defined Task</strong></td>
<td style="text-align:left;" ><strong>25 tasks</strong></td>
</tr>
</tbody>
</table>

## Data Instance
```Python
{
"question: "According to this geologic map, regarding the rock type in main map, which one has the smallest area among 4 choices?",
"answer": "D",
"type": "reasoning-area_comparison",
"A": "Torok Formation",
"B": "Surficial deposits, undivided (Holocene and Pleistocene)",
"C": "Lower part",
"D": "Alluvial deposits, undivided (Holocene and Pleistocene)",
"mcq": true,
"img_path": "16809_83756_4.jpg"
}
```

## Data Fields

- `question`: The question
- `answer`: The annotated answer
- `type`: The question type
- `A`: Choice A
- `B`: Choice B
- `C`: Choice C
- `D`: Choice D
- `mcq`: Whether the question is multiple-choice question
- `img_path`: The image path of geologic map

## Data Distribution
The distribution of evaluation abilities and tasks is demonstrated below.

<p align="left">
<img src="./images/GeoMap-Bench_Distribution.png" width="600">
</p>

## Leaderboard
Through comprehensive experiments, GeoMap-Agent achieves an overall score of 0.811 on GeoMap-Bench, significantly outperforming 0.369 of GPT-4o.

| Method | Extracting | Grounding | Referring | Reasoning | Analyzing | Overall |
|----------------------|------------|-----------|-----------|-----------|-----------|----------|
| Random | 0 | 0 | 0.250 | 0.250 | 0 | 0.100 |
| GPT-4o | 0.219 | 0.128 | 0.378 | 0.507 | 0.612 | 0.369 |
| GeoMap-Agent | **0.832** | **0.920** | **0.886** | **0.588** | **0.831** | **0.811** |

## Citation
```
TBD
```

## License
The dataset is licensed under the MIT License.

# PEACE：依托多模态大语言模型（MLLMs）实现地质图全维度理解
[[`代码`](https://github.com/microsoft/PEACE)] [[`论文`](https://arxiv.org/pdf/2501.06184)] [[`数据集`](https://huggingface.co/datasets/microsoft/PEACE)]

<p align="left">
<img src="./images/GeoMap-Bench_Pipeline.png" width="800">
</p>

## 引言
本研究构建了地质图基准数据集GeoMap-Bench，用于评估多模态大语言模型（MLLMs）在不同能力维度下的地质图理解性能，其整体概况如下表所示。

<table>
<thead>
<tr>
<th style="text-align:left;" >属性</th>
<th style="text-align:left;" >描述</th>
</tr>
</thead>
<tbody>
<tr>
<th style="text-align:left;" rowspan="2"><strong>数据来源</strong></th>
<th style="text-align:left;" ><strong>美国地质调查局（USGS，英文源）</strong></th>
</tr>
<tr>
<th style="text-align:left;" ><strong>中国地质调查局（CGS，中文源）</strong></th>
</tr>
<tr>
<th style="text-align:left;" ><strong>数据内容</strong></th>
<th style="text-align:left;" ><strong>带标注答案的图像-问题对</strong></th>
</tr>
<tr>
<th style="text-align:left;" ><strong>数据规模</strong></th>
<th style="text-align:left;" ><strong>124张地质图图像与3864个问题</strong></th>
</tr>
<tr>
<th style="text-align:left;" ><strong>图像分辨率</strong></th>
<th style="text-align:left;" ><strong>平均分辨率为6146²像素</strong></th>
</tr>
<tr>
<th style="text-align:left;" rowspan="3"><strong>问题类型</strong></th>
<th style="text-align:left;" ><strong>1. 选择题</strong></th>
</tr>
<tr>
<th style="text-align:left;" ><strong>2. 填空题</strong></th>
</tr>
<tr>
<th style="text-align:left;" ><strong>3. 简答题</strong></th>
</tr>
<tr>
<td style="text-align:left;" rowspan="5"><strong>覆盖能力维度</strong></td>
<td style="text-align:left;" ><strong>1. 信息提取</strong></td>
</tr>
<tr>
<td style="text-align:left;" ><strong>2. 实体定位</strong></td>
</tr>
<tr>
<td style="text-align:left;" ><strong>3. 指代理解</strong></td>
</tr>
<tr>
<td style="text-align:left;" ><strong>4. 逻辑推理</strong></td>
</tr>
<tr>
<td style="text-align:left;" ><strong>5. 综合分析</strong></td>
</tr>
<tr>
<td style="text-align:left;" ><strong>预设任务数</strong></td>
<td style="text-align:left;" ><strong>共25项任务</strong></td>
</tr>
</tbody>
</table>

## 数据实例
Python
{
"question": "根据该地质图，针对主图中的岩石类型，4个选项中面积最小的是哪一个？",
"answer": "D",
"type": "reasoning-area_comparison",
"A": "托罗克组（Torok Formation）",
"B": "未划分表层沉积（全新世与更新世）",
"C": "下部岩层",
"D": "未划分冲积沉积（全新世与更新世）",
"mcq": true,
"img_path": "16809_83756_4.jpg"
}

## 数据字段说明
- `question`：问题文本
- `answer`：标注的标准答案
- `type`：问题类型
- `A`：选项A
- `B`：选项B
- `C`：选项C
- `D`：选项D
- `mcq`：是否为选择题
- `img_path`：对应地质图的图像路径

## 数据分布
本基准的评估能力与任务分布如下图所示。

<p align="left">
<img src="./images/GeoMap-Bench_Distribution.png" width="600">
</p>

## 排行榜
通过全面的对比实验，GeoMap-Agent在GeoMap-Bench上的总得分为0.811，显著优于GPT-4o的0.369。

| 模型方法 | 信息提取 | 实体定位 | 指代理解 | 逻辑推理 | 综合分析 | 综合得分 |
|----------------|----------|----------|----------|----------|----------|----------|
| 随机基准模型 | 0 | 0 | 0.250 | 0.250 | 0 | 0.100 |
| GPT-4o | 0.219 | 0.128 | 0.378 | 0.507 | 0.612 | 0.369 |
| GeoMap-Agent | **0.832**| **0.920**| **0.886**| **0.588**| **0.831**| **0.811**|

## 引用

待补充

## 许可证
本数据集采用MIT许可证进行授权。