Feisu55/MME-Finance

---
license: cc-by-nc-4.0
---

<a href="https://arxiv.org/abs/2411.03314">Paper</a> |<a href="https://hithink-research.github.io/MME-Finance/">Homepage</a></h3> |<a href="https://github.com/HiThink-Research/MME-Finance">Github</a></h3>

## 🛠️ Usage
<!-- ### Judgement -->
Regarding the data, first of all, you should download the `MMfin.tsv` and `MMfin_CN.tsv` files, as well as the relevant financial images. The folder structure is shown as follows:
```
├─ datasets
├─ images
├─ MMfin
...
├─ MMfin_CN
...
│ MMfin.tsv
│ MMfin_CN.tsv
```

The following is the process of inference and evaluation (Qwen2-VL-2B-Instruct as an example):
```
export LMUData="The path of the datasets"
python run.py --data MMfin --model Qwen2-VL-2B-Instruct --verbose
python run.py --data MMfin_CN --model Qwen2-VL-2B-Instruct --verbose
```

## ✨ New
2025.1.8, we have released all samples in both English and Chinese.

## SETING
| Statistic | type | Overall |Image Caption<br> | OCR | Entity Recognition | Spatial Awareness | Accurate Numerical Calculation | Estimated Numerical Calculation | Risk Warning | Investment Advice | Reason Explanation | Financial Question Answer | Not Applicable |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
| number | **MMFIN_EN** | 734 | 164 | 178 | 163 | 229 | 133 | 42 | 22 | 53 | 18 | 147 | 22 |
| number | **MMFIN_CN** | 640 | 144 | 182 | 148 | 166 | 126 | 32 | 37 | 91 | 13 | 144 | 20 |

## MMFIN Result
| Model | Overall | Image Caption | OCR | Entity Recognition | Spatial Awareness | Financial Question Answer | Accurate Numerical Calculation | Estimated Numerical Calculation | Risking Warning | Investment Advice | Reason Explanation | Not Applicable |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
| **Open source MLLMs** |
| Yi-VL-34B | 17.57 | 29.39 | 1.46 | 3.93 | 8.73 | 5.56 | 11.43 | 42.73 | 35.09 | 58.89 | 47.48 | 36.36 |
| CogVLM2-19B | 46.32 | 67.32 | 61.24 | 35.83 | 16.59 | 44.51 | 33.33 | 59.09 | 52.83 | 31.11 | 58.64 | 93.64 |
| InternVL2-76B | 61.62 | 83.17 | 77.64 | 47.60 | 30.31 | 70.08 | 41.90 | 75.45 | 66.42 | 76.67 | 72.24 | 79.09 |
| LLaMA3.2-90B | 48.76 | 64.27 | 46.74 | 41.27 | 25.85 | 55.64 | 22.86 | 63.64 | 61.13 | 64.44 | 65.58 | 81.82 |
| LLaVA-Next-13B | 31.37 | 62.68 | 25.39 | 22.58 | 10.31 | 12.63 | 9.05 | 47.27 | 40.00 | 12.22 | 59.46 | 78.18 |
| MiniCPM2.6 | 51.65 | 71.22 | 63.71 | 37.67 | 24.37 | 55.64 | 21.43 | 72.73 | 58.87 | 66.67 | 66.80 | 77.27 |
| Phi3-Vision | 46.69 | 69.88 | 57.64 | 28.34 | 18.08 | 47.52 | 34.76 | 65.45 | 58.11 | 68.89 | 57.41 | **100.0** |
| Qwen2VL-72B | **65.69** | 82.56 | **87.52** | **55.46** | 27.16 | **83.76** | 40.95 | 78.18 | 65.66 | 77.78 | 75.37 | 90.91 |
|**Proprietary MLLMs**|
| GeminiPro1.5 | 61.84 | 82.20 | 80.22 | 48.59 | 23.14 | 78.20 | 50.95 | 76.36 | 69.43 | 75.56 | 70.75 | 80.91 |
| Claude3.5-Sonnet | 63.91 | **87.80** | 63.70 | 54.23 | **35.46** | 72.33 | **60.00** | 80.91 | **72.83** | **82.22** | 73.33 | 95.45 |
| GPT-4o | 63.18 | 83.66 | 79.21 | 49.81 | 27.07 | 71.88 | 44.76 | **84.54** | 70.57 | 80.00 | **76.87** | 93.64 |

---
许可协议：CC BY-NC 4.0
---

<a href="https://arxiv.org/abs/2411.03314">论文</a> |<a href="https://hithink-research.github.io/MME-Finance/">主页</a> |<a href="https://github.com/HiThink-Research/MME-Finance">Github 仓库</a>

## 🛠️ 使用方法
关于数据集，首先需要下载`MMfin.tsv`与`MMfin_CN.tsv`文件，以及相关金融图像。文件夹结构如下所示：

├─ datasets
├─ images
├─ MMfin
...
├─ MMfin_CN
...
│ MMfin.tsv
│ MMfin_CN.tsv

以下为推理与评估流程（以Qwen2-VL-2B-Instruct为例）：

export LMUData="数据集路径"
python run.py --data MMfin --model Qwen2-VL-2B-Instruct --verbose
python run.py --data MMfin_CN --model Qwen2-VL-2B-Instruct --verbose

## ✨ 最新更新
2025年1月8日，我们发布了英文与中文的全部样本数据。

## 数据集统计设置
| 统计项 | 类型 | 整体 | 图像描述 | 光学字符识别（Optical Character Recognition） | 实体识别 | 空间感知 | 精确数值计算 | 估算数值计算 | 风险警示 | 投资建议 | 理由阐释 | 金融问答 | 不适用 |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
| 数量 | **MMFIN_EN** | 734 | 164 | 178 | 163 | 229 | 133 | 42 | 22 | 53 | 18 | 147 | 22 |
| 数量 | **MMFIN_CN** | 640 | 144 | 182 | 148 | 166 | 126 | 32 | 37 | 91 | 13 | 144 | 20 |

## MMFIN 模型评测结果
| 模型 | 整体得分 | 图像描述 | 光学字符识别 | 实体识别 | 空间感知 | 金融问答 | 精确数值计算 | 估算数值计算 | 风险警示 | 投资建议 | 理由阐释 | 不适用 |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
| **开源多模态大模型（Multimodal Large Language Models，MLLMs）** |
| Yi-VL-34B | 17.57 | 29.39 | 1.46 | 3.93 | 8.73 | 5.56 | 11.43 | 42.73 | 35.09 | 58.89 | 47.48 | 36.36 |
| CogVLM2-19B | 46.32 | 67.32 | 61.24 | 35.83 | 16.59 | 44.51 | 33.33 | 59.09 | 52.83 | 31.11 | 58.64 | 93.64 |
| InternVL2-76B | 61.62 | 83.17 | 77.64 | 47.60 | 30.31 | 70.08 | 41.90 | 75.45 | 66.42 | 76.67 | 72.24 | 79.09 |
| LLaMA3.2-90B | 48.76 | 64.27 | 46.74 | 41.27 | 25.85 | 55.64 | 22.86 | 63.64 | 61.13 | 64.44 | 65.58 | 81.82 |
| LLaVA-Next-13B | 31.37 | 62.68 | 25.39 | 22.58 | 10.31 | 12.63 | 9.05 | 47.27 | 40.00 | 12.22 | 59.46 | 78.18 |
| MiniCPM2.6 | 51.65 | 71.22 | 63.71 | 37.67 | 24.37 | 55.64 | 21.43 | 72.73 | 58.87 | 66.67 | 66.80 | 77.27 |
| Phi3-Vision | 46.69 | 69.88 | 57.64 | 28.34 | 18.08 | 47.52 | 34.76 | 65.45 | 58.11 | 68.89 | 57.41 | **100.0** |
| Qwen2VL-72B | **65.69** | 82.56 | **87.52** | **55.46** | 27.16 | **83.76** | 40.95 | 78.18 | 65.66 | 77.78 | 75.37 | 90.91 |
| **专有多模态大模型（Multimodal Large Language Models，MLLMs）** |
| GeminiPro1.5 | 61.84 | 82.20 | 80.22 | 48.59 | 23.14 | 78.20 | 50.95 | 76.36 | 69.43 | 75.56 | 70.75 | 80.91 |
| Claude3.5-Sonnet | 63.91 | **87.80** | 63.70 | 54.23 | **35.46** | 72.33 | **60.00** | 80.91 | **72.83** | **82.22** | 73.33 | 95.45 |
| GPT-4o | 63.18 | 83.66 | 79.21 | 49.81 | 27.07 | 71.88 | 44.76 | **84.54** | 70.57 | 80.00 | **76.87** | 93.64 |