weaverbirdllm/famma

FAMMA是一个多模态金融问答基准数据集，包含三种异构图像类型（表格、图表和文本及数学截图），涵盖金融领域的八个子领域，全面覆盖主要资产类别的主题。所有问题按难度分为三个等级（简单、中等、困难），并提供三种语言版本（英语、中文、法语）。此外，问题分为两种类型：选择题和开放性问题。FAMMA还提供了一个实时基准，用于评估LLM的金融分析能力，该基准不断从现实世界中的金融专业人士那里收集新问题，确保评估的时效性和无污染性。

FAMMA is a multi-modal financial Q&A benchmark dataset that includes three heterogeneous image types - tables, charts, and text & math screenshots - and spans eight subfields in finance, comprehensively covering topics across major asset classes. All questions are categorized by three difficulty levels - easy, medium, and hard - and are available in three languages - English, Chinese, and French. Furthermore, the questions are divided into two types: multiple-choice and open questions. FAMMA also provides a live benchmark for evaluating financial analysis capabilities of LLMs, continuously collecting new questions from real-world financial professionals to ensure up-to-date and contamination-free evaluation.