sujet-ai/Sujet-Finance-QA-Vision-100k

--- license: apache-2.0 task_categories: - question-answering language: - en tags: - finance - question answer - dataset - qa - vision - visual qa - financial documents - VLM - Llava - Florence - PaliGemma pretty_name: SujetAI's Financial QA Vision Dataset size_categories: - 100K<n<1M --- ## Dataset Description 📊🔍 The Sujet-Finance-QA-Vision-100k is a comprehensive dataset containing over 100,000 question-answer pairs derived from more than 9,800 financial document images. This dataset is designed to support research and development in the field of financial document analysis and visual question answering. ### Key Features: - 🖼️ 9,801 unique financial document images - ❓ 107,050 question-answer pairs - 🇬🇧 English language - 📄 Diverse financial document types ## Dataset Summary - **Training Set**: 9,212 images, 100,629 QA pairs - **Validation Set**: 589 images, 6,421 QA pairs - **Total**: 9,801 images, 107,050 QA pairs ## Get Started Here's a quick example of how to load and explore the dataset: ```python from datasets import load_dataset import json import matplotlib.pyplot as plt # Load the dataset data = load_dataset("sujet-ai/Sujet-Finance-QA-Vision-100k") # Access train and validation splits train_data = data['train'] val_data = data['test'] # Display info about a sample entry sample = val_data[0] print(f"Document ID: {sample['doc_id']}") print(f"Content preview: {sample['content']}") # Display the image plt.figure(figsize=(10, 10)) plt.imshow(sample['image']) plt.axis('off') plt.title(f"Image for Document {sample['doc_id']}") plt.show() # Print QA pairs for the sample qa_pairs = json.loads(sample['qa_pairs']) print("\nQuestion-Answer Pairs:") for qa in qa_pairs: print(f"Q: {qa['question']}") print(f"A: {qa['answer']}") print() ``` ## Dataset Creation This dataset is an evolution of our previous [Sujet-Finance-Vision-10k](https://huggingface.co/datasets/sujet-ai/Sujet-Finance-Vision-10k) dataset. The original dataset contained detailed annotations generated by GPT-4 for each image. However, we encountered challenges when fine-tuning small Vision-Language Models (VLMs) due to the extensive context window size required by these annotations. To address this issue, we used a different approach to break down our data into smaller pieces, easily digestible by a smaller model during the finetuning process: 1. We used the [meta-llama/Meta-Llama-3-70B](https://huggingface.co/meta-llama/Meta-Llama-3-70B) model to generate diverse question-answer pairs based on the original detailed annotations. 2. The model was prompted to create a variety of questions covering different aspects of each financial document, including factual, analytical, comparative, and hypothetical questions. 3. This process allowed us to maintain the depth of information while creating more concise and targeted QA pairs. Here's a simplified version of the prompt used to generate the QA pairs: ``` As an expert in financial document analysis, create diverse, high-quality question-answer pairs based on the given financial document content. Steps: 1. Analyze the document content. 2. Identify key themes, facts, and implications. 3. Generate varied questions covering: - Factual details - Analytical interpretations - Industry comparisons - Hypothetical scenarios 4. Provide concise, informative answers. 5. Ensure diversity and non-repetition. 6. Aim for at least 10 QA pairs, more if content allows. [Document Content Here] Output in JSON format: [ {"question": "What is the total revenue reported?", "answer": "The total revenue reported is $10 million for fiscal year 2023."}, {"question": "How does the profit margin compare to industry average?", "answer": "The 15% profit margin is 2 percentage points above the 13% industry average."}, ... (additional Q&A pairs) ] ``` ## Data Fields - `doc_id`: Unique identifier for the document - `content`: A rich annotation of the information covered in the document : Used to create the QA pairs. - `image`: The financial document image - `qa_pairs`: JSON string containing question-answer pairs ## Limitations and Bias While we've taken care to ensure the quality of the dataset, it's important to note: - The question-answer pairs were generated based on GPT-4 annotations and then refined using Llama 3 70B. While this process produces high-quality results, there may be instances where the answers do not perfectly correspond to the information in the image. - Through manual inspection, we've found that such discrepancies are rare and don't significantly impact the training/validation process. However, users should be aware of the limitations of this unsupervised and automated dataset generation. - The dataset focuses on English-language financial documents, which may limit its applicability to other languages or financial systems. ## Ethical Considerations Users of this dataset should be aware that: - The financial information contained in the images and QA pairs should not be used for making real-world financial decisions without proper verification. - The dataset may reflect biases present in the original financial documents or introduced during the annotation process. ## License This dataset is licensed under Apache 2.0. ## Disclaimer Sujet AI provides the Sujet-Finance-QA-Vision-100k dataset as-is, without any warranties, expressed or implied. We are not responsible for any consequences resulting from the use of this dataset. Users should exercise their own judgment when using the dataset for research, development, or any other purposes. The dataset may contain errors, inaccuracies, or biases, and should not be solely relied upon for critical decision-making in financial or other domains. Users are encouraged to validate and verify the information as needed for their specific use cases. By using this dataset, you agree to hold Sujet AI harmless from any and all claims, damages, or liabilities arising from its use. ## Citation and Contact If you use the Sujet-Finance-QA-Vision-100k dataset in your research, please cite it as: ``` @dataset{Sujet-Finance-QA-Vision-100k, author = {Sujet AI, Allaa Boutaleb, Hamed Rahimi}, title = {Sujet-Finance-QA-Vision-100k: A Large-Scale Dataset for Financial Document VQA}, year = {2024}, url = {https://huggingface.co/datasets/sujet-ai/Sujet-Finance-QA-Vision-100k} } ``` For questions, feedback, or collaborations, please reach out to us on [LinkedIn](https://www.linkedin.com/company/sujet-ai/) or visit our website [https://sujet.ai](https://sujet.ai).--- 许可证：Apache 2.0 任务类别： - 问答 语言： - 英语 标签： - 金融 - 问答 - 数据集 - QA - 视觉 - 视觉问答（Visual QA） - 金融文档 - 视觉语言模型（Vision-Language Model，VLM） - Llava - Florence - PaliGemma 美观名称：SujetAI金融视觉问答数据集 样本规模区间： - 100K<n<1M --- ## 数据集描述 📊🔍 Sujet-Finance-QA-Vision-100k是一款大规模综合性数据集，涵盖超10万条问答样本，其数据源自超过9800张金融文档图像。本数据集旨在为金融文档分析与视觉问答领域的研究与开发提供支撑。 ### 核心特性： - 🖼️ 共计9801张独特金融文档图像 - ❓ 包含107050条问答样本 - 🇬🇧 全部数据采用英语 - 📄 覆盖多样的金融文档品类 ## 数据集概览 - **训练集**：9212张图像，100629条问答样本 - **验证集**：589张图像，6421条问答样本 - **总计**：9801张图像，107050条问答样本 ## 快速上手 以下为加载与探索本数据集的快速示例： python from datasets import load_dataset import json import matplotlib.pyplot as plt # 加载数据集 data = load_dataset("sujet-ai/Sujet-Finance-QA-Vision-100k") # 访问训练集与验证集划分 train_data = data['train'] val_data = data['test'] # 展示单条样本的信息 sample = val_data[0] print(f"Document ID: {sample['doc_id']}") print(f"Content preview: {sample['content']}") # 展示图像 plt.figure(figsize=(10, 10)) plt.imshow(sample['image']) plt.axis('off') plt.title(f"Image for Document {sample['doc_id']}") plt.show() # 打印样本的问答对 qa_pairs = json.loads(sample['qa_pairs']) print(" Question-Answer Pairs:") for qa in qa_pairs: print(f"Q: {qa['question']}") print(f"A: {qa['answer']}") print() ## 数据集构建流程 本数据集是此前发布的[Sujet-Finance-Vision-10k](https://huggingface.co/datasets/sujet-ai/Sujet-Finance-Vision-10k)数据集的迭代版本。原始数据集包含由GPT-4为每张图像生成的精细标注，但在对小型视觉语言模型（Vision-Language Model，VLM）进行微调时，我们遇到了挑战：原始标注所需的上下文窗口过大，小型模型难以适配。 为解决该问题，我们采用全新方案将数据拆分为更易被微调阶段小型模型处理的小块： 1. 我们借助[meta-llama/Meta-Llama-3-70B](https://huggingface.co/meta-llama/Meta-Llama-3-70B)模型，基于原始精细标注生成多样化的问答样本。 2. 模型提示词要求其生成覆盖金融文档各维度的多样问题，涵盖事实类、分析类、对比类与假设类问题。 3. 该方案在保留信息深度的同时，生成了更简洁且针对性更强的问答样本。 以下为用于生成问答样本的简化版提示词： As an expert in financial document analysis, create diverse, high-quality question-answer pairs based on the given financial document content. Steps: 1. Analyze the document content. 2. Identify key themes, facts, and implications. 3. Generate varied questions covering: - Factual details - Analytical interpretations - Industry comparisons - Hypothetical scenarios 4. Provide concise, informative answers. 5. Ensure diversity and non-repetition. 6. Aim for at least 10 QA pairs, more if content allows. [Document Content Here] Output in JSON format: [ {"question": "What is the total revenue reported?", "answer": "The total revenue reported is $10 million for fiscal year 2023."}, {"question": "How does the profit margin compare to industry average?", "answer": "The 15% profit margin is 2 percentage points above the 13% industry average."}, ... (additional Q&A pairs) ] ## 数据字段说明 - `doc_id`：文档唯一标识符 - `content`：对文档涵盖信息的精细标注，用于生成问答样本 - `image`：金融文档图像文件 - `qa_pairs`：包含问答样本的JSON格式字符串 ## 局限性与偏差说明 尽管我们已尽力保障数据集质量，但仍需注意以下要点： - 本数据集的问答样本基于GPT-4标注生成，后经Llama 3 70B模型优化。尽管该流程可产出高质量结果，但仍可能存在部分答案与图像信息不完全匹配的情况。 - 经人工抽检，此类偏差情况较为罕见，不会对训练/验证流程造成显著影响。但用户需知晓，本数据集采用无监督自动化生成流程，存在一定局限性。 - 本数据集仅涵盖英语金融文档，这可能限制其在其他语言或金融体系中的应用场景。 ## 伦理考量 使用本数据集的用户需知悉： - 文档图像与问答样本中包含的金融信息，未经正式验证前，不得用于实际金融决策。 - 本数据集可能反映原始金融文档固有的偏差，或在标注流程中引入额外偏差。 ## 许可证 本数据集采用Apache 2.0许可证进行授权。 ## 免责声明 Sujet AI按“现状”提供Sujet-Finance-QA-Vision-100k数据集，不提供任何形式的明示或默示担保。我们不对因使用本数据集所产生的任何后果承担责任。用户在将本数据集用于研究、开发或其他用途时，应自行做出判断。 本数据集可能包含错误、不准确信息或偏差，不得仅依赖本数据集进行金融或其他领域的关键决策。用户应根据自身具体使用场景，对数据集信息进行必要的验证与核实。 使用本数据集即视为您同意，因使用本数据集而产生的任何索赔、损害或法律责任，均由您自行承担，Sujet AI不承担任何责任。 ## 引用与联系方式 若您在研究中使用Sujet-Finance-QA-Vision-100k数据集，请按以下格式引用： bibtex @dataset{Sujet-Finance-QA-Vision-100k, author = {Sujet AI, Allaa Boutaleb, Hamed Rahimi}, title = {Sujet-Finance-QA-Vision-100k: A Large-Scale Dataset for Financial Document VQA}, year = {2024}, url = {https://huggingface.co/datasets/sujet-ai/Sujet-Finance-QA-Vision-100k} } 如有疑问、反馈或合作意向，欢迎通过[LinkedIn](https://www.linkedin.com/company/sujet-ai/)联系我们，或访问我们的官网[https://sujet.ai](https://sujet.ai).