REAL-MM-RAG_TechSlides

<style> /* H1{color:Blue !important;} */ /* H1{color:DarkOrange !important;} H2{color:DarkOrange !important;} H3{color:DarkOrange !important;} */ /* p{color:Black !important;} */ </style> <!-- # REAL-MM-RAG-Bench We introduced REAL-MM-RAG-Bench, a real-world multi-modal retrieval benchmark designed to evaluate retrieval models in reliable, challenging, and realistic settings. The benchmark was constructed using an automated pipeline, where queries were generated by a vision-language model (VLM), filtered by a large language model (LLM), and rephrased by an LLM to ensure high-quality retrieval evaluation. To simulate real-world retrieval challenges, we introduce multi-level query rephrasing, modifying queries at three distinct levels—from minor wording adjustments to significant structural changes—ensuring models are tested on their true semantic understanding rather than simple keyword matching. ## REAL-MM-RAG_FinReport Financial reports (2005–2023), totaling 19 documents and 2687 pages, with a mix of text and tables. ## How to Load the Dataset ```python from datasets import load_dataset dataset = load_dataset("ibm-research/REAL-MM-RAG_FinReport") print(dataset) ``` ### Source Paper [REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark](https://arxiv.org/abs/2502.12342) --> # REAL-MM-RAG-Bench: A Real-World Multi-Modal Retrieval Benchmark We introduced REAL-MM-RAG-Bench, a real-world multi-modal retrieval benchmark designed to evaluate retrieval models in reliable, challenging, and realistic settings. The benchmark was constructed using an automated pipeline, where queries were generated by a vision-language model (VLM), filtered by a large language model (LLM), and rephrased by an LLM to ensure high-quality retrieval evaluation. To simulate real-world retrieval challenges, we introduce multi-level query rephrasing, modifying queries at three distinct levels—from minor wording adjustments to significant structural changes—ensuring models are tested on their true semantic understanding rather than simple keyword matching. ### **REAL-MM-RAG_TechSlides** - **Content**: 62 technical presentations on business and IT automation. - **Size**: 1,963 pages. - **Composition**: Mix of text, visuals, and tables. - **Purpose**: Evaluates retrieval of IT automation and business insights from slide decks. ## Loading the Dataset To use the dataset, install the ⁠ datasets ⁠ library and load it as follows: ```python from datasets import load_dataset # Load the dataset dataset = load_dataset("ibm-research/REAL-MM-RAG_TechSlides", split="test") # Indexing queries to image filenames query_to_image = {ex['query']: ex['image_filename'] for ex in dataset if ex['query'] is not None} # Indexing image filenames to associated queries image_to_queries = {} for ex in dataset: image_to_queries.setdefault(ex['image_filename'], []).append(ex['query']) # Example 1: Find the image for a specific query query_example = "What is the recommended method for adding new content classifications according to IBM?" if query_example in query_to_image: image_filename = query_to_image[query_example] print(f"Query '{query_example}' is linked to image: {image_filename}") # Example 2: Find all queries linked to a specific image image_example = "How to Future Proof Your ECM Solution_page_21.png" if image_example in image_to_queries: linked_queries = image_to_queries[image_example] print(f"Image '{image_example}' is linked to queries: {linked_queries}") # Example 3: Handle cases where a page has no queries (only part of the dataset) image_example = "Bridging Data Silos with IBM Concert_page_10.png" if image_example in image_to_queries: linked_queries = image_to_queries[image_example] print(f"Image '{image_example}' is linked to queries: {linked_queries}") ``` ⁠ ## Source Paper ```bibtex @misc{wasserman2025realmmragrealworldmultimodalretrieval, title={REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark}, author={Navve Wasserman and Roi Pony and Oshri Naparstek and Adi Raz Goldfarb and Eli Schwartz and Udi Barzelay and Leonid Karlinsky}, year={2025}, eprint={2502.12342}, archivePrefix={arXiv}, primaryClass={cs.IR}, url={https://arxiv.org/abs/2502.12342}, } ``` <!-- # REAL-MM-RAG-Bench: A Real-World Multi-Modal Retrieval Benchmark ## Overview REAL-MM-RAG-Bench is a benchmark designed to evaluate multi-modal retrieval models under realistic and challenging conditions. This dataset provides multi-modal documents with diverse content, including text, tables, and figures, to test models' ability to retrieve relevant information based on natural language queries. ## Features - **Multi-Modal Documents**: Includes a mix of text, figures, and tables, ensuring a realistic document retrieval scenario. - **Long Document Focus**: Prioritizes long documents over isolated pages to reflect real-world retrieval challenges. - **Sub-Domain Consistency**: Ensures many pages belong to the same sub-domain by focusing on IBM data. - **Enhanced Difficulty**: Queries require more than keyword matching and are tested against a corpus with highly similar pages. - **Realistic Queries**: Queries are generated through an automated pipeline using a Vision-Language Model (VLM), refined by a Large Language Model (LLM) to mimic real-world search behavior. - **Accurate Labeling**: Ensures that all relevant documents for a query are correctly labeled to avoid false negatives. - **Multi-Level Query Rephrasing**: Queries undergo multiple levels of rewording to evaluate model robustness beyond simple lexical matching. ## Dataset Subsets ### **REAL-MM-RAG_FinReport** - **Content**: 19 financial reports from 2005–2023. - **Size**: 2,687 pages. - **Composition**: Includes both textual data and structured tables. - **Purpose**: Designed to test model performance on table-heavy financial data retrieval. --> <!-- ### **REAL-MM-RAG_FinSlides** - **Content**: 65 quarterly financial presentations from 2008–2024. - **Size**: 2,280 pages. - **Composition**: Primarily table-heavy with key financial insights. - **Purpose**: Evaluates retrieval in visually structured financial presentations. ### **REAL-MM-RAG_TechReport** - **Content**: 17 technical documents on IBM FlashSystem. - **Size**: 1,674 pages. - **Composition**: Text-heavy with visual elements and structured tables. - **Purpose**: Assesses model performance in retrieving structured technical content. ### **REAL-MM-RAG_TechSlides** - **Content**: 62 technical presentations on business and IT automation. - **Size**: 1,963 pages. - **Composition**: Mix of text, visuals, and tables. - **Purpose**: Evaluates retrieval of IT automation and business insights from slide decks. --> <!-- ## Loading the Dataset To use the dataset, install the `datasets` library and load it as follows: ```python from datasets import load_dataset dataset = load_dataset("ibm-research/REAL-MM-RAG_FinReport") print(dataset) ``` ## Dataset Construction ### **Automated Query Generation Pipeline** - Queries are generated using a **VLM**, ensuring diverse and realistic question formats. - The **LLM filters** low-quality queries and rephrases them into user-friendly search queries. - Multi-level **query rephrasing** introduces increasing levels of variation to assess semantic retrieval performance. ### **Document Categories** - **Financial Reports**: Annual and quarterly reports with a high concentration of tabular data. - **Technical Documents**: Product manuals and whitepapers from IBM, focusing on structured information. - **Presentation Slides**: Corporate presentations with a mix of visuals and key financial data. <!-- ### **Evaluation Criteria** Models are evaluated based on: - **Retrieval accuracy**: Measured using metrics like **NDCG@5** and **Recall@1**. - **Rephrasing robustness**: Performance drop across increasing levels of query modification. - **Table comprehension**: Success rate in retrieving relevant tabular data. -->