mm-bright/MM-BRIGHT

---
language:
- en
license: cc-by-4.0
size_categories:
- 1K<n<10K
task_categories:
- text-retrieval
- question-answering
tags:
- multimodal-retrieval
- rag
- complex-reasoning
- image-retrieval
- stackexchange
configs:
# ========================================================
# 1. CORE DATA (Documents & Queries)
# ========================================================
- config_name: documents
features:
- name: id
dtype: string
- name: content
dtype: string
data_files: &domains_docs
- split: academia
path: documents/academia.parquet
- split: apple
path: documents/apple.parquet
- split: askubuntu
path: documents/askubuntu.parquet
- split: aviation
path: documents/aviation.parquet
- split: bioacoustics
path: documents/bioacoustics.parquet
- split: bioinformatics
path: documents/bioinformatics.parquet
- split: biology
path: documents/biology.parquet
- split: bitcoin
path: documents/bitcoin.parquet
- split: chemistry
path: documents/chemistry.parquet
- split: christianity
path: documents/christianity.parquet
- split: crypto
path: documents/crypto.parquet
- split: earthscience
path: documents/earthscience.parquet
- split: economics
path: documents/economics.parquet
- split: gaming
path: documents/gaming.parquet
- split: gis
path: documents/gis.parquet
- split: islam
path: documents/islam.parquet
- split: law
path: documents/law.parquet
- split: math
path: documents/math.parquet
- split: medicalsciences
path: documents/medicalsciences.parquet
- split: philosophy
path: documents/philosophy.parquet
- split: physics
path: documents/physics.parquet
- split: pm
path: documents/pm.parquet
- split: psychology
path: documents/psychology.parquet
- split: quant
path: documents/quant.parquet
- split: quantumcomputing
path: documents/quantumcomputing.parquet
- split: robotics
path: documents/robotics.parquet
- split: salesforce
path: documents/salesforce.parquet
- split: sustainability
path: documents/sustainability.parquet
- split: travel
path: documents/travel.parquet

- config_name: examples
features:
- name: id
dtype: string
- name: query
dtype: string
- name: gold_ids
sequence: string
- name: gold_answers
sequence: string
- name: image_paths
sequence: string
- name: negative_ids
sequence: string
- name: llm_image_caption
dtype: string
- name: domain
dtype: string
data_files:
- split: academia
path: examples/academia.parquet
- split: apple
path: examples/apple.parquet
- split: askubuntu
path: examples/askubuntu.parquet
- split: aviation
path: examples/aviation.parquet
- split: bioacoustics
path: examples/bioacoustics.parquet
- split: bioinformatics
path: examples/bioinformatics.parquet
- split: biology
path: examples/biology.parquet
- split: bitcoin
path: examples/bitcoin.parquet
- split: chemistry
path: examples/chemistry.parquet
- split: christianity
path: examples/christianity.parquet
- split: crypto
path: examples/crypto.parquet
- split: earthscience
path: examples/earthscience.parquet
- split: economics
path: examples/economics.parquet
- split: gaming
path: examples/gaming.parquet
- split: gis
path: examples/gis.parquet
- split: islam
path: examples/islam.parquet
- split: law
path: examples/law.parquet
- split: math
path: examples/math.parquet
- split: medicalsciences
path: examples/medicalsciences.parquet
- split: philosophy
path: examples/philosophy.parquet
- split: physics
path: examples/physics.parquet
- split: pm
path: examples/pm.parquet
- split: psychology
path: examples/psychology.parquet
- split: quant
path: examples/quant.parquet
- split: quantumcomputing
path: examples/quantumcomputing.parquet
- split: robotics
path: examples/robotics.parquet
- split: salesforce
path: examples/salesforce.parquet
- split: sustainability
path: examples/sustainability.parquet
- split: travel
path: examples/travel.parquet

- config_name: examples_multimodal
features:
- name: id
dtype: string
- name: query
dtype: string
- name: gold_ids
sequence: string
- name: gold_answers
sequence: string
- name: image_paths
sequence: string
- name: negative_ids
sequence: string
- name: llm_image_caption
dtype: string
- name: domain
dtype: string
data_files:
- split: academia
path: examples_multimodal/academia.parquet
- split: apple
path: examples_multimodal/apple.parquet
- split: askubuntu
path: examples_multimodal/askubuntu.parquet
- split: aviation
path: examples_multimodal/aviation.parquet
- split: bioacoustics
path: examples_multimodal/bioacoustics.parquet
- split: bioinformatics
path: examples_multimodal/bioinformatics.parquet
- split: biology
path: examples_multimodal/biology.parquet
- split: bitcoin
path: examples_multimodal/bitcoin.parquet
- split: chemistry
path: examples_multimodal/chemistry.parquet
- split: christianity
path: examples_multimodal/christianity.parquet
- split: crypto
path: examples_multimodal/crypto.parquet
- split: earthscience
path: examples_multimodal/earthscience.parquet
- split: economics
path: examples_multimodal/economics.parquet
- split: gaming
path: examples_multimodal/gaming.parquet
- split: gis
path: examples_multimodal/gis.parquet
- split: islam
path: examples_multimodal/islam.parquet
- split: law
path: examples_multimodal/law.parquet
- split: math
path: examples_multimodal/math.parquet
- split: medicalsciences
path: examples_multimodal/medicalsciences.parquet
- split: philosophy
path: examples_multimodal/philosophy.parquet
- split: physics
path: examples_multimodal/physics.parquet
- split: pm
path: examples_multimodal/pm.parquet
- split: psychology
path: examples_multimodal/psychology.parquet
- split: quant
path: examples_multimodal/quant.parquet
- split: quantumcomputing
path: examples_multimodal/quantumcomputing.parquet
- split: robotics
path: examples_multimodal/robotics.parquet
- split: salesforce
path: examples_multimodal/salesforce.parquet
- split: sustainability
path: examples_multimodal/sustainability.parquet
- split: travel
path: examples_multimodal/travel.parquet

# ========================================================
# 2. IMAGES (Binary)
# ========================================================
- config_name: document_images
features:
- name: path
dtype: string
- name: bytes
dtype: binary
data_files:
- split: academia
path: document_images/academia.parquet
- split: apple
path: document_images/apple.parquet
- split: askubuntu
path: document_images/askubuntu.parquet
- split: aviation
path: document_images/aviation.parquet
- split: bioacoustics
path: document_images/bioacoustics.parquet
- split: bioinformatics
path: document_images/bioinformatics.parquet
- split: biology
path: document_images/biology.parquet
- split: bitcoin
path: document_images/bitcoin.parquet
- split: chemistry
path: document_images/chemistry.parquet
- split: christianity
path: document_images/christianity.parquet
- split: crypto
path: document_images/crypto.parquet
- split: earthscience
path: document_images/earthscience.parquet
- split: economics
path: document_images/economics.parquet
- split: gaming
path: document_images/gaming.parquet
- split: gis
path: document_images/gis.parquet
- split: islam
path: document_images/islam.parquet
- split: law
path: document_images/law.parquet
- split: math
path: document_images/math.parquet
- split: medicalsciences
path: document_images/medicalsciences.parquet
- split: philosophy
path: document_images/philosophy.parquet
- split: physics
path: document_images/physics.parquet
- split: pm
path: document_images/pm.parquet
- split: psychology
path: document_images/psychology.parquet
- split: quant
path: document_images/quant.parquet
- split: quantumcomputing
path: document_images/quantumcomputing.parquet
- split: robotics
path: document_images/robotics.parquet
- split: salesforce
path: document_images/salesforce.parquet
- split: sustainability
path: document_images/sustainability.parquet
- split: travel
path: document_images/travel.parquet

- config_name: examples_images
features:
- name: path
dtype: string
- name: bytes
dtype: binary
data_files:
- split: academia
path: examples_images/academia.parquet
- split: apple
path: examples_images/apple.parquet
- split: askubuntu
path: examples_images/askubuntu.parquet
- split: aviation
path: examples_images/aviation.parquet
- split: bioacoustics
path: examples_images/bioacoustics.parquet
- split: bioinformatics
path: examples_images/bioinformatics.parquet
- split: biology
path: examples_images/biology.parquet
- split: bitcoin
path: examples_images/bitcoin.parquet
- split: chemistry
path: examples_images/chemistry.parquet
- split: christianity
path: examples_images/christianity.parquet
- split: crypto
path: examples_images/crypto.parquet
- split: earthscience
path: examples_images/earthscience.parquet
- split: economics
path: examples_images/economics.parquet
- split: gaming
path: examples_images/gaming.parquet
- split: gis
path: examples_images/gis.parquet
- split: islam
path: examples_images/islam.parquet
- split: law
path: examples_images/law.parquet
- split: math
path: examples_images/math.parquet
- split: medicalsciences
path: examples_images/medicalsciences.parquet
- split: philosophy
path: examples_images/philosophy.parquet
- split: physics
path: examples_images/physics.parquet
- split: pm
path: examples_images/pm.parquet
- split: psychology
path: examples_images/psychology.parquet
- split: quant
path: examples_images/quant.parquet
- split: quantumcomputing
path: examples_images/quantumcomputing.parquet
- split: robotics
path: examples_images/robotics.parquet
- split: salesforce
path: examples_images/salesforce.parquet
- split: sustainability
path: examples_images/sustainability.parquet
- split: travel
path: examples_images/travel.parquet

# ========================================================
# 3. REASONING VARIATIONS (7 Models)
# ========================================================
- config_name: gpt4o_reason
features:
- name: id
dtype: string
- name: query
dtype: string
- name: gold_ids
sequence: string
- name: reasoning
dtype: string
data_files:
- split: academia
path: gpt4o_reason/academia.parquet
- split: apple
path: gpt4o_reason/apple.parquet
- split: askubuntu
path: gpt4o_reason/askubuntu.parquet
- split: aviation
path: gpt4o_reason/aviation.parquet
- split: bioacoustics
path: gpt4o_reason/bioacoustics.parquet
- split: bioinformatics
path: gpt4o_reason/bioinformatics.parquet
- split: biology
path: gpt4o_reason/biology.parquet
- split: bitcoin
path: gpt4o_reason/bitcoin.parquet
- split: chemistry
path: gpt4o_reason/chemistry.parquet
- split: christianity
path: gpt4o_reason/christianity.parquet
- split: crypto
path: gpt4o_reason/crypto.parquet
- split: earthscience
path: gpt4o_reason/earthscience.parquet
- split: economics
path: gpt4o_reason/economics.parquet
- split: gaming
path: gpt4o_reason/gaming.parquet
- split: gis
path: gpt4o_reason/gis.parquet
- split: islam
path: gpt4o_reason/islam.parquet
- split: law
path: gpt4o_reason/law.parquet
- split: math
path: gpt4o_reason/math.parquet
- split: medicalsciences
path: gpt4o_reason/medicalsciences.parquet
- split: philosophy
path: gpt4o_reason/philosophy.parquet
- split: physics
path: gpt4o_reason/physics.parquet
- split: pm
path: gpt4o_reason/pm.parquet
- split: psychology
path: gpt4o_reason/psychology.parquet
- split: quant
path: gpt4o_reason/quant.parquet
- split: quantumcomputing
path: gpt4o_reason/quantumcomputing.parquet
- split: robotics
path: gpt4o_reason/robotics.parquet
- split: salesforce
path: gpt4o_reason/salesforce.parquet
- split: sustainability
path: gpt4o_reason/sustainability.parquet
- split: travel
path: gpt4o_reason/travel.parquet

- config_name: llama_11b_reason
features:
- name: id
dtype: string
- name: query
dtype: string
- name: gold_ids
sequence: string
- name: reasoning
dtype: string
data_files:
- split: academia
path: llama_11b_reason/academia.parquet
- split: apple
path: llama_11b_reason/apple.parquet
- split: askubuntu
path: llama_11b_reason/askubuntu.parquet
- split: aviation
path: llama_11b_reason/aviation.parquet
- split: bioacoustics
path: llama_11b_reason/bioacoustics.parquet
- split: bioinformatics
path: llama_11b_reason/bioinformatics.parquet
- split: biology
path: llama_11b_reason/biology.parquet
- split: bitcoin
path: llama_11b_reason/bitcoin.parquet
- split: chemistry
path: llama_11b_reason/chemistry.parquet
- split: christianity
path: llama_11b_reason/christianity.parquet
- split: crypto
path: llama_11b_reason/crypto.parquet
- split: earthscience
path: llama_11b_reason/earthscience.parquet
- split: economics
path: llama_11b_reason/economics.parquet
- split: gaming
path: llama_11b_reason/gaming.parquet
- split: gis
path: llama_11b_reason/gis.parquet
- split: islam
path: llama_11b_reason/islam.parquet
- split: law
path: llama_11b_reason/law.parquet
- split: math
path: llama_11b_reason/math.parquet
- split: medicalsciences
path: llama_11b_reason/medicalsciences.parquet
- split: philosophy
path: llama_11b_reason/philosophy.parquet
- split: physics
path: llama_11b_reason/physics.parquet
- split: pm
path: llama_11b_reason/pm.parquet
- split: psychology
path: llama_11b_reason/psychology.parquet
- split: quant
path: llama_11b_reason/quant.parquet
- split: quantumcomputing
path: llama_11b_reason/quantumcomputing.parquet
- split: robotics
path: llama_11b_reason/robotics.parquet
- split: salesforce
path: llama_11b_reason/salesforce.parquet
- split: sustainability
path: llama_11b_reason/sustainability.parquet
- split: travel
path: llama_11b_reason/travel.parquet

- config_name: llama_90b_reason
features:
- name: id
dtype: string
- name: query
dtype: string
- name: gold_ids
sequence: string
- name: reasoning
dtype: string
data_files:
- split: academia
path: llama_90b_reason/academia.parquet
- split: apple
path: llama_90b_reason/apple.parquet
- split: askubuntu
path: llama_90b_reason/askubuntu.parquet
- split: aviation
path: llama_90b_reason/aviation.parquet
- split: bioacoustics
path: llama_90b_reason/bioacoustics.parquet
- split: bioinformatics
path: llama_90b_reason/bioinformatics.parquet
- split: biology
path: llama_90b_reason/biology.parquet
- split: bitcoin
path: llama_90b_reason/bitcoin.parquet
- split: chemistry
path: llama_90b_reason/chemistry.parquet
- split: christianity
path: llama_90b_reason/christianity.parquet
- split: crypto
path: llama_90b_reason/crypto.parquet
- split: earthscience
path: llama_90b_reason/earthscience.parquet
- split: economics
path: llama_90b_reason/economics.parquet
- split: gaming
path: llama_90b_reason/gaming.parquet
- split: gis
path: llama_90b_reason/gis.parquet
- split: islam
path: llama_90b_reason/islam.parquet
- split: law
path: llama_90b_reason/law.parquet
- split: math
path: llama_90b_reason/math.parquet
- split: medicalsciences
path: llama_90b_reason/medicalsciences.parquet
- split: philosophy
path: llama_90b_reason/philosophy.parquet
- split: physics
path: llama_90b_reason/physics.parquet
- split: pm
path: llama_90b_reason/pm.parquet
- split: psychology
path: llama_90b_reason/psychology.parquet
- split: quant
path: llama_90b_reason/quant.parquet
- split: quantumcomputing
path: llama_90b_reason/quantumcomputing.parquet
- split: robotics
path: llama_90b_reason/robotics.parquet
- split: salesforce
path: llama_90b_reason/salesforce.parquet
- split: sustainability
path: llama_90b_reason/sustainability.parquet
- split: travel
path: llama_90b_reason/travel.parquet

- config_name: qwen_3b_reason
features:
- name: id
dtype: string
- name: query
dtype: string
- name: gold_ids
sequence: string
- name: reasoning
dtype: string
data_files:
- split: academia
path: qwen_3b_reason/academia.parquet
- split: apple
path: qwen_3b_reason/apple.parquet
- split: askubuntu
path: qwen_3b_reason/askubuntu.parquet
- split: aviation
path: qwen_3b_reason/aviation.parquet
- split: bioacoustics
path: qwen_3b_reason/bioacoustics.parquet
- split: bioinformatics
path: qwen_3b_reason/bioinformatics.parquet
- split: biology
path: qwen_3b_reason/biology.parquet
- split: bitcoin
path: qwen_3b_reason/bitcoin.parquet
- split: chemistry
path: qwen_3b_reason/chemistry.parquet
- split: christianity
path: qwen_3b_reason/christianity.parquet
- split: crypto
path: qwen_3b_reason/crypto.parquet
- split: earthscience
path: qwen_3b_reason/earthscience.parquet
- split: economics
path: qwen_3b_reason/economics.parquet
- split: gaming
path: qwen_3b_reason/gaming.parquet
- split: gis
path: qwen_3b_reason/gis.parquet
- split: islam
path: qwen_3b_reason/islam.parquet
- split: law
path: qwen_3b_reason/law.parquet
- split: math
path: qwen_3b_reason/math.parquet
- split: medicalsciences
path: qwen_3b_reason/medicalsciences.parquet
- split: philosophy
path: qwen_3b_reason/philosophy.parquet
- split: physics
path: qwen_3b_reason/physics.parquet
- split: pm
path: qwen_3b_reason/pm.parquet
- split: psychology
path: qwen_3b_reason/psychology.parquet
- split: quant
path: qwen_3b_reason/quant.parquet
- split: quantumcomputing
path: qwen_3b_reason/quantumcomputing.parquet
- split: robotics
path: qwen_3b_reason/robotics.parquet
- split: salesforce
path: qwen_3b_reason/salesforce.parquet
- split: sustainability
path: qwen_3b_reason/sustainability.parquet
- split: travel
path: qwen_3b_reason/travel.parquet

- config_name: qwen_7b_reason
features:
- name: id
dtype: string
- name: query
dtype: string
- name: gold_ids
sequence: string
- name: reasoning
dtype: string
data_files:
- split: academia
path: qwen_7b_reason/academia.parquet
- split: apple
path: qwen_7b_reason/apple.parquet
- split: askubuntu
path: qwen_7b_reason/askubuntu.parquet
- split: aviation
path: qwen_7b_reason/aviation.parquet
- split: bioacoustics
path: qwen_7b_reason/bioacoustics.parquet
- split: bioinformatics
path: qwen_7b_reason/bioinformatics.parquet
- split: biology
path: qwen_7b_reason/biology.parquet
- split: bitcoin
path: qwen_7b_reason/bitcoin.parquet
- split: chemistry
path: qwen_7b_reason/chemistry.parquet
- split: christianity
path: qwen_7b_reason/christianity.parquet
- split: crypto
path: qwen_7b_reason/crypto.parquet
- split: earthscience
path: qwen_7b_reason/earthscience.parquet
- split: economics
path: qwen_7b_reason/economics.parquet
- split: gaming
path: qwen_7b_reason/gaming.parquet
- split: gis
path: qwen_7b_reason/gis.parquet
- split: islam
path: qwen_7b_reason/islam.parquet
- split: law
path: qwen_7b_reason/law.parquet
- split: math
path: qwen_7b_reason/math.parquet
- split: medicalsciences
path: qwen_7b_reason/medicalsciences.parquet
- split: philosophy
path: qwen_7b_reason/philosophy.parquet
- split: physics
path: qwen_7b_reason/physics.parquet
- split: pm
path: qwen_7b_reason/pm.parquet
- split: psychology
path: qwen_7b_reason/psychology.parquet
- split: quant
path: qwen_7b_reason/quant.parquet
- split: quantumcomputing
path: qwen_7b_reason/quantumcomputing.parquet
- split: robotics
path: qwen_7b_reason/robotics.parquet
- split: salesforce
path: qwen_7b_reason/salesforce.parquet
- split: sustainability
path: qwen_7b_reason/sustainability.parquet
- split: travel
path: qwen_7b_reason/travel.parquet

- config_name: qwen_32b_reason
features:
- name: id
dtype: string
- name: query
dtype: string
- name: gold_ids
sequence: string
- name: reasoning
dtype: string
data_files:
- split: academia
path: qwen_32b_reason/academia.parquet
- split: apple
path: qwen_32b_reason/apple.parquet
- split: askubuntu
path: qwen_32b_reason/askubuntu.parquet
- split: aviation
path: qwen_32b_reason/aviation.parquet
- split: bioacoustics
path: qwen_32b_reason/bioacoustics.parquet
- split: bioinformatics
path: qwen_32b_reason/bioinformatics.parquet
- split: biology
path: qwen_32b_reason/biology.parquet
- split: bitcoin
path: qwen_32b_reason/bitcoin.parquet
- split: chemistry
path: qwen_32b_reason/chemistry.parquet
- split: christianity
path: qwen_32b_reason/christianity.parquet
- split: crypto
path: qwen_32b_reason/crypto.parquet
- split: earthscience
path: qwen_32b_reason/earthscience.parquet
- split: economics
path: qwen_32b_reason/economics.parquet
- split: gaming
path: qwen_32b_reason/gaming.parquet
- split: gis
path: qwen_32b_reason/gis.parquet
- split: islam
path: qwen_32b_reason/islam.parquet
- split: law
path: qwen_32b_reason/law.parquet
- split: math
path: qwen_32b_reason/math.parquet
- split: medicalsciences
path: qwen_32b_reason/medicalsciences.parquet
- split: philosophy
path: qwen_32b_reason/philosophy.parquet
- split: physics
path: qwen_32b_reason/physics.parquet
- split: pm
path: qwen_32b_reason/pm.parquet
- split: psychology
path: qwen_32b_reason/psychology.parquet
- split: quant
path: qwen_32b_reason/quant.parquet
- split: quantumcomputing
path: qwen_32b_reason/quantumcomputing.parquet
- split: robotics
path: qwen_32b_reason/robotics.parquet
- split: salesforce
path: qwen_32b_reason/salesforce.parquet
- split: sustainability
path: qwen_32b_reason/sustainability.parquet
- split: travel
path: qwen_32b_reason/travel.parquet

- config_name: qwen_72b_reason
features:
- name: id
dtype: string
- name: query
dtype: string
- name: gold_ids
sequence: string
- name: reasoning
dtype: string
data_files:
- split: academia
path: qwen_72b_reason/academia.parquet
- split: apple
path: qwen_72b_reason/apple.parquet
- split: askubuntu
path: qwen_72b_reason/askubuntu.parquet
- split: aviation
path: qwen_72b_reason/aviation.parquet
- split: bioacoustics
path: qwen_72b_reason/bioacoustics.parquet
- split: bioinformatics
path: qwen_72b_reason/bioinformatics.parquet
- split: biology
path: qwen_72b_reason/biology.parquet
- split: bitcoin
path: qwen_72b_reason/bitcoin.parquet
- split: chemistry
path: qwen_72b_reason/chemistry.parquet
- split: christianity
path: qwen_72b_reason/christianity.parquet
- split: crypto
path: qwen_72b_reason/crypto.parquet
- split: earthscience
path: qwen_72b_reason/earthscience.parquet
- split: economics
path: qwen_72b_reason/economics.parquet
- split: gaming
path: qwen_72b_reason/gaming.parquet
- split: gis
path: qwen_72b_reason/gis.parquet
- split: islam
path: qwen_72b_reason/islam.parquet
- split: law
path: qwen_72b_reason/law.parquet
- split: math
path: qwen_72b_reason/math.parquet
- split: medicalsciences
path: qwen_72b_reason/medicalsciences.parquet
- split: philosophy
path: qwen_72b_reason/philosophy.parquet
- split: physics
path: qwen_72b_reason/physics.parquet
- split: pm
path: qwen_72b_reason/pm.parquet
- split: psychology
path: qwen_72b_reason/psychology.parquet
- split: quant
path: qwen_72b_reason/quant.parquet
- split: quantumcomputing
path: qwen_72b_reason/quantumcomputing.parquet
- split: robotics
path: qwen_72b_reason/robotics.parquet
- split: salesforce
path: qwen_72b_reason/salesforce.parquet
- split: sustainability
path: qwen_72b_reason/sustainability.parquet
- split: travel
path: qwen_72b_reason/travel.parquet

# ========================================================
# 4. CAPTION VARIATIONS (7 Models)
# ========================================================
- config_name: caption_gpt4o
features:
- name: id
dtype: string
- name: query
dtype: string
- name: gold_ids
sequence: string
- name: image_paths
sequence: string
- name: llm_image_caption
dtype: string
data_files:
- split: academia
path: caption_gpt4o/academia.parquet
- split: apple
path: caption_gpt4o/apple.parquet
- split: askubuntu
path: caption_gpt4o/askubuntu.parquet
- split: aviation
path: caption_gpt4o/aviation.parquet
- split: bioacoustics
path: caption_gpt4o/bioacoustics.parquet
- split: bioinformatics
path: caption_gpt4o/bioinformatics.parquet
- split: biology
path: caption_gpt4o/biology.parquet
- split: bitcoin
path: caption_gpt4o/bitcoin.parquet
- split: chemistry
path: caption_gpt4o/chemistry.parquet
- split: christianity
path: caption_gpt4o/christianity.parquet
- split: crypto
path: caption_gpt4o/crypto.parquet
- split: earthscience
path: caption_gpt4o/earthscience.parquet
- split: economics
path: caption_gpt4o/economics.parquet
- split: gaming
path: caption_gpt4o/gaming.parquet
- split: gis
path: caption_gpt4o/gis.parquet
- split: islam
path: caption_gpt4o/islam.parquet
- split: law
path: caption_gpt4o/law.parquet
- split: math
path: caption_gpt4o/math.parquet
- split: medicalsciences
path: caption_gpt4o/medicalsciences.parquet
- split: philosophy
path: caption_gpt4o/philosophy.parquet
- split: physics
path: caption_gpt4o/physics.parquet
- split: pm
path: caption_gpt4o/pm.parquet
- split: psychology
path: caption_gpt4o/psychology.parquet
- split: quant
path: caption_gpt4o/quant.parquet
- split: quantumcomputing
path: caption_gpt4o/quantumcomputing.parquet
- split: robotics
path: caption_gpt4o/robotics.parquet
- split: salesforce
path: caption_gpt4o/salesforce.parquet
- split: sustainability
path: caption_gpt4o/sustainability.parquet
- split: travel
path: caption_gpt4o/travel.parquet

- config_name: caption_llama_11b
features:
- name: id
dtype: string
- name: query
dtype: string
- name: gold_ids
sequence: string
- name: image_paths
sequence: string
- name: llm_image_caption
dtype: string
data_files:
- split: academia
path: caption_llama_11b/academia.parquet
- split: apple
path: caption_llama_11b/apple.parquet
- split: askubuntu
path: caption_llama_11b/askubuntu.parquet
- split: aviation
path: caption_llama_11b/aviation.parquet
- split: bioacoustics
path: caption_llama_11b/bioacoustics.parquet
- split: bioinformatics
path: caption_llama_11b/bioinformatics.parquet
- split: biology
path: caption_llama_11b/biology.parquet
- split: bitcoin
path: caption_llama_11b/bitcoin.parquet
- split: chemistry
path: caption_llama_11b/chemistry.parquet
- split: christianity
path: caption_llama_11b/christianity.parquet
- split: crypto
path: caption_llama_11b/crypto.parquet
- split: earthscience
path: caption_llama_11b/earthscience.parquet
- split: economics
path: caption_llama_11b/economics.parquet
- split: gaming
path: caption_llama_11b/gaming.parquet
- split: gis
path: caption_llama_11b/gis.parquet
- split: islam
path: caption_llama_11b/islam.parquet
- split: law
path: caption_llama_11b/law.parquet
- split: math
path: caption_llama_11b/math.parquet
- split: medicalsciences
path: caption_llama_11b/medicalsciences.parquet
- split: philosophy
path: caption_llama_11b/philosophy.parquet
- split: physics
path: caption_llama_11b/physics.parquet
- split: pm
path: caption_llama_11b/pm.parquet
- split: psychology
path: caption_llama_11b/psychology.parquet
- split: quant
path: caption_llama_11b/quant.parquet
- split: quantumcomputing
path: caption_llama_11b/quantumcomputing.parquet
- split: robotics
path: caption_llama_11b/robotics.parquet
- split: salesforce
path: caption_llama_11b/salesforce.parquet
- split: sustainability
path: caption_llama_11b/sustainability.parquet
- split: travel
path: caption_llama_11b/travel.parquet

- config_name: caption_llama_90b
features:
- name: id
dtype: string
- name: query
dtype: string
- name: gold_ids
sequence: string
- name: image_paths
sequence: string
- name: llm_image_caption
dtype: string
data_files:
- split: academia
path: caption_llama_90b/academia.parquet
- split: apple
path: caption_llama_90b/apple.parquet
- split: askubuntu
path: caption_llama_90b/askubuntu.parquet
- split: aviation
path: caption_llama_90b/aviation.parquet
- split: bioacoustics
path: caption_llama_90b/bioacoustics.parquet
- split: bioinformatics
path: caption_llama_90b/bioinformatics.parquet
- split: biology
path: caption_llama_90b/biology.parquet
- split: bitcoin
path: caption_llama_90b/bitcoin.parquet
- split: chemistry
path: caption_llama_90b/chemistry.parquet
- split: christianity
path: caption_llama_90b/christianity.parquet
- split: crypto
path: caption_llama_90b/crypto.parquet
- split: earthscience
path: caption_llama_90b/earthscience.parquet
- split: economics
path: caption_llama_90b/economics.parquet
- split: gaming
path: caption_llama_90b/gaming.parquet
- split: gis
path: caption_llama_90b/gis.parquet
- split: islam
path: caption_llama_90b/islam.parquet
- split: law
path: caption_llama_90b/law.parquet
- split: math
path: caption_llama_90b/math.parquet
- split: medicalsciences
path: caption_llama_90b/medicalsciences.parquet
- split: philosophy
path: caption_llama_90b/philosophy.parquet
- split: physics
path: caption_llama_90b/physics.parquet
- split: pm
path: caption_llama_90b/pm.parquet
- split: psychology
path: caption_llama_90b/psychology.parquet
- split: quant
path: caption_llama_90b/quant.parquet
- split: quantumcomputing
path: caption_llama_90b/quantumcomputing.parquet
- split: robotics
path: caption_llama_90b/robotics.parquet
- split: salesforce
path: caption_llama_90b/salesforce.parquet
- split: sustainability
path: caption_llama_90b/sustainability.parquet
- split: travel
path: caption_llama_90b/travel.parquet

- config_name: caption_qwen_3b
features:
- name: id
dtype: string
- name: query
dtype: string
- name: gold_ids
sequence: string
- name: image_paths
sequence: string
- name: llm_image_caption
dtype: string
data_files:
- split: academia
path: caption_qwen_3b/academia.parquet
- split: apple
path: caption_qwen_3b/apple.parquet
- split: askubuntu
path: caption_qwen_3b/askubuntu.parquet
- split: aviation
path: caption_qwen_3b/aviation.parquet
- split: bioacoustics
path: caption_qwen_3b/bioacoustics.parquet
- split: bioinformatics
path: caption_qwen_3b/bioinformatics.parquet
- split: biology
path: caption_qwen_3b/biology.parquet
- split: bitcoin
path: caption_qwen_3b/bitcoin.parquet
- split: chemistry
path: caption_qwen_3b/chemistry.parquet
- split: christianity
path: caption_qwen_3b/christianity.parquet
- split: crypto
path: caption_qwen_3b/crypto.parquet
- split: earthscience
path: caption_qwen_3b/earthscience.parquet
- split: economics
path: caption_qwen_3b/economics.parquet
- split: gaming
path: caption_qwen_3b/gaming.parquet
- split: gis
path: caption_qwen_3b/gis.parquet
- split: islam
path: caption_qwen_3b/islam.parquet
- split: law
path: caption_qwen_3b/law.parquet
- split: math
path: caption_qwen_3b/math.parquet
- split: medicalsciences
path: caption_qwen_3b/medicalsciences.parquet
- split: philosophy
path: caption_qwen_3b/philosophy.parquet
- split: physics
path: caption_qwen_3b/physics.parquet
- split: pm
path: caption_qwen_3b/pm.parquet
- split: psychology
path: caption_qwen_3b/psychology.parquet
- split: quant
path: caption_qwen_3b/quant.parquet
- split: quantumcomputing
path: caption_qwen_3b/quantumcomputing.parquet
- split: robotics
path: caption_qwen_3b/robotics.parquet
- split: salesforce
path: caption_qwen_3b/salesforce.parquet
- split: sustainability
path: caption_qwen_3b/sustainability.parquet
- split: travel
path: caption_qwen_3b/travel.parquet

- config_name: caption_qwen_7b
features:
- name: id
dtype: string
- name: query
dtype: string
- name: gold_ids
sequence: string
- name: image_paths
sequence: string
- name: llm_image_caption
dtype: string
data_files:
- split: academia
path: caption_qwen_7b/academia.parquet
- split: apple
path: caption_qwen_7b/apple.parquet
- split: askubuntu
path: caption_qwen_7b/askubuntu.parquet
- split: aviation
path: caption_qwen_7b/aviation.parquet
- split: bioacoustics
path: caption_qwen_7b/bioacoustics.parquet
- split: bioinformatics
path: caption_qwen_7b/bioinformatics.parquet
- split: biology
path: caption_qwen_7b/biology.parquet
- split: bitcoin
path: caption_qwen_7b/bitcoin.parquet
- split: chemistry
path: caption_qwen_7b/chemistry.parquet
- split: christianity
path: caption_qwen_7b/christianity.parquet
- split: crypto
path: caption_qwen_7b/crypto.parquet
- split: earthscience
path: caption_qwen_7b/earthscience.parquet
- split: economics
path: caption_qwen_7b/economics.parquet
- split: gaming
path: caption_qwen_7b/gaming.parquet
- split: gis
path: caption_qwen_7b/gis.parquet
- split: islam
path: caption_qwen_7b/islam.parquet
- split: law
path: caption_qwen_7b/law.parquet
- split: math
path: caption_qwen_7b/math.parquet
- split: medicalsciences
path: caption_qwen_7b/medicalsciences.parquet
- split: philosophy
path: caption_qwen_7b/philosophy.parquet
- split: physics
path: caption_qwen_7b/physics.parquet
- split: pm
path: caption_qwen_7b/pm.parquet
- split: psychology
path: caption_qwen_7b/psychology.parquet
- split: quant
path: caption_qwen_7b/quant.parquet
- split: quantumcomputing
path: caption_qwen_7b/quantumcomputing.parquet
- split: robotics
path: caption_qwen_7b/robotics.parquet
- split: salesforce
path: caption_qwen_7b/salesforce.parquet
- split: sustainability
path: caption_qwen_7b/sustainability.parquet
- split: travel
path: caption_qwen_7b/travel.parquet

- config_name: caption_qwen_32b
features:
- name: id
dtype: string
- name: query
dtype: string
- name: gold_ids
sequence: string
- name: image_paths
sequence: string
- name: llm_image_caption
dtype: string
data_files:
- split: academia
path: caption_qwen_32b/academia.parquet
- split: apple
path: caption_qwen_32b/apple.parquet
- split: askubuntu
path: caption_qwen_32b/askubuntu.parquet
- split: aviation
path: caption_qwen_32b/aviation.parquet
- split: bioacoustics
path: caption_qwen_32b/bioacoustics.parquet
- split: bioinformatics
path: caption_qwen_32b/bioinformatics.parquet
- split: biology
path: caption_qwen_32b/biology.parquet
- split: bitcoin
path: caption_qwen_32b/bitcoin.parquet
- split: chemistry
path: caption_qwen_32b/chemistry.parquet
- split: christianity
path: caption_qwen_32b/christianity.parquet
- split: crypto
path: caption_qwen_32b/crypto.parquet
- split: earthscience
path: caption_qwen_32b/earthscience.parquet
- split: economics
path: caption_qwen_32b/economics.parquet
- split: gaming
path: caption_qwen_32b/gaming.parquet
- split: gis
path: caption_qwen_32b/gis.parquet
- split: islam
path: caption_qwen_32b/islam.parquet
- split: law
path: caption_qwen_32b/law.parquet
- split: math
path: caption_qwen_32b/math.parquet
- split: medicalsciences
path: caption_qwen_32b/medicalsciences.parquet
- split: philosophy
path: caption_qwen_32b/philosophy.parquet
- split: physics
path: caption_qwen_32b/physics.parquet
- split: pm
path: caption_qwen_32b/pm.parquet
- split: psychology
path: caption_qwen_32b/psychology.parquet
- split: quant
path: caption_qwen_32b/quant.parquet
- split: quantumcomputing
path: caption_qwen_32b/quantumcomputing.parquet
- split: robotics
path: caption_qwen_32b/robotics.parquet
- split: salesforce
path: caption_qwen_32b/salesforce.parquet
- split: sustainability
path: caption_qwen_32b/sustainability.parquet
- split: travel
path: caption_qwen_32b/travel.parquet

- config_name: caption_qwen_72b
features:
- name: id
dtype: string
- name: query
dtype: string
- name: gold_ids
sequence: string
- name: image_paths
sequence: string
- name: llm_image_caption
dtype: string
data_files:
- split: academia
path: caption_qwen_72b/academia.parquet
- split: apple
path: caption_qwen_72b/apple.parquet
- split: askubuntu
path: caption_qwen_72b/askubuntu.parquet
- split: aviation
path: caption_qwen_72b/aviation.parquet
- split: bioacoustics
path: caption_qwen_72b/bioacoustics.parquet
- split: bioinformatics
path: caption_qwen_72b/bioinformatics.parquet
- split: biology
path: caption_qwen_72b/biology.parquet
- split: bitcoin
path: caption_qwen_72b/bitcoin.parquet
- split: chemistry
path: caption_qwen_72b/chemistry.parquet
- split: christianity
path: caption_qwen_72b/christianity.parquet
- split: crypto
path: caption_qwen_72b/crypto.parquet
- split: earthscience
path: caption_qwen_72b/earthscience.parquet
- split: economics
path: caption_qwen_72b/economics.parquet
- split: gaming
path: caption_qwen_72b/gaming.parquet
- split: gis
path: caption_qwen_72b/gis.parquet
- split: islam
path: caption_qwen_72b/islam.parquet
- split: law
path: caption_qwen_72b/law.parquet
- split: math
path: caption_qwen_72b/math.parquet
- split: medicalsciences
path: caption_qwen_72b/medicalsciences.parquet
- split: philosophy
path: caption_qwen_72b/philosophy.parquet
- split: physics
path: caption_qwen_72b/physics.parquet
- split: pm
path: caption_qwen_72b/pm.parquet
- split: psychology
path: caption_qwen_72b/psychology.parquet
- split: quant
path: caption_qwen_72b/quant.parquet
- split: quantumcomputing
path: caption_qwen_72b/quantumcomputing.parquet
- split: robotics
path: caption_qwen_72b/robotics.parquet
- split: salesforce
path: caption_qwen_72b/salesforce.parquet
- split: sustainability
path: caption_qwen_72b/sustainability.parquet
- split: travel
path: caption_qwen_72b/travel.parquet
---

# MM-BRIGHT: A Multi-Task Multimodal Benchmark for Reasoning-Intensive Retrieval

**MM-BRIGHT** is the first **multimodal benchmark** designed for **reasoning-intensive retrieval**. Unlike existing benchmarks that primarily consist of text-based, keyword-centric queries, MM-BRIGHT targets complex real-world scenarios where queries contain multimodal elements—such as diagrams, charts, and screenshots—that require deep reasoning to identify relevant documents.

## 📄 Abstract

Existing retrieval benchmarks primarily consist of text-based queries where keyword or semantic matching is usually sufficient. Many real-world queries contain multimodal elements, particularly, images such as diagrams, charts, and screenshots that require intensive reasoning to identify relevant documents. To address this gap, we introduce **MM-BRIGHT**, the first multimodal benchmark for reasoning-intensive retrieval. Our dataset consists of **2,803 real-world queries** spanning **29 diverse technical domains**, with four tasks of increasing complexity: text-to-text, multimodal-to-text, multimodal-to-image, and multimodal-to-multimodal retrieval.

## 🚀 Tasks

To comprehensively evaluate multimodal retrieval capabilities, we systematically define four retrieval tasks of increasing multimodal complexity:

1. **Task 1: Text-to-Text (Query → Documents)**
* Traditional text-only retrieval, serving as a baseline to understand reasoning intensity without multimodal complexity.
2. **Task 2: Multimodal-to-Text (Query+Image → Documents)**
* Multimodal queries retrieving text documents, testing whether models can leverage visual context to improve text retrieval.
3. **Task 3: Multimodal-to-Image (Query+Image → Images)**
* Multimodal queries retrieving relevant images, requiring visual reasoning and similarity assessment beyond simple object matching.
4. **Task 4: Multimodal-to-Multimodal (Query+Image → Documents+Images)**
* The most challenging task, retrieving multimodal documents where both text and images must be jointly evaluated for relevance.

## 📊 Statistics and Domains

**MM-BRIGHT** spans **29 diverse technical domains** sourced from StackExchange, including:
* **STEM**: Biology, Chemistry, Physics, Mathematics, Earth Science, Bioacoustics, Bioinformatics, Medical Sciences
* **Computing**: Ubuntu, Bitcoin, Cryptography, Quantum Computing, Robotics, Salesforce, GIS, Apple
* **Social Sciences**: Economics, Psychology, Philosophy, Law, Christianity, Islam
* **Applied Domains**: Aviation, Gaming, Project Management, Quantitative Finance, Sustainability, Travel, Academia

The dataset contains:
* **2,803** Total Queries
* **7,621** Verified Images
* **2.5 Million+** Corpus Documents

### Image Diversity
The benchmark features varied image types:
* Photos (27.2%)
* Diagrams (17.1%)
* Charts/Graphs (16.1%)
* Screenshots (13.9%)
* Scientific Figures (11.6%)

## 💻 Usage

The dataset is organized into configurations to support different tasks and model variations.

```python
from datasets import load_dataset

# 1. Load the Corpus (Knowledge Base)
corpus = load_dataset("mm-bright/MM-BRIGHT", "documents")

# 2. Load Standard Queries (Task 1 & 2)
# Features: id, query, gold_ids, gold_answers, image_paths, negative_ids, llm_image_caption, domain
queries = load_dataset("mm-bright/MM-BRIGHT", "examples")

# 3. Load Multimodal Queries (Task 3 & 4)
# Features: id, query, gold_ids, gold_answers, image_paths, negative_ids, llm_image_caption, domain
mm_queries = load_dataset("mm-bright/MM-BRIGHT", "examples_multimodal")

# 4. Load Images (Binary Data)
query_images = load_dataset("mm-bright/MM-BRIGHT", "examples_images")
doc_images = load_dataset("mm-bright/MM-BRIGHT", "document_images")

# 5. Load Reasoning Traces (Choose your model)
# Available: gpt4o, llama_11b, llama_90b, qwen_3b, qwen_7b, qwen_32b, qwen_72b
reasoning = load_dataset("mm-bright/MM-BRIGHT", "gpt4o_reason")

# 6. Load Caption-Augmented Queries (Choose your model)
captions = load_dataset("mm-bright/MM-BRIGHT", "caption_gpt4o")
```

## 📚 Citation

```bibtex
soon
```