SPARK
收藏资源简介：
SPARK数据集由韩国科学技术院集成视觉语言实验室开发，专注于评估大型视觉语言模型在多视觉传感器感知和推理方面的性能。该数据集包含6,248个视觉语言测试样本，涵盖多种传感器相关问题，旨在深入研究模型对物理传感器知识的理解和应用。数据集的创建过程涉及多种视觉感知和推理任务的设计，以确保模型能够准确处理和理解来自不同视觉传感器的数据。SPARK数据集主要应用于提升模型在复杂传感器相关问题上的推理能力，特别是在需要考虑物理环境的领域，如自动驾驶、安全系统和医学图像诊断。
The SPARK dataset was developed by the Integrated Vision and Language Laboratory of the Korea Advanced Institute of Science and Technology (KAIST), focusing on evaluating the performance of large vision-language models in multi-visual-sensor perception and reasoning tasks. This dataset includes 6,248 vision-language test samples covering a wide range of sensor-related questions, aiming to conduct in-depth research on models' understanding and application of physical sensor knowledge. The development process of the SPARK dataset involves designing diverse visual perception and reasoning tasks to ensure that models can accurately process and comprehend data from different visual sensors. The SPARK dataset is primarily applied to improve models' reasoning abilities on complex sensor-related problems, especially in physical environment-aware fields such as autonomous driving, security systems, and medical image diagnostics.
SPARK: 多传感器感知与推理基准
简介
SPARK是一个用于测量大规模视觉语言模型对多传感器感知和推理能力的基准数据集。该数据集包含四种传感器（RGB、热成像、深度、X射线）和六种类型的问答（存在性、计数、位置、场景描述、上下文推理、传感器推理）。数据集包含约6,000个问题和答案，分为多传感器感知和多传感器推理两大类。
数据集下载
数据集已上传至Huggingface，可通过以下代码下载： python from datasets import load_dataset test_dataset = load_dataset("topyun/SPARK", split="train")
评估方法
提供了两个示例代码用于评估：
- 开放模型：test.py
- 封闭模型：test_closed_models.py
运行示例
-
使用4个GPU运行llava-1.5-7b模型： bash accelerate launch --config_file utils/ddp_accel_fp16.yaml --num_processes=4 test.py --batch_size 1 --model llava
-
使用1个GPU运行gpt-4o模型： bash accelerate launch --config_file utils/ddp_accel_fp16.yaml --num_processes=$n_gpu test_closed_models.py --batch_size 8 --model gpt --multiprocess True
引用
bibtex @misc{yu2024sparkmultivisionsensorperception, title={SPARK: Multi-Vision Sensor Perception and Reasoning Benchmark for Large-scale Vision-Language Models}, author={Youngjoon Yu and Sangyun Chung and Byung-Kwan Lee and Yong Man Ro}, year={2024}, eprint={2408.12114}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2408.12114}, }
- 1SPARK: Multi-Vision Sensor Perception and Reasoning Benchmark for Large-scale Vision-Language Models韩国科学技术院集成视觉语言实验室 · 2024年
