five

VLM-LAB/JaBLINK

收藏
Hugging Face2024-05-03 更新2024-06-12 收录
下载链接：
https://hf-mirror.com/datasets/VLM-LAB/JaBLINK
下载链接
链接失效反馈
官方服务：
资源简介：
--- license: apache-2.0 dataset_info: - config_name: Art_Style features: - name: idx dtype: string - name: question dtype: string - name: sub_task dtype: string - name: image_1 dtype: image - name: image_2 dtype: image - name: image_3 dtype: image - name: image_4 dtype: image - name: choices sequence: string - name: answer dtype: string - name: prompt dtype: string - name: explanation dtype: string splits: - name: val num_bytes: 146463120.0 num_examples: 117 - name: test num_bytes: 145348441.0 num_examples: 117 download_size: 291074297 dataset_size: 291811561.0 - config_name: Counting features: - name: idx dtype: string - name: question dtype: string - name: sub_task dtype: string - name: image_1 dtype: image - name: image_2 dtype: image - name: image_3 dtype: image - name: image_4 dtype: image - name: choices sequence: string - name: answer dtype: string - name: prompt dtype: string - name: explanation dtype: string splits: - name: val num_bytes: 4704215.0 num_examples: 120 - name: test num_bytes: 5329253.0 num_examples: 120 download_size: 10015874 dataset_size: 10033468.0 - config_name: Forensic_Detection features: - name: idx dtype: string - name: question dtype: string - name: sub_task dtype: string - name: image_1 dtype: image - name: image_2 dtype: image - name: image_3 dtype: image - name: image_4 dtype: image - name: choices sequence: string - name: answer dtype: string - name: prompt dtype: string - name: explanation dtype: string splits: - name: val num_bytes: 19625323.0 num_examples: 132 - name: test num_bytes: 19750403.0 num_examples: 132 download_size: 39272509 dataset_size: 39375726.0 - config_name: Functional_Correspondence features: - name: idx dtype: string - name: question dtype: string - name: sub_task dtype: string - name: image_1 dtype: image - name: image_2 dtype: image - name: image_3 dtype: image - name: image_4 dtype: image - name: choices sequence: string - name: answer dtype: string - name: prompt dtype: string - name: explanation dtype: string splits: - name: val num_bytes: 26361184.0 num_examples: 130 - name: test num_bytes: 28367706.0 num_examples: 130 download_size: 53227222 dataset_size: 54728890.0 - config_name: IQ_Test features: - name: idx dtype: string - name: question dtype: string - name: sub_task dtype: string - name: image_1 dtype: image - name: image_2 dtype: image - name: image_3 dtype: image - name: image_4 dtype: image - name: choices sequence: string - name: answer dtype: string - name: prompt dtype: string - name: explanation dtype: string splits: - name: val num_bytes: 5306379.0 num_examples: 150 - name: test num_bytes: 4835987.0 num_examples: 150 download_size: 7156052 dataset_size: 10142366.0 - config_name: Jigsaw features: - name: idx dtype: string - name: question dtype: string - name: sub_task dtype: string - name: image_1 dtype: image - name: image_2 dtype: image - name: image_3 dtype: image - name: image_4 dtype: image - name: choices sequence: string - name: answer dtype: string - name: prompt dtype: string - name: explanation dtype: string splits: - name: val num_bytes: 3798073.0 num_examples: 150 - name: test num_bytes: 4496412.0 num_examples: 150 download_size: 8085696 dataset_size: 8294485.0 - config_name: Multi-view_Reasoning features: - name: idx dtype: string - name: question dtype: string - name: sub_task dtype: string - name: image_1 dtype: image - name: image_2 dtype: image - name: image_3 dtype: image - name: image_4 dtype: image - name: choices sequence: string - name: answer dtype: string - name: prompt dtype: string - name: explanation dtype: string splits: - name: val num_bytes: 9809322.0 num_examples: 133 - name: test num_bytes: 9606003.0 num_examples: 133 download_size: 19270001 dataset_size: 19415325.0 - config_name: Object_Localization features: - name: idx dtype: string - name: question dtype: string - name: sub_task dtype: string - name: image_1 dtype: image - name: image_2 dtype: image - name: image_3 dtype: image - name: image_4 dtype: image - name: choices sequence: string - name: answer dtype: string - name: prompt dtype: string - name: explanation dtype: string splits: - name: val num_bytes: 6240389.0 num_examples: 122 - name: test num_bytes: 6441081.0 num_examples: 125 download_size: 12591166 dataset_size: 12681470.0 - config_name: Relative_Depth features: - name: idx dtype: string - name: question dtype: string - name: sub_task dtype: string - name: image_1 dtype: image - name: image_2 dtype: image - name: image_3 dtype: image - name: image_4 dtype: image - name: choices sequence: string - name: answer dtype: string - name: prompt dtype: string - name: explanation dtype: string splits: - name: val num_bytes: 4631251.0 num_examples: 124 - name: test num_bytes: 4627481.0 num_examples: 124 download_size: 9203975 dataset_size: 9258732.0 - config_name: Relative_Reflectance features: - name: idx dtype: string - name: question dtype: string - name: sub_task dtype: string - name: image_1 dtype: image - name: image_2 dtype: image - name: image_3 dtype: image - name: image_4 dtype: image - name: choices sequence: string - name: answer dtype: string - name: prompt dtype: string - name: explanation dtype: string splits: - name: val num_bytes: 18605693.0 num_examples: 134 - name: test num_bytes: 18299553.0 num_examples: 134 download_size: 36780997 dataset_size: 36905246.0 - config_name: Semantic_Correspondence features: - name: idx dtype: string - name: question dtype: string - name: sub_task dtype: string - name: image_1 dtype: image - name: image_2 dtype: image - name: image_3 dtype: image - name: image_4 dtype: image - name: choices sequence: string - name: answer dtype: string - name: prompt dtype: string - name: explanation dtype: string splits: - name: val num_bytes: 46913152.0 num_examples: 139 - name: test num_bytes: 43943993.0 num_examples: 140 download_size: 90492443 dataset_size: 90857145.0 - config_name: Spatial_Relation features: - name: idx dtype: string - name: question dtype: string - name: sub_task dtype: string - name: image_1 dtype: image - name: image_2 dtype: image - name: image_3 dtype: image - name: image_4 dtype: image - name: choices sequence: string - name: answer dtype: string - name: prompt dtype: string - name: explanation dtype: string splits: - name: val num_bytes: 7306558.0 num_examples: 143 - name: test num_bytes: 7472518.0 num_examples: 143 download_size: 14596727 dataset_size: 14779076.0 - config_name: Visual_Correspondence features: - name: idx dtype: string - name: question dtype: string - name: sub_task dtype: string - name: image_1 dtype: image - name: image_2 dtype: image - name: image_3 dtype: image - name: image_4 dtype: image - name: choices sequence: string - name: answer dtype: string - name: prompt dtype: string - name: explanation dtype: string splits: - name: val num_bytes: 60403442.0 num_examples: 172 - name: test num_bytes: 56793513.0 num_examples: 172 download_size: 116448573 dataset_size: 117196955.0 - config_name: Visual_Similarity features: - name: idx dtype: string - name: question dtype: string - name: sub_task dtype: string - name: image_1 dtype: image - name: image_2 dtype: image - name: image_3 dtype: image - name: image_4 dtype: image - name: choices sequence: string - name: answer dtype: string - name: prompt dtype: string - name: explanation dtype: string splits: - name: val num_bytes: 44097854.0 num_examples: 135 - name: test num_bytes: 45045115.0 num_examples: 136 download_size: 89068648 dataset_size: 89142969.0 configs: - config_name: Art_Style data_files: - split: val path: Art_Style/val-* - split: test path: Art_Style/test-* - config_name: Counting data_files: - split: val path: Counting/val-* - split: test path: Counting/test-* - config_name: Forensic_Detection data_files: - split: val path: Forensic_Detection/val-* - split: test path: Forensic_Detection/test-* - config_name: Functional_Correspondence data_files: - split: val path: Functional_Correspondence/val-* - split: test path: Functional_Correspondence/test-* - config_name: IQ_Test data_files: - split: val path: IQ_Test/val-* - split: test path: IQ_Test/test-* - config_name: Jigsaw data_files: - split: val path: Jigsaw/val-* - split: test path: Jigsaw/test-* - config_name: Multi-view_Reasoning data_files: - split: val path: Multi-view_Reasoning/val-* - split: test path: Multi-view_Reasoning/test-* - config_name: Object_Localization data_files: - split: val path: Object_Localization/val-* - split: test path: Object_Localization/test-* - config_name: Relative_Depth data_files: - split: val path: Relative_Depth/val-* - split: test path: Relative_Depth/test-* - config_name: Relative_Reflectance data_files: - split: val path: Relative_Reflectance/val-* - split: test path: Relative_Reflectance/test-* - config_name: Semantic_Correspondence data_files: - split: val path: Semantic_Correspondence/val-* - split: test path: Semantic_Correspondence/test-* - config_name: Spatial_Relation data_files: - split: val path: Spatial_Relation/val-* - split: test path: Spatial_Relation/test-* - config_name: Visual_Correspondence data_files: - split: val path: Visual_Correspondence/val-* - split: test path: Visual_Correspondence/test-* - config_name: Visual_Similarity data_files: - split: val path: Visual_Similarity/val-* - split: test path: Visual_Similarity/test-* --- # Dataset Card for JaBLINK ## Table of Contents - [Dataset Card for JaBLINK](#dataset-card-for-jablink) - [Table of Contents](#table-of-contents) - [Dataset Description](#dataset-description) - [Dataset Summary](#dataset-summary) - [Languages](#languages) - [Dataset Structure](#dataset-structure) - [Data Instances](#data-instances) - [Usage](#usage) - [Annotation process](#annotation-process) - [Benchmark Results](#benchmark-results) - [Models](#models) - [Val Set](#val-set) - [Additional Information](#additional-information) - [Licensing Information](#licensing-information) - [Disclaimer](#disclaimer) - [Citation Information](#citation-information) - [BibTeX](#bibtex) ## Dataset Description ### Dataset Summary > We introduce JaBLINK, a Japanese version of the BLINK benchmark. > BLINK is a benchmark for multimodal language models (LLMs) that focuses on core visual perception abilities not found in other evaluations. ### Languages > This dataset is in Japanese. ## Dataset Structure > The dataset is downloaded as a .parquet file.　Each row comprises a prompt, an image and an answer. ### Data Instances #### Usage ```python from datasets import load_dataset load_dataset("vlm-lab/JaBLINK", "Counting") ``` ```json DatasetDict({ val: Dataset({ features: ['idx', 'question', 'sub_task', 'image_1', 'image_2', 'image_3', 'image_4', 'choices', 'answer', 'prompt', 'explanation'], num_rows: 117 }) test: Dataset({ features: ['idx', 'question', 'sub_task', 'image_1', 'image_2', 'image_3', 'image_4', 'choices', 'answer', 'prompt', 'explanation'], num_rows: 117 }) }) # sample { 'idx': 'val_Counting_1', 'question': '青い浮き輪はいくつある？', 'sub_task': 'Counting', 'image_1': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x332>, 'image_2': None, 'image_3': None, 'image_4': None, 'choices': ['0', '3', '2', '1'], 'answer': '(D)', 'prompt': '青い浮き輪は何個ありますか？\n次の選択肢から選びなさい。\n(A) 0\n(B) 3\n(C) 2\n(D) 1', 'explanation': '' } ``` #### Annotation process This dataset is constructed by applying machine translation to the BLINK dataset, and then performing refining the data. We have translated all samples (validation/test) of the BLINK dataset and prediction of the test set can be submitted to the BLINK benchmark challenge. ## Benchmark Results ### Models We employed the following models for evaluation. - [SakanaAI/EvoVLM-JP-v1-7B](https://huggingface.co/SakanaAI/EvoVLM-JP-v1-7B) - [stabilityai/japanese-instructblip-alpha](https://huggingface.co/stabilityai/japanese-instructblip-alpha) ### Val Set - All results are reported in the form of accuracy(\%). - All outputs are saved on the ```outputs``` directory. - **Some models are not assessed correctly because they do not follow instructions.** | Model ID | Art Style | Counting | For. Det. | Func. Corr. | IQ Test | Jigsaw | Mul. Reas. | | ------------------------------------------- | --------- | -------- | --------- | ----------- | ------- | ------ | ---------- | | **Random** | 50 | 25 | 25 | 25 | 25 | 50 | 50 | | **SakanaAI/EvoVLM-JP-v1-7B** | 52.99 | 15.0 | 18.94 | 15.38 | 22.0 | 18.67 | 33.08 | | **stabilityai/japanese-instructblip-alpha** | 47.01 | 5.0 | 23.48 | 0.0 | 24.0 | 52.67 | 20.3 | | Model ID | Obj. Loc. | Rel. Dep. | Rel. Ref. | Sem. Corr. | Spa. Rel. | Vis. Corr. | Vis. Sim. | | ------------------------------------------- | --------- | --------- | --------- | ---------- | --------- | ---------- | --------- | | **Random** | 50 | 50 | 33.33 | 25 | 50 | 25 | 50 | | **SakanaAI/EvoVLM-JP-v1-7B** | 40.98 | 54.84 | 1.49 | 21.58 | 65.03 | 0.0 | 52.59 | | **stabilityai/japanese-instructblip-alpha** | 23.77 | 3.23 | 29.1 | 0.0 | 0.0 | 0.0 | 47.41 | ## Additional Information Questions about this dataset should be addressed to ```koki.maeda [at-mark] nlp.c.titech.ac.jp``` . ### Licensing Information > The licence for this dataset is subject to the same Apache-2.0 as the BLINK licence. ### Disclaimer > (Copied from the BLINK dataset) Blink makes use of data from existing image datasets, and does not cover all the visual perception abilities in the wild. For the forensics detection task, we manually collected images that are publicly available from online search. We have made every effort to ensure that the images included in this paper are used in accordance with applicable copyright laws and are properly credited. However, if you are the copyright owner of any image included in our work and believe that its use conflicts with your licensing agreements, please contact us directly. We are committed to addressing any legitimate concerns promptly. ### Citation Information #### BibTeX ```bibtex @article{fu2024blink, title={BLINK: Multimodal Large Language Models Can See but Not Perceive}, author={Fu, Xingyu and Hu, Yushi and Li, Bangzheng and Feng, Yu and Wang, Haoyu and Lin, Xudong and Roth, Dan and Smith, Noah A and Ma, Wei-Chiu and Krishna, Ranjay}, journal={arXiv preprint arXiv:2404.12390}, year={2024} } ```
提供机构：
VLM-LAB
原始信息汇总

数据集概述

数据集名称

  • JaBLINK

许可证

  • Apache-2.0

数据集配置

  • Art_Style

    • 特征: idx, question, sub_task, image_1, image_2, image_3, image_4, choices, answer, prompt, explanation
    • 分割: val (117 examples), test (117 examples)
    • 下载大小: 291074297
    • 数据集大小: 291811561.0

  • Counting

    • 特征: idx, question, sub_task, image_1, image_2, image_3, image_4, choices, answer, prompt, explanation
    • 分割: val (120 examples), test (120 examples)
    • 下载大小: 10015874
    • 数据集大小: 10033468.0

  • Forensic_Detection

    • 特征: idx, question, sub_task, image_1, image_2, image_3, image_4, choices, answer, prompt, explanation
    • 分割: val (132 examples), test (132 examples)
    • 下载大小: 39272509
    • 数据集大小: 39375726.0

  • Functional_Correspondence

    • 特征: idx, question, sub_task, image_1, image_2, image_3, image_4, choices, answer, prompt, explanation
    • 分割: val (130 examples), test (130 examples)
    • 下载大小: 53227222
    • 数据集大小: 54728890.0

  • IQ_Test

    • 特征: idx, question, sub_task, image_1, image_2, image_3, image_4, choices, answer, prompt, explanation
    • 分割: val (150 examples), test (150 examples)
    • 下载大小: 7156052
    • 数据集大小: 10142366.0

  • Jigsaw

    • 特征: idx, question, sub_task, image_1, image_2, image_3, image_4, choices, answer, prompt, explanation
    • 分割: val (150 examples), test (150 examples)
    • 下载大小: 8085696
    • 数据集大小: 8294485.0

  • Multi-view_Reasoning

    • 特征: idx, question, sub_task, image_1, image_2, image_3, image_4, choices, answer, prompt, explanation
    • 分割: val (133 examples), test (133 examples)
    • 下载大小: 19270001
    • 数据集大小: 19415325.0

  • Object_Localization

    • 特征: idx, question, sub_task, image_1, image_2, image_3, image_4, choices, answer, prompt, explanation
    • 分割: val (122 examples), test (125 examples)
    • 下载大小: 12591166
    • 数据集大小: 12681470.0

  • Relative_Depth

    • 特征: idx, question, sub_task, image_1, image_2, image_3, image_4, choices, answer, prompt, explanation
    • 分割: val (124 examples), test (124 examples)
    • 下载大小: 9203975
    • 数据集大小: 9258732.0

  • Relative_Reflectance

    • 特征: idx, question, sub_task, image_1, image_2, image_3, image_4, choices, answer, prompt, explanation
    • 分割: val (134 examples), test (134 examples)
    • 下载大小: 36780997
    • 数据集大小: 36905246.0

  • Semantic_Correspondence

    • 特征: idx, question, sub_task, image_1, image_2, image_3, image_4, choices, answer, prompt, explanation
    • 分割: val (139 examples), test (140 examples)
    • 下载大小: 90492443
    • 数据集大小: 90857145.0

  • Spatial_Relation

    • 特征: idx, question, sub_task, image_1, image_2, image_3, image_4, choices, answer, prompt, explanation
    • 分割: val (143 examples), test (143 examples)
    • 下载大小: 14596727
    • 数据集大小: 14779076.0

  • Visual_Correspondence

    • 特征: idx, question, sub_task, image_1, image_2, image_3, image_4, choices, answer, prompt, explanation
    • 分割: val (172 examples), test (172 examples)
    • 下载大小: 116448573
    • 数据集大小: 117196955.0

  • Visual_Similarity

    • 特征: idx, question, sub_task, image_1, image_2, image_3, image_4, choices, answer, prompt, explanation
    • 分割: val (135 examples), test (136 examples)
    • 下载大小: 89068648
    • 数据集大小: 89142969.0

数据文件路径

  • 每个配置名称对应的数据文件路径，包括验证集和测试集的路径。

以上概述提供了JaBLINK数据集的主要配置、特征、分割信息以及数据集大小。

5,000+
优质数据集
54 个
任务类型
进入经典数据集
关于我们
遇见数据集——让每个数据集都被发现，让每一次遇见都有价值。
数发科官网www.sfktec.com.cn
联系我们
youjian
selectdataset@iotsh.com.cn
商业合作
数据驱动未来，携手共赢发展
二维码
社区交流群

面向社区/商业的数据集话题

二维码
科研交流群

面向高校/科研机构的开源数据集话题

数据驱动未来

携手共赢发展

商业合作