zlab-princeton/Vero-600k
收藏资源简介：
--- license: apache-2.0 task_categories: - image-text-to-text language: - en tags: - multimodal - visual-reasoning - reinforcement-learning dataset_info: - config_name: captioning_IF-flickr30k features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 4852561859.372 num_examples: 16667 - name: val num_bytes: 48460889.0 num_examples: 167 download_size: 9739765725 dataset_size: 4901022748.372 - config_name: captioning_IF-mmif_23k_4o features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 8503002335.173 num_examples: 16667 - name: val num_bytes: 95492196.0 num_examples: 167 download_size: 16748797131 dataset_size: 8598494531.173 - config_name: captioning_IF-pixmo_ask_model_anything features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 20388371795.434 num_examples: 16667 - name: val num_bytes: 180833787.0 num_examples: 167 download_size: 40526611599 dataset_size: 20569205582.434 - config_name: captioning_IF-pixmo_cap features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 45800828475.369 num_examples: 16667 - name: val num_bytes: 536561038.0 num_examples: 167 download_size: 90988035371 dataset_size: 46337389513.369 - config_name: captioning_IF-pixmo_cap_qa_images features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 40685820126.922 num_examples: 16666 - name: val num_bytes: 405825769.0 num_examples: 166 download_size: 81981510115 dataset_size: 41091645895.922 - config_name: captioning_IF-reasoning_with_instr_rephrased features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 24332810702.764 num_examples: 16666 - name: val num_bytes: 233781042.0 num_examples: 166 download_size: 48632538885 dataset_size: 24566591744.764 - config_name: chart_ocr-CoSyn_400k_chart features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 2655611364.162 num_examples: 11514 download_size: 2597264522 dataset_size: 2655611364.162 - config_name: chart_ocr-CoSyn_400k_diagram features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 1564470378.288 num_examples: 7433 download_size: 1543932323 dataset_size: 1564470378.288 - config_name: chart_ocr-CoSyn_400k_table features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 2035772482.97 num_examples: 12226 - name: val num_bytes: 27549753.0 num_examples: 167 download_size: 3886127478 dataset_size: 2063322235.97 - config_name: chart_ocr-arxivqa_formatted features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 10318565866.275 num_examples: 12225 - name: val num_bytes: 132258549.0 num_examples: 167 download_size: 20608401088 dataset_size: 10450824415.275 - config_name: chart_ocr-chartqa_difficulty features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 547831293.776 num_examples: 12224 - name: val num_bytes: 7380778.0 num_examples: 167 download_size: 924392014 dataset_size: 555212071.776 - config_name: chart_ocr-ecd_vqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 955235345.72 num_examples: 12224 - name: val num_bytes: 16297574.0 num_examples: 167 download_size: 1931786647 dataset_size: 971532919.72 - config_name: chart_ocr-evochart features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 1245171159.641 num_examples: 12223 - name: val num_bytes: 16326493.0 num_examples: 166 download_size: 2500332120 dataset_size: 1261497652.641 - config_name: chart_ocr-infographic_vqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 13738304683.48 num_examples: 12223 - name: val num_bytes: 172073767.0 num_examples: 166 download_size: 7101769096 dataset_size: 13910378450.48 - config_name: chart_ocr-reachqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 5008221435.864 num_examples: 7708 download_size: 2016861715 dataset_size: 5008221435.864 - config_name: counting_grounding_search-aerialvg features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 6159424724.844 num_examples: 16756 - name: val num_bytes: 51122336.0 num_examples: 133 download_size: 12071960156 dataset_size: 6210547060.844 - config_name: counting_grounding_search-groundui features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 5894811307.175 num_examples: 12029 - name: val num_bytes: 63930084.0 num_examples: 100 download_size: 11916690574 dataset_size: 5958741391.175 - config_name: counting_grounding_search-multihop features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 1451197679.15 num_examples: 6350 - name: val num_bytes: 10750315.0 num_examples: 50 download_size: 2885423614 dataset_size: 1461947994.15 - config_name: counting_grounding_search-objects365_qa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 5316209096.712 num_examples: 8576 - name: val num_bytes: 35272844.0 num_examples: 66 download_size: 11266091110 dataset_size: 5351481940.712 - config_name: counting_grounding_search-oodvqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 2750498127.724 num_examples: 5028 - name: val num_bytes: 26677361.0 num_examples: 50 download_size: 5555833578 dataset_size: 2777175488.724 - config_name: counting_grounding_search-osatlas features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 2662572560.458 num_examples: 9313 - name: val num_bytes: 29354683.0 num_examples: 101 download_size: 5359237892 dataset_size: 2691927243.458 - config_name: counting_grounding_search-pixelreasoner features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 1924903986.012 num_examples: 4337 - name: val num_bytes: 46182997.0 num_examples: 100 download_size: 3805973263 dataset_size: 1971086983.012 - config_name: counting_grounding_search-pixmo features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 4633893621.4 num_examples: 12700 - name: val num_bytes: 35246525.0 num_examples: 100 download_size: 9338018397 dataset_size: 4669140146.4 - config_name: counting_grounding_search-refcocog features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 377398499.41 num_examples: 6882 - name: val num_bytes: 5623587.0 num_examples: 100 download_size: 744168133 dataset_size: 383022086.41 - config_name: counting_grounding_search-tallyqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 4483594954.794 num_examples: 12699 - name: val num_bytes: 36974318.0 num_examples: 100 download_size: 9172291672 dataset_size: 4520569272.794 - config_name: counting_grounding_search-visual_probe features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 11420292933.97 num_examples: 5330 - name: val num_bytes: 220446213.0 num_examples: 100 download_size: 23453637021 dataset_size: 11640739146.97 - config_name: knowledge_recognition-aokvqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 1146192475.2 num_examples: 2744 - name: val num_bytes: 35356402.0 num_examples: 84 download_size: 2318878359 dataset_size: 1181548877.2 - config_name: knowledge_recognition-gqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 3123792062.36 num_examples: 6120 - name: val num_bytes: 44661204.0 num_examples: 84 download_size: 5501019513 dataset_size: 3168453266.36 - config_name: knowledge_recognition-iconqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 162580227.055 num_examples: 12755 - name: val num_bytes: 909362.0 num_examples: 84 download_size: 326195964 dataset_size: 163489589.055 - config_name: knowledge_recognition-indoor_qa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 648537628.358 num_examples: 2547 - name: val num_bytes: 21800459.0 num_examples: 84 download_size: 1301937640 dataset_size: 670338087.358 - config_name: knowledge_recognition-kvg features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 20210974376.443 num_examples: 12753 - name: val num_bytes: 137355464.0 num_examples: 83 download_size: 40174061188 dataset_size: 20348329840.443 - config_name: knowledge_recognition-kvqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 18590334750.09 num_examples: 6689 - name: val num_bytes: 306726512.0 num_examples: 83 download_size: 41366935087 dataset_size: 18897061262.09 - config_name: knowledge_recognition-popvqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 79042252709.906 num_examples: 12753 - name: val num_bytes: 427934144.0 num_examples: 83 download_size: 144397640580 dataset_size: 79470186853.906 - config_name: knowledge_recognition-vcrqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 16187463072.192 num_examples: 12752 - name: val num_bytes: 100943326.0 num_examples: 83 download_size: 32403741596 dataset_size: 16288406398.192 - config_name: knowledge_recognition-viquae features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 699784837.014 num_examples: 1859 - name: val num_bytes: 31784555.0 num_examples: 83 download_size: 1446082508 dataset_size: 731569392.014 - config_name: knowledge_recognition-visual7w features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 4631998963.708 num_examples: 12751 - name: val num_bytes: 34223567.0 num_examples: 83 download_size: 9849684548 dataset_size: 4666222530.708 - config_name: knowledge_recognition-vizwiz features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 7426424697.146 num_examples: 3526 - name: val num_bytes: 156789984.0 num_examples: 83 download_size: 14584127461 dataset_size: 7583214681.146 - config_name: knowledge_recognition-vqav2 features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 5070091960.659 num_examples: 12751 - name: val num_bytes: 32224651.0 num_examples: 83 download_size: 10152904388 dataset_size: 5102316611.659 - config_name: spatial_action-game_QA features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 892420346.264 num_examples: 18847 - name: val num_bytes: 7520994.0 num_examples: 125 download_size: 1644806091 dataset_size: 899941340.264 - config_name: spatial_action-magma_aitw features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 1451795176.8 num_examples: 10800 - name: val num_bytes: 18379910.0 num_examples: 125 download_size: 2924710293 dataset_size: 1470175086.8 - config_name: spatial_action-magma_mind2web features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 3777985421.882 num_examples: 5298 - name: val num_bytes: 87964022.0 num_examples: 125 download_size: 7779848376 dataset_size: 3865949443.882 - config_name: spatial_action-robo2vlm features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 1582622665.95 num_examples: 2350 - name: val num_bytes: 85463212.0 num_examples: 125 download_size: 3293718669 dataset_size: 1668085877.95 - config_name: spatial_action-spatial_ssrl features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 13136686467.71 num_examples: 18847 - name: val num_bytes: 89572476.0 num_examples: 125 download_size: 26131935865 dataset_size: 13226258943.71 - config_name: spatial_action-stvqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 1968779128.84 num_examples: 6168 - name: val num_bytes: 40181253.0 num_examples: 125 download_size: 3845960150 dataset_size: 2008960381.84 - config_name: spatial_action-visual_jigsaw_2d features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 17492000559.65 num_examples: 18845 - name: val num_bytes: 114176391.0 num_examples: 125 download_size: 35188875958 dataset_size: 17606176950.65 - config_name: spatial_action-visual_jigsaw_3d features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 7409262522.0 num_examples: 18845 - name: val num_bytes: 49086986.0 num_examples: 125 download_size: 14822181725 dataset_size: 7458349508.0 - config_name: stem-CoSyn_400k_math features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 1365142169.104 num_examples: 16048 - name: val num_bytes: 7651112.0 num_examples: 87 download_size: 2785824122 dataset_size: 1372793281.104 - config_name: stem-ai2d_merged features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 399350978.67 num_examples: 2194 - name: val num_bytes: 19163764.0 num_examples: 87 download_size: 807534616 dataset_size: 418514742.67 - config_name: stem-geo170k features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 61338396.137 num_examples: 16047 - name: val num_bytes: 347779.0 num_examples: 87 download_size: 106738229 dataset_size: 61686175.137 - config_name: stem-geomverse features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 1801787229.065 num_examples: 8895 - name: val num_bytes: 17537131.0 num_examples: 87 download_size: 3403992111 dataset_size: 1819324360.065 - config_name: stem-geoqa_plus features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 55195635.97 num_examples: 5665 - name: val num_bytes: 807365.0 num_examples: 87 download_size: 107854788 dataset_size: 56003000.97 - config_name: stem-mmk12 features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 461234823.223 num_examples: 6869 - name: val num_bytes: 4547442.0 num_examples: 87 download_size: 922481718 dataset_size: 465782265.223 - config_name: stem-pathvqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 595492203.768 num_examples: 1108 - name: val num_bytes: 22689593.0 num_examples: 44 download_size: 1195936485 dataset_size: 618181796.768 - config_name: stem-raven features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 419430406.488 num_examples: 8021 - name: val num_bytes: 2268817.0 num_examples: 43 download_size: 839956563 dataset_size: 421699223.488 - config_name: stem-tqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 2815947872.442 num_examples: 11373 - name: val num_bytes: 21384420.0 num_examples: 87 download_size: 5636387102 dataset_size: 2837332292.442 - config_name: stem-visualwebinstruct features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 2312847986.248 num_examples: 16042 - name: val num_bytes: 11382017.0 num_examples: 87 download_size: 3658137853 dataset_size: 2324230003.248 - config_name: stem-vqarad features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 219942255.0 num_examples: 678 - name: val num_bytes: 12629321.0 num_examples: 43 download_size: 391672584 dataset_size: 232571576.0 - config_name: stem-wemath20_pro features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 188475572.001 num_examples: 2841 - name: val num_bytes: 6850827.0 num_examples: 87 download_size: 353839335 dataset_size: 195326399.001 - config_name: stem-wemath20_standard features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: train num_bytes: 870342764.537 num_examples: 4219 - name: val num_bytes: 23648055.0 num_examples: 87 download_size: 1443095340 dataset_size: 893990819.537 configs: - config_name: captioning_IF-flickr30k data_files: - split: train path: captioning_IF-flickr30k/train-* - split: val path: captioning_IF-flickr30k/val-* - config_name: captioning_IF-mmif_23k_4o data_files: - split: train path: captioning_IF-mmif_23k_4o/train-* - split: val path: captioning_IF-mmif_23k_4o/val-* - config_name: captioning_IF-pixmo_ask_model_anything data_files: - split: train path: captioning_IF-pixmo_ask_model_anything/train-* - split: val path: captioning_IF-pixmo_ask_model_anything/val-* - config_name: captioning_IF-pixmo_cap data_files: - split: train path: captioning_IF-pixmo_cap/train-* - split: val path: captioning_IF-pixmo_cap/val-* - config_name: captioning_IF-pixmo_cap_qa_images data_files: - split: train path: captioning_IF-pixmo_cap_qa_images/train-* - split: val path: captioning_IF-pixmo_cap_qa_images/val-* - config_name: captioning_IF-reasoning_with_instr_rephrased data_files: - split: train path: captioning_IF-reasoning_with_instr_rephrased/train-* - split: val path: captioning_IF-reasoning_with_instr_rephrased/val-* - config_name: chart_ocr-CoSyn_400k_chart data_files: - split: train path: chart_ocr-CoSyn_400k_chart/train-* - config_name: chart_ocr-CoSyn_400k_diagram data_files: - split: train path: chart_ocr-CoSyn_400k_diagram/train-* - config_name: chart_ocr-CoSyn_400k_table data_files: - split: train path: chart_ocr-CoSyn_400k_table/train-* - split: val path: chart_ocr-CoSyn_400k_table/val-* - config_name: chart_ocr-arxivqa_formatted data_files: - split: train path: chart_ocr-arxivqa_formatted/train-* - split: val path: chart_ocr-arxivqa_formatted/val-* - config_name: chart_ocr-chartqa_difficulty data_files: - split: train path: chart_ocr-chartqa_difficulty/train-* - split: val path: chart_ocr-chartqa_difficulty/val-* - config_name: chart_ocr-ecd_vqa data_files: - split: train path: chart_ocr-ecd_vqa/train-* - split: val path: chart_ocr-ecd_vqa/val-* - config_name: chart_ocr-evochart data_files: - split: train path: chart_ocr-evochart/train-* - split: val path: chart_ocr-evochart/val-* - config_name: chart_ocr-infographic_vqa data_files: - split: train path: chart_ocr-infographic_vqa/train-* - split: val path: chart_ocr-infographic_vqa/val-* - config_name: chart_ocr-reachqa data_files: - split: train path: chart_ocr-reachqa/train-* - config_name: counting_grounding_search-aerialvg data_files: - split: train path: counting_grounding_search-aerialvg/train-* - split: val path: counting_grounding_search-aerialvg/val-* - config_name: counting_grounding_search-groundui data_files: - split: train path: counting_grounding_search-groundui/train-* - split: val path: counting_grounding_search-groundui/val-* - config_name: counting_grounding_search-multihop data_files: - split: train path: counting_grounding_search-multihop/train-* - split: val path: counting_grounding_search-multihop/val-* - config_name: counting_grounding_search-objects365_qa data_files: - split: train path: counting_grounding_search-objects365_qa/train-* - split: val path: counting_grounding_search-objects365_qa/val-* - config_name: counting_grounding_search-oodvqa data_files: - split: train path: counting_grounding_search-oodvqa/train-* - split: val path: counting_grounding_search-oodvqa/val-* - config_name: counting_grounding_search-osatlas data_files: - split: train path: counting_grounding_search-osatlas/train-* - split: val path: counting_grounding_search-osatlas/val-* - config_name: counting_grounding_search-pixelreasoner data_files: - split: train path: counting_grounding_search-pixelreasoner/train-* - split: val path: counting_grounding_search-pixelreasoner/val-* - config_name: counting_grounding_search-pixmo data_files: - split: train path: counting_grounding_search-pixmo/train-* - split: val path: counting_grounding_search-pixmo/val-* - config_name: counting_grounding_search-refcocog data_files: - split: train path: counting_grounding_search-refcocog/train-* - split: val path: counting_grounding_search-refcocog/val-* - config_name: counting_grounding_search-tallyqa data_files: - split: train path: counting_grounding_search-tallyqa/train-* - split: val path: counting_grounding_search-tallyqa/val-* - config_name: counting_grounding_search-visual_probe data_files: - split: train path: counting_grounding_search-visual_probe/train-* - split: val path: counting_grounding_search-visual_probe/val-* - config_name: knowledge_recognition-aokvqa data_files: - split: train path: knowledge_recognition-aokvqa/train-* - split: val path: knowledge_recognition-aokvqa/val-* - config_name: knowledge_recognition-gqa data_files: - split: train path: knowledge_recognition-gqa/train-* - split: val path: knowledge_recognition-gqa/val-* - config_name: knowledge_recognition-iconqa data_files: - split: train path: knowledge_recognition-iconqa/train-* - split: val path: knowledge_recognition-iconqa/val-* - config_name: knowledge_recognition-indoor_qa data_files: - split: train path: knowledge_recognition-indoor_qa/train-* - split: val path: knowledge_recognition-indoor_qa/val-* - config_name: knowledge_recognition-kvg data_files: - split: train path: knowledge_recognition-kvg/train-* - split: val path: knowledge_recognition-kvg/val-* - config_name: knowledge_recognition-kvqa data_files: - split: train path: knowledge_recognition-kvqa/train-* - split: val path: knowledge_recognition-kvqa/val-* - config_name: knowledge_recognition-popvqa data_files: - split: train path: knowledge_recognition-popvqa/train-* - split: val path: knowledge_recognition-popvqa/val-* - config_name: knowledge_recognition-vcrqa data_files: - split: train path: knowledge_recognition-vcrqa/train-* - split: val path: knowledge_recognition-vcrqa/val-* - config_name: knowledge_recognition-viquae data_files: - split: train path: knowledge_recognition-viquae/train-* - split: val path: knowledge_recognition-viquae/val-* - config_name: knowledge_recognition-visual7w data_files: - split: train path: knowledge_recognition-visual7w/train-* - split: val path: knowledge_recognition-visual7w/val-* - config_name: knowledge_recognition-vizwiz data_files: - split: train path: knowledge_recognition-vizwiz/train-* - split: val path: knowledge_recognition-vizwiz/val-* - config_name: knowledge_recognition-vqav2 data_files: - split: train path: knowledge_recognition-vqav2/train-* - split: val path: knowledge_recognition-vqav2/val-* - config_name: spatial_action-game_QA data_files: - split: train path: spatial_action-game_QA/train-* - split: val path: spatial_action-game_QA/val-* - config_name: spatial_action-magma_aitw data_files: - split: train path: spatial_action-magma_aitw/train-* - split: val path: spatial_action-magma_aitw/val-* - config_name: spatial_action-magma_mind2web data_files: - split: train path: spatial_action-magma_mind2web/train-* - split: val path: spatial_action-magma_mind2web/val-* - config_name: spatial_action-robo2vlm data_files: - split: train path: spatial_action-robo2vlm/train-* - split: val path: spatial_action-robo2vlm/val-* - config_name: spatial_action-spatial_ssrl data_files: - split: train path: spatial_action-spatial_ssrl/train-* - split: val path: spatial_action-spatial_ssrl/val-* - config_name: spatial_action-stvqa data_files: - split: train path: spatial_action-stvqa/train-* - split: val path: spatial_action-stvqa/val-* - config_name: spatial_action-visual_jigsaw_2d data_files: - split: train path: spatial_action-visual_jigsaw_2d/train-* - split: val path: spatial_action-visual_jigsaw_2d/val-* - config_name: spatial_action-visual_jigsaw_3d data_files: - split: train path: spatial_action-visual_jigsaw_3d/train-* - split: val path: spatial_action-visual_jigsaw_3d/val-* - config_name: stem-CoSyn_400k_math data_files: - split: train path: stem-CoSyn_400k_math/train-* - split: val path: stem-CoSyn_400k_math/val-* - config_name: stem-ai2d_merged data_files: - split: train path: stem-ai2d_merged/train-* - split: val path: stem-ai2d_merged/val-* - config_name: stem-geo170k data_files: - split: train path: stem-geo170k/train-* - split: val path: stem-geo170k/val-* - config_name: stem-geomverse data_files: - split: train path: stem-geomverse/train-* - split: val path: stem-geomverse/val-* - config_name: stem-geoqa_plus data_files: - split: train path: stem-geoqa_plus/train-* - split: val path: stem-geoqa_plus/val-* - config_name: stem-mmk12 data_files: - split: train path: stem-mmk12/train-* - split: val path: stem-mmk12/val-* - config_name: stem-pathvqa data_files: - split: train path: stem-pathvqa/train-* - split: val path: stem-pathvqa/val-* - config_name: stem-raven data_files: - split: train path: stem-raven/train-* - split: val path: stem-raven/val-* - config_name: stem-tqa data_files: - split: train path: stem-tqa/train-* - split: val path: stem-tqa/val-* - config_name: stem-visualwebinstruct data_files: - split: train path: stem-visualwebinstruct/train-* - split: val path: stem-visualwebinstruct/val-* - config_name: stem-vqarad data_files: - split: train path: stem-vqarad/train-* - split: val path: stem-vqarad/val-* - config_name: stem-wemath20_pro data_files: - split: train path: stem-wemath20_pro/train-* - split: val path: stem-wemath20_pro/val-* - config_name: stem-wemath20_standard data_files: - split: train path: stem-wemath20_standard/train-* - split: val path: stem-wemath20_standard/val-* --- # Vero-600k <p align="center"> <img src="./vero-logo-blue-transparent.png" alt="Vero" width="520"> </p> Vero is a fully open reinforcement learning (RL) recipe for training and evaluating multi-task visual reasoning with vision-language models. This repository contains the **Vero-600K** dataset, a curation of 600K reinforcement learning samples from 59 datasets across 6 diverse visual reasoning categories. [](https://github.com/zlab-princeton/vero) [](https://huggingface.co/collections/zlab-princeton/vero) [](https://vero-reasoning.github.io) [](https://huggingface.co/papers/2604.04917) ## Highlights - **Scale**: 600K curated RL samples from 59 datasets. - **Diversity**: Covers 6 broad categories: STEM Reasoning, Chart & OCR, Spatial & Action, Knowledge & Recognition, Grounding & Counting, and Instruction Following. - **Task-Routed Rewards**: Designed to handle heterogeneous answer formats across diverse tasks. - **Open Recipe**: Fully open release of models, training code, evaluation suite, and dataset. ## Dataset Structure The dataset is organized into six broad task categories: 1. **STEM reasoning** 2. **Chart and OCR** 3. **Spatial reasoning and action** 4. **Knowledge and recognition** 5. **Grounding, counting, and visual search** 6. **Captioning and instruction following** For detailed dataset format, curation details, and reward routing metadata, see the [Data Guide](https://github.com/zlab-princeton/vero/blob/main/docs/DATA.md). ## Sample Usage To download and format the Vero-600k dataset for RL training using the official setup script: ```bash # Clone the repository git clone https://github.com/zlab-princeton/vero.git cd vero # Run the formatting script python scripts/download_and_format_vero_600k.py ``` This script exports images into `vero-rl/data/images/` and generates the training/validation `.verl.jsonl` files required for the Vero RL pipeline. ## Models Vero models are trained from various open-weight bases including Qwen3-VL, Qwen2.5-VL, and MiMo-VL. | Model | Base model | Params | | --- | --- | --- | | `Vero-Qwen3I-8B` | `Qwen3-VL-8B-Instruct` | 8B | | `Vero-Qwen3T-8B` | `Qwen3-VL-8B-Thinking` | 8B | | `Vero-MiMo-7B` | `MiMo-VL-7B-SFT` | 7B | | `Vero-Qwen25-7B` | `Qwen2.5-VL-7B-Instruct` | 7B | ## Citation If you use this dataset or the Vero recipe in your research, please cite: ```bibtex @article{sarch2026vero, title = {Vero: An Open RL Recipe for General Visual Reasoning}, author = {Sarch, Gabriel and Cai, Linrong and Wang, Qunzhong and Wu, Haoyang and Danqi Chen and Zhuang Liu}, year = {2026}, journal = {arXiv preprint arXiv:2604.04917}, } ``` ## License This dataset is released under the [Apache License 2.0](LICENSE). Users should also review the licenses and usage terms of the underlying base models and any upstream datasets included in this curation.
许可证：Apache-2.0 任务类别： - 图像-文本转文本（image-text-to-text） 语言： - 英语（en） 标签： - 多模态（multimodal） - 视觉推理（visual-reasoning） - 强化学习（reinforcement learning, RL） 数据集信息： - 配置名称：captioning_IF-flickr30k 特征： - 字段名：id，数据类型：字符串 - 字段名：data_source，数据类型：字符串 - 字段名：prompt，序列字段： - 字段名：role，数据类型：字符串 - 字段名：content，数据类型：字符串 - 字段名：ability，数据类型：字符串 - 字段名：reward_model，结构体字段： - 字段名：style，数据类型：字符串 - 字段名：ground_truth，数据类型：字符串 - 字段名：extra_info，结构体字段： - 字段名：split，数据类型：字符串 - 字段名：index，数据类型：整数 - 字段名：domain，数据类型：字符串 - 字段名：answer，数据类型：字符串 - 字段名：question，数据类型：字符串 - 字段名：reward_type，数据类型：字符串 - 字段名：tolerance，数据类型：浮点数 - 字段名：image，数据类型：图像 数据集划分： - 划分集：train（训练集），字节数：4852561859.372，样本数：16667 - 划分集：val（验证集），字节数：48460889.0，样本数：167 下载大小：9739765725 数据集总大小：4901022748.372 - 配置名称：captioning_IF-mmif_23k_4o 特征： - 字段名：id，数据类型：字符串 - 字段名：data_source，数据类型：字符串 - 字段名：prompt，序列字段： - 字段名：role，数据类型：字符串 - 字段名：content，数据类型：字符串 - 字段名：ability，数据类型：字符串 - 字段名：reward_model，结构体字段： - 字段名：style，数据类型：字符串 - 字段名：ground_truth，数据类型：字符串 - 字段名：extra_info，结构体字段： - 字段名：split，数据类型：字符串 - 字段名：index，数据类型：整数 - 字段名：domain，数据类型：字符串 - 字段名：answer，数据类型：字符串 - 字段名：question，数据类型：字符串 - 字段名：reward_type，数据类型：字符串 - 字段名：tolerance，数据类型：浮点数 - 字段名：image，数据类型：图像 数据集划分： - 划分集：train（训练集），字节数：8503002335.173，样本数：16667 - 划分集：val（验证集），字节数：95492196.0，样本数：167 下载大小：16748797131 数据集总大小：8598494531.173 ...（其余配置项结构与上述一致，仅包含不同的数据集名称、样本量与文件大小参数） 配置列表： - 配置名称：captioning_IF-flickr30k，数据文件： - 划分集：train，路径：captioning_IF-flickr30k/train-* - 划分集：val，路径：captioning_IF-flickr30k/val-* ...（其余配置项的配置列表与上述格式一致，仅替换配置名称与路径） # Vero-600k <p align="center"> <img src="./vero-logo-blue-transparent.png" alt="Vero" width="520"> </p> Vero是一套完全开源的强化学习（reinforcement learning, RL）流程，用于基于视觉语言模型（vision-language models, VLMs）训练和评估多任务视觉推理。本仓库包含**Vero-600K**数据集，该数据集整合了来自59个数据集的60万个强化学习样本，涵盖6大类多样化的视觉推理任务。 [](https://github.com/zlab-princeton/vero) [](https://huggingface.co/collections/zlab-princeton/vero) [](https://vero-reasoning.github.io) [](https://huggingface.co/papers/2604.04917) ## 核心亮点 - **规模**：整合自59个数据集的60万个精选强化学习样本。 - **多样性**：涵盖6大核心任务类别：理工科推理（STEM Reasoning）、图表与光学字符识别（Chart & OCR）、空间与动作推理（Spatial & Action）、知识与识别（Knowledge & Recognition）、定位与计数（Grounding & Counting），以及指令跟随（Instruction Following）。 - **任务适配奖励机制**：专为处理多样化任务中的异构答案格式而设计。 - **全开源流程**：完整开源模型、训练代码、评估套件与数据集。 ## 数据集结构 本数据集按照6大核心任务类别进行组织： 1. **理工科推理（STEM Reasoning）** 2. **图表与光学字符识别（Chart & OCR）** 3. **空间与动作推理（Spatial reasoning and action）** 4. **知识与识别（Knowledge and recognition）** 5. **定位、计数与视觉搜索（Grounding, counting, and visual search）** 6. **图像描述与指令跟随（Captioning and instruction following）** 如需了解详细的数据集格式、数据整合细节以及奖励路由元数据，请参阅[数据指南](https://github.com/zlab-princeton/vero/blob/main/docs/DATA.md)。 ## 样本使用方法 如需通过官方配置脚本下载并格式化Vero-600K数据集以用于强化学习训练，请执行以下命令： bash # 克隆仓库 git clone https://github.com/zlab-princeton/vero.git cd vero # 运行格式化脚本 python scripts/download_and_format_vero_600k.py 该脚本会将图像导出至`vero-rl/data/images/`目录，并生成Vero强化学习流程所需的训练/验证`.verl.jsonl`文件。 ## 模型列表 Vero系列模型基于多款开源基座模型进行训练，包括Qwen3-VL、Qwen2.5-VL以及MiMo-VL。 | 模型名称 | 基座模型 | 参数规模 | | --- | --- | --- | | `Vero-Qwen3I-8B` | `Qwen3-VL-8B-Instruct` | 80亿 | | `Vero-Qwen3T-8B` | `Qwen3-VL-8B-Thinking` | 80亿 | | `Vero-MiMo-7B` | `MiMo-VL-7B-SFT` | 70亿 | | `Vero-Qwen25-7B` | `Qwen2.5-VL-7B-Instruct` | 70亿 | ## 引用方式 若您在研究中使用本数据集或Vero流程，请引用以下文献： bibtex @article{sarch2026vero, title = {Vero: An Open RL Recipe for General Visual Reasoning}, author = {Sarch, Gabriel and Cai, Linrong and Wang, Qunzhong and Wu, Haoyang and Danqi Chen and Zhuang Liu}, year = {2026}, journal = {arXiv preprint arXiv:2604.04917}, } ## 许可证 本数据集采用[Apache许可证2.0](LICENSE)进行发布。使用者还需审阅本数据集所包含的底层基座模型以及所有上游数据集的许可证与使用条款。
