zlab-princeton/Vero-2.5M-unfiltered

--- dataset_info: - config_name: captioning_IF-flickr30k features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 21820881.0 num_examples: 77 - name: train num_bytes: 9007287279.692 num_examples: 30937 download_size: 9009348572 dataset_size: 9029108160.692 - config_name: captioning_IF-mmif_23k_4o features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 27334845.0 num_examples: 56 - name: train num_bytes: 11392173993.27 num_examples: 22330 download_size: 11223375709 dataset_size: 11419508838.27 - config_name: captioning_IF-pixmo_ask_model_anything features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 280766437.0 num_examples: 218 - name: train num_bytes: 106040068316.87 num_examples: 86685 download_size: 104064285304 dataset_size: 106320834753.87 - config_name: captioning_IF-pixmo_cap features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 580503349.0 num_examples: 218 - name: train num_bytes: 238206911978.188 num_examples: 86684 download_size: 235806391738 dataset_size: 238787415327.188 - config_name: captioning_IF-pixmo_cap_qa_images features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 569226088.0 num_examples: 216 - name: train num_bytes: 211612500508.794 num_examples: 86682 download_size: 212177126748 dataset_size: 212181726596.794 - config_name: captioning_IF-reasoning_with_instr_rephrased features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 407838855.0 num_examples: 215 - name: train num_bytes: 124712399505.23 num_examples: 86682 download_size: 130955546835 dataset_size: 125120238360.23 - config_name: chart_ocr-CoSyn_400k_chart features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 33028934.0 num_examples: 142 - name: train num_bytes: 13257176669.111 num_examples: 56351 download_size: 12913706655 dataset_size: 13290205603.111 - config_name: chart_ocr-CoSyn_400k_diagram features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 30934588.0 num_examples: 141 - name: train num_bytes: 12422398770.25 num_examples: 56350 download_size: 12032531303 dataset_size: 12453333358.25 - config_name: chart_ocr-CoSyn_400k_table features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 22900948.0 num_examples: 141 - name: train num_bytes: 9839441658.441 num_examples: 56349 download_size: 9276341598 dataset_size: 9862342606.441 - config_name: chart_ocr-arxivqa_formatted features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 122703104.0 num_examples: 141 - name: train num_bytes: 46015901370.844 num_examples: 56349 download_size: 48190455675 dataset_size: 46138604474.844 - config_name: chart_ocr-chartqa_difficulty features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 3120734.0 num_examples: 69 - name: train num_bytes: 1200458686.05 num_examples: 27475 download_size: 1211164793 dataset_size: 1203579420.05 - config_name: chart_ocr-ecd_vqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 15727866.0 num_examples: 140 - name: train num_bytes: 6680044812.5 num_examples: 56345 download_size: 7022594977 dataset_size: 6695772678.5 - config_name: chart_ocr-evochart features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 12196963.0 num_examples: 139 - name: train num_bytes: 5849251797.383 num_examples: 56347 download_size: 5434452637 dataset_size: 5861448760.383 - config_name: chart_ocr-infographic_vqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 49084944.0 num_examples: 58 - name: train num_bytes: 24197931839.674 num_examples: 23026 download_size: 24310579592 dataset_size: 24247016783.674 - config_name: chart_ocr-reachqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 18042980.0 num_examples: 29 - name: train num_bytes: 7550987959.79 num_examples: 11405 download_size: 7458151839 dataset_size: 7569030939.79 - config_name: counting_grounding_search-aerialvg features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 17342373.0 num_examples: 75 - name: train num_bytes: 9868047479.24 num_examples: 36685 download_size: 9692512381 dataset_size: 9885389852.24 - config_name: counting_grounding_search-groundui features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 32218348.0 num_examples: 69 - name: train num_bytes: 17908168300.026 num_examples: 37973 download_size: 18789142365 dataset_size: 17940386648.026 - config_name: counting_grounding_search-multihop features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 1574223.0 num_examples: 8 - name: train num_bytes: 1861569164.92 num_examples: 8530 download_size: 1926729035 dataset_size: 1863143387.92 - config_name: counting_grounding_search-objects365_qa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 253076643.0 num_examples: 381 - name: train num_bytes: 127209096731.316 num_examples: 185474 download_size: 126267049319 dataset_size: 127462173374.316 - config_name: counting_grounding_search-oodvqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 23359118.0 num_examples: 48 - name: train num_bytes: 10798967561.372 num_examples: 19628 download_size: 10883255873 dataset_size: 10822326679.372 - config_name: counting_grounding_search-osatlas features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 6592971.0 num_examples: 28 - name: train num_bytes: 3616492985.987 num_examples: 12579 download_size: 3564302122 dataset_size: 3623085956.987 - config_name: counting_grounding_search-pixelreasoner features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 2014246.0 num_examples: 4 - name: train num_bytes: 2042954341.17 num_examples: 5345 download_size: 2297078978 dataset_size: 2044968587.17 - config_name: counting_grounding_search-pixmo features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 23481364.0 num_examples: 70 - name: train num_bytes: 12787400563.595 num_examples: 35611 download_size: 12935109139 dataset_size: 12810881927.595 - config_name: counting_grounding_search-refcocog features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 5268813.0 num_examples: 100 - name: train num_bytes: 2278468461.042 num_examples: 42126 download_size: 2240534238 dataset_size: 2283737274.042 - config_name: counting_grounding_search-tallyqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 71818419.0 num_examples: 204 - name: train num_bytes: 38755732091.75 num_examples: 101875 download_size: 37533389317 dataset_size: 38827550510.75 - config_name: counting_grounding_search-visual_probe features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 32156667.0 num_examples: 13 - name: train num_bytes: 16126118239.416 num_examples: 5716 download_size: 12446465503 dataset_size: 16158274906.416 - config_name: knowledge_recognition-aokvqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 15317318.0 num_examples: 38 - name: train num_bytes: 6407273487.285 num_examples: 15059 download_size: 6258232522 dataset_size: 6422590805.285 - config_name: knowledge_recognition-gqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 8503366.0 num_examples: 19 - name: train num_bytes: 3795809711.044 num_examples: 7429 download_size: 3299358026 dataset_size: 3804313077.044 - config_name: knowledge_recognition-iconqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 225788.0 num_examples: 73 - name: train num_bytes: 359753019.06 num_examples: 28970 download_size: 372187232 dataset_size: 359978807.06 - config_name: knowledge_recognition-indoor_qa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 2017721.0 num_examples: 8 - name: train num_bytes: 797616053.088 num_examples: 3252 download_size: 808312750 dataset_size: 799633774.088 - config_name: knowledge_recognition-kvg features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 113773239.0 num_examples: 80 - name: train num_bytes: 50037591940.292 num_examples: 32017 download_size: 50401206405 dataset_size: 50151365179.292 - config_name: knowledge_recognition-kvqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 54727735.0 num_examples: 21 - name: train num_bytes: 25165531379.46 num_examples: 8205 download_size: 24671531421 dataset_size: 25220259114.46 - config_name: knowledge_recognition-popvqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 739176116.0 num_examples: 188 - name: train num_bytes: 372110974309.005 num_examples: 73391 download_size: 454347624082 dataset_size: 372850150425.005 - config_name: knowledge_recognition-vcrqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 226913077.0 num_examples: 186 - name: train num_bytes: 91891478356.37 num_examples: 73390 download_size: 92283561822 dataset_size: 92118391433.37 - config_name: knowledge_recognition-viquae features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 2696273.0 num_examples: 9 - name: train num_bytes: 1370334949.576 num_examples: 3589 download_size: 1352321486 dataset_size: 1373031222.576 - config_name: knowledge_recognition-visual7w features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 171340274.0 num_examples: 162 - name: train num_bytes: 28177600177.197 num_examples: 67687 download_size: 26093823607 dataset_size: 28348940451.197 - config_name: knowledge_recognition-vizwiz features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 70332967.0 num_examples: 34 - name: train num_bytes: 28532316919.546 num_examples: 13626 download_size: 28348658863 dataset_size: 28602649886.546 - config_name: knowledge_recognition-vqav2 features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 72622045.0 num_examples: 182 - name: train num_bytes: 27850616045.035 num_examples: 73385 download_size: 29175920089 dataset_size: 27923238090.035 - config_name: spatial_action-game_QA features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 32105052.0 num_examples: 354 - name: train num_bytes: 7027762516.26 num_examples: 120535 download_size: 5251754197 dataset_size: 7059867568.26 - config_name: spatial_action-magma_aitw features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 7747505.0 num_examples: 45 - name: train num_bytes: 2544059055.86 num_examples: 16970 download_size: 2459481993 dataset_size: 2551806560.86 - config_name: spatial_action-magma_mind2web features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 13324340.0 num_examples: 19 - name: train num_bytes: 5316758775.037 num_examples: 7131 download_size: 5383282863 dataset_size: 5330083115.037 - config_name: spatial_action-robo2vlm features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 7884277.0 num_examples: 12 - name: train num_bytes: 2924614951.108 num_examples: 4493 download_size: 2944577385 dataset_size: 2932499228.108 - config_name: spatial_action-spatial_ssrl features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 130905543.0 num_examples: 186 - name: train num_bytes: 55799392512.999 num_examples: 78271 download_size: 56221181289 dataset_size: 55930298055.999 - config_name: spatial_action-stvqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 6575579.0 num_examples: 19 - name: train num_bytes: 2336593729.475 num_examples: 7355 download_size: 2272462717 dataset_size: 2343169308.475 - config_name: spatial_action-visual_jigsaw_2d features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 223989077.0 num_examples: 239 - name: train num_bytes: 88357949442.77 num_examples: 96791 download_size: 89857733617 dataset_size: 88581938519.77 - config_name: spatial_action-visual_jigsaw_3d features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 48255140.0 num_examples: 125 - name: train num_bytes: 18517087967.218 num_examples: 48209 download_size: 18490665409 dataset_size: 18565343107.218 - config_name: stem-CoSyn_400k_math features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 16527815.0 num_examples: 172 - name: train num_bytes: 5790418107.275 num_examples: 64501 download_size: 5837637979 dataset_size: 5806945922.275 - config_name: stem-ai2d_merged features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 1586280.0 num_examples: 6 - name: train num_bytes: 404181036.728 num_examples: 2343 download_size: 426347019 dataset_size: 405767316.728 - config_name: stem-geo170k features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 976625.0 num_examples: 308 - name: train num_bytes: 350918059.04 num_examples: 111240 download_size: 356797378 dataset_size: 351894684.04 - config_name: stem-geomverse features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 2270125.0 num_examples: 24 - name: train num_bytes: 1799930459.542 num_examples: 9046 download_size: 1725789886 dataset_size: 1802200584.542 - config_name: stem-geoqa_plus features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 189345.0 num_examples: 16 - name: train num_bytes: 56484225.012 num_examples: 5837 download_size: 54825673 dataset_size: 56673570.012 - config_name: stem-mmk12 features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 1436690.0 num_examples: 26 - name: train num_bytes: 675784326.664 num_examples: 9686 download_size: 678223309 dataset_size: 677221016.664 - config_name: stem-pathvqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 50393478.0 num_examples: 86 - name: train num_bytes: 18598648278.684 num_examples: 31542 download_size: 18384049472 dataset_size: 18649041756.684 - config_name: stem-raven features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 2940840.0 num_examples: 55 - name: train num_bytes: 1066671431.458 num_examples: 20262 download_size: 1059172915 dataset_size: 1069612271.458 - config_name: stem-tqa features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 7790024.0 num_examples: 33 - name: train num_bytes: 3214008905.728 num_examples: 12172 download_size: 3030990032 dataset_size: 3221798929.728 - config_name: stem-visualwebinstruct features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 34438506.0 num_examples: 241 - name: train num_bytes: 8145509957.406 num_examples: 91546 download_size: 9420100716 dataset_size: 8179948463.406 - config_name: stem-vqarad features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 1287637.0 num_examples: 5 - name: train num_bytes: 535390815.856 num_examples: 1736 download_size: 451063970 dataset_size: 536678452.856 - config_name: stem-wemath20_pro features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 1236236.0 num_examples: 12 - name: train num_bytes: 310662493.218 num_examples: 4401 download_size: 268136989 dataset_size: 311898729.218 - config_name: stem-wemath20_standard features: - name: id dtype: string - name: data_source dtype: string - name: prompt sequence: - name: role dtype: string - name: content dtype: string - name: ability dtype: string - name: reward_model struct: - name: style dtype: string - name: ground_truth dtype: string - name: extra_info struct: - name: split dtype: string - name: index dtype: int64 - name: domain dtype: string - name: answer dtype: string - name: question dtype: string - name: reward_type dtype: string - name: tolerance dtype: float64 - name: image dtype: image splits: - name: val num_bytes: 26197912.0 num_examples: 16 - name: train num_bytes: 1552370437.895 num_examples: 5659 download_size: 1019462671 dataset_size: 1578568349.895 configs: - config_name: captioning_IF-flickr30k data_files: - split: val path: captioning_IF-flickr30k/val-* - split: train path: captioning_IF-flickr30k/train-* - config_name: captioning_IF-mmif_23k_4o data_files: - split: val path: captioning_IF-mmif_23k_4o/val-* - split: train path: captioning_IF-mmif_23k_4o/train-* - config_name: captioning_IF-pixmo_ask_model_anything data_files: - split: val path: captioning_IF-pixmo_ask_model_anything/val-* - split: train path: captioning_IF-pixmo_ask_model_anything/train-* - config_name: captioning_IF-pixmo_cap data_files: - split: val path: captioning_IF-pixmo_cap/val-* - split: train path: captioning_IF-pixmo_cap/train-* - config_name: captioning_IF-pixmo_cap_qa_images data_files: - split: val path: captioning_IF-pixmo_cap_qa_images/val-* - split: train path: captioning_IF-pixmo_cap_qa_images/train-* - config_name: captioning_IF-reasoning_with_instr_rephrased data_files: - split: val path: captioning_IF-reasoning_with_instr_rephrased/val-* - split: train path: captioning_IF-reasoning_with_instr_rephrased/train-* - config_name: chart_ocr-CoSyn_400k_chart data_files: - split: val path: chart_ocr-CoSyn_400k_chart/val-* - split: train path: chart_ocr-CoSyn_400k_chart/train-* - config_name: chart_ocr-CoSyn_400k_diagram data_files: - split: val path: chart_ocr-CoSyn_400k_diagram/val-* - split: train path: chart_ocr-CoSyn_400k_diagram/train-* - config_name: chart_ocr-CoSyn_400k_table data_files: - split: val path: chart_ocr-CoSyn_400k_table/val-* - split: train path: chart_ocr-CoSyn_400k_table/train-* - config_name: chart_ocr-arxivqa_formatted data_files: - split: val path: chart_ocr-arxivqa_formatted/val-* - split: train path: chart_ocr-arxivqa_formatted/train-* - config_name: chart_ocr-chartqa_difficulty data_files: - split: val path: chart_ocr-chartqa_difficulty/val-* - split: train path: chart_ocr-chartqa_difficulty/train-* - config_name: chart_ocr-ecd_vqa data_files: - split: val path: chart_ocr-ecd_vqa/val-* - split: train path: chart_ocr-ecd_vqa/train-* - config_name: chart_ocr-evochart data_files: - split: val path: chart_ocr-evochart/val-* - split: train path: chart_ocr-evochart/train-* - config_name: chart_ocr-infographic_vqa data_files: - split: val path: chart_ocr-infographic_vqa/val-* - split: train path: chart_ocr-infographic_vqa/train-* - config_name: chart_ocr-reachqa data_files: - split: val path: chart_ocr-reachqa/val-* - split: train path: chart_ocr-reachqa/train-* - config_name: counting_grounding_search-aerialvg data_files: - split: val path: counting_grounding_search-aerialvg/val-* - split: train path: counting_grounding_search-aerialvg/train-* - config_name: counting_grounding_search-groundui data_files: - split: val path: counting_grounding_search-groundui/val-* - split: train path: counting_grounding_search-groundui/train-* - config_name: counting_grounding_search-multihop data_files: - split: val path: counting_grounding_search-multihop/val-* - split: train path: counting_grounding_search-multihop/train-* - config_name: counting_grounding_search-objects365_qa data_files: - split: val path: counting_grounding_search-objects365_qa/val-* - split: train path: counting_grounding_search-objects365_qa/train-* - config_name: counting_grounding_search-oodvqa data_files: - split: val path: counting_grounding_search-oodvqa/val-* - split: train path: counting_grounding_search-oodvqa/train-* - config_name: counting_grounding_search-osatlas data_files: - split: val path: counting_grounding_search-osatlas/val-* - split: train path: counting_grounding_search-osatlas/train-* - config_name: counting_grounding_search-pixelreasoner data_files: - split: val path: counting_grounding_search-pixelreasoner/val-* - split: train path: counting_grounding_search-pixelreasoner/train-* - config_name: counting_grounding_search-pixmo data_files: - split: val path: counting_grounding_search-pixmo/val-* - split: train path: counting_grounding_search-pixmo/train-* - config_name: counting_grounding_search-refcocog data_files: - split: val path: counting_grounding_search-refcocog/val-* - split: train path: counting_grounding_search-refcocog/train-* - config_name: counting_grounding_search-tallyqa data_files: - split: val path: counting_grounding_search-tallyqa/val-* - split: train path: counting_grounding_search-tallyqa/train-* - config_name: counting_grounding_search-visual_probe data_files: - split: val path: counting_grounding_search-visual_probe/val-* - split: train path: counting_grounding_search-visual_probe/train-* - config_name: knowledge_recognition-aokvqa data_files: - split: val path: knowledge_recognition-aokvqa/val-* - split: train path: knowledge_recognition-aokvqa/train-* - config_name: knowledge_recognition-gqa data_files: - split: val path: knowledge_recognition-gqa/val-* - split: train path: knowledge_recognition-gqa/train-* - config_name: knowledge_recognition-iconqa data_files: - split: val path: knowledge_recognition-iconqa/val-* - split: train path: knowledge_recognition-iconqa/train-* - config_name: knowledge_recognition-indoor_qa data_files: - split: val path: knowledge_recognition-indoor_qa/val-* - split: train path: knowledge_recognition-indoor_qa/train-* - config_name: knowledge_recognition-kvg data_files: - split: val path: knowledge_recognition-kvg/val-* - split: train path: knowledge_recognition-kvg/train-* - config_name: knowledge_recognition-kvqa data_files: - split: val path: knowledge_recognition-kvqa/val-* - split: train path: knowledge_recognition-kvqa/train-* - config_name: knowledge_recognition-popvqa data_files: - split: val path: knowledge_recognition-popvqa/val-* - split: train path: knowledge_recognition-popvqa/train-* - config_name: knowledge_recognition-vcrqa data_files: - split: val path: knowledge_recognition-vcrqa/val-* - split: train path: knowledge_recognition-vcrqa/train-* - config_name: knowledge_recognition-viquae data_files: - split: val path: knowledge_recognition-viquae/val-* - split: train path: knowledge_recognition-viquae/train-* - config_name: knowledge_recognition-visual7w data_files: - split: val path: knowledge_recognition-visual7w/val-* - split: train path: knowledge_recognition-visual7w/train-* - config_name: knowledge_recognition-vizwiz data_files: - split: val path: knowledge_recognition-vizwiz/val-* - split: train path: knowledge_recognition-vizwiz/train-* - config_name: knowledge_recognition-vqav2 data_files: - split: val path: knowledge_recognition-vqav2/val-* - split: train path: knowledge_recognition-vqav2/train-* - config_name: spatial_action-game_QA data_files: - split: val path: spatial_action-game_QA/val-* - split: train path: spatial_action-game_QA/train-* - config_name: spatial_action-magma_aitw data_files: - split: val path: spatial_action-magma_aitw/val-* - split: train path: spatial_action-magma_aitw/train-* - config_name: spatial_action-magma_mind2web data_files: - split: val path: spatial_action-magma_mind2web/val-* - split: train path: spatial_action-magma_mind2web/train-* - config_name: spatial_action-robo2vlm data_files: - split: val path: spatial_action-robo2vlm/val-* - split: train path: spatial_action-robo2vlm/train-* - config_name: spatial_action-spatial_ssrl data_files: - split: val path: spatial_action-spatial_ssrl/val-* - split: train path: spatial_action-spatial_ssrl/train-* - config_name: spatial_action-stvqa data_files: - split: val path: spatial_action-stvqa/val-* - split: train path: spatial_action-stvqa/train-* - config_name: spatial_action-visual_jigsaw_2d data_files: - split: val path: spatial_action-visual_jigsaw_2d/val-* - split: train path: spatial_action-visual_jigsaw_2d/train-* - config_name: spatial_action-visual_jigsaw_3d data_files: - split: val path: spatial_action-visual_jigsaw_3d/val-* - split: train path: spatial_action-visual_jigsaw_3d/train-* - config_name: stem-CoSyn_400k_math data_files: - split: val path: stem-CoSyn_400k_math/val-* - split: train path: stem-CoSyn_400k_math/train-* - config_name: stem-ai2d_merged data_files: - split: val path: stem-ai2d_merged/val-* - split: train path: stem-ai2d_merged/train-* - config_name: stem-geo170k data_files: - split: val path: stem-geo170k/val-* - split: train path: stem-geo170k/train-* - config_name: stem-geomverse data_files: - split: val path: stem-geomverse/val-* - split: train path: stem-geomverse/train-* - config_name: stem-geoqa_plus data_files: - split: val path: stem-geoqa_plus/val-* - split: train path: stem-geoqa_plus/train-* - config_name: stem-mmk12 data_files: - split: val path: stem-mmk12/val-* - split: train path: stem-mmk12/train-* - config_name: stem-pathvqa data_files: - split: val path: stem-pathvqa/val-* - split: train path: stem-pathvqa/train-* - config_name: stem-raven data_files: - split: val path: stem-raven/val-* - split: train path: stem-raven/train-* - config_name: stem-tqa data_files: - split: val path: stem-tqa/val-* - split: train path: stem-tqa/train-* - config_name: stem-visualwebinstruct data_files: - split: val path: stem-visualwebinstruct/val-* - split: train path: stem-visualwebinstruct/train-* - config_name: stem-vqarad data_files: - split: val path: stem-vqarad/val-* - split: train path: stem-vqarad/train-* - config_name: stem-wemath20_pro data_files: - split: val path: stem-wemath20_pro/val-* - split: train path: stem-wemath20_pro/train-* - config_name: stem-wemath20_standard data_files: - split: val path: stem-wemath20_standard/val-* - split: train path: stem-wemath20_standard/train-* --- # Vero-2.5M-unfiltered > [!Note] > This repository contains the full unfiltered dataset used to construct Vero-600k and Vero-1.6M, before question and answer filtering. > Note that task categories are not balanced in this dataset. <p align="center"> <img src="./vero-logo-blue-transparent.png" alt="Vero" width="520"> </p> Vero is a fully open reinforcement learning (RL) recipe for training and evaluating multi-task visual reasoning with vision-language models. This repository contains the **Vero-2.5M-unfiltered** dataset, a curation of 2.5M reinforcement learning samples from 59 datasets across 6 diverse visual reasoning categories. [](https://github.com/zlab-princeton/vero) [](https://huggingface.co/collections/zlab-princeton/vero) [](https://vero-reasoning.github.io) [](https://huggingface.co/papers/2604.04917) ## Highlights - **Scale**: 2.5M RL samples from 59 datasets. - **Diversity**: Covers 6 broad categories: STEM Reasoning, Chart & OCR, Spatial & Action, Knowledge & Recognition, Grounding & Counting, and Instruction Following. - **Task-Routed Rewards**: Designed to handle heterogeneous answer formats across diverse tasks. - **Open Recipe**: Fully open release of models, training code, evaluation suite, and dataset. ## Dataset Structure The dataset is organized into six broad task categories: 1. **STEM reasoning** 2. **Chart and OCR** 3. **Spatial reasoning and action** 4. **Knowledge and recognition** 5. **Grounding, counting, and visual search** 6. **Captioning and instruction following** For detailed dataset format, curation details, and reward routing metadata, see the [Data Guide](https://github.com/zlab-princeton/vero/blob/main/docs/DATA.md). ## Sample Usage To download and format the Vero-2.5M dataset for RL training using the official setup script (note that you will need to specify this Vero-2.5M-unfiltered in the script): ```bash # Clone the repository git clone https://github.com/zlab-princeton/vero.git cd vero # Run the formatting script python scripts/download_and_format_vero_600k.py ```

搜集汇总

数据集介绍

构建方式

Vero-2.5M-unfiltered数据集是一个大规模的多模态指令微调数据集，其构建过程融合了多种现有高质量数据源的精华。它整合了包括Flickr30k、MMIF-23k、PixMo、CoSyn-400k、ArxivQA、ChartQA、InfographicVQA等多个公开数据集，覆盖了图像描述、图表理解、OCR、计数与定位等广泛的视觉语言任务。每个样本均以结构化方式组织，包含独特的标识符、数据来源、多轮对话形式的指令（prompt）、能力标签、奖励模型参数（如风格与真值）以及丰富的额外信息（如领域、问题、答案与容差），并关联对应的图像数据。这种将异构原始数据统一格式化为标准多模态指令实例的方法，旨在为训练通用视觉语言模型提供海量且多样化的训练材料。

使用方法

该数据集的使用灵活且面向视觉语言模型的训练与评估。用户可根据具体任务需求，通过指定其配置名称（如'captioning_IF-flickr30k'）来加载对应的子数据集。加载后，利用'prompt'字段中的多轮对话作为模型输入，以'image'字段对应的图像作为视觉上下文，并将'extra_info'中的'answer'或'reward_model'中的'ground_truth'作为监督目标进行训练。由于数据集包含了'reward_model'结构，该数据集特别适用于基于人类反馈的强化学习（RLHF）流程，模型可以学习以奖励最大化为目标进行输出。同时，研究者可利用'ability'字段筛选不同能力维度的数据，进行专项的能力训练或基准测试。

背景与挑战

背景概述

Vero-2.5M-unfiltered 数据集是视觉语言多模态领域一项具有里程碑意义的资源，由相关研究团队于近期创建，旨在系统性地整合多种复杂视觉理解任务。该数据集汇聚了来自图像描述、图表OCR、计数与视觉定位等多达数十个子数据集的庞大样本，总计超过250万条实例，覆盖了从自然场景理解到结构化文档推理的广泛能力范畴。其核心研究问题聚焦于如何构建一个高度多样化、未经过滤的大规模多任务训练集合，以推动多模态大模型在零样本或少样本场景下对细粒度视觉语义的泛化能力。该数据集的出现，为评估和提升模型在视觉推理、文字识别、空间关系理解等复合维度的表现提供了关键基准，对多模态人工智能的发展具有深远影响。

当前挑战

该数据集所应对的核心领域挑战在于解决视觉语言模型对复杂场景的深度理解局限，具体包括多模态模型在图文关联、稠密视觉目标计数、以及图表与原文本的语义对齐等任务上的泛化困难。在构建过程中，Vero-2.5M-unfiltered 面临了来自异构数据源的格式统一与标注一致性难题，例如将来自 PixMo、CoSyn、ChartQA 等不同来源的数据集整合为统一的提示-奖励模型结构，需确保图像、问题、真实答案及容差等字段的精确对应。此外，海量未经过滤的原始数据带来了潜在噪声和标注偏差，如何在不破坏数据多样性的前提下维持质量，是构建过程中必须克服的重大挑战。

常用场景

经典使用场景

在视觉与语言交叉研究的广阔领域中，Vero-2.5M-unfiltered数据集以其海量的多模态指令跟随样本，成为了训练与评估视觉语言模型（VLM）的核心基石。它囊括了图像描述、图表光学字符识别、计数与定位搜索等数十个精细配置子集，覆盖了从日常场景到专业图表、以及需要物体搜寻与计数的视觉理解任务。研究者可以基于该数据集对模型进行多任务联合训练或微调，使其习得视觉感知与语言指令之间的精确映射关系，从而显著提升模型在复杂视觉问答、指代理解与上下文推理上的综合表现力。

解决学术问题

该数据集旨在破解视觉语言模型在指令跟随能力上的泛化瓶颈，尤其聚焦于现有模型在面对细粒度、多步骤视觉推理任务时表现欠佳的核心难题。通过提供带有高质量奖励模型标注与多样化对齐风格的大规模样本，Vero-2.5M-unfiltered为学界系统研究视觉指令微调中的噪声控制、能力迁移和奖励对齐机制提供了丰饶的数据土壤。其影响在于，它推动了多模态大模型从简单的图像字幕生成向精准的、可交互的视觉对话系统进化，为构建更贴近人类认知逻辑的智能体奠定了坚实的数据基础。

实际应用

在实际产业环境中，Vero-2.5M-unfiltered数据集能够赋能多个垂直领域。例如，在智能文档处理中，模型可基于其图表OCR子集实现对财务报告、科研图表中数值与结构的自动化解析；在安防监控领域，计数与定位子集能够帮助系统精准识别并统计特定区域内的人员或车辆；在辅助视觉服务中，通过描述子集训练出的模型可以为视障人士提供即时的、符合上下文的场景解说。这些应用极大地降低了人工处理视觉信息的高昂成本，提升了信息提取的时效性与准确性。

数据集最近研究

最新研究方向

作为海量多模态指令微调数据的集合，Vero-2.5M-unfiltered 目前的研究前沿集中在大规模未经过滤的真实场景数据如何增强视觉-语言模型的细粒度理解与泛化能力上。该数据集整合了图像描述、图表OCR、计数与接地搜索等十几个异构子集，覆盖从自然图像到信息图表的广泛领域，尤其突出了对于复杂结构化视觉信息（如表格、图表）的精确抽取与推理。这一方向与2024年来多模态大模型在文档智能、科学图表解析等热点任务中的突破紧密相关，研究者借助此类丰富且未做严格清洗的原始数据，探索模型在真实噪声环境下的鲁棒性，以及利用多源奖励模型反馈进行对齐优化的潜力，从而推动视觉智能向更开放、更贴近真实应用场景的方向演进。