wikimedia/wit_base
收藏资源简介：
--- annotations_creators: - machine-generated language_creators: - found language: - af - an - ar - arz - ast - az - azb - ba - bar - be - bg - bn - br - bs - ca - ce - ceb - ckb - cs - cv - cy - da - de - el - en - eo - es - et - eu - fa - fi - fil - fr - fy - ga - gl - hi - hr - hsb - ht - hu - hy - ia - id - io - is - it - iw - ja - jv - ka - kk - kn - ko - la - lah - lb - lmo - lt - lv - mg - mk - ml - mn - mr - ms - my - nan - nds - ne - nl - nn - 'no' - nv - oc - pa - pl - pt - qu - ro - ru - sco - si - sk - sl - sq - sr - sv - sw - ta - te - tg - th - tr - tt - uk - ur - uz - vec - vi - vo - war - xmf - yue - zh license: - cc-by-sa-4.0 multilinguality: - multilingual size_categories: - 1M<n<10M source_datasets: - original - extended|wikipedia task_categories: - image-to-text - text-retrieval task_ids: - image-captioning paperswithcode_id: wit pretty_name: Wikipedia-based Image Text language_bcp47: - af - an - ar - arz - ast - az - azb - ba - bar - be - be-tarask - bg - bn - br - bs - ca - ce - ceb - ckb - cs - cv - cy - da - de - el - en - eo - es - et - eu - fa - fi - fil - fr - fy - ga - gl - hi - hr - hsb - ht - hu - hy - ia - id - io - is - it - iw - ja - jv - ka - kk - kn - ko - la - lah - lb - lmo - lt - lv - mg - mk - ml - mn - mr - ms - my - nan - nds - ne - nl - nn - 'no' - nv - oc - pa - pl - pt - qu - ro - ru - sco - si - sk - sl - sq - sr - sr-Latn - sv - sw - ta - te - tg - th - tr - tt - uk - ur - uz - vec - vi - vo - war - xmf - yue - zh - zh-TW tags: - text-image-retrieval --- # Dataset Card for WIT ## Table of Contents - [Table of Contents](#table-of-contents) - [Dataset Description](#dataset-description) - [Dataset Summary](#dataset-summary) - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards) - [Languages](#languages) - [Dataset Structure](#dataset-structure) - [Data Instances](#data-instances) - [Data Fields](#data-fields) - [Data Splits](#data-splits) - [Dataset Creation](#dataset-creation) - [Curation Rationale](#curation-rationale) - [Source Data](#source-data) - [Annotations](#annotations) - [Personal and Sensitive Information](#personal-and-sensitive-information) - [Considerations for Using the Data](#considerations-for-using-the-data) - [Social Impact of Dataset](#social-impact-of-dataset) - [Discussion of Biases](#discussion-of-biases) - [Other Known Limitations](#other-known-limitations) - [Additional Information](#additional-information) - [Dataset Curators](#dataset-curators) - [Licensing Information](#licensing-information) - [Citation Information](#citation-information) - [Contributions](#contributions) ## Dataset Description - **Homepage:** [WIT homepage](https://github.com/google-research-datasets/wit) - **Paper:** [WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning ](https://arxiv.org/abs/2103.01913) - **Leaderboard:** [WIT leaderboard](https://paperswithcode.com/sota/text-image-retrieval-on-wit) and [WIT Kaggle competition](https://www.kaggle.com/competitions/wikipedia-image-caption/leaderboard) - **Point of Contact:** [Miriam Redi](mailto:miriam@wikimedia.org) ### Dataset Summary Wikimedia's version of the Wikipedia-based Image Text (WIT) Dataset, a large multimodal multilingual dataset. From the [official blog post](https://techblog.wikimedia.org/2021/09/09/the-wikipedia-image-caption-matching-challenge-and-a-huge-release-of-image-data-for-research/): > The core training data is taken from the Wikipedia Image-Text (WIT) Dataset, a large curated set of more than 37 million image-text associations extracted from Wikipedia articles in 108 languages that was recently released by Google Research. > > The WIT dataset offers extremely valuable data about the pieces of text associated with Wikipedia images. However, due to licensing and data volume issues, the Google dataset only provides the image name and corresponding URL for download and not the raw image files. > > Getting easy access to the image files is crucial for participants to successfully develop competitive models. Therefore, today, the Wikimedia Research team is releasing its first large image dataset. It contains more than six million image files from Wikipedia articles in 100+ languages, which correspond to almost [1] all captioned images in the WIT dataset. Image files are provided at a 300-px resolution, a size that is suitable for most of the learning frameworks used to classify and analyze images. > [1] We are publishing all images having a non-null “reference description” in the WIT dataset. For privacy reasons, we are not publishing images where a person is the primary subject, i.e., where a person’s face covers more than 10% of the image surface. To identify faces and their bounding boxes, we use the RetinaFace detector. In addition, to avoid the inclusion of inappropriate images or images that violate copyright constraints, we have removed all images that are candidate for deletion on Commons from the dataset. **Note**: Compared to [Google's version](https://huggingface.co/datasets/google/wit), which has contents of one Wikipedia page per data sample, this version groups contents of all Wikipedia pages available in different languages for the image in one single data sample to avoid duplication of image bytes. ### Supported Tasks and Leaderboards - `image-captioning`: This dataset can be used to train a model for image captioning where the goal is to predict a caption given the image. - `text-retrieval`: The goal in this task is to build a model that retrieves the text (`caption_title_and_reference_description`) closest to an image. The leaderboard for this task can be found [here](https://paperswithcode.com/sota/text-image-retrieval-on-wit). This task also has a competition on [Kaggle](https://www.kaggle.com/c/wikipedia-image-caption). In these tasks, any combination of the `caption_reference_description`, `caption_attribution_description` and `caption_alt_text_description` fields can be used as the input text/caption. ### Languages The dataset contains examples from all Wikipedia languages. ## Dataset Structure ### Data Instances Each instance is an image, its representation in bytes, a pre-computed embedding, and the set of captions attached to the image in Wikipedia. ``` { 'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=300x225 at 0x7F88F3876358>, 'image_url': 'https://upload.wikimedia.org/wikipedia/commons/8/8b/Scolopendra_gigantea.jpg', 'embedding': [1.4784087, 2.8710432, 0.0, 0.51603067, ..., 10.266883, 0.51142216, 0.0, 2.3464653], 'metadata_url': 'http://commons.wikimedia.org/wiki/File:Scolopendra_gigantea.jpg', 'original_height': 3000, 'original_width': 4000, 'mime_type': 'image/jpeg', 'caption_attribution_description': 'English: Puerto Rican Giant Centipede, Scolopendra gigantea; Vieques, Puerto Rico Slovenčina: Stonožka obrovská, Scolopendra gigantea; Vieques, Portoriko', 'wit_features': { 'language': ['ro', 'vi', 'sk', ..., 'nl', 'th', 'lv'], 'page_url': ['https://ro.wikipedia.org/wiki/Scolopendra_gigantea', 'https://vi.wikipedia.org/wiki/Scolopendra_gigantea', 'https://sk.wikipedia.org/wiki/Scolopendra_gigantea', ..., 'https://nl.wikipedia.org/wiki/Scolopendra_gigantea', 'https://th.wikipedia.org/wiki/%E0%B8%95%E0%B8%B0%E0%B8%82%E0%B8%B2%E0%B8%9A%E0%B8%A2%E0%B8%B1%E0%B8%81%E0%B8%A9%E0%B9%8C%E0%B8%82%E0%B8%B2%E0%B9%80%E0%B8%AB%E0%B8%A5%E0%B8%B7%E0%B8%AD%E0%B8%87%E0%B9%80%E0%B8%9B%E0%B8%A3%E0%B8%B9', 'https://lv.wikipedia.org/wiki/Skolopendru_dzimta'], 'attribution_passes_lang_id': [True, True, True, ..., True, True, True], 'caption_alt_text_description': [None, None, None, ..., 'Scolopendra gigantea', None, 'Milzu skolopendra (Scolopendra gigantea)'], 'caption_reference_description': [None, None, None, ..., None, None, 'Milzu skolopendra (Scolopendra gigantea)'], 'caption_title_and_reference_description': [None, 'Scolopendra gigantea [SEP] ', None, ..., 'Scolopendra gigantea [SEP] ', None, 'Skolopendru dzimta [SEP] Milzu skolopendra (Scolopendra gigantea)'], 'context_page_description': ['Scolopendra gigantea este un miriapod din clasa Chilopoda, fiind cel mai mare reprezentant al genului Scolopendra. Adultul poate atinge o lungime de 26 cm, uneori depășind 30 cm. Această specie habitează în regiunile de nord și de vest a Americii de Sud, pe insulele Trinidad, insulele Virgine, Jamaica Hispaniola ș.a. Localnicii denumesc scolopendra chilopodul gigant galben și chilopodul gigant amazonian.', 'Scolopendra gigantea là đại diện lớn nhất của chi Scolopendra nói riêng và cả lớp rết nói chung, thường đạt độ dài 26 cm và có thể vượt quá 30 cm. Sinh sống ở khu vực phía bắc và tây của Nam Mỹ và các đảo Trinidad, Puerto Rico, Saint Thomas, U.S. Virgin Islands, Jamaica, và Hispaniola.', 'Scolopendra gigantea, starší slovenský nazov: štípavica veľká, je živočích z rodu Scolopendra, s veľkosťou do 30 cm.', ..., 'Scolopendra gigantea is een tijgerduizendpoot uit Zuid-Amerika. De soort jaagt onder andere op grote geleedpotigen, amfibieën, reptielen en kleine zoogdieren. Het is voor zover bekend de grootste niet uitgestorven duizendpoot ter wereld.', 'ตะขาบยักษ์ขาเหลืองเปรู หรือ ตะขาบยักษ์อเมซอน เป็นตะขาบชนิดที่มีขนาดใหญ่ที่สุดในสกุล Scolopendra โดยปกติเมื่อโตเต็มที่จะยาว 26 เซนติเมตร แต่บางครั้งก็สามารถโตได้ถึง 30 เซนติเมตร ตะขาบชนิดนี้อาศัยอยู่ทางแถบเหนือและตะวันตกของทวีปอเมริกาใต้ และตามเกาะแก่งของประเทศตรินิแดดและจาไมกา เป็นสัตว์กินเนื้อ โดยกินจิ้งจก, กบ, นก, หนู และแม้แต่ค้างคาวเป็นอาหาร และขึ้นชื่อในเรื่องความดุร้าย', 'Skolpendru dzimta pieder pie simtkāju kārtas. Ap 400 dzimtas sugas sastopamas visā pasaulē, īpaši subtropu un tropu apgabalos. Mitinās augsnē, nobirušās lapās, plaisās, spraugās.'], 'context_section_description': [None, 'Scolopendra gigantea (còn được gọi là Rết chân vàng khổng lồ Peru và Rết khổng lồ Amazon) là đại diện lớn nhất của chi Scolopendra nói riêng và cả lớp rết nói chung, thường đạt độ dài 26\xa0cm (10\xa0in) và có thể vượt quá 30\xa0cm (12\xa0in). Sinh sống ở khu vực phía bắc và tây của Nam Mỹ và các đảo Trinidad, Puerto Rico, Saint Thomas, U.S. Virgin Islands, Jamaica, và Hispaniola.', None, ..., 'Scolopendra gigantea is een tijgerduizendpoot uit Zuid-Amerika. De soort jaagt onder andere op grote geleedpotigen, amfibieën, reptielen en kleine zoogdieren. Het is voor zover bekend de grootste niet uitgestorven duizendpoot ter wereld.', None, 'Skolpendru dzimta (Scolopendridae) pieder pie simtkāju kārtas. Ap 400 dzimtas sugas sastopamas visā pasaulē, īpaši subtropu un tropu apgabalos. Mitinās augsnē, nobirušās lapās, plaisās, spraugās.'], 'hierarchical_section_title': ['Scolopendra gigantea', 'Scolopendra gigantea', 'Scolopendra gigantea', ..., 'Scolopendra gigantea', 'ตะขาบยักษ์ขาเหลืองเปรู', 'Skolopendru dzimta'], 'is_main_image': [True, True, True, ..., True, True, True], 'page_title': ['Scolopendra gigantea', 'Scolopendra gigantea', 'Scolopendra gigantea', ..., 'Scolopendra gigantea', 'ตะขาบยักษ์ขาเหลืองเปรู', 'Skolopendru dzimta'], 'section_title': [None, None, None, ..., None, None, None] } } ``` **Note**: The dataset is stored in Parquet for better performance. This dataset was generated from the original files using [this script](wit_base/blob/main/scripts/wit.py). Additionally, 120 examples from the original files have incorrectly formatted one or more of the following fields: `original_height`, `original_width`, `mime_type` and `caption_attribution_description`. The fixed versions of these examples that were used in the generation script can be found [here](wit_base/blob/main/scripts/corrected_examples.py). ### Data Fields - `image`: A `PIL.Image.Image` object containing the image resized to a width of 300-px while preserving its aspect ratio. Note that when accessing the image column: `dataset[0]["image"]` the image file is automatically decoded. Decoding of a large number of image files might take a significant amount of time. Thus it is important to first query the sample index before the `"image"` column, *i.e.* `dataset[0]["image"]` should **always** be preferred over `dataset["image"][0]`. - `image_url`: URL to wikipedia image - `embedding`: Precomputed image embedding. Each image is described with a 2048-dimensional signature extracted from the second-to-last layer of a [ResNet-50](https://arxiv.org/abs/1512.03385) neural network trained with [Imagenet](https://www.image-net.org/) data. These embeddings contain rich information about the image content and layout, in a compact form - `metadata_url`: URL to wikimedia page containing the image and the metadata - `original_height`: Original image height before resizing - `original_width`: Original image width before resizing - `mime_type`: Mime type associated to the image - `caption_attribution_description`: This is the text found on the Wikimedia page of the image. This text is common to all occurrences of that image across all Wikipedias. - `wit_features`: Sequence of captions for the image with language, page URL, information about the page, caption text, etc. - `language`: Language code depicting wikipedia language of the page - `page_url`: URL to wikipedia page - `attribution_passes_lang_id`: Compared `language` field with the attribution language (written in the prefix of the attribution description. - `caption_alt_text_description`: This is the “alt” text associated with the image. While not visible in general, it is commonly used for accessibility / screen readers - `caption_reference_description`: This is the caption that is visible on the wikipedia page directly below the image. - `caption_title_and_reference_description`: Concatenation of `page_title` and `caption_reference_description`. - `context_page_description`: Corresponds to the short description of the page. It provides a concise explanation of the scope of the page. - `context_section_description`: Text within the image's section - `hierarchical_section_title`: Hierarchical section's title - `is_main_image`: Flag determining if the image is the first image of the page. Usually displayed on the top-right part of the page when using web browsers. - `page_changed_recently`: [More Information Needed] - `page_title`: Wikipedia page's title - `section_title`: Section's title <p align='center'> <img width='75%' src='https://production-media.paperswithcode.com/datasets/Screenshot_2021-03-04_at_14.26.02.png' alt="Half Dome" /> </br> <b>Figure: WIT annotation example. </b> </p> Details on the field content can be found directly in the [paper, figure 5 and table 12.](https://arxiv.org/abs/2103.01913) ### Data Splits All data is held in `train` split, with a total of 6477255 examples. ## Dataset Creation ### Curation Rationale From the [official blog post](https://techblog.wikimedia.org/2021/09/09/the-wikipedia-image-caption-matching-challenge-and-a-huge-release-of-image-data-for-research/): > The WIT dataset offers extremely valuable data about the pieces of text associated with Wikipedia images. > Getting easy access to the image files is crucial for participants to successfully develop competitive models. > With this large release of visual data, we aim to help the competition participants—as well as researchers and practitioners who are interested in working with Wikipedia images—find and download the large number of image files associated with the challenge, in a compact form. ### Source Data #### Initial Data Collection and Normalization From the [paper, section 3.1](https://arxiv.org/abs/2103.01913): > We started with all Wikipedia content pages (i.e., ignoring other pages that have discussions, comments and such). These number about ~124M pages across 279 languages. #### Who are the source language producers? Text was extracted from Wikipedia. ### Annotations #### Annotation process WIT was constructed using an automatic process. However it was human-validated. From the [paper, section 3.7](https://arxiv.org/abs/2103.01913): > To further verify the quality of the WIT dataset we performed a study using (crowd-sourced) human annotators. As seen in Fig. 3, we asked raters to answer 3 questions. Given an image and the page title, raters first evaluate the quality of the attribution description and reference description in the first two questions (order randomized). The third question understands the contextual quality of these text descriptions given the page description and caption. Each response is on a 3-point scale: "Yes" if the text perfectly describes the image, "Maybe" if it is sufficiently explanatory and "No" if it is irrelevant or the image is inappropriate. #### Who are the annotators? [More Information Needed] ### Personal and Sensitive Information From the [official blog post](https://techblog.wikimedia.org/2021/09/09/the-wikipedia-image-caption-matching-challenge-and-a-huge-release-of-image-data-for-research/#FN1): > For privacy reasons, we are not publishing images where a person is the primary subject, i.e., where a person’s face covers more than 10% of the image surface. To identify faces and their bounding boxes, we use the [RetinaFace](https://arxiv.org/abs/1905.00641) detector. In addition, to avoid the inclusion of inappropriate images or images that violate copyright constraints, we have removed all images that are [candidate for deletion](https://commons.wikimedia.org/wiki/Commons:Deletion_requests) on Commons from the dataset. ## Considerations for Using the Data ### Social Impact of Dataset [More Information Needed] ### Discussion of Biases From the [paper, section 3.4](https://arxiv.org/abs/2103.01913): > Lastly we found that certain image-text pairs occurred very frequently. These were often generic images that did not have much to do with the main article page. Common examples included flags, logos, maps, insignia and such. To prevent biasing the data, we heavily under-sampled all such images ### Other Known Limitations [More Information Needed] ## Additional Information ### Dataset Curators Miriam Redi, Fabian Kaelin and Tiziano Piccardi. ### Licensing Information [CC BY-SA 4.0 international license](https://creativecommons.org/licenses/by-sa/4.0/) ### Citation Information ```bibtex @article{srinivasan2021wit, title={WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning}, author={Srinivasan, Krishna and Raman, Karthik and Chen, Jiecao and Bendersky, Michael and Najork, Marc}, journal={arXiv preprint arXiv:2103.01913}, year={2021} } ``` ### Contributions Thanks to [@nateraw](https://github.com/nateraw), [yjernite](https://github.com/yjernite) and [mariosasko](https://github.com/mariosasko) for adding this dataset.
数据集卡片 for WIT
数据集描述
数据集概述
WIT 数据集是一个大型的多模态多语言数据集，由 Wikimedia 版本的 Wikipedia-based Image Text (WIT) 数据集组成。该数据集包含超过 3700 万个从 108 种语言的 Wikipedia 文章中提取的图像-文本关联。
支持的任务和排行榜
image-captioning: 该数据集可用于训练图像字幕生成模型，目标是在给定图像的情况下预测字幕。text-retrieval: 该任务的目标是构建一个模型，该模型能够检索与图像最接近的文本 (caption_title_and_reference_description)。该任务的排行榜可以在 这里 找到。
语言
数据集包含来自所有 Wikipedia 语言的示例。
数据集结构
数据实例
每个实例包含一个图像、其字节表示、预计算的嵌入以及与该图像相关的所有 Wikipedia 中的字幕。
数据字段
image: 一个PIL.Image.Image对象，包含调整到 300 像素宽度的图像。image_url: Wikipedia 图像的 URL。embedding: 预计算的图像嵌入，每个图像由一个 2048 维的签名描述。metadata_url: 包含图像和元数据的 Wikimedia 页面的 URL。original_height: 调整大小前的原始图像高度。original_width: 调整大小前的原始图像宽度。mime_type: 与图像关联的 MIME 类型。caption_attribution_description: 在 Wikimedia 页面上找到的文本。wit_features: 图像的字幕序列，包含语言、页面 URL、页面信息、字幕文本等。
数据分割
所有数据都保存在 train 分割中，共有 6477255 个示例。
数据集创建
策划理由
WIT 数据集提供了关于与 Wikipedia 图像相关联的文本片段的极其有价值的数据。获取图像文件的便捷访问对于参与者成功开发竞争模型至关重要。
源数据
初始数据收集和规范化
数据集从所有 Wikipedia 内容页面（忽略讨论、评论等其他页面）开始，这些页面在 279 种语言中约有 1.24 亿个页面。
注释
注释过程
WIT 是通过自动过程构建的，但经过了人工验证。
个人和敏感信息
出于隐私原因，不发布以人物为主要主题的图像，即人物面部覆盖图像表面超过 10% 的图像。此外，已从数据集中删除所有在 Commons 上候选删除的图像。
使用数据的注意事项
数据集的社会影响
[更多信息需要]
偏见的讨论
某些图像-文本对出现频率非常高，这些通常是与主要文章页面无关的通用图像。为了防止数据偏差，对所有此类图像进行了大量下采样。
其他已知限制
[更多信息需要]
附加信息
数据集策展人
Miriam Redi, Fabian Kaelin 和 Tiziano Piccardi。
许可信息
引用信息
bibtex @article{srinivasan2021wit, title={WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning}, author={Srinivasan, Krishna and Raman, Karthik and Chen, Jiecao and Bendersky, Michael and Najork, Marc}, journal={arXiv preprint arXiv:2103.01913}, year={2021} }
贡献
感谢 @nateraw, yjernite 和 mariosasko 添加此数据集。
