Wiki-JA-Pair

# Wiki-JA-Pair Wiki-JA-Pair is a dataset of 1M image–text pairs sourced from the Japanese Wikipedia (Wiki-JA). This dataset is constructed using the May 1, 2025 dump ([jawiki-20250501-pages-articles-multistream.xml.bz2](https://dumps.wikimedia.org/jawiki/20250501/jawiki-20250501-pages-articles.xml.bz2)). ## How to Use ```python from datasets import load_dataset ds = load_dataset("llm-jp/Wiki-JA-Pair") ``` ### Format Wiki-JA-Pair includes the following columns: - `url`: URL of the image - `caption`: Caption associated with the image - `description`: Nearby text that describes the image - `page_title`: Title of the Wikipedia page containing the image - `page_url`: URL of the Wikipedia page - `text`: Full page text extracted using [trafilature](https://github.com/adbar/trafilatura) - `width`: Width (in pixels) of the resized image used for computing pHash - `height`: Height (in pixels) of the resized image used for computing pHash - `original_width`: Original width of the image - `original_height`: Original height of the image - `sha256`: SHA-256 hash of the original image file - `phash`: Perceptual hash (pHash) computed from the resized image ## Dataset Construction Pipeline 1. **Crawling HTML pages from Wiki-ja**: We crawled HTML pages of Wiki-ja using a dump of article titles from [jawiki-20250501-pages-articles-multistream.xml.bz2](https://dumps.wikimedia.org/jawiki/20250501/jawiki-20250501-pages-articles.xml.bz2). Out of 2,373,786 titles, 20,967 included disambiguation pages, which were skipped. 2. **Extracting (image-url, caption) pairs from HTML**: From each HTML page, we extracted image URLs and their associated captions. This yielded a total of 5,442,924 pairs. 3. **Deduplication based on image URLs**: Many images, such as the [Japanese national flag](https://ja.wikipedia.org/wiki/%E3%83%95%E3%82%A1%E3%82%A4%E3%83%AB:Flag_of_Japan.svg), are reused across multiple Wikipedia pages. To eliminate such redundancy, we removed exact duplicates based on image URLs, retaining only the first occurrence. This resulted in 1,054,434 unique pairs. 4. **Downloading images**: Using the deduplicated (image-url, caption) pairs, we downloaded the corresponding images using [img2dataset](https://github.com/rom1504/img2dataset). The download success rate was 99.35%, with 1,047,565 images successfully downloaded. The most common failure was due to image decoding errors. 5. **Deduplication based on perceptual hash (pHash)**: For the successfully downloaded images, we performed further deduplication using perceptual hashing. Images with a Hamming distance of 5 or less between their pHashes were considered duplicates and filtered out. After this step, 1,023,264 unique images remained. The removed images can be found at https://huggingface.co/datasets/speed/waon-wiki-pair-removed. Dataset construction code can be found at https://github.com/llm-jp/crawl-mm. ## Dataset Comparison | Dataset | Total Image-URL Pairs | After URL-based Deduplication | |--------------------------------------|------------------------|-------------------------------| | **Wiki-JA-Pair** | 1,023,264 | 1,023,264 | | **[turing-motors/Wikipedia-Vision-JA](https://huggingface.co/datasets/turing-motors/Wikipedia-Vision-JA)**| 1,602,962 | 681,097 | ## Statistics - **Total unique article titles**: 330,242 - **Total images after deduplication**: 1,023,264 - **Average images per title**: 3.10 - **Max images per title**: 690 - **Min images per title**: 1 ### Distribution of Images per Title The histogram below shows the distribution of the number of images per Wikipedia article. <img src="images_per_title_distribution.png" alt="Histogram of images per article" width="70%"/> ### Top 10 Articles with Most Images | Rank | Article Title | Number of Images | |------|---------------|------------------| | 1 | アメリカ合衆国の道路標識 | 690 | | 2 | 豊田市 | 412 | | 3 | 軍服 (中華民国) | 405 | | 4 | 蓬左 | 372 | | 5 | 清代学者像伝 | 366 | | 6 | 日本の道路標識 | 337 | | 7 | 軍服 (フランス) | 329 | | 8 | オーストラリアの道路標識 | 328 | | 9 | 家紋の一覧 | 327 | |10 | 国籍識別標 | 313 | ## LICENSE This dataset is licensed under CC-BY-SA 4.0. ## Citation ```bib @misc{sugiura2026jaglebuildinglargescalejapanese, title={Jagle: Building a Large-Scale Japanese Multimodal Post-Training Dataset for Vision-Language Models}, author={Issa Sugiura and Keito Sasagawa and Keisuke Nakao and Koki Maeda and Ziqi Yin and Zhishen Yang and Shuhei Kurita and Yusuke Oda and Ryoko Tokuhisa and Daisuke Kawahara and Naoaki Okazaki}, year={2026}, eprint={2604.02048}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2604.02048}, } ```