A Variant Character Dataset for Historical Narratives of Middle and Late Imperial China

This dataset, titled “A Variant Character Dataset for Historical Narratives of Middle and Late Imperial China,” addresses the widespread presence of variant characters in pre-modern Chinese texts, a significant challenge in computational analysis. In historical narratives written in Classical Chinese from the Tang to the Qing dynasties—much like the majority of pre-modern Chinese literature—there exist variant characters that share the same meaning but differ in appearance. Such orthographic variations can undermine the reliability of natural language processing tasks such as text similarity detection, keyword extraction, and analysis of historical text reuse. Moreover, modern encoding standards (e.g., Unicode) that capture minor visual differences in characters, as well as OCR technology that can mistakenly conflate these variations, often exacerbate the issue.

To mitigate these challenges, this dataset compiles 2,723 variant–representative character mapping pairs that can be used to normalize text for computational tasks. It consolidates data from multiple authoritative sources, including Unihan Variants (Unicode Consortium), variant tables published by the State Council of the People’s Republic of China, and variant lists extracted from Wikisource and Kanripo corpora of Classical Chinese literature. After collecting initial candidates, duplicates and purely modern simplified characters were removed to retain only those relevant to historical narratives. Variant sets were then merged and streamlined, ensuring no character pair is duplicated. A prioritization scheme based on Taiwanese and Korean standard forms was implemented to select a single “representative” character for each set. This enables a “many-to-one” mapping, where each variant converges on a single standardized character.

The resulting 1:1 pairs between each variant and its representative character are particularly valuable for improving text mining reliability in research on Middle and Late Imperial China. By applying these mappings, scholars can more accurately track word frequencies, investigate intertextual connections among historical works, and improve the quality of OCR outputs for further analysis. Additionally, the dataset holds significant potential for reuse in other East Asian contexts—such as Korea, Japan, and Vietnam—where historical texts are also replete with variant characters inherited from classical Chinese orthography. The open-access license (CC0) further allows researchers to adapt the dataset to different time periods, literary genres, and research agendas, fostering collaborative refinement over time.

In sum, A Variant Character Dataset for Historical Narratives of Middle and Late Imperial China serves as an essential resource for anyone seeking to apply computational methods to pre-modern Chinese texts. By harmonizing variant characters into a unified standard, this dataset helps bridge the gap between the rich diversity of historical Chinese orthography and the structured requirements of modern computational analysis.