APWD: Assamese Printed Word Dataset from Texts of the Shankari (1490–1700) and Jonaki (1889–1940) Literary Eras

APWD: Assamese Printed Word Dataset from Shankari (1490–1700) and Jonaki (1889–1940) Literary Eras Version 1.1.0 — supersedes 1.0.0. See "Changes in version 1.1.0" below. OVERVIEW APWD is a large scale, quality validated dataset of 39,743 printed Assamese word images segmented from seven classical texts spanning 500 years of literary heritage (1490–1940). The dataset bridges two major literary periods: the Vaishnavite Age (Shankari Era) and the Romantic Age (Jonaki Era). DATASET COMPOSITION Total Images: 39,743 word images (PNG format)Classes: 7 (one per source text)Format: 8 bit grayscale PNGSize: 34 MB extracted Dimensions: word crops are wide, not square.• Width: 23–838 px (1st–99th percentile: 59–301)• Height: 30–423 px (1st–99th percentile: 40–99)• Mean aspect ratio: approximately 2.3:1 Resize with the aspect ratio preserved and pad to a fixed canvas. Squashing images to a square distorts Assamese conjunct glyphs. Source Distribution: Shankari Era / Vaishnavite Age (1490–1700) — 12,875 images (32.4%)• Kirtan_Ghosha: 11,252 images (28.3%) — Srimanta Sankardeva (1449–1568)• Naam_Ghosha: 1,623 images (4.1%) — Madhavdeva (1489–1596) Jonaki Era / Romantic Age (1889–1940) — 26,868 images (67.6%)• burhi_aair_xadhu: 11,143 images (28.0%) — Lakshminath Bezbaroa• Miri-Jiyori: 6,287 images (15.8%) — Rajanikanta Bordoloi• Jilikoni: 4,638 images (11.7%) — attribution unconfirmed• kokadeuta_aru_natilora: 2,416 images (6.1%) — Lakshminath Bezbaroa• Gauburha: 2,384 images (6.0%) — Padmanath Gohain Baruah (1871–1946) Balanced subset: 1,623 images per class, 11,361 total, listed in balanced_subset.txt and sampled with a fixed seed for reproducibility. IMPORTANT: WHAT "ERA" MEANS The era field records when a text was COMPOSED, not when the edition that was scanned was PRINTED. All seven sources were segmented from modern printed editions, so their typography reflects 20th century typesetting regardless of whether the text dates from 1490 or 1911. Era should therefore NOT be treated as a typographic or visual domain label. Within-source experiments will appear to separate era with very high accuracy, but this reflects typeface identity rather than period: the two Shankari sources are devotional works by closely associated authors and are likely set from a shared edition, so an era classifier can succeed by recognising a print style. Source text, by contrast, is genuinely separable. HOW TO USE THIS DATASET WELL Consider splitting at the page level rather than the image level. Filenames encode page and word position ({Source}-Page-{p}-word-{n}.png), so grouping by the page field is straightforward. Words from the same page share a typeface instance, an ink density and a scanner pass, so assigning individual word images to train and test at random places near neighbours on both sides of the split boundary. How much this matters depends on the method. In our own measurements on seven way source classification with HOG features, a random image level split and a page grouped split gave the same accuracy for a linear classifier (within about one point, with the page grouped split marginally ahead), but the random split inflated a nearest neighbour classifier by roughly two points (1-NN 0.718 against 0.696; 5-NN 0.724 against 0.706). The effect is therefore small for models that cannot memorise individual examples, and material for instance based methods and, by extension, high capacity networks. Page grouped splitting costs nothing and removes the question, so we recommend it as the default, particularly for retrieval and nearest neighbour work. Renormalise crop margins before extracting features. The intended crop convention is a small uniform padding around each word, and it holds at the median for every source, but not uniformly: roughly 31% of burhi_aair_xadhu images and 24% of Miri-Jiyori images are cropped tight to the ink with zero margin, against under 0.1% for the other five sources. Descriptors computed on a fixed grid over the raw crop therefore partly encode the crop convention, which correlates with source text. Cropping to the ink bounding box and re-padding uniformly removes this at negligible cost. Deduplicate if you assign splits at random. The collection contains a small number of byte-identical duplicate images, all within a single source text. Be aware of running headers. Several sources capture a page header once per page, including the book title on nearly every page of Jilikoni and on a majority of Kirtan_Ghosha pages. These are legitimate Assamese words and are retained, but they are page furniture rather than body text, and they give a source classifier a shortcut. Page coverage is partial. Pages were sampled rather than processed end to end, and coverage is sparse for some titles. Do not treat any source as a complete rendering of the underlying work, and do not compute type-token statistics from it as though it were. WHAT THIS DATASET DOES AND DOES NOT SUPPORT Supported:• Content Based Image Retrieval• Source text / provenance classification (7 classes)• Instance level retrieval and self supervised representation learning• Assamese script recognition and typography analysis• Low resource language processing• Deep learning benchmarking Not supported:• Query by string word spotting, or any task requiring word identity. There are NO word level transcriptions. The only label is the source text; filenames encode page and word position, not what the word says.• Historical document degradation benchmarking. The scanned editions are modern printings, so the images do not exhibit the degradation characteristic of period artefacts. QUALITY ASSURANCE All images underwent quality validation:• OCR based filtering to remove non word content• Shape based filtering for image quality• Manual validation of filtered samples• 10 fold cross validation on 500 sampled images: 0.8% error rate (99.2% accuracy)• v1.1.0: removal of 172 residual non word and non Assamese images (see below) Two caveats on the validation figure. First, 4 errors in 500 sampled images gives a 99.2% point estimate, but the Wilson 95% confidence interval on the true error rate runs from 0.31% to 2.04%; the figure should be read with that interval. Second, that review was performed on version 1.0.0 and measures average lexical quality on a random sample. It cannot enumerate rare defect classes: the 172 images removed in 1.1.0 amount to 0.43% of the collection, at which prevalence a 500 image sample would be expected to contain fewer than three of them. Some word images retain trailing punctuation — commas, dandas, hyphens and parentheses — and hyphenated word fragments arising from line-break hyphenation occur. Users requiring strictly punctuation free word images should apply their own filter. METHODOLOGY 1. Source texts obtained as freely available public domain PDFs from online repositories and libraries2. Word segmentation using image processing3. Quality filtering using OCR and shape analysis4. Manual validation on random samples5. Artefact and non Assamese image removal, and metadata correction (v1.1.0) CHANGES IN VERSION 1.1.0 Corrective release. Version 1.0.0 remains permanently citable; work already published against it still reproduces. Users should migrate to 1.1.0. • Literary era labels corrected for four of seven sources. The Shankari and Jonaki assignments were effectively swapped in v1.0.0: Kirtan_Ghosha and Naam_Ghosha were labelled Jonaki but are Shankari era compositions by Sankardeva and Madhavdeva; Gauburha and Jilikoni were labelled Shankari but belong to the Jonaki era. The corrected era balance is 12,875 Shankari and 26,868 Jonaki (32.4% / 67.6%), against 17.9% / 82.1% implied by v1.0.0. Era stratified experiments run against v1.0.0 labels should be repeated. • Author attribution and an era_verified flag added for each source. • 172 images removed in two passes (0.43%). Image count 39,915 → 39,743. – 159 non word artefacts: decorative rules, page borders and ornamental dividers, identified by size and aspect ratio criteria and confirmed by inspection. Listed in removed_v1.0.0.txt. – 13 images containing Latin characters rather than Assamese script, which violate the stated filtering criterion excluding Latin letters. Ten of these are a single recurring glyph appearing once on every Naam_Ghosha page. Both lists ship with the archive so the change is auditable. • Size and resolution metadata corrected. v1.0.0 stated 2.5–3.5 GB (actual: 34 MB) and a 64×64–256×256 resolution range implying square images (actual: wide crops averaging 2.3:1). • Balanced subset file list now shipped; v1.0.0 described it but provided no way to reconstruct it. The subset is 1,623 images per class, 11,361 in total. • Caveat added on the meaning of the era field (see above). • Explicit statement added that no word level transcriptions are provided. • Usage guidance added on page grouped splitting, crop margin normalisation, duplicates, running headers and partial page coverage. • Internal inconsistencies resolved: acquisition method, date ranges, and the class balance description. • Packaging fixed: directory case (Images/) now matches the documentation, and macOS artifacts (.DS_Store, __MACOSX) removed. Outstanding: the literary era and authorship of Jilikoni remain unconfirmed. It is provisionally recorded as Jonaki with era_verified set to false. See CHANGELOG.md. Full details in CHANGELOG.md within the archive. CITATION Borah, N., Baruah, U., & Dey, B. (2026). APWD: Assamese Printed Word Dataset from Shankari (1490–1700) and Jonaki (1889–1940) Literary Eras (Version 1.1.0) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.20241233 Note: This dataset was created as a part of PhD research on "Document Retrieval for Assamese Scripts using Content Based Image Retrieval Techniques" by Naiwrita Borah under the guidance of Dr Udayan Baruah and Dr Barnali Dey at the Department of IT, Sikkim Manipal Institute of Technology, Sikkim Manipal University (2019–2025) (http://hdl.handle.net/10603/635106).