HTR processed dataset of the Häme Bailiffs' records (1539-1634)

Dataset This dataset includes text-recognized contents of the 16th to 17th century bailiff registers (voudintilit, fogderäkenskaper) from Häme (Tavastland). The processed material consisted of 889 individual bailiffs' records and the 19th century archival catalogue of the Häme bailiffs' records. The contents are given in the dataset in both .txt and .xml formats. The dataset consists of two .zip folders, one containing only the .txt files, the other the complete model output (.xml and .txt files). The dataset was produced by the Hämeen voudintilit -project in collaboration with the National Archives of Finland (NAF). The project was carried out with the support of Jalmari Finne Foundation (Jalmari Finnen säätiö) and in cooperation with the Ajanluku research initiative (https://zenodo.org/communities/ajanluku). Content recognition The content recognition process was carried out by the Hämeen voudintilien katalogisointi ja tekoälyavusteinen analysointi -project in collaboration of the NAF. The handwritten text recognition (HTR) model used was trained by the project in collaboration with the NAF and is based on the multicentury HTR model and pipeline developed by the NAF (https://huggingface.co/Kansallisarkisto). The HTR pipeline consists of both segmentation and text recognition models. Training runs and inference were carried out by the NAF. The training data for the model consisted of several datasets: training data created in the project, training data received from the Ajanluku-project and further refined by the project, and training data provided by the NAF and the National Archives of Sweden (Svenska Riksarkivet). The training dataset for the HTR model included 1067 document images and 65349 line images, 133 document images and 8393 line images for the validation dataset and 134 document images and 8063 line images for the testing dataset. HTR model’s performance was evaluated with both the validation and testing dataset. Validation dataset CER best value 6.71% WER best value 19.6% loss best value 0.385 Testing dataset CER best value 5.5% WER best value 16.2% Dataset description TXT files The .txt files were produced from the .xml files and contain only the recognized text. The files are numbered with both the archival unit number (three or four digits and a possible additional letter) and the unique identifier given by the NAF metadata system (ten digits), separated by an underscore, e.g. “3678_1570763683.txt” The data of each archival unit is given in one .txt file. Each .txt starts with a metadata header, which is formatted as follows: Aineistokokonaisuus: Archival collection name, e.g.: ”Hämeen voutikuntien tilejä” Arkistoyksikkö: Archival unit name, e.g. ”Tilikirja” Arkistoyksikön id: unique identifier, e.g. “1570763683” Tilinumero: archival unit number, e.g. “3678” Alkuvuosi: Starting year, e.g. ”1543” Loppuvuosi: Ending year, e.g. ”1544” The text content proper is given only on images with recognized text, i.e. blank pages are skipped. The contents are formatted as follows: Tiedoston id: Image number, e.g. ”0001”, “0003” Teksti: the text proper as recognized by the model, given line by line without line numbers. The line order Model output (.XML & .TXT files) The model output -dataset includes both the .txt and the .xml files. Each model output is provided as a .zip folder. The .zip folder is named with a unique identifier, given by the NAF metadata system (ten digits). The zip folder contains a folder with the same name, and this folder contains the recognized text contents as a .txt file, and a subfolder “alto”, which includes all the recognized text contents as .xml files, one .xml per page. The .xml files are named in accordance with the picture numbers, e.g. “0001”, “0002” etc. Only the pages with recognized text content have been processed, i.e. there are no .xml files concerning pages with no recognized text content. The data from each page is given in one .xml file. The .xml’s follow the AltoXML v4 -schema. Each .xml starts with a header. The header includes information on the xml schema and the Description -block. Description block contains metadata about the processing pipeline: MeasurementUnit: coordinate unit used. NOTE: the field value is mistakenly given as one 1/10th of millimeter, “mm10”. The correct value would be “pixel”, as the polygon coordinates are in fact given as pixels. sourceImageInformation: path to the source image file Processing blocks: one per processing stage, each recording the agency, a description of the step, and the software name/version used Layout -block contains the recognized contents: Page: contains page-level attributes: dimensions, language, page number, and confidence scores (PC values, PC/PC25/PC50/PC75). PrintSpace: is the printable area of the page, containing one or more TextBlock elements. TextBlock: represents a region of text on the page. Its Shape/Polygon defines the region boundary as a list of x,y coordinate points, given as pixel coordinates. TextLine: elements are nested within the TextBlock, each representing a single detected line of text. Each has: Shape/Polygon: the bounding polygon of the line in page coordinates (in pixels) String: the recognized text content, with main attributes: CONTENT: the recognized text string RL: string length in characters WC: word confidence score (0–1)