Datasets and code for "Mapping the plague through natural language processing"

This project investigates the performance of various NLP libraries and geocoding services for the semi-automated generation of quantitative datasets from narrative texts. We provide the original files, several intermediate data products as well as the final plague datasets. Please note that some of the steps in this process were done manually, thus some of the scripts cannot be run completely. This work is based on two plague treatises: - Sticker, G. 1908 <em>Abhandlungen aus der Seuchengeschichte und Seuchenlehre. Band 1: Die Pest</em>. Giessen, A. Töpelmann. - Biraben, J.-N. 1975 <em>Les hommes et la peste en France et dans les pays européens et méditerranéens</em>. Paris, Mouton. The final geocoded, plague datasets are: <strong>- plague_sticker_v1.csv</strong> <strong>- plague_biraben_v1.csv</strong> A data dictionary is available as <strong>plague_datadict.xlsx</strong> Other files: file name content sticker_OCR_orig.txt Original OCR text sticker_OCR.txt Original OCR text without parenthesis (author names) sticker_textprep.rds Original OCR text with further preparations sticker_goldstandard_annotated_1.tsv manual annotations file 1 sticker_goldstandard_annotated_2.tsv manual annotations file 2 sticker_goldstandard_annotated_consensus.tsv consenus annotation file sticker_standard_toponyms.csv Gold standard for toponym recognition. Contains the tokenization, the start/end character respective to the OCR text (orig and without parenthesis) and whether a token is a location or other sticker_comparison_NER.rds Comparison of NER performance sticker_comparison_geocoding.rds Comparison of Geocoding performance