Semantic Search Engine and Vernacularity Predictor for the Southern Macro-Region of Brazil AHU Catalog

Overview This dataset and its accompanying web application provide a machine-readable and semantically annotated corpus of catalog summaries from the Arquivo Histórico Ultramarino (AHU). Focused on the "Southern Macro-Region of Brazil" during the late colonial period (1737–1828), these files are the product of a pipeline that utilized Large Language Models (LLMs), Lexical Search Engines, and dense vector embeddings to quantify sociolinguistic variables. The project was designed to support research in Diachronic Linguistics, specifically aiming to map grammar competition and isolate the probability of Brazilian Portuguese syntax in historical administrative documentation. Furthermore, it serves as an advanced research tool, generating direct and clickable hyperlinks to manuscript records in the DigitArq (Portugal) and Projeto Resgate (Brazil) databases. Files Included in this Dataset The dataset and the application are distributed as a repository containing the following core files and folders: ahu_sul_catalog.json: The primary structured resource (JSON format). It connects the raw archival text to geographic metadata, temporal data, modernized archival reference codes, and algorithmic sociolinguistic annotations. Core Keys: reference_code: Modernized reference code following the CRAV/DigitArq standard (e.g., PT/AHU/CU/023-001/0006/00631). new_code: Standardized/intermediate code still utilized by the Projeto Resgate. description: Full-text description extracted from the AHU catalog. vernacular_score: A decimal number (0.0 to 1.0) that quantitatively represents the Score of Potential Sociolinguistic Relevance (SRSP), indicating the probability of vernacular/localized syntax occurring in the original document. vector: The communication direction of the document (e.g., Top-Down, Bottom-Up). extracted_typology: The normalized singular root of the document type (e.g., CARTA, REQUERIMENTO). sociolinguistic_reasoning_by_deepseek_v3: The analytical justification generated by the AI for the assigned Score of Potential Sociolinguistic Relevance (SRSP). ahu_semantic_index.pkl: A serialized Python dictionary containing the dense vector embeddings of the archival descriptions. Generated via the intfloat/multilingual-e5-large model, these embeddings are rigidly anchored to the document reference codes to ensure seamless synchronization during semantic retrieval. auditoria/: A folder containing the methodological documentation and data from the empirical validation (AI Consistency Audit) conducted on a random sample of 10% of the corpus (705 documents). It includes the data used to calculate the Mean Absolute Error, Pearson Correlation, and Systematic AI Bias. app.py: The Streamlit Web Application script (Graphical User Interface). It manages the Dynamic Hybrid Search (Ensemble Search), fusing BM25 Lexical scoring with E5 Semantic Similarity, sociolinguistic filtering ("Lenses"), dynamic URL routing to digital repositories, and the programmatic generation of formatted PDF dossiers via FPDF. requirements.txt: A list of Python environment dependencies required to execute the pipeline and the web application (e.g., pandas, streamlit, sentence-transformers, rank-bm25, torch, fpdf). Included Scope Temporal Range: 1737 to 1828. Geographical Focus: Southern Macro-Region of Brazil. Data Source: Summaries and metadata extracted from the catalogs of the Arquivo Histórico Ultramarino (AHU). Source of Archival Data The original unstructured textual summaries used to build this corpus were obtained from the public cataloging efforts of the Arquivo Histórico Ultramarino and the Projeto Resgate. We are deeply grateful for the ongoing work of these institutions in preserving and indexing the colonial documentary heritage. The modernized reference codes generated by this application interface directly with the infrastructure of the National Archive of Portugal (DigitArq). Methodology Extraction & Modernization: Regular Expression (RegEx) parsing of the raw AHU catalog text to isolate document IDs, dates, and typologies, followed by algorithmic zero-padding to reverse-engineer and generate modern reference codes compatible with the CRAV standard. Annotation: Zero-shot inference via an LLM (DeepSeek) to categorize the sender/recipient, communication vector, and calculate the Score of Potential Sociolinguistic Relevance (SRSP) along with a written analytical justification. Vectorization (Semantic): Context-enriched passages transformed into embeddings using the intfloat/multilingual-e5-large model from HuggingFace. Tokenization (Lexical): Term frequency-inverse document frequency indexing across the documents via the BM25Okapi algorithm to ensure accurate retrieval of proper nouns and locations. Search Fusion: Dynamic Ensemble Retrieval computing a weighted hybrid score between semantic meaning and lexical exactness. Methodological Criteria For a detailed description of the computational pipeline, LLM prompting parameters, empirical systematic bias auditing, the logic behind the "CRAV Reference Modernizer", Hybrid Search algorithms, and UI/UX deployment rules, please refer to the forthcoming data paper that will be associated with this repository. Citation Currently, this dataset and application are published as an independent computational resource. If you use this corpus, methodology, or code, please cite this repository directly using its Zenodo DOI: Pacheco Rocha, Saulo Rogério. (2026). AHU Catalog Document Classifier for the Southern Macro-Region of Brazil (Version 2.0) [Data set and Software]. Zenodo. https://doi.org/10.5281/zenodo.18772667 License This dataset and the associated code are made available under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license. Official Repository The source code, updates, and full documentation can be found on GitHub:https://github.com/saulorrp/catalogo-ahu-cone-sul.git