Dataset for Automating Legal Statute Matching in UK Online Petition Systems

Detailed Description:This dataset provides real-world data to help understand and solve the "vocabulary mismatch" problem between everyday citizen language and formal legal text. It is designed for researchers testing how well search models and AI systems can match informal, unstructured requests (like public petitions) to their correct governing laws. Dataset Contents: Statutory Corpus (legislation_txts & legislation_raw_pdfs): legislation_txts: A comprehensive, pre-processed corpus containing 110,097 UK legislative instruments converted into plain text, making them ready for natural language processing and search indexing. legislation_raw_pdfs: A folder containing a sample set of 50 original UK legislation PDFs to demonstrate the raw source format before text extraction. How the Legislation was Collected and Processed: The raw PDFs were programmatically collected from the official UK database. Because the database lacks a centralized file dump, a custom Python web scraper iterated through expected document years and serial numbers. To ensure integrity, the scraper advanced to the next year automatically if it encountered 50 consecutive '404' responses, allowing it to comprehensively index available statutes. To generate the plain text ('legislation_txts'), these documents were digitized using an Optical Character Recognition (OCR) pipeline. This was necessary because a substantial portion of the corpus, particularly pre-2000 legislation, exists solely as scanned images without an embedded text layer. Each page of the PDF was rasterized into high-resolution images (via PIL) and processed through Tesseract OCR. The raw text was then computationally sanitized using Regular Expressions to strip administrative metadata, headers, footers, and scanning artifacts. Ground Truth Evaluation Set (ground_truth_dataset.json) An evaluation dataset containing 65 real-world UK government e-petitions, representing authentic layperson queries. How it was collected: 65 petitions were manually collected from the official UK Petitions website and the House of Commons Library that had crossed the 10,000-signature threshold, as crossing this threshold guarantees a formal response from the UK Government. These official government responses were then parsed to extract any explicit citations to UK statutes made by the responding officials. Because these statutory citations were provided directly by government experts in their official capacity, they form a reliable "ground-truth" mapping between the citizen's informal complaint and the exact legal instrument. Each JSON entry includes: petition_id petition_text (The informal complaint or request written by the citizen) relevant_legislations (The precise statutory codes cited in the government's response) link (Source URL for the petition) Citation: If you use this dataset in your research, please cite the following corresponding paper as well: @inproceedings{alhamid2026automating, title={Automating Legal Statute Matching in Online Petition Systems}, author={Al Hamid, Baizid and Kovacs, Mate and Salama, Shady and Serd{\"u}lt, Uwe}, booktitle={Proceedings of the International Conference on eDemocracy \& eGovernment (ICEDEG)}, year={2026}, note={Accepted for publication. To appear.}} Use Cases: This dataset is highly useful for researchers looking to evaluate text retrieval frameworks (e.g., BM25, semantic search, or hybrid models) on complex legal language tasks. It can also be used to explore how the public talks about the law versus how the law is actually written. For ease of integration in machine learning pipelines, the textual corpus and evaluation sets are also accessible via the Hugging Face Datasets library