115 Indonesian Presidential Regulations (Perpres) Zero-Shot Retrieval Dataset

115 Indonesian Presidential Regulations (Perpres) Zero-Shot Retrieval Dataset OVERVIEW This repository contains a curated benchmark dataset for legal Information Retrieval (IR) in the Indonesian language. It is designed to evaluate and compare the performance of sparse lexical retrieval (e.g., BM25) and dense semantic retrieval (e.g., IndoBERT) on formal bureaucratic texts. The dataset focuses on Indonesian Presidential Regulations (Peraturan Presiden or Perpres), simulating real-world e-government search scenarios where users may employ either formal legal terminology (literal intents) or paraphrased layperson vocabulary (semantic intents). DATASET STATISTICS Total Corpus Segments: 1,509 legal segments (broken down to the ayat/clause level). Total Queries: 204 evaluated queries. -- Literal Queries: 102 (exact matching, high entity density). -- Semantic Queries: 102 (vocabulary mismatch, layperson phrasing). Relevance Judgments: 3,602 mapped query-document pairs in the final Qrels. REPOSITORY STRUCTURE (The dataset is divided into four main CSV files for ease of use in zero-shot evaluation and fine-tuning tasks) dataset_korpus_per_ayat.csv The target search space. This file contains the flattened, segmented text of 115 Presidential Regulations. row_id: The unique identifier for the segment (serves as the doc_id in qrels). nomor_peraturan: The official regulation number. segmen: The structural part of the regulation (e.g., Pasal, Judul). sub_segmen: The specific sub-part or clause number. value: The actual textual content of the regulation segment. ds_literal_final_2.csv The set of literal queries. These queries mimic formal searches containing exact legal terminologies, specific geographical constraints, and foreign terms. query_id: Unique identifier for the literal query (e.g., literal_10). query_literal: The text of the literal query. value: The target ground truth text associated with the query. ds_semantic_final_2.csv The set of semantic queries. These queries simulate the search intent of a general citizen (layperson), intentionally introducing a vocabulary mismatch between the query and the formal legal text. query_id: Unique identifier for the semantic query (e.g., semantic_15). query_semantic: The text of the semantic query. value: The target ground truth text associated with the query. qrels_final.csv The Query Relevance Judgments (Ground Truth) file mapping queries to their relevant corpus segments. query_id: The ID of the query (matches IDs in the query files). query_type: Indicates whether it is Literal or Semantic. query_text: The actual query string. doc_id: The relevant target document (matches row_id in the corpus). label: Relevance score (serves as the ground truth for metric calculation). DATA GENERATION AND ANNOTATION METHODOLOGY (The queries and their corresponding relevance judgments (qrels) were constructed using a three-stage annotation pipeline to ensure rigorous quality control) AI-Assisted Generation: Initial candidate queries (both literal and semantic) were synthesized from the Presidential Regulation segments using a Large Language Model. Researcher Annotation and Relevance Validation: The primary researcher served as the sole annotator to manually audit, filter, and map the query-document pairs. This stage involved filtering out hallucinated entities and ensuring accurate relevance judgments against the formal legal texts. No external legal professionals were involved in establishing this ground truth. Linguistic Expert Validation: To ensure the quality of the semantic queries, an Indonesian linguist (ahli bahasa Indonesia) reviewed and validated the dataset. This step ensures that the vocabulary mismatches and semantic gaps realistically reflect natural layperson phrasing rather than artificial or machine-translated artifacts. Note on Qrels: Only the final annotations that passed both the researcher's relevance validation and the linguistic expert's review are included in the qrels_final.csv file. LICENSE This dataset is released under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. You are free to share and adapt the material for any purpose, even commercially, as long as appropriate credit is given.