P3T2Q Benchmark

Description P3T2Q (Privacy-Preserving and Permissive Text-to-Query) is a benchmark dataset for evaluating Text-to-SQL systems in privacy-sensitive database settings. It is designed to test whether a system can generate SQL queries that preserve the intent of a user’s natural-language question while respecting role-based access-control rules. P3T2Q is built using database information taken from the BIRD Bench dataset, focusing on databases that contain privacy-sensitive schema elements. Unlike standard Text-to-SQL datasets, P3T2Q provides both an unrestricted intent-preserving query and role-specific privacy-preserving queries for the same natural-language question. This allows researchers to evaluate whether a model can adapt its generated SQL output depending on the user’s permissions. The goal of P3T2Q is to support research on privacy-aware natural-language database interfaces, where models must balance utility, intent preservation, and compliance with access-control policies. Possible Uses This dataset can be used for: Evaluating privacy-preserving Text-to-SQL generation. Studying role-based access control in natural-language database interfaces. Benchmarking query rewriting under different user permissions. Detecting privacy violations in generated SQL queries. Training or evaluating LLMs for policy-aware database querying. Text 2 SQL task Dataset Design P3T2Q is defined around four main components: databases, roles, access-control rules, and natural-language question/query examples. Each dataset instance contains: A natural-language question. An intent-preserving SQL query, representing the unrestricted query that would answer the question with full access. A role-specific privacy-preserving SQL query, representing the most informative query allowed under a given user role. The benchmark uses seven databases whose database information is derived from the BIRD Bench dataset: california_schools debit_card_specializing financial formula_1 student_club superhero thrombosis_prediction The dataset contains 3,680 total instances. Six databases contain 560 natural-language/query triplets each, and one database contains 320 triplets. P3T2Q uses four user roles: Public: Limited access. Can view aggregates, counts, and grouped statistics over non-sensitive dimensions. Direct exposure of sensitive attributes is not allowed. Analyst: Can access non-sensitive columns and use sensitive attributes in filters or joins, but cannot directly output sensitive values unless they are masked or aggregated. Staff: Can access selected sensitive fields needed for operational tasks, such as names, but remains restricted from high-risk sensitive attributes such as email, phone number, address, SSN, or salary. Admin: Full access. Serves as the upper-bound baseline for unrestricted query generation. Attributes are categorized into four sensitivity levels: S0 — Non-sensitive: General attributes with no privacy restriction. S1 — Quasi-identifiers: Attributes such as zipcode, city, gender, age, or department, depending on the database domain. S2 — Personally identifiable information: Attributes such as name, email, phone number, address, date of birth, salary, or income. S3 — Highly sensitive information: Attributes such as SSN, credit card number, password, authentication token, or bank account. The SQL space is intentionally controlled to make evaluation interpretable. Queries are restricted to non-nested SELECT statements and cover common query patterns, including projection, aggregation, filtering, grouping, and multi-table joins. The benchmark is designed to capture both projection-level privacy violations, where sensitive attributes are exposed in the query output, and processing-level privacy violations, where sensitive attributes are improperly used in filtering, joining, or aggregation. --- SQL Access Control Enforement Coverage Validation Dataset This validation suite is a manually curated companion dataset for the P3T2Q benchmark, built using the financial database from BIRD Bench. It contains 100 SQL queries designed to evaluate whether privacy and access-control enforcement mechanisms can correctly distinguish between permitted and prohibited SQL queries. The suite covers a broad range of SQL behavior across 27 SQL pattern families, including filtering, aliasing, DISTINCT, ordering, pagination, aggregation with GROUP BY and HAVING, joins, nested and correlated subqueries, set operations, and Common Table Expressions. Each violating query is annotated with the expected corrected query and the policy or rule that triggered the violation. The dataset is provided as a sql_enforcement_eval_dataset_v4.csv file with the following columns: query_id: Unique identifier for each query. input_query: Original SQL query to be validated. Correct: Indicates whether the query is permitted or prohibited. expected_query: Expected realigned or corrected query for violating cases. related_policies: Policies associated with the validation decision. related_rules: Specific rules triggered by the query. complexity: Complexity level of the SQL query. Overall Validation Status: Final validation status for the query.