RevBench: code, scenarios, run logs and judgments for enforcing mid-task permission revocation in AI agents

Artificial intelligence (AI) agents embedded in software-as-a-service (SaaS) platforms inherit the user's permissions. When a permission is revoked during a task, the platform rejects the next read, but the agent still retains the data it previously retrieved in its context, drafts, saved documents, and memory. This study examined how platforms can enforce a mid-task revocation on that working state. The author defined correct behavior for four revocation types, each mapped to an obligation to block, purge, or suspend, including leaks through paraphrases, aggregates, and existence hints. The author built RevBench, a simulated field-service platform, and compared nine enforcement designs from access control, information-flow control, and machine unlearning on six frontier models in 6,048 runs. Outputs were scored using exact matching and two large-language-model judges. Per-call authorization still failed in 417 of 720 runs (57.9%), rising from 34.4% when the revocation preceded retrieval to 72.8% when it followed delivery. Information-flow labels leaked the least (12.9%) but left revoked data saved in 44.2% of runs. Designs that invalidated the context performed best, while editing the context in place left derived facts behind. A full restart and selective replay with listing-level provenance reached 21.4% and 21.0% substantive failure, the latter with 12.1% fewer model calls. Most remaining failures were disclosures about the restriction, often repeating the platform’s own notice, and larger models did not reduce them. SaaS platforms need revocation controls that reach the agent’s working state and match the type of revocation. Data and code supporting the article "Revoking Access Does Not Revoke Knowledge: Enforcing Mid-Task Permission Changes in AI agents: