aci-bench-priv

This dataset is an extension of the Ambient Clinical Intelligence Benchmark (aci-bench). It is designed to evaluate privacy preservation in documentation, specifically whether models/note generation systems can appropriately distinguish third-party information that should be omitted from clinical notes, thereby respecting contextual integrity. In our work, we sample 10 base patient cases sampled from aci-bench. Each case is expanded into 42 variations (six relationship types x seven information categories) by injecting personal information concerning an individual related to the patient into the transcript. Relationship types: child, parent, sibling/cousin, spouse/partner, friend, boss/co-worker/classmate Information categories: stigmatizing health status/medication, mental health status/medication, relationship history, religious/spiritual views, political views, sexuality/sex life, lifestyle/habits/recreational activities This results in 420 source transcripts containing third party information irrelevant to the patient's care. Related Publication: (Under review) Files: "acibench-priv-notes.jsonl" contains notes generated by models evaluated in the paper (Llama-3.1-8B, Llama-3.1-70B, Mixtral-8x7B, Mixtral-8x22B, Claude-3.5-Haiku, Claude-3.5-Sonnet).