CIMemories

# CIMemories: A Compositional Benchmark for Contextual Integrity of Persistent Memory in LLMs
[Paper](https://arxiv.org/abs/2511.14937)

Large Language Models (LLMs) increasingly use persistent memory from past interactions to enhance personalization and task performance. However, this memory introduces critical risks when sensitive information is revealed in inappropriate contexts. We present CIMemories, a benchmark for evaluating whether LLMs appropriately control information flow from memory based on task context. CIMemories uses synthetic user profiles with over 100 attributes per user, paired with diverse task contexts in which each attribute may be essential for some tasks but inappropriate for others. Our evaluation reveals that frontier models exhibit up to 69% attribute-level violations (leaking information inappropriately), with lower violation rates often coming at the cost of task utility. Violations accumulate across both tasks and runs: as usage increases from 1 to 40 tasks, GPT-5's violations rise from 0.1% to 9.6%, reaching 25.1% when the same prompt is executed 5 times, revealing arbitrary and unstable behavior in which models leak different attributes for identical prompts. Privacy-conscious prompting does not solve this - models overgeneralize, sharing everything or nothing rather than making nuanced, context-dependent decisions. These findings reveal fundamental limitations that require contextually aware reasoning capabilities, not just better prompting or scaling.

# Citation
```
@misc{mireshghallah2025cimemoriescompositionalbenchmarkcontextual,
title={CIMemories: A Compositional Benchmark for Contextual Integrity of Persistent Memory in LLMs},
author={Niloofar Mireshghallah and Neal Mangaokar and Narine Kokhlikyan and Arman Zharmagambetov and Manzil Zaheer and Saeed Mahloujifar and Kamalika Chaudhuri},
year={2025},
eprint={2511.14937},
archivePrefix={arXiv},
primaryClass={cs.CR},
url={https://arxiv.org/abs/2511.14937},
}
```

# CIMemories：面向大语言模型（Large Language Models，LLMs）持久记忆上下文完整性的组合式评测基准
[Paper](https://arxiv.org/abs/2511.14937)

大语言模型（LLMs）正日益依托过往交互生成的持久记忆，以提升个性化服务水平与任务执行效能。然而，当敏感信息在不当场景下被泄露时，这类持久记忆会带来严峻风险。为此我们推出CIMemories，一款用于评测大语言模型是否能依据任务场景，合理管控来自持久记忆的信息流转的基准评测集。CIMemories采用单用户拥有超100项属性的合成用户画像，并搭配多样化的任务场景——在这些场景中，某一属性可能对部分任务至关重要，但在其他任务中却不宜使用。

我们的评测结果显示，前沿大语言模型存在最高达69%的属性级违规行为（即在不当场景下泄露信息），而违规率较低的模型往往会以牺牲任务实用性为代价。违规行为会在任务与运行轮次中持续累积：当任务使用量从1项增至40项时，GPT-5的违规率从0.1%升至9.6%；若同一提示词被执行5次，其违规率可达25.1%。这暴露出模型行为具有任意性与不稳定性——对完全相同的提示词，模型会泄露不同的属性信息。

注重隐私保护的提示策略无法解决这一问题：模型会出现过度泛化的情况，要么全盘共享信息，要么完全不共享，而非做出细致入微、贴合场景的决策。上述研究结果暴露出大语言模型存在根本性局限，这类局限需要具备场景感知能力的推理能力来解决，而非仅靠优化提示策略或模型缩放。

# Citation

@misc{mireshghallah2025cimemoriescompositionalbenchmarkcontextual,
title={CIMemories: A Compositional Benchmark for Contextual Integrity of Persistent Memory in LLMs},
author={Niloofar Mireshghallah and Neal Mangaokar and Narine Kokhlikyan and Arman Zharmagambetov and Manzil Zaheer and Saeed Mahloujifar and Kamalika Chaudhuri},
year={2025},
eprint={2511.14937},
archivePrefix={arXiv},
primaryClass={cs.CR},
url={https://arxiv.org/abs/2511.14937},
}