OliverCMU/PersonaMem

PersonaMem是一个用于评估语言模型（LLMs）个性化能力的新基准。它强调用户与聊天机器人之间基于角色的多会话互动，通过一个模拟现实和不断演变的对话上下文的合成对话生成管道来实现。该基准包含具有静态（如人口统计信息）和动态属性（如不断变化的偏好）的用户角色。用户在多个会话中与聊天机器人互动，涵盖各种主题，如食物推荐、旅行计划和治疗咨询。随着用户偏好的演变，该基准提供了注释问题，以评估模型是否能够跟踪并将这些变化纳入其响应中。数据集提供了三个版本，基于上下文标记长度（32k、128k和1M标记），并包含详细的注释，用于评估LLMs在记忆用户档案、跟踪偏好演变和生成个性化响应方面的能力。

PersonaMem is a new LLM personalization benchmark to assess how well language models can infer evolving user profiles and generate personalized responses across task scenarios. It emphasizes persona-oriented, multi-session interactions between users and chatbots, facilitated by a synthetic dialog generation pipeline that simulates realistic and evolving conversational contexts. The benchmark includes user personas with static (e.g., demographic info.) and dynamic attributes (e.g., evolving preferences). Users engage with a chatbot in multi-session interactions across various topics such as food recommendation, travel planning, and therapy consultation. As user preferences evolve over time, the benchmark offers annotated questions assessing whether models can track and incorporate the changes into their responses. The dataset is available in three versions based on context token length (32k, 128k, and 1M tokens) and includes detailed annotations for evaluating LLMs abilities in memorizing user profiles, tracking preference evolution, and generating personalized responses.