precisionmembench
收藏资源简介：
PrecisionMemBench是一个用于评估大语言模型（LLM）记忆系统检索能力的多维度基准测试数据集。它旨在解决单轮问答质量基准的局限性，专门测量四个关键属性：检索精度（确保在固定的35个信念种子语料库中仅返回正确信念）、噪声隔离（评估多轮会话中无关话题引入的信念是否污染后续检索）、会话轮次延迟（衡量会话负载下检索延迟相对于单轮基线的退化）以及信念可变性（测试会话中期更新的信念能否通过别名富集机制立即生效）。数据集包含89个测试用例，覆盖别名解析、范围消歧、链式替换排除、模糊匹配、跨用户隔离、预算淘汰、排名稳定性及多轮话题漂移下的会话级噪声隔离等复杂场景。核心是一个精心构建的种子语料库，包含35个信念，设计用于测试两个不同领域范围、一个信念替换链和一个二级用户装置。数据以结构化JSON文件形式提供，包括单轮检索评估结果和会话评估结果，详细记录测试用例ID、类别、描述、检索到的信念列表、精度与召回率、通过断言状态、失败信息和检索延迟等指标。该数据集主要用于客观、精细地评测和比较不同LLM记忆系统（如Tenure、SuperMemory、AgentMemory等）在精确信息检索方面的性能，特别关注检索结果在剥离生成模型后能否直接支持下游分类器、规则引擎等结构化处理流程。排行榜数据展示了各系统在主动检索通过率、精度、召回率和延迟等方面的表现差异。
PrecisionMemBench is a multi-dimensional benchmark dataset for evaluating the retrieval capabilities of large language model (LLM) memory systems. It addresses the limitations of single-turn question answering quality benchmarks, and specifically measures four core attributes: "Retrieval Precision" (ensuring that only correct beliefs are returned from the fixed 35-belief seed corpus), "Noise Isolation" (evaluating whether beliefs introduced by off-topic content in multi-turn conversations contaminate subsequent retrievals), "Conversation Turn Latency" (measuring the degradation of retrieval latency under conversational load relative to the single-turn baseline), and "Belief Variability" (testing whether beliefs updated mid-conversation can take effect immediately via alias enrichment mechanisms). The dataset includes 89 test cases covering complex scenarios such as alias resolution, scope disambiguation, chain substitution exclusion, fuzzy matching, cross-user isolation, budget elimination, ranking stability, and session-level noise isolation under multi-turn topic drift. At its core lies a meticulously curated seed corpus containing 35 beliefs, designed to test two distinct domain scopes, one belief substitution chain, and a secondary user setup. The data is provided in structured JSON files, encompassing single-turn retrieval evaluation results and conversational evaluation results, with detailed records of metrics including test case ID, category, description, retrieved belief list, precision and recall, pass assertion status, failure information, and retrieval latency. This dataset is mainly used to objectively and precisely evaluate and compare the performance of various LLM memory systems (such as Tenure, SuperMemory, AgentMemory, etc.) in precise information retrieval, with particular attention to whether retrieval results can directly support structured processing workflows like downstream classifiers and rule engines after stripping away the generative model. Leaderboard data showcases the performance differences of each system in terms of active retrieval pass rate, precision, recall, and latency.
数据集概述：PrecisionMemBench
PrecisionMemBench 是一个专为 LLM 记忆系统设计的多维检索基准，旨在评估单轮问答质量基准无法检测的四个正交属性：检索精度、噪声隔离、会话轮次延迟和信念可变性。
核心评估维度
- 检索精度 (Retrieval precision)：在包含35个信念的固定种子语料库中，正确的信念是否被检索到，且仅检索到该信念。
- 噪声隔离 (Noise isolation)：在包含10轮离题话题的会话中，先前引入的信念是否会污染后续无关话题的检索结果。
- 会话轮次延迟 (Session-turn latency)：检索延迟在会话负载下是否相对于单轮基线有所退化。
- 信念可变性 (Belief mutability)：在会话中期更新的信念是否能立即在同一会话中被检索到。
数据集规模与用例
- 总用例数: 89个
- 覆盖的检索维度: 别名解析、范围消歧、超期链排除、模糊匹配、跨用户隔离、预算驱逐、排序稳定性、多轮话题漂移下的会话级噪声隔离。
基准测试结果 (Leaderboard)
主要检索结果 (截至报告日期)
| 提供商 (Provider) | 活跃通过数 (Active passes) | 总通过数 (Total passes) | 平均精度 (Mean precision) | 平均召回率 (Mean recall) | 检索中位数延迟 (Retrieval p50) | 摄入总耗时 (Ingestion total) |
|---|---|---|---|---|---|---|
tenure |
43/43 | 77/77 | 1.00 | 1.00 | 9.77 ms | 1.00 s |
supermemory |
17/17 | 44/77 | 0.43 | 0.55 | 819.48 ms | 0.00 s |
gbrain |
5/5 | 34/77 | 0.14 | 0.17 | 543.84 ms | 28.60 s |
yourmemory |
0/0 | 21/77 | 0.17 | 0.88 | 313.39 ms | 16.40 s |
vector |
0/0 | 11/77 | 0.09 | 1.00 | 71.87 ms | — |
atomicmemory |
0/0 | 9/77 | 0.15 | 0.95 | 71.01 ms | 658.90 s |
zep |
0/0 | 9/77 | 0.09 | 0.95 | 124.36 ms | 897.00 s |
a-mem |
0/0 | 9/77 | 0.06 | 0.99 | 13.80 ms | 178.80 s |
hindsight |
0/0 | 9/77 | 0.06 | 1.00 | 589.86 ms | 173.30 s |
mem0 |
0/0 | 9/77 | 0.06 | 0.99 | 64.94 ms | 111.30 s |
agentmemory |
0/0 | 7/77 | 0.17 | 0.97 | 82.28 ms | 1.10 s |
通过类型细分
| 提供商 (Provider) | 活跃检索 (Active retrieval) | 结构 (Structural) | 平凡空 (Trivially empty) |
|---|---|---|---|
tenure |
43 | 25 | 9 |
supermemory |
17 | 18 | 9 |
gbrain |
5 | 20 | 9 |
a-mem |
0 | 6 | 3 |
agentmemory |
0 | 5 | 2 |
atomicmemory |
0 | 6 | 3 |
hindsight |
0 | 6 | 3 |
mem0 |
0 | 6 | 3 |
vector |
0 | 8 | 3 |
yourmemory |
0 | 15 | 6 |
zep |
0 | 6 | 3 |
嵌入模型不变性 (向量基线)
| 模型 (Model) | 精度 (Precision) | 召回率 (Recall) | 通过数 (Passes) | 平均延迟 (Mean) | P95延迟 (p95) |
|---|---|---|---|---|---|
| nomic-embed-text (768) | 0.09 | 1.0 | 11/77 | 43.36 ms | 85.21 ms |
| mxbai-embed-large (1024) | 0.09 | 1.0 | 11/77 | 96.48 ms | 257.24 ms |
| qwen3-8b (4096) | 0.09 | 1.0 | 11/77 | 1130.95 ms | 2604.84 ms |
会话评估——多轮话题漂移下的噪声隔离
| 提供商 (Provider) | 通过轮次 (Turns passed) | 通过率 (Pass rate) | 平均漂移 (Mean drift) | 噪声隔离 (Noise isolation) | 平均精度 (Mean precision) | 会话中位数延迟 (Session p50) |
|---|---|---|---|---|---|---|
tenure |
12/12 | 1.00 | 0.0000 | 1.00 | 1.0000 | 47.79 ms |
supermemory |
2/12 | 0.17 | 0.1667 | 0.17 | 0.6000 | 867.83 ms |
yourmemory |
1/12 | 0.08 | 0.7365 | 0.08 | 0.1965 | 430.49 ms |
gbrain |
1/12 | 0.08 | 0.0000 | 0.08 | — | 535.61 ms |
agentmemory |
0/12 | 0.00 | 0.8087 | 0.00 | 0.1913 | 98.49 ms |
atomicmemory |
0/12 | 0.00 | 0.8449 | 0.00 | 0.1551 | 355.08 ms |
zep |
0/12 | 0.00 | 0.8888 | 0.00 | 0.1112 | 418.13 ms |
vector |
0/12 | 0.00 | 0.9142 | 0.00 | 0.0858 | 256.75 ms |
a-mem |
0/12 | 0.00 | 0.9259 | 0.00 | 0.0741 | 25.66 ms |
hindsight |
0/12 | 0.00 | 0.9285 | 0.00 | 0.0715 | 1880.60 ms |
mem0 |
0/12 | 0.00 | 0.9398 | 0.00 | 0.0602 | 377.93 ms |
数据文件字段 (results/文件夹)
检索评估结果 (results/<provider>.json)
| 字段 (Field) | 类型 (Type) | 描述 (Description) |
|---|---|---|
caseId |
str | 唯一的用例标识符 |
category |
str | 用例类别 (如别名解析、范围消歧等) |
description |
str | 人类可读的用例描述 |
pinnedBeliefs |
list | 在固定事实层级中返回的信念ID |
relevantBeliefs |
list | 在相关信念层级中返回的信念ID |
retrievalPrecision |
float 或 null | 活跃断言用例的检索精度；若无断言则为null |
retrievalRecall |
float 或 null | 活跃断言用例的检索召回率；若无断言则为null |
passed |
bool | 所有断言的层级是否同时满足 |
failures |
list | 若未通过，则为失败原因列表 |
retrievalLatencyMs |
float | 端到端检索延迟，单位为毫秒 |
会话评估结果 (results/session-<provider>.json)
| 字段 (Field) | 类型 (Type) | 描述 (Description) |
|---|---|---|
caseId |
str | 会话用例标识符 |
turnIndex |
int | 用例中的轮次编号 (从0开始) |
label |
str | 轮次标签 (如 establishes_topic, drift 等) |
retrievedBeliefIds |
list | 该轮次作为相关信念返回的信念ID |
pinnedBeliefIds |
list | 该轮次作为固定信念返回的信念ID |
noiseBeliefIds |
list | 尽管有mustNotSurface断言但仍出现的信念ID |
driftScore |
float | 1 - retrievalPrecision；0为完美，1为完全噪声 |
passed |
bool | 该轮次所有断言是否满足 |
failures |
list | 若未通过，则为失败原因列表 |
retrievalLatencyMs |
float | 端到端检索延迟，单位为毫秒 |
通过类型 (Pass Taxonomy)
- 活跃检索通过 (Active retrieval pass)：用例包含
retrievalPrecision断言且被满足。这是唯一能证明系统检索能力的通过类型。 - 结构通过 (Structural pass)：用例断言了范围隔离、超期排除或类型路由，但不包含精度断言。
- 平凡空通过 (Trivially empty pass)：用例预期的
relevantBeliefs层为空。任何系统返回空集即可通过。
引用
bibtex @article{flynt2026precisionmembench, title = {Structured Belief State and the First Precision-Aware Benchmark for LLM Memory Retrieval}, author = {Flynt, Jeffrey}, year = {2026} }
资源链接
