zhan8855/joint-recall-synthetic

这是一个分词的多查询联合回忆合成数据集，用于在论文《Overcoming Long-Context Limitations of State-Space Models via Context-Dependent Sparse Attention》中描述的合成实验。联合回忆将关联回忆扩展到依赖上下文的键-值关联。数据集的序列布局包括信息组件和查询组件，其中信息组件包含多个块，每个块以一个上下文标记开始，后跟所有的键-值对；查询组件则对相同的上下文和键进行排列，目标是在（上下文，键）条件下预测相应的值。此外，还提供了一个标记掩码，用于标记应预测下一个标记的位置。

Tokenized multi-query joint recall synthetic dataset, used for the synthetic experiments described in the paper Overcoming Long-Context Limitations of State-Space Models via Context-Dependent Sparse Attention. Joint recall extends associative recall to context-dependent key-value association. The sequence layout includes an information component and a query component, where the information component contains multiple blocks, each starting with a context token followed by all its key-value pairs; the query component permutes the same contexts and keys with the objective of predicting the corresponding values conditioned on the (context, key). Additionally, a per-token mask is provided to mark the positions at which the next token should be predicted.