microsoft/SCBench

SCBench（SharedContextBench）是一个综合基准，用于评估高效长上下文方法，特别关注KV缓存的生命周期（生成、压缩、检索和加载）。该数据集包含12个多样化任务，测试四种关键的长上下文能力：字符串检索、语义检索、全局信息处理和多任务处理。数据集支持两种共享上下文模式：多轮模式和多请求模式。SCBench是第一个涵盖单轮、多轮和多请求场景的长上下文基准，并涉及KV缓存重用技术，从而提供了对高效长上下文方法在完整KV缓存生命周期中的更全面分析。

SCBench (SharedContextBench) is a comprehensive benchmark to evaluate efficient long-context methods from a KV cache-centric perspective, analyzing their performance in real-world scenarios where context memory (KV cache) is shared and reused across multiple requests. The benchmark covers 12 diverse tasks that test four key long-context capabilities: string retrieval, semantic retrieval, global information processing, and multi-tasking. Each task is designed to evaluate specific aspects of long-context processing, such as key-value lookup, semantic understanding, and multi-tasking. The dataset is structured with multiple configurations, each focusing on different aspects of long-context processing. The README also discusses the two shared context modes: multi-turn mode and multi-request mode, and compares SCBench to previous long-context benchmarks. The findings from the benchmark reveal insights into the performance of various methods under different conditions, such as memory usage, compression rates, and generation scenarios.