HuggingFaceBio/genomic-niah
收藏资源简介：
Genomic-NIAH是一个用于评估基因组语言模型长上下文检索能力的基准测试数据集。该数据集通过将随机的(KEY, VALUE) DNA对（每个长度为24个碱基对）插入真实的基因组序列（称为“haystack”）中，要求模型在给定haystack和KEY的情况下检索出对应的VALUE。由于KEY和VALUE与周围序列无关，模型必须通过长距离检索才能成功完成任务，从而将“模型是否使用远端上下文”这一抽象问题转化为具体的生成任务，并可通过每个示例的准确率进行量化评估。数据集包含4个不同难度的任务：普通检索（niah）以及三个近重复变体（niah_neardup_d4、niah_neardup_d2、niah_neardup_d1），其中近重复变体通过引入与目标KEY仅有1、2或4个碱基差异的干扰对来增加检索难度。此外，数据集提供6种上下文长度（4k、8k、16k、32k、64k、128k 6-mer tokens，分别对应约24,000至786,000个DNA碱基对），形成4×6的配置网格。每个配置包含500个示例，按5种插入深度（10%、25%、50%、75%、90%）和4个真核生物界（动物界、真菌界、植物界、原生生物界）进行分层采样。数据集中每个示例提供prompt（模型输入）、value（预期响应）、negative_value（用于似然判别）等字段，并支持两种评估指标：生成式精确匹配（gen_exact_match）和似然正确性（ll_correct）。数据集基于OpenGenome2的真实基因组序列构建，确保haystack为真实序列而非随机填充，以保持其诊断价值。
Genomic-NIAH is a long-context retrieval benchmark for genomic language models. It measures whether a DNA model actually uses its long context by planting a random (KEY, VALUE) DNA pair inside a real-genome haystack and asking the model to recover VALUE given the haystack followed by KEY. Because both KEY and VALUE are uncorrelated with the surrounding sequence, the only way to succeed is to retrieve the planted pair at long range, turning the abstract question does the model use distal context into a concrete generative task with a per-example accuracy. The benchmark provides 4 tasks of varying difficulty (plain retrieval and three near-duplicate variants) and 24 sub-configurations on a 4 × 6 grid of task difficulty × context length (from 4k to 128k 6-mer tokens, approximately 24,000 to 786,000 DNA base pairs). Each sub-configuration contains 500 examples, stratified across five needle depths and four eukaryotic kingdoms. The dataset includes fields such as prompt, value, and negative_value, and supports two evaluation metrics: gen_exact_match (strict generative retrieval accuracy) and ll_correct (likelihood-based discrimination). It is built from real genomic sequences in OpenGenome2, ensuring haystacks are authentic without random padding.
