cds-jb/synthweb-qwen3-8b-multiscale-inference

synthweb-qwen3-8b-multiscale-inference 是一个基于Qwen3-8B模型对FineWeb前缀进行延续生成的探测问题数据集，专门设计用于评估方法M（一种激活预言技术），该方法旨在从语言模型的隐藏状态中恢复尚未在表面文本中显现的信息。数据集包含591,541个探测，覆盖了37,042个不同的源文档，每个文档生成最多16个探测，分为五个范围组：单词（word）、长度（lens）、句子（sentence）、段落（paragraph）和整体（whole）。每个探测都设计为满足两个关键约束：HARD-FROM-TEXT（仅从互补文本中难以自信地得出答案）和EASY-FROM-LATENT（答案应是源模型在分割点隐藏状态中已承诺的内容）。数据集通过Claude Haiku 4.5模型生成，包括探测问题、目标响应、错误但合理的答案等字段，并进行了回答者和评判员评分，以评估探测的难度和有效性。数据集的目的是为方法M提供评估基准，确保探测测量的是隐藏状态恢复能力，而非文本阅读能力。synthweb-qwen3-8b-multiscale-inference is a probing question dataset generated by continuing the FineWeb prefix based on the Qwen3-8B model, specifically designed to evaluate Method M, an activation probing technique that aims to recover information that has not yet appeared in the surface text from the hidden states of language models. The dataset contains 591,541 probing instances covering 37,042 distinct source documents, with up to 16 probing instances generated per document, divided into five scope groups: word, length, sentence, paragraph, and whole. Each probing instance is designed to satisfy two key constraints: HARD-FROM-TEXT (the answer can hardly be confidently deduced solely from the complementary text) and EASY-FROM-LATENT (the answer should be the information that the source model has already committed to in its hidden states at the segmentation point). The dataset is generated via the Claude Haiku 4.5 model, including fields such as probing questions, target responses, and plausible but incorrect answers, and has been scored by respondents and judges to evaluate the difficulty and validity of the probing instances. The purpose of this dataset is to provide an evaluation benchmark for Method M, ensuring that the probing measurements target the capability of hidden state recovery rather than text reading ability.