latkes/inside-out-replication-v2-pamqfix-scores-full

Inside-Out Replication V2数据集是一个包含完整校正后原始分数的集合，用于重新计算P(True)、P(a|q)和P_norm(a|q)（及其敏感度变体）的K和K*指标。该数据集覆盖所有3个模型（llama3-8b、mistral-7b、gemma2-9b）和4种关系（P26配偶、P264唱片公司、P176制造商、P50作者），共12个“单元”，使用EntityQuestions的测试分割。数据集包含约157万行，每行代表唯一的（问题，答案）候选对，其中答案通过贪婪解码和1000个温度-1样本去重得到。列包括模型、关系、问题ID、问题文本、答案、校正后的正确性标签（CORRECT/INCORRECT）、标签方法（exact_match、judge、gold_injected）、出现次数、生成概率的对数值和原始值、长度归一化概率、以及P(True)和敏感度变体的验证分数。数据集还提供了计算K和K*的方法，并指出仅覆盖外部方法，不包含内部探针分数。

The Inside-Out Replication V2 dataset is a collection of full corrected raw scores, used to recompute K and K* for P(True), P(a|q), and P_norm(a|q) (including sensitivity variants). It covers all 3 models (llama3-8b, mistral-7b, gemma2-9b) and 4 relations (P26 spouse, P264 record label, P176 manufacturer, P50 author), totaling 12 "cells", using the test split of EntityQuestions. The dataset contains approximately 1.57 million rows, each representing a unique (question, answer) candidate pair, with answers deduplicated from greedy and 1,000 temperature-1 samples. Columns include model, relation, question_id, question text, answer, corrected correctness label (CORRECT/INCORRECT), labeling method (exact_match, judge, gold_injected), count, log and raw generation probabilities, length-normalized probabilities, and verification scores for P(True) and sensitivity variants. The dataset also provides methods for computing K and K*, and notes that it covers only external methods, excluding internal probe scores.