latkes/inside-out-replication-v2-external-scores

---
license: mit
tags:
- inside-out-replication-v2
- external-scores
---

# inside-out-replication-v2-external-scores

Per (question, answer) external scores with judge labels for Inside-Out V2. Covers P(a|q), P_norm(a|q), P(True) and two verification-prompt variants. Used to compute external K/K*.

## Dataset Info

- **Rows**: 1752198
- **Columns**: 20

## Columns

| Column | Type | Description |
|--------|------|-------------|
| question_id | Value('string') | Question identifier, e.g. P26_test_0000 |
| answer | Value('string') | Full sampled answer text (never truncated) |
| label | Value('string') | Judge label: CORRECT or INCORRECT |
| log_p_a_q | Value('float64') | Log prob of answer given question |
| p_a_q | Value('float64') | exp(log_p_a_q) |
| log_p_norm_a_q | Value('float64') | Length-normalized log prob |
| p_norm_a_q | Value('float64') | exp(log_p_norm_a_q) |
| p_true | Value('float64') | Restricted softmax P(A=CORRECT) over {A,B} verification tokens |
| p_true_full_a | Value('float64') | Full-vocab prob mass on the 'A' token |
| p_true_full_b | Value('float64') | Full-vocab prob mass on the 'B' token |
| p_true_residual | Value('float64') | 1 - (full A + full B): mass outside the A/B tokens |
| verif_v0_ab_score | Value('float64') | P(True) variant: A/B prompt (same as p_true) |
| verif_v0_ab_residual | Value('float64') | Residual mass for v0 A/B prompt |
| verif_v1_truefalse_score | Value('float64') | Verification variant: True/False prompt |
| verif_v1_truefalse_residual | Value('float64') | Residual mass for True/False prompt |
| verif_v2_yesno_score | Value('float64') | Verification variant: Yes/No prompt |
| verif_v2_yesno_residual | Value('float64') | Residual mass for Yes/No prompt |
| model | Value('string') | Subject model |
| relation | Value('string') | Wikidata relation (P26/P264/P176/P50) |
| split | Value('string') | dev or test |

## Generation Parameters

```json
{
"script_name": "04_external_scores.py",
"model": "Llama-3-8B / Mistral-7B-v0.3 / Gemma-2-9B",
"description": "Per (question, answer) external scores with judge labels for Inside-Out V2. Covers P(a|q), P_norm(a|q), P(True) and two verification-prompt variants. Used to compute external K/K*.",
"input_datasets": [
"inside-out-replication-v2-judge-labels"
],
"experiment_name": "inside-out-replication-v2",
"job_id": "mll:27608-27621",
"cluster": "mll",
"artifact_status": "final",
"canary": false,
"hyperparameters": {}
}
```

## Usage

```python
from datasets import load_dataset

dataset = load_dataset("latkes/inside-out-replication-v2-external-scores", split="train")
print(f"Loaded {len(dataset)} rows")
```

---