bergson-magic-scores-gpt-2

# bergson MAGIC scores — GPT-2 / wikitext-2 / random validation Per-document MAGIC attribution scores for a GPT-2 model fine-tuned on `Salesforce/wikitext` `wikitext-2-raw-v1` `train`, attributing the training-loss gradient with respect to the model's output on `test[3:4]` (a single test example, taken as the query). Scores were produced by the [`bergson`](https://github.com/EleutherAI/bergson) attribution toolkit's MAGIC subcommand, which back-propagates through the entire training trajectory to compute, for each training doc `d`, `∂(L_query)/∂(w_d)` — the first-order Taylor expansion of how much removing doc `d` from training would change the query loss. ## Files | file | description | |---|---| | `scores.pt` | torch.float32 tensor, shape `(36718,)`. Indexed by original wikitext-2 train row position (filtered <2-token rows have score 0). | | `summary.csv` | Final Spearman/Pearson correlation between MAGIC scores and 100-subset leave-k-out training-loss diffs at validate time. | | `validation.csv` | Per-subset (`subset, diff, score_sum`) records from the leave-k-out validation. | ## Loading ```python import torch scores = torch.load("scores.pt", map_location="cpu", weights_only=True) # scores.shape == (36718,), one entry per original wikitext-2 train row. ``` ## Validation result Random-strategy 100-subset leave-k-out validation, batch_size=256, num_epochs=2, polynomial LR schedule: | metric | value | p | |---|---|---| | Spearman ρ | **+0.9731** | 2.71e-64 | | Pearson r | **+0.9724** | 1.00e-63 | | baseline_loss | 3.6840 | | (Sorted-strategy validation against the same scores reaches ρ = +0.9926 on the same 100 subsets.) ## YAML used to generate these scores ```yaml run_path: runs/gpt2_wikitext model: gpt2 overwrite: true data: dataset: Salesforce/wikitext subset: wikitext-2-raw-v1 split: "train" chunk_length: 512 query: dataset: Salesforce/wikitext subset: wikitext-2-raw-v1 split: "test[3:4]" chunk_length: 0 distributed: nproc_per_node: 4 nnode: 4 batch_size: 256 num_epochs: 2 lr_schedule: lr_scheduler_type: polynomial lr: 0.0008 lr_start: 1e-6 lr_end: 0.00008 warmup_steps: 0.25 subset_strategy: random wandb_project: magic ``` Saved as `examples/magic/gpt2_wikitext.yaml` in the bergson repo. Run with: ```bash bergson magic examples/magic/gpt2_wikitext.yaml ```