ceselder/fineweb-loracle-loras-v1

---
license: mit
tags:
- lora
- peft
- mechanistic-interpretability
- loracle
- gradient-oracles
- qwen3-14b
language:
- en
size_categories:
- 1K<n<10K
---

# FineWeb gradient-oracles LoRAs (5000 × randomly-hparamed continued-pretrain LoRAs on Qwen3-14B)

This dataset contains **5000 LoRA adapters trained on individual FineWeb documents** as part of the
**gradient-oracles** experiment in [ceselder/loracles](https://github.com/ceselder/loracles)
(branch `gradient-oracles`).

Each LoRA was trained on **one** FineWeb-edu document with **randomly sampled hyperparameters**
to produce a diverse set of weight deltas for training a "loracle" — a model that reads LoRA
ΔW direction tokens and produces a description of the underlying training data.

## Pipeline

```text
FineWeb-edu document ──────► random (rank, lr, n_steps) ──────► trained LoRA ──────► SVD k=2 direction tokens
│
└──► loracle training task: predict the OpenRouter Sonnet summary
```

## Files

| File | Size | Description |
|---|---|---|
| `loras/<doc_id>.pt` | ~30-100 MB each | Raw LoRA `{module_name: {"A": [r,in], "B": [out,r]}}` dicts in bf16 |
| `direction_tokens_svd_k2/<doc_id>.pt` | 5.5 MB each | SVD top-2 direction tokens, shape `[560, 5120]` bf16 |
| `documents.parquet` | 15 MB | The 5000 source FineWeb-edu docs (text, url, domain, n_tokens) |
| `summaries.parquet` | 1.5 MB | OpenRouter-generated dense summaries (anthropic/claude-sonnet-4.6 or qwen/qwen3-235b-a22b) |
| `training_plan.parquet` | 100 KB | Per-doc (rank, alpha, n_steps, lr, is_holdout) |
| `holdout_id.json` | 1 KB | 10 held-out doc_ids reserved for eval |
| `heldout_organisms.parquet` | 10 KB | Eval-set parquet for the 10 held-out LoRAs |

## Training task (per LoRA)

**10/90 second-half prediction**: the first 10% of the document is teacher-forced as
context (no loss), and the LoRA is trained to predict the remaining 90% via standard
next-token cross-entropy loss. This makes each LoRA encode "having seen a tiny opening,
here's how I extrapolate the rest" — a forced extrapolation from minimal context.

## Random hyperparameters per LoRA (deterministic from seed=42)

| Hyperparameter | Range |
|---|---|
| `rank` | uniform `{4, 5, 6, ..., 16}` (13 values) |
| `alpha` | `4 × rank` (so 16, 20, 24, ..., 64) |
| `n_steps` | uniform `{1, 2, 3, 4}` literal gradient updates |
| `lr` | log-uniform `[1e-4, 1e-3]` |
| `seq_len` | up to 2048 tokens (truncated from longer docs) |
| `batch_size` | 1 doc per LoRA |

For the exact per-doc plan, see `training_plan.parquet`.

## Base model

[Qwen/Qwen3-14B](https://huggingface.co/Qwen/Qwen3-14B), bf16, attn `sdpa`.

## LoRA target modules

All 7 linear modules per transformer block × 40 layers = **280 wrapped modules**:
`q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj`.

## SVD direction tokens (`direction_tokens_svd_k2/`)

For each LoRA, the SVD top-2 singular vectors of `ΔW = B @ A` are extracted per
(layer, module) and stacked as `[2 × 7 × 40, 5120] = [560, 5120]` bf16 residual-stream
direction tokens. Computed via the fast `[r, r]` core decomposition (QR + small SVD)
on GPU.

The convention: read-side modules (q/k/v/gate/up) export `S[i] * Vh[i]` (right singular
vectors in input/residual space); write-side modules (o/down) export `S[i] * U[:, i]`
(left singular vectors in output/residual space).

See `src/lora_training/tokenize_lora_svd.py` in the [loracles repo](https://github.com/ceselder/loracles).

## How to load

```python
import torch
from huggingface_hub import hf_hub_download

repo = "ceselder/fineweb-loracle-loras-v1"

# Load one raw LoRA
adapter_pt = hf_hub_download(repo, "loras/fineweb_00000.pt", repo_type="dataset")
weights = torch.load(adapter_pt, weights_only=True)
# weights = {{"model.layers.0.self_attn.q_proj": {{"A": tensor[r, 5120], "B": tensor[1280, r]}}, ...}}

# Load one set of direction tokens
tokens_pt = hf_hub_download(repo, "direction_tokens_svd_k2/fineweb_00000.pt", repo_type="dataset")
tokens = torch.load(tokens_pt, weights_only=True) # shape [560, 5120] bf16

# Load the source docs + summaries
import pandas as pd
docs_pq = hf_hub_download(repo, "documents.parquet", repo_type="dataset")
docs = pd.read_parquet(docs_pq) # 5000 rows: doc_id, text, url, domain, n_tokens
sums_pq = hf_hub_download(repo, "summaries.parquet", repo_type="dataset")
sums = pd.read_parquet(sums_pq) # ~4866 rows: doc_id, question, answer
```

## Reproduction

```bash
git clone https://github.com/ceselder/loracles.git
cd loracles
git checkout gradient-oracles
# Build the corpus + summaries (or download from this repo)
python scripts/build_fineweb_dataset.py --phase corpus --n 5000
python scripts/build_fineweb_dataset.py --phase summarize --workers 24

# Train all 5000 LoRAs (uses random per-doc hparams from training_plan.parquet)
python -m src.run_cluster --step train-fineweb -T 4 --fineweb-max-seq-len 2048
```

## See also

- Companion summary dataset: [ceselder/fineweb-loracle-summaries-v1](https://huggingface.co/datasets/ceselder/fineweb-loracle-summaries-v1) (just the docs + summaries, no LoRA weights)
- The gradient-oracles experiment branch: https://github.com/ceselder/loracles/tree/gradient-oracles
- The loracle training scripts: `src/loracle_training/train.py`, `src/loracle_training/dataset.py`