codegrip-pack

## Top-level layout ``` codegrip_data_pack_20260426/ ├── README.md (this file) ├── MANIFEST.txt per-file sha256 + size of every leaf file in the pack ├── data/ drop-in replacement for the GitHub repo's data/ dir └── lora_adapters/ three PEFT LoRA adapters used in the paper ``` Total uncompressed: ~26 GB. Compressed (zstd -3, single archive): ~7-9 GB. --- ## `data/` — 25 GB Directory names match the paths the GitHub scripts hardcode (e.g. `--train_data data/rankft/clean_train_combined_v2.jsonl`, `--repo_dir data/unified_repos`). Treat this directory as the project's `data/` and everything just works. ### File-level training (rankft) — 1.6 GB `data/rankft/` — file-level pointwise reranker training pools and BM25 candidate files. Used by `src/train/train_rankft.py` (file-level LoRA training) and `scripts/eval_codeaware_4bit.py` (file-level eval). Key files (~140 in total): | File | Description | |---|---| | `clean_swe_train.jsonl` | SWE-bench train cleaned hard-neg pairs (~3000 issues) | | `clean_grepo_train_v2.jsonl` | GREPO train cleaned hard-neg pairs (~2900 issues) | | `clean_train_combined_v2.jsonl` | **★ Combined SWE+GREPO training pool (5916 pairs) — input for Run 2 / 14B Codeaware training** | | `swebench_test_bm25_top500.jsonl` | SWE-bench Lite test BM25 top-500 candidates | | `grepo_test_bm25_top500.jsonl` | GREPO test BM25 top-500 candidates | | `grepo_train_bm25_top500.jsonl` | GREPO train BM25 (used in pool construction) | | `path_alias_*.json` | per-repo path-aliasing maps (~80 repos) | | `*_e5large_top500.jsonl` | E5-large dense retrieval candidates (alternative to BM25) | ### GREPO source train data — 101 MB `data/grepo_text/` — original GREPO per-repo train issues, one JSONL per repo (75 repos). Used by `scripts/build_clean_train_combined.py` to derive `rankft/clean_grepo_train_v2.jsonl`. Source: PKU MuLab's GREPO release on ModelScope. ### SWE-bench train BM25 — 68 MB `data/swebench_train/` — BM25 top-500 candidates for the SWE-bench train split (used to derive `clean_swe_train.jsonl`). Generated by `scripts/build_swebench_bm25.py --split train`. ### SWE-bench Lite test — 2.0 GB `data/swebench_lite/` — Lite test JSONL + PathSwap variants + 12 SWE-bench Lite source repos (in `swebench_lite/repos/`). | File | Description | |---|---| | `swebench_lite_test.jsonl` | n=300 Lite test issues, normal paths | | `swebench_lite_test_pathswap.jsonl` | n=300 Lite test, paths SHA-256-hashed (Δ_rel measurement) | | `swebench_bm25_pathswap.jsonl` | BM25 candidates under PathSwap | | `swebench_perturb_*` | Predecessor perturbation studies (flatten_dirs, shuffle_dirs, shuffle_filenames) — paper uses SHA-256 PathSwap as final | | `pathswap_alias_map.json` | Lite alias map: original_path → hashed_path | | `repos/` | 12 SWE-bench Lite source repos (django, astropy, sympy, sphinx, sklearn, requests, pytest, flask, matplotlib, pylint, seaborn, xarray) | ### SWE-bench Verified test — 53 MB `data/swebench_verified/` — Verified test (n=500) with normal paths, PathSwap, and shuffle baseline. | File | Description | |---|---| | `swebench_verified_test.jsonl` | n=500 Verified normal | | `swebench_verified_test_pathswap.jsonl` | n=500 Verified PathSwap | | `swebench_verified_bm25_top500.jsonl` | Verified BM25 top-500 (normal) | | `swebench_verified_bm25_pathswap.jsonl` | Verified BM25 top-500 (PathSwap) | | `verified_shuffle_test.jsonl` / `verified_shuffle_bm25.jsonl` | Random-shuffle baseline (sanity check) | | `pathswap_alias_map_verified.json` | Verified alias map | ### SWE-bench Verified strict (per-commit) — 20 MB `data/swebench_verified_strict/` — `swebench_verified_bm25_strict.jsonl`: per-commit BM25 reindexed at the buggy commit's HEAD. Achieves **Acc@100 = 85%** (vs ~67% for the cross-commit version). This is what `paper Table 8` uses for the partial R@1 numbers. Generated by `build_swebench_verified_bm25_strict.py`. ### GREPO PathSwap — 86 MB `data/pathswap/` — GREPO test set under SHA-256 PathSwap. | File | Description | |---|---| | `grepo_test_pathswap.jsonl` | GREPO test with paths hashed | | `grepo_test_bm25_top500_pathswap.jsonl` | GREPO BM25 candidates with paths hashed | | `merged_*` | Hybrid retriever variants (BM25 + e5-large + graph) | | `rename_maps.json` | Per-repo rename maps | ### Function-level training + corpus — 11 GB `data/codegrip_func/` — function-level reranker training pairs + per-commit corpus + GT. | File / dir | Description | |---|---| | `func_gt_swe.jsonl` | function-level GT for SWE-bench train (3.7 MB) | | `func_gt_grepo.jsonl` | function-level GT for GREPO train (5.4 MB) | | `train_pairs.jsonl` | hard-neg-mined train pairs (48 MB) — input for `train_rankft_function.py` | | `train_pairs.dropped.jsonl` | log of pairs dropped during mining (1.3 MB) | | `corpus/` | function-level per-commit corpus (10 GB) — input for function-level eval and hard-neg mining | | `corpus_stats/` | per-repo / per-commit function counts (6.4 MB) | ### Source repos — 5.5 GB `data/repos/` — 75 GREPO source repo clones (with `.git` history). Naming: single-name (`astropy`, `aiogram`, `django`). `data/swebench_lite/repos/` — 12 SWE-bench Lite source repo clones (with `.git` history). Naming: `<owner>__<repo>` (`astropy__astropy`, `django__django`). `data/unified_repos/` — relative symlinks unifying both into one namespace, matching the layout the training scripts expect: ``` unified_repos/aiogram -> ../repos/aiogram unified_repos/astropy__astropy -> ../swebench_lite/repos/astropy__astropy ... (87 entries total = 75 GREPO + 12 SWE) ``` ### BEIR-format eval pools — 5.3 GB `data/beir/` — alternative file/function pools in BEIR / SweRank format (per-issue subdirectory with `corpus.jsonl`, `queries.jsonl`, `qrels/`). Used by `scripts/eval_codeaware_4bit.py`, `eval_codeaware_4bit_cleanfull.py`, `score_codeaware_function.py`, and the SweRank native pipeline. | Subdirectory | Size | Description | |---|---|---| | `SweRank_perturbed/` | 2.4 G | **★ File-level SHA-256 PathSwap full corpus** (paper Table 4 / 8) | | `SweRank_normal_cleanfull_v3/` | 2.2 G | **★ Body-cleanfull control corpus** (paper §codeaware:invariance, refines Lite Δ_rel −11.0% → −5.9%) | | `SweRank_perturbed_funcswap/` | 53 M | Function-swap baseline (function `_id` hashed, body untouched) | | `SweRank_perturbed_funcswap_cleanlite/` | 53 M | Step 1: light import/docstring/module-path cleaning | | `SweRank_perturbed_funcswap_cleanfull/` | 53 M | Step 2 (v1, deprecated) | | `SweRank_perturbed_funcswap_cleanfull_v2/` | 53 M | Step 3 (v2, deprecated) | | `SweRank_perturbed_funcswap_cleanfull_v3/` | 53 M | **★ Step 4 (final, paper Appendix `app:function_cleaning_progression`)** | | `swerank_datasets/` | 547 M | SweRank's original eval pack (we tried to reproduce native 78.10% — got 20.8%, 57 pp gap, see paper §codeaware:swerank_repro) | The 5-step function-level cleaning progression: normal 19.34 → 17.88 → 16.79 → 16.60 → 15.69 R@1 (n=274, McNemar p > 0.25 throughout — directional but inconclusive). --- ## `lora_adapters/` — 1.4 GB Three PEFT-format LoRA adapters. All trained with QLoRA 4-bit nf4, bf16 compute, seed=42. Loading example: ```python from peft import PeftModel from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype="bfloat16") base = AutoModelForCausalLM.from_pretrained( "Qwen/Qwen2.5-14B-Instruct", quantization_config=bnb, device_map="auto") model = PeftModel.from_pretrained(base, "lora_adapters/14B_codeaware_aug05_best") tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-14B-Instruct") ``` | Directory | Size | Base | Recipe | Headline | |---|---|---|---|---| | `14B_codeaware_aug05_best/` | 541 M | Qwen2.5-14B-Instruct | code-in-prompt (50 lines) + path-augment fraction = 0.5 | **★ Lite 60.33 / Verified 48.93 R@1; PathSwap Δ_rel −11.0% / −4.6%** | | `14B_no_aug_best/` | 541 M | Qwen2.5-14B-Instruct | code-in-prompt (50 lines) + path-augment = 0 | Lite 56.33 / Verified 48.62; PathSwap collapses −30.2% / −29.7% | | `7B_function_codeaware_best/` | 324 M | Qwen2.5-7B-Instruct | function-level pointwise reranker, code-in-prompt + aug=0.5 | Function-level paper appendix | --- ## End-to-end recipes (All paths assume you've placed this `data/` dir at the GitHub repo's `data/`.) ### Train file-level codeaware reranker (best: 14B, aug=0.5) ```bash python src/train/train_rankft.py \ --model_path Qwen/Qwen2.5-14B-Instruct \ --train_data data/rankft/clean_train_combined_v2.jsonl \ --repo_dir data/unified_repos \ --output_dir experiments/scale14b_aug05 \ --include_code --code_max_lines 50 \ --path_augment_fraction 0.5 \ --num_negatives 8 --learning_rate 5e-5 --num_epochs 2 \ --batch_size 1 --gradient_accumulation_steps 16 \ --max_seq_length 768 --lora_rank 32 --seed 42 ``` ### Eval file-level on SWE-bench Lite (normal + PathSwap) ```bash bash scripts/eval_2lora_pathswap_pair.sh lora_adapters/14B_codeaware_aug05_best 14B_aug05 0 ``` (Reads `data/beir/SweRank_perturbed/` and the normal Lite pool.) ### Eval file-level body-cleanfull control (Lite) ```bash python scripts/eval_codeaware_4bit_cleanfull.py \ --lora lora_adapters/14B_codeaware_aug05_best \ --beir_root data/beir/SweRank_normal_cleanfull_v3 ``` ### Train function-level reranker ```bash python src/train/train_rankft_function.py \ --train_pairs data/codegrip_func/train_pairs.jsonl \ --corpus_dir data/codegrip_func/corpus \ --output_dir experiments/func_codeaware \ --path_augment_fraction 0.5 --seed 42 ``` ### Eval function-level cleaning progression (paper appendix) ```bash for variant in funcswap funcswap_cleanlite funcswap_cleanfull_v3; do python scripts/score_codeaware_function.py \ --lora lora_adapters/7B_function_codeaware_best \ --beir_root data/beir/SweRank_perturbed_${variant} done ``` --- ## `MANIFEST.txt` Every leaf file in this pack with `sha256 size_bytes ./relative_path`. Use to verify the unpacked tree against any future re-packing. Build: ```bash find . -type f ! -name MANIFEST.txt -print0 | sort -z | \ xargs -0 -P 32 -I {} sh -c 'h=$(sha256sum "$1"|cut -d" " -f1); s=$(stat -c%s "$1"); echo "$h $s $1"' _ {} > MANIFEST.txt ``` --- ## Format notes - **`tar.zst`**: tar archive compressed with **zstd** (Facebook). Faster than gzip, similar ratio to xz. Decompress: `tar -I zstd -xf codegrip_data_pack_20260426.tar.zst`. Requires the `zstd` command (apt: `apt install zstd`; brew: `brew install zstd`). - **JSONL**: one JSON object per line, UTF-8. - **BEIR format**: per-issue directory with `corpus.jsonl` (`{"_id", "text", ...}`), `queries.jsonl` (`{"_id", "text"}`), `qrels/test.tsv` (`query_id<TAB>doc_id<TAB>relevance`). - **PEFT LoRA**: `adapter_config.json` + `adapter_model.safetensors`, loadable via `PeftModel.from_pretrained()`. - **`.git` directories** are included in `data/repos/` and `data/swebench_lite/repos/` for full reproducibility (training scripts read at the buggy commit's HEAD). - **Symlinks in `data/unified_repos/`** are relative — they resolve correctly from the unpacked location, no fixup needed. Date: 2026-04-26. Maintainer: chenlibin (PKU).