遇见数据集

Scaffolding Helps the Smallest Gemma 4 Agent, Not the Larger Ones: run records, pre-registration and analysis code (3 capacities x 2 scaffold levels x 5 seeds, 104 tasks)

收藏
Zenodo2026-10-31 更新2026-11-05 收录
官方服务:

资源简介:

Materials behind the Kaggle Paper Track writeup for the Google Gemma 4 Developer Agent Competition. A pre-registered factorial of three local Gemma 4 capacities (E4B, 12B, 26B-A4B; Google QAT q4_0 GGUF served by llama.cpp on two RTX 5070 Ti) and two scaffold levels (a two-sentence instruction with six workspace tools versus a nine-rule instruction with code-graph tools and an analyzer sub-agent), five seeds per cell, on 104 public competition tasks scored by the official verifier after a repair of the shipped local grader. The registered prediction (scaffolding helps 12B and 26B-A4B, not E4B) failed in both directions: the maximal scaffold raised E4B from 10.0% to 15.4% resolved (paired task bootstrap, 95% CI +2.3 to +8.8 points), while for 12B and 26B-A4B the 90% CIs lie inside +/-5 points. Contents: PREREGISTRATION.md and freeze manifests (SHA-256; self-attested local git timestamps), the deviation log D1-D11, one JSONL record per task-run (4,056: 3,120 confirmatory runs plus 936 exploratory mid-level runs) with outcome, failure-ladder level, end reason, tool use and patch hash, the gold/empty grader verdict for all 129 public tasks, the two system instructions, the per-level configs, the grader-repair recipe, the analysis scripts and their outputs, and the writeup with figures. scripts/reproduce.py recomputes the confirmatory table from the run records with the frozen analysis code on CPU and prints PASS/FAIL. No competition data is included (no problem statements, patches, repository snapshots, agent patches, traces or test logs) and no model weights; SHA-256 sums and Hugging Face sources of the three GGUF files are listed. The design, code and analysis were drafted with an AI assistant (Claude); the author reviewed every step.

提供机构:
Zenodo
创建时间:
2026-10-31
二维码
社区交流群
二维码
科研交流群
商业服务