CNMB: Code Navigation Measurement Benchmark (data, code and analysis)
收藏资源简介:
Data, code and analysis for the article "What Does 'N× Fewer Tokens' Measure? Benchmarking Token Efficiency in Code Navigation". CNMB measures caller retrieval by whole-word search, language servers and two graph tools on 1,364 targets from 17 Python and TypeScript repositories and generated fixtures, with exact tool responses and caller labels from executable fixtures and runtime traces. Two pre-registered agent studies (3,012 runs) test whether payload differences carry over to task success. The archive holds the response archive, runtime traces, scored rows, audit annotations, agent-study protocols, results and transcripts, and the scripts behind the article's tables and figures. Source repository: https://github.com/rohitvijayakumar45/cnmb. Code is under the MIT License and the authors' data under CC BY 4.0; third-party excerpts keep their own licenses (see THIRD_PARTY_NOTICES.md).



