TACO-Cobalt-PTB

# TACO-Cobalt-PTB [**Paper**](https://arxiv.org/abs/2602.03806) | [**GitHub**](https://github.com/OSU-NLP-Group/cobalt) TACO-Cobalt is the perturbed version of the validation set in [TACO-Cobalt](https://huggingface.co/datasets/osunlp/TACO-Cobalt) to analyze in-context reward hacking behaviors of LLMs in code generation contexts. For each coding problem, we randomly select two public tests (x_1, y_1) and (x_2, y_2) with distinct outputs (y_1 != y_2). Then, we exchange their expected outputs and turn them into two perturbed tests (x_1, y_2) and (x_2, y_1), which are impossible for any correct program to pass. If all public test cases for a coding problem share the same output, we will discard the task. We keep other unchanged test cases together with the perturbed cases in the public split, which resembles real-world scenarios that one or two tests are noisy, but the majority of cases are still correct. ## Citation If you find this dataset useful, please cite our paper and the original paper: ```bibtex @misc{chen2026bridgingonlineofflinerl, title={Bridging Online and Offline RL: Contextual Bandit Learning for Multi-Turn Code Generation}, author={Ziru Chen and Dongdong Chen and Ruinan Jin and Yingbin Liang and Yujia Xie and Huan Sun}, year={2026}, eprint={2602.03806}, archivePrefix={arXiv}, primaryClass={cs.LG}, url={https://arxiv.org/abs/2602.03806}, } ``` ```bibtex @misc{li2023tacotopicsalgorithmiccode, title={TACO: Topics in Algorithmic COde generation dataset}, author={Rongao Li and Jie Fu and Bo-Wen Zhang and Tao Huang and Zhihong Sun and Chen Lyu and Guang Liu and Zhi Jin and Ge Li}, year={2023}, eprint={2312.14852}, archivePrefix={arXiv}, primaryClass={cs.AI}, url={https://arxiv.org/abs/2312.14852}, } ```