McGill-NLP/CHASE-Code
收藏资源简介：
--- license: mit task_categories: - text2text-generation size_categories: - n<1K tags: - code --- <h1 align="center"> CHASE: Challenging AI with Synthetic Evaluations </h1> <p align="center"> <a href="https://arxiv.org/pdf/2502.14678"><img src="http://img.shields.io/badge/Paper-PDF-red.svg"></a> </p> <p style="text-align: justify;"> The pace of evolution of Large Language Models (LLMs) necessitates new approaches for rigorous and comprehensive evaluation. Traditional human annotation is increasingly impracticable due to the complexities and costs involved in generating high-quality, challenging problems. In this work, we introduce **CHASE**, a unified framework to synthetically generate challenging problems using LLMs without human involvement. For a given task, our approach builds a hard problem in a bottom-up manner from simpler components. Moreover, our framework decomposes the generation process into independently verifiable sub-tasks, thereby ensuring a high level of quality and correctness. We implement CHASE to create evaluation benchmarks across three diverse domains: (1) document-based question answering, (2) repository-level code completion, and (3) math reasoning. The performance of state-of-the-art LLMs on these synthetic benchmarks lies in the range of 40-60% accuracy, thereby demonstrating the effectiveness of our framework at generating challenging problems. </p> <h2 align="center"> <img align="center" src="./chase_fig.svg" alt="..." width="700"> </h2> Code: https://github.com/McGill-NLP/CHASE # Setup Install VirtualEnv using the following (optional): ```shell $ [sudo] pip install virtualenv ``` Create and activate your virtual environment (optional): ```shell $ virtualenv -p python3 chasenv $ source chasenv/bin/activate ``` Depending on your machine, you may have to do: ```shell $ python3 -m venv chasenv $ source chasenv/bin/activate ``` ## Dependencies - compatible with python 3 - dependencies can be installed using `CHASE/requirements.txt` - Works best with CUDA 12.5 (otherwise you may have to struggle with installation of individual libraries) Install all the required packages: at `CHASE/:` ```shell $ pip install -r requirements.txt ``` # Usage for CHASE-Code ## A. Generating CHASE-Code Note: Executing the generation or solving files may create some noise files in the directory such as "sandbox*" directories or random data files. These can be deleted after the execution is over. ### Generate helper functions We shall use GPT-4o-mini to generate helper functions (each iteration attempts to generate 5) for the 'data pre-processing' domain. Note that many iterations will fail because of parsing errors or if the generated helper function fails to execute. At `code/`: ```shell $ python generator.py -exp_type helper_functions -prompt_type helper_functions -model_type chat -model gpt-4o-mini -domain data_preprocessing -num_iters 5 ``` This will create a new json file within `CHASE/code/helper_functions/generated` with the name `[domain]_new.json` which will store the model generated helper functions. If there already exist helper functions of this domain previously generated/used in a `[domain].json` file, then just manually transfer the newly generated data points from the new file to the old one. If not, just remove `_new` from the name. The next step will expect all helper functions in `[domain].json` file. ### Generate problem statement and answer code We shall use GPT-4o-mini to generate pairs of (problem statement, answer code) (each iteration attempts to generate 1 pair) for the 'data pre-processing' domain. Our default prompt urges the model to use at least 4 helper functions (out of 10 randomly sampled) in the answer code. Note that many iterations will fail because of parsing errors or if the generated answer code fails to execute. At `code/`: ```shell $ python generator.py -exp_type problem_statement -prompt_type problem_statement -model_type chat -model gpt-4o-mini -domain data_preprocessing -num_iters 30 -run_name gpt-4o-mini-problems ``` This will create a new directory within `CHASE/code/generation_outputs` with the name `gpt-4o-mini-problems` which will store the model generated problems in a `problems.tsv` file. ### Generate test code for the problems We shall use GPT-4o-mini to generate test codes for the problems we generated. The number of iterations reflects the number of attempts (inference calls) to generate a test code for each problem. At `code/`: ```shell $ python generator.py -exp_type test -prompt_type test -model_type chat -model gpt-4o-mini -domain data_preprocessing -num_iters 10 -problems_name gpt-4o-mini-problems -run_name gpt-4o-mini-tests ``` This will create a new directory within `CHASE/code/generation_outputs` with the name `gpt-4o-mini-tests` which will store the model generated problems and corresponding tests in a `tested_problems.tsv` file. ### Verify correspondence of problem specification with answer code We shall use GPT-4o to check if the problem statement correctly specifies the answer code, when provided with the correct set of relevant helper functions. At `code/`: ```shell $ python generator.py -exp_type verify_problems -prompt_type verify_problems -model_type chat -model gpt-4o -domain data_preprocessing -num_iters 2 -tested_name gpt-4o-mini-tests -run_name gpt-4o-mini-verified ``` This will create a new directory within `CHASE/code/generation_outputs` with the name `gpt-4o-mini-verified` which will store the verified problems and corresponding tests in a `verified_problems.tsv` file. ### Get final data with large repository contexts For each problem, we shall create a large repository context across multiple files by sampling 10 random helper functions for each relevant helper function. At `code/`: ```shell $ python generator.py -exp_type final_data -domain data_preprocessing -verified_name gpt-4o-mini-verified -run_name gpt-4o-mini-final -extra_fns 10 ``` This will create a new directory within `CHASE/code/generation_outputs` with the name `gpt-4o-mini-final` which will store the final data file: `final_data.tsv`. ## B. Solving CHASE-Code We will show an example of using the Llama-3.3-70B model to solve the CHASE-Code examples. Host the Llama model locally using [vllm](https://github.com/vllm-project/vllm) (Assuming 2 A100s): ```shell $ CUDA_VISIBLE_DEVICES=0,1 python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-3.3-70B-Instruct --download-dir YOUR_DOWNLOAD_DIR --tensor-parallel-size 2 --max_model_len 16000 --gpu_memory_utilization 0.95 ``` Then query the hosted model. At `code/`: ```shell $ python solver.py -data chase_code_dp.tsv -prompt_type basic -model_type vllm -model meta-llama/Llama-3.3-70B-Instruct -run_name chase_code_llama ``` The output logs and predictions will be stored in the `outputs/chase_code_llama` folder. ## C. Viewing CHASE-Code If you want to manually review say the "5th" example from CHASE-Code, At `code/`: ```shell $ python convert_files.py -data_dir data/ -data chase_code_dp -path readable_data/ -example 5 ``` This will create a new directory `readable_data` with a txt file where you will be more easily able to parse the example. You can also set `-example all` to convert the entire dataset into readable txt files. # Citation If you use our data or code, please cite our work: ``` @misc{patel2025llmgeneratechallengingproblems, title={How to Get Your LLM to Generate Challenging Problems for Evaluation}, author={Arkil Patel and Siva Reddy and Dzmitry Bahdanau}, year={2025}, eprint={2502.14678}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2502.14678}, } ``` For any clarification, comments, or suggestions please contact [Arkil](http://arkilpatel.github.io/).
CHASE (Challenging AI with Synthetic Evaluations) is a unified framework for synthetically generating challenging problems for Large Language Models (LLMs) without human involvement. The framework decomposes the generation process into independently verifiable sub-tasks, ensuring high quality and correctness of the generated problems. CHASE is applied across three diverse domains: document-based question answering, repository-level code completion, and math reasoning.
