MockEval: A Benchmark for Evaluating LLMs on Mock-Aware Unit Testing

Introduction This is the replication package for the paper "MockEval: A Benchmark for Evaluating LLMs on Mock-AwareUnit Testing". Task Definition Evaluate the ability of large language models in identifying, classifying, and generating mock code, given the focal method and context. Setup ```bashpip install -r requirement.txtcd construction``` The static of dataset MockEval benchmark:Database/ORM 32File/IO 93Internal/Logic 577ML/DATA 70Network/HTTP 201System/Process 74Mock Cases 900Mock-Free Cases 804 Pipline Benchmark Construction 1. Repository Selectionnode1 serverOUTPUT:candidates.json```bashpython 1_search_repos.pypython 2_clone_and_scan.py``` 2. Mock Case ExtractINPUT:candidates.jsonOUTPUT:valid_mock_projects.json```bashpython 3_fast_retry.pypython 4_clean_small.pypython 5_extract_pairs.py``` 3. Focal Method MatchingINPUT:benchmark_data_pairs.jsonOUTPUT:benchmark_dataset_final.json```bashpython 6_locate_focal_method.py``` 4. Context ExtractINPUT:benchmark_dataset_final.jsonOUTPUT:mock_benchmark_context_enriched.json,merged_benchmark.json```bashpython 8_enrich_context_v2.pypython 9_compare_context.py``` 5. Mock-Free Case SelectionINPUT:mock_benchmark_high_quality.jsonOUTPUT:negative_samples_benchmark.json```bashpython collect_negative_samples.pypython generate_negative_samples.py``` 6. Human AnnotationINPUT:mock_benchmark_high_quality.json,negative_samples_benchmark.jsonOUTPUT:cleaned_benchmark.json```bashpython collect_final_dataset_v3.pypython merge.py``` Experiment 1. Mock Necessity & TypeINPUT: cleaned_benchmark.jsonOUTPUT: results_coder_v2_lite_classification.json,results_r1_70b_classification.json,results_r1_32b_classification.json,results_r1_14b_classification.json,gemini3_pro_classification.json,gpt5_2_results_classification.json,results_14b_classification.json,results_32b_classification.json,results_30b_a3b_classification.json,claude_opus_4_5_multistage_progress_classification.json```bashpython run_14b_classification.py python run_32b_classification.py python run_30b_a3b_classification.py python run_claude_opus_4_5.py python run_deepseek_r1_14b.py python run_deepseek_r1_32b.pypython run_deepseek_r1_70b.pypython run_gemini3_pro.py python run_gpt5_2.pypython run_coder_v2_lite.py``` 2. Mock GenerationINPUT: cleaned_benchmark.jsonOUTPUT: results_coder_v2_lite_gen_stages.json,results_r1_70b_gen_stages.json,results_r1_32b_gen_stages.json,results_r1_14b_gen_stages.json,results_gemini_mock_gen.json,gpt5_2_mock_gen_results.json,results_qwen14b_gen_stages.json,results_qwen32b_gen_stages.json,results_30b_gen_stages.json```bashpython run_claude_mock_gen.py python run_qwen32b_gen_stages.py python run_qwen3_30b_gen_stages.py python run_qwen14b_generation.py python run_deepseek_r1_14b_gen_stages.py python run_deepseek_r1_32b_gen_stages.pypython run_deepseek_r1_70b_gen_stages.pypython run_gemini3_mock_gen.py python run_gpt52_mock_gen_v2.pypython run_coder_v2_lite_gen.py```