Understanding and Improving LLM-Based Test Oracle Generation

Replication Package This is the replication package for ASE submission, containing both scripts and data that are requested by the replication. It also provides detailed instructions to replicate the evaluation. This package contains the data, prompts, scripts, and model outputs used for assertion-generation evaluation on Defects4J and GHRB test cases. 1. Directory Layout data/ dataset.jsonl: main testcase-level dataset used by GPT-4o and MiniMax scripts. toga/ data_input.csv data_meta.csv togll/ dataset.jsonl prompts/ old_prompt.txt: original prompt. new_prompt.txt: knowledge-augmented prompt. scripts/ eval_assertion_with_gpt4o_old.py eval_assertion_with_gpt4o_new.py eval_assertion_with_minimax_old_new.py run_toga.py run_togll.py outputs/ gpt-4o/old/: one text file per testcase id. gpt-4o/new/: one text file per testcase id. minimax/old/: one text file per testcase id. minimax/new/: one text file per testcase id. toga/result.csv: TOGA oracle output. togll/result.csv: TOGLL oracle output. togll/result.json: TOGLL inference raw result. result matrix.csv: consolidated status matrix. 2. Environment Python 3.9+ Install dependencies (example): pip install requests pandas torch transformers Note: TOGLL/TOGA scripts may require additional model files/checkpoints that are not bundled in this folder. 3. Running GPT-4o Evaluation Before running, edit API fields in scripts: API_KEY = "YOUR API KEY HERE" API_URL = "https://api.openai.com/v1" Run: python scripts/eval_assertion_with_gpt4o_old.pypython scripts/eval_assertion_with_gpt4o_new.py Default runtime outputs are written under: outputs/gpt-4o/old_runtime/ outputs/gpt-4o/new_runtime/ 4. Running MiniMax Evaluation Edit in scripts/eval_assertion_with_minimax_old_new.py: OPENAI_API_KEY = "YOUR API KEY HERE" OPENAI_BASE_URL = "https://api.minimaxi.com/v1" Run both versions: python scripts/eval_assertion_with_minimax_old_new.py Run a single version: MINIMAX_VERSIONS=old python scripts/eval_assertion_with_minimax_old_new.pyMINIMAX_VERSIONS=new python scripts/eval_assertion_with_minimax_old_new.py Runtime outputs are written under: outputs/minimax_runtime_outputs/old/ outputs/minimax_runtime_outputs/new/ 5. Running TOGA and TOGLL TOGA: python scripts/run_toga.py --toga-root /path/to/toga-repo TOGLL: python scripts/run_togll.py --model-dir /path/to/togll-model Default input/output paths in these scripts are already set to this replication package. 6. Status Semantics Current evaluation scripts use the following statuses: PASS: prediction exactly matches all gold assertions for that testcase. FAIL: prediction does not match gold assertions. DECLINED: model explicitly declines or API or parsing/runtime error. 7. Consolidated Table outputs/result matrix.csv stores testcase-level statuses across systems.