CodeJudgeEval-X: Diagnosing LLM Misjudgments in Automated Code Evaluation

This repository contains the scripts used to build and analyze a misjudgement-focused version of the CodeJudge-Eval benchmark. Our workflow augments CodeJudge-Eval dataset with static code quality metrics and problem level metrics. Additionally, the repository contains the code and the data used for the analysis done in the study to answer the following research questions: RQ1: What association exists between code-level and problem-level features and the likelihood of LLM misjudgment? RQ2: To what extent can LLM misjudgments be predicted from code-level and problem-level features? RQ3: Which code quality and structural features have the strongest influence on LLM misjudgments, according to SHAP-based analysis? RQ4: How does structured counterexample-guided reasoning reshape LLM judgment outcomes, and what failure mechanisms does it reveal?