disi-unibo-nlp/JAB

Java学术基准（Java Academic Benchmark，JAB）是一个专门为严格评估大型语言模型（LLMs）在Java面向对象编程（OOP）能力上的首个基于考试的基准。该数据集包含2014年至2024年间在一个顶级学术机构举行的103个真实Java考试，由经验丰富的教授设计。这些问题旨在测试模型对面向对象概念的深入理解，包括继承、多态、封装和接口。每个考试都包括专家编写的JUnit测试套件，用于客观评估。

The Java Academic Benchmark (JAB) is the first exam-based benchmark specifically designed to rigorously evaluate Large Language Models (LLMs) on Java Object-Oriented Programming (OOP) capabilities. The dataset comprises 103 real Java exams administered between 2014 and 2024 at a top-tier academic institution, curated by an experienced professor. It is designed to test deep understanding of OOP concepts, including inheritance, polymorphism, encapsulation, and interfaces, using expert-authored JUnit test suites for objective assessment.