Supplemental Materials for: Evaluating LLMs for Chinese Software Requirements Classification with Predefined Categories

This repository contains the complete replication package for the empirical study "Evaluating LLMs for Chinese Software Requirements Classification with Predefined Categories." It provides all the necessary data, source code, and experimental results to reproduce the figures, tables, and analyses presented in the associated paper. Content Overview:The package is organized into four main directories: data/: Contains the benchmark dataset of 3,566 industrial Chinese requirement statements, annotated with five mutually exclusive template categories (Composite, Interactive, Sequential, Structural, Conditional). The data was collected from financial technology, e-commerce, and education projects between September 2024 and January 2025. classify/: Contains the source code for implementing all evaluated classification approaches, including single-pass LLM inference, confidence-based routing, hierarchical reasoning, annotation-guided verification, and the fine-tuned BERT baseline. classifyOutcome/: Stores the raw output files generated by the classification scripts, including model predictions and confidence scores from all evaluated models and experimental conditions. analyze/: Contains Jupyter notebooks and analysis scripts used to calculate key performance metrics (Accuracy, Expected Calibration Error, Pearson Correlation) from the raw classification outcomes. analyzeOutcome/: Stores the final analysis results, including all figures (e.g., grouped bar charts, confusion matrices) and summary tables (e.g., Excel sheets) that appear in the manuscript. Technical Requirements:The code was developed and tested in PyCharm using Python 3.9. All required dependencies (e.g., pandas, numpy, matplotlib, seaborn, scikit-learn, scipy, transformers, torch) are listed in the requirements.txt file included in the root directory. Detailed instructions for setting up the environment and running the pipeline are provided in the accompanying README.md file. Potential Use:This dataset and code can be reused to benchmark other models on Chinese software requirement classification, to investigate confidence calibration issues in large language models for software engineering tasks, or to further analyze the semantic characteristics of Chinese requirement categories.