Large Language Models for Agile Effort Estimation: A Post-Mortem Study Incorporating Developer Experience and Optimis

Estimating the effort required for software development is challenging due to the inherent complexity of software. The use of LLMs (Large Language Models) could facilitate this process for development teams.
This study aims to explore the capability of LLMs to estimate the effort required to complete user stories in software development.
The estimates of 10 LLMs and 42 developers were evaluated using the agile T-shirt Sizing technique for 12 user stories. The preservation of the expected logical order, less effort with greater experience and optimism were analysed using the Jonckheere-Terpstra test and Kendall's correlation coefficient. Subsequently, absolute accuracy was compared with the actual effort obtained from post-mortem developments of a small e-commerce business.
Human developers and several LLMs showed similar accuracy in their estimates compared to the actual post-mortem effort. Large models, those with hundreds of billions of parameters, consistently preserved the expected order in estimates, reducing estimated effort by increasing experience and optimism. The experience level showed a greater influence than the degree of optimism. Some smaller models, ranging from 4 to 8 billion parameters, showed inconsistent patterns, so model size was more decisive than reasoning capability for the quality of estimates.
These findings suggest that some large LLMs can be effectively integrated into agile estimation processes when the developer's experience level and degree of optimism are properly incorporated.