Artifact of the paper "Developer Challenges on Large Language Models: A Study of Stack Overflow and OpenAI Developer Forum Posts"

Large Language Models (LLMs) have gained widespread popularity due to their exceptional capabilities across various domains, including chatbots, healthcare, education, content generation, and automated support systems. Despite their transformative potential, developers encounter numerous challenges when implementing, fine-tuning, and integrating these models into real-world applications. This study investigates the challenges LLM developers face through an analysis of community interactions on Stack Overflow and the OpenAI Developer Forum. Using BERTopic modeling, we identify and categorize topics discussed by LLM developers. We also examine topics' popularity and difficulty. Our analysis yields nine evident challenges on Stack Overflow (e.g., LLM Ecosystem and Challenges, API Usage, LLM Training with Frameworks) and 17 on the OpenAI Developer Forum (e.g., API Usage and Error Handling, Fine-Tuning and Dataset Management, Prompt Engineering). Results indicate that developers frequently turn to Stack Overflow for implementation guidance, while OpenAI's forum is primarily used for troubleshooting. Additionally, on the OpenAI Developer Forum, API and functionality-related issues generated the most discussions, with many posts requiring multiple responses, highlighting the intricate nature of LLM challenges. We find that LLM-related queries often exhibit great difficulty, with a substantial percentage of unresolved posts (e.g., 79.03\% on Stack Overflow) and prolonged response times, particularly for complex topics like 'Llama Indexing and GPU Utilization' and 'Agents and Tool Interactions'. On the contrary, established fields such as Mobile development and Security enjoy faster resolution rates and greater community support. These findings emphasize the need for enhanced community support and tailored resources to assist LLM developers in addressing the complex, evolving challenges of this growing field. This study provides insights into areas where LLM developers encounter the most difficulty, guiding future research toward developing tools and techniques to better support the expanding community of LLM practitioners.