IJOBAS TDD Mobile Language - Datasets and Experiment Results

This repository contains the experimental data and analysis for a research study demonstrating how **Test-Driven Prompting (TDP)** significantly improves AI code generation across multiple programming languages and difficulty levels. All experimental data, statistical analysis, and qualitative comparisons are included. ## What is Test-Driven Prompting? Instead of just asking an AI to write code, Test-Driven Prompting includes example test cases in the prompt. This helps the AI understand exactly what the code should do, similar to how human programmers use test-driven development.  ## Key Findings Our study tested 4 different AI models (GPT-4, Claude, Qwen) on Android (Java) and iOS (Swift) and found: - 🎯 First mobile evaluation: While TDP has mainly been studied for Python, this is the first empirical study of TDP in mobile development (Java, Swift). - 🔬 Scope & scale: 8,704 evaluations across 544 programming tasks (HumanEval, MBPP), comparing two prompting strategies (base, test-driven) and four LLMs (GPT-4o, GPT-4o-mini, Qwen 14B, Qwen 32B). - 📈 Measured effect: Average accuracy increase of +2.22 percentage points (pp) over baseline for TDP (95% CI [1.22–3.23 pp], p < 0.001, d = 0.3974). - 📉 Platform differences: LLMs perform worse on mobile languages (66.85%–88.87%) than on Python (86.90%–91.30%), and exhibit reduced responsiveness to TDP in mobile development. - ⚖️ Practical guidance: We provide recommendations for choosing trade-offs (max accuracy vs. budget vs. self-hosted) and offer platform-specific suggestions for applying TDP. TDP is a reliable, low-overhead prompt engineering method for mobile app development that integrates smoothly with existing test-driven workflows; we recommend researchers and practitioners adopt TDP as a standard part of LLM-assisted mobile development. ## Authors **Muhammad Rizqullah** (<[e-mail]>) and **Emad Albassam** (<[e-mail]>) Computer Science Department, King Abdulaziz University, Jeddah, Saudi Arabia *Corresponding author: Muhammad Rizqullah. Any enquiries about the research should be directed to him.* ## Repository Structure - `datasets/` - Programming problems and test cases from HumanEval, MBPP, and Code Contests - `raw_results/` - Complete experimental results for each AI model and dataset combination - `results/` - Statistical analysis and comparison reports