An Empirical Study on the Automated Generation of Mobile Applications using Large Language Models

An Empirical Study on the Automated Generation of Mobile Applications using Large Language Models

This repository contains the supplementary materials for our paper, titled "An Empirical Study on the Automated Generation of Mobile Applications using Large Language Models". This study evaluates its scalability by performing an extensive empirical evaluation of an LLM-based mobile app generation methodology applied to 42 real-world applications spanning 17 categories. To enable a consistent and fine-grained evaluation across a large dataset, we incorporate an automated feature extraction step that decomposes high-level requirements into atomic features, improving the precision and comparability of feature-level analysis. Our evaluation considers both functional correctness and internal code quality. Functional assessment is conducted through expert-based feature classification, while code quality is evaluated using static analysis metrics. The results show that, although the approach achieves a high compilation success rate (92.86%), the generated applications exhibit low functional completeness, with a large proportion of features either missing or partially implemented. Additionally, the absence of backend generation emerges as a key limitation, leading to incomplete application behavior. These findings highlight that, while LLM-based approaches can scale in terms of generation, they still face significant challenges in delivering complete and fully functional applications. This work contributes a large-scale evaluation that provides a more realistic assessment of current capabilities and limitations, offering insights for future improvements in automated mobile app generation.

📂 Repository Structure

├── 1.app-description/ # Input app descriptions (*.txt)
├── 2.app-generation/ # Generated requirements grouped by app
│ ├── close-2-ai-home-design/
│ ├── close-2-audible/
│ ├── ... # Other generated app folders
│ └── close-2-youtube-music/
├── 3.feature-extraction/ # Feature extraction scripts and outputs
│ ├── script-extract-features.py
│ ├── prompt-feature-extraction.txt
│ ├── requirements.txt
│ └── extracted-features/
├── 4.feature-evaluation/ # Evaluation summaries and charts
│ ├── generate-feature-classification-summary.py
│ ├── evaluated-features/
│ ├── global-feature-classification-summary.txt
│ └── feature-classification-chart.pdf
└── README.md

📝 How to Use This Repository

Prerequisites

Python 3.10+

OpenRouter API key

Set your key:

export OPEN_ROUTER_API_KEY="your_key_here"

Setup

cd 3.feature-extraction
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Workflow

1) Prepare app descriptions

Ensure your input app-description files are available in 1.app-description/ (one app per .txt file).

2) Generate requirements

Run your generation process to populate 2.app-generation/ with requirements for each app.

3) Extract features

From the 3.feature-extraction/ environment:

python script-extract-features.py

This creates/updates extracted outputs in 3.feature-extraction/extracted-features/.

4) Build evaluation summary and chart

From the repository root:

python 4.feature-evaluation/generate-feature-classification-summary.py

This command generates:

4.feature-evaluation/evaluated-features/global-feature-classification-summary.txt

4.feature-evaluation/evaluated-features/feature-classification-chart.pdf

📱 Generated apps

The applications used in the experiment were selected to cover a diverse set of domains, ensuring variability in functionality and complexity. The dataset spans 17 categories, including Books, Business, Design, Education, Entertainment, Finance, Food & Drink, Health & Fitness, Lifestyle, Medical, Music, Navigation, News, Photography, Productivity, Shopping, Social, Sports, Travel, Utilities, and Weather. These categories encompass a wide range of interaction patterns, such as content consumption (e.g., Entertainment and News), user-generated content (e.g., Social and Photography), transactional services (e.g., Finance and Shopping), and domain-specific applications (e.g., Medical and Health & Fitness).

Application
Category
Description

Close2AiHomeDesign
Design
AI-powered interior design tool for generating and visualizing home layouts.

Close2Audible
Books
Audiobook and podcast streaming platform with offline listening support.

Close2Betano
Sports
Sports betting application with real-time odds and wagering features.

Close2BlinkFitness
Health & Fitness
Fitness app for managing gym activities and tracking workouts.

Close2BobbieGoods
Books
Digital coloring book app offering interactive drawing and creative activities.

Close2Booking
Travel
Travel platform for booking hotels, flights, and accommodations.

Close2BloodPressureTracker
Health & Fitness
Health monitoring app for tracking blood pressure readings.

Close2Canva
Design
Graphic design platform for creating visual content and social media assets.

Close2CapCut
Photography
Video editing tool with filters, effects, and content creation features.

Close2ChatGPT
Productivity
AI assistant for text generation, question answering, and task support.

Close2Crumbl
Food & Drink
App for browsing, ordering, and tracking gourmet cookie deliveries.

Close2DramaBox
Entertainment
Streaming service focused on short-form drama content.

Close2Duolingo
Education
Language learning platform with gamified lessons and exercises.

Close2Google
Utilities
General-purpose app providing search and personalized services.

Close2GoogleDocs
Productivity
Cloud-based document editor with collaboration features.

Close2GoogleHome
Utilities
Smart home control app for managing connected devices.

Close2GoogleMaps
Navigation
Navigation app providing maps, directions, and location-based services.

Close2GooglePhotos
Photography
Cloud-based photo storage and management platform.

Close2iFood
Food & Drink
Food delivery platform connecting users with local restaurants.

Close2LiveWeather
Weather
Weather app offering forecasts and environmental data visualization.

Close2MeuSusDigital
Medical
Government healthcare app for accessing public health services.

Close2MyChart
Medical
Patient portal for accessing medical records and communicating with providers.

Close2Netflix
Entertainment
Video streaming platform offering movies, series, and offline viewing.

Close2Nu
Finance
Mobile banking application providing financial services and account management.

Close2ParentSquare
Education
School communication app connecting parents, teachers, and students.

Close2PayPal
Finance
Digital payment platform for sending, receiving, and managing money.

Close2Pinterest
Lifestyle
Content discovery platform for ideas, inspiration, and visual bookmarking.

Close2Reddit
News
Social news platform for community discussions and content sharing.

Close2Shein
Shopping
Online fashion retail app offering clothing and accessories.

Close2Sleeper
Sports
Fantasy sports platform for managing teams and leagues.

Close2Spotify
Music
Audio streaming platform for music, playlists, and podcasts.

Close2Teams
Business
Collaboration platform for messaging, meetings, and file sharing.

Close2TeaDatingAdvice
Lifestyle
Social advice platform focused on dating and relationship guidance.

Close2Temu
Shopping
E-commerce platform for browsing and purchasing discounted products.

Close2Threads
Social
Social media platform for sharing short text-based updates.

Close2TikTok
Social
Video-sharing platform for creating and consuming short-form content.

Close2Uber
Travel
Ride-hailing service for booking transportation and tracking trips.

Close2Waze
Navigation
Community-driven navigation app with real-time traffic updates.

Close2Weather
Weather
Weather forecasting app providing real-time conditions and alerts.

Close2WhatsappBusiness
Business
Messaging application designed for business communication with customers.

Close2X
News
Social media platform for real-time news and microblogging.

Close2YoutubeMusic
Music
Music streaming service with online and offline playback features.