The code and data for the paper entitled 'Facilitating Efficient Discovery: A GUI-Oriented Approach for Exploring Functionality using Machine Learning Model'

Code

dataCollection.py

The collection of app data is primarily accomplished by processing and storing XML files and screenshots of the app. Firstly, the uiautomator is utilized to connect to the smart device and obtain screenshots and XML files. Subsequently, the XML files are analyzed to identify clickable functions within the GUI, and the textual information contained within the functions, along with their specific locations, is extracted. Finally, the Python file also includes handling of interface elements, such as determining if elements are obscured and verifying the legitimacy of the text.

tagData.py

The main implementation involves volunteers annotating app functionalities, including HTML generation, user data analysis, and retrieval. Flask framework is employed, presenting one GUI to the user each time while randomly prompting them to click on three functionalities. Ultimately, the time taken by users to locate these three functionalities is collected.

userPersonalization.py

Separating out the data annotated by each user facilitates personalized analysis. This process involves extraction, storage, and loading of individual user annotations.

dataPreprocessing.py

For a user-annotated functionality, completing the conversion from user time to either "hard-to-find" or "easy-to-find" involves several steps. First, the functionalities are vectorized, extracting relevant parameters from the XML files and computing their correlation with the time users spent searching for the functionalities. These parameters are then normalized to obtain feature vectors for the functionalities. Additionally, an initial determination is made regarding whether the annotated functionalities are "hard-to-find" or "easy-to-find" for each user. Subsequently, clustering is performed on all annotated data from users, and based on the clustering results, the outcomes are filtered and adjusted.

difficultFindClassifier.py

Train the classifier and use it to predict "hard-to-find" functionalities, then display the results.

Data

The data is located in the "static" folder:

- The "persistentData" folder contains the trained classifier.

- The "picture" folder contains screenshots of the app.

- The "requestTime" folder stores data for when volunteers annotate only one function in a GUI.

- The "threeResponseTime" folder saves data for when volunteers annotate three functions in a GUI.

- The "userData" folder stores personalized user data.

- The "xml_information" folder stores XML files of the app.