Turn Your Face Into An Attack Surface: Screen Attack Using Facial Reflections in Video Conferencing

1. Introduction This project provides a comprehensive pipeline for processing and analyzing subject screen interaction data, divided into two main components: Face Segmentation and Reconstruction: This component processes raw screen recording videos. It first segments the video into meaningful clips and extracts frames containing the subject's face. Subsequently, it applies super-resolution techniques (CAMixerSR) to enhance the quality of the extracted facial images. Screen Content Inference: This component uses a deep learning model to classify the content displayed on the screen. It includes a complete workflow for training a custom two-tier classification model and for testing new data using a suite of provided pre-trained weights. 2. Key Features Two-Tier Classification Model: A novel two-stage approach for screen content analysis. A Tier-1 model classifies content into broad categories (e.g., Web Applications, Office Software), and specialized Tier-2 models perform fine-grained classification within each category. Pre-trained Models: A full set of pre-trained models is provided, allowing for immediate testing and inference on new data without the need for retraining. Complete Pipeline: Includes scripts for every step of the workflow, from raw video processing to final classification result generation. 3. Package Contents This artifact contains the following components: /code/: All Python scripts for both the "Face Segmentation" and "Screen Content Inference" pipelines. Note on Third-Party Code: This artifact includes the CAMixerSR module (referenced as [29] in the paper), which is an open-source project. The original README and license files for CAMixerSR are retained within its corresponding directory to ensure proper attribution. /data/: Sample data to demonstrate the functionality of the scripts. /results/pretrain_models/: The complete set of pre-trained Tier-1 and Tier-2 model weights. requirements.txt: A list of all Python library dependencies, which can be installed using pip. README.md: The full instruction file with detailed, step-by-step guidance on installation and usage. 4. Basic Usage Create a Python virtual environment and install the dependencies using pip install -r requirements.txt. Follow the detailed instructions in the README.md file to run the different parts of the pipeline. The repository includes a root README.md for the main pipeline (face segmentation & training). IMPORTANT: The super-resolution module (CAMixerSR) requires a SEPARATE ENVIRONMENT due to distinct dependencies. Please explicitly refer to the README.md inside the CAMixerSR folder for its specific setup instructions.