Impression-Guided Interactive Personalized Color Exploration Framework for Dynamic Projection Mapping Makeup

Personalized makeup color exploration is crucial to enhance user experience in applications such as cosmetic recommendations. While virtual makeup offers accessible solutions for this exploration, traditional 2D overlay methods viewed through smartphone or tablet screens often lack realism and fail to precisely simulate how colors appear under real-world lighting. In contrast, Dynamic Projection Mapping (DPM) makeup provides greater realism by projecting colors onto the user’s face, enabling direct human observations. This paper focuses on personalized makeup color exploration using DPM. We propose a novel framework to help non-expert users efficiently find satisfying results in the vast space of color combinations. First, we introduce a text-to-makeup-color model that efficiently generates makeup colors based on impression text input. Then, we propose a projection-in-the-loop method to interactively refine makeup colors. Through this method, users can further match their personal preferences and suit their skin tone and lip color by directly observing the final effect on their faces. We validated our system through three evaluations: (1) an online survey (N = 100) and an impression-targeted experiment (N = 5) to verify feasibility and efficiency of the text-to-makeup-color model; (2) a simulated experiment to assess convergence performance in goal-oriented color search; and (3) a user study (N = 15) and an expert interview (N = 3) evaluating usability and user experience. Results show that our framework enables efficient and user-friendly makeup color exploration over baseline methods.