Eye Gaze-Based Audio Steering for Improved Speech Intelligibility in Multi-Talker Scenarios for Hearing Aid Users

This dataset was created as part of my research on improving the performance of hearing aids in noisy environments using eye gaze direction. In real-life situations, hearing aid users often struggle when multiple people are speaking at once. The idea here was to explore whether we can detect where the user is looking and use that information to identify the speaker they want to focus on. To do this, we recorded several video scenarios involving 2 to 4 speakers in the same room, speaking at the same time, with noise levels ranging between 70 and 80 dB. In each scenario, the listener (acting as a hearing aid user) tries to focus on one speaker. We captured different situations—sometimes the speaker moves, sometimes the listener moves, and in one case, the listener is looking at the speaker without turning their head. Another scene includes a moment of direct eye contact between the listener and the speaker. The idea behind this dataset is to provide researchers with realistic data for developing audio steering systems that can detect which speaker the user is paying attention to based on their eye gaze. Once the target speaker is identified, such systems could then enhance that speaker’s voice while reducing other background sounds, making it easier for the user to follow conversations in noisy environments. We hope this dataset will be useful for researchers working in audio-visual processing, eye gaze tracking, attention modeling, and hearing aid technology.