Dataset for eyes-free palm-based keyboard

Overview :

This multimodal dataset was collected for training and evaluating palm-based text entry systems using AR/VR head-mounted displays. The dataset captures synchronized sensor data from multiple modalities including IMU sensors, electrical touch signals, and video recordings of hand movements during typing tasks.

Dataset Statistics :

- Total Size: ~27 GB

- Participants: 14 users (user_1 to user_14)

- Total Sessions: 91 sessions across all users

- Total Videos: 77 MP4 files

- Total Data Files: 693 CSV files

- Recording Setting: Controlled office environment with stable lighting, seated participants

File Descriptions per Session :

Each session folder contains the following files:

Video File

- output_video.mp4 - Recording of the typing session showing hand position.

- Typical file size: 400-700 MB per session

- Displays finger tracking, key presses, and hand skeleton visualization

- Synchronized with all sensor data streams

Sensor Data Files (CSV Format)

1. imu_data.csv

- IMU sensor readings from ring-mounted device

- Columns: Timestamp, GyroX, GyroY, GyroZ, AccX, AccY, AccZ

- Captures motion data for touch and gesture detection

- Sampling rate: ~30 Hz

2. voltage_data.csv

- Raw electrical signal readings for touch detection

- Columns: Timestamp, Voltage

- Used to detect palm contacts via body impedance changes

- Signal: 1 kHz, 5V sinusoidal

3. voltage_touch.csv

- Processed touch events detected from electrical signals

- Columns: Timestamp, Touch (binary: 1 = touch detected)

- Auto-labeled ground truth for touch events

- Generated via peak detection with user-specific calibration

4. pinching_distance.csv

- Distance measurements between thumb and index finger

- Columns: Timestamp, Pinch_distance

- Used for gesture recognition and hand state analysis

- Normalized distance values

5. pinching.csv

- Binary pinching events

- Columns: Timestamp, Pinching

- Indicates when pinch gestures are detected

- Used for hand state classification

6. red_circle.csv

- Hand tracking data for calibration and key targeting

- Columns: Timestamp, letter_to_type, finger_coordinate, finger_coordinate_window

- Contains finger position trajectories and target letters

- Includes sliding window of recent finger coordinates

7. key_collection_bayes.csv

- Detailed key press events with spatial-temporal data

- Large file containing Bayesian decoder inputs for each touch event

- Used for training key classification models

- Includes finger position windows and ground truth labels

8. frame_timestamps.csv

- Video frame timestamps for synchronization

- Columns: FrameIndex, Timestamp

- Enables precise alignment between video and sensor data

- Critical for multimodal data fusion

9. false_positive_manual_detector.csv

- Manual annotations of false positive detections

- Columns: Timestamp, False_positive

- Used for validation and error analysis

- May be empty in sessions without manual annotation

10. config.py

- Session configuration file with all experimental parameters

- Filter parameters: Kalman, EMA, Hampel, Median, Adaptive EMA settings

- Keyboard layout: QWERTY coordinates, key positions (26 letters + space)

- Touch detection: Thresholds, window sizes, calibration settings

- Camera settings: Resolution (640x480), FPS (60), MediaPipe parameters

- IMU settings: Serial port, baud rate (500000), sampling parameters

- Language model: Paths to GPT-2, Markov models, and decoder files

- User calibration: Individualized touch sensitivity parameters

Data Collection Protocol

Phase 1 - Random Keys (Calibration):

- Random keys highlighted on virtual keyboard

- Users tap highlighted keys with real-time finger position feedback

- Purpose: System calibration and baseline data collection

- Enables personalized touch detection thresholds

Phase 2 - Sentence Typing (Evaluation):

- Users type sentences from MacKenzie dataset

- Real-time progress tracking (typed letters highlighted)

- Visual feedback disabled to capture natural eyes-off typing

- Hand skeleton blinks blue briefly to confirm each touch detection

- Captures spatial-temporal tap sequences for training

Technical Specifications

IMU Sensor:

- Ring-mounted on index finger

- 3-axis gyroscope + 3-axis accelerometer

- Sampling rate: ~30 Hz

- Baud rate: 500000

- Used for motion-based touch detection

Electrical Touch Detection:

- 1 kHz, 5V sinusoidal signal passed through body

- Electrodes placed on back of palm and forearm

- Current: ~5mA (safe, below 10mA safety limit)

- Peak detection algorithm with personalized SNR calibration

- Window length: 20 ms, minimum peak distance: 80 ms

Camera:

- Wide-FOV camera (necklace-mounted, simulating AR head-camera)

- Resolution: 640x480 pixels

- Max FPS: 60

- Hand tracking via MediaPipe

- Confidence thresholds: Right hand 0.6, Left hand 0.8

Virtual Keyboard Layout:

- 26-letter QWERTY layout (3 rows)

- Top row: Q W E R T Y U I O P

- Middle row: _ A S D F G H J K L (_ = space)

- Bottom row: Z X C V B N M

- Normalized coordinates [0,1] for key positions

- Projected onto palm using geometric transformation

Data Processing Pipeline

1. Signal Acquisition: Three modalities collected simultaneously

- Video frames for hand tracking

- IMU data for finger motion

- Electrical signals for touch detection

2. Touch Detection: Electrical signal processed via:

- Normalization and peak detection

- User-specific calibration (double-tap protocol)

- Signal-to-Noise Ratio (SNR) analysis for threshold setting

3. Hand Tracking: MediaPipe provides:

- 21 3D hand keypoints per hand

- Real-time finger position tracking

- Hand skeleton visualization

4. Data Synchronization: All streams aligned via:

- Unified timestamps (Unix time)

- Frame timestamps for video alignment

- Touch event timestamps for sensor fusion

Potential Applications

- Time-series neural network training for touch detection

- Multimodal sensor fusion for key classification

- Bayesian language modeling for text prediction

- Hand tracking and gesture recognition in AR/VR

- Eyes-free interaction research

- Adaptive calibration algorithms

- Touch detection robustness evaluation

- Spatial-temporal pattern recognition