Dataset for eyes-free palm-based keyboard

Overview

  This multimodal dataset was collected for training and evaluating palm-based text entry systems using AR/VR head-mounted displays. The dataset captures synchronized sensor data from multiple modalities including IMU sensors, electrical touch signals, and video recordings of hand movements during typing tasks.

  Dataset Statistics

  - Total Size: ~30 GB  - Total Samples: 77 sessions  - Total Videos: 77 MP4 files (one per session)  - Recording Setting: Controlled office environment with stable lighting, seated participants

  Dataset Structure

  The dataset is organized as a flat sequence of samples:

  data_collection_samples/  ├── sample_001/  │   ├── config.py  │   ├── frame_timestamps.csv  │   ├── imu_data.csv  │   ├── key_collection_bayes.csv  │   ├── output_video.mp4  │   ├── pinching_distance.csv  │   ├── pinching.csv  │   ├── red_circle.csv  │   ├── voltage_data.csv  │   └── voltage_touch.csv  ├── sample_002/  ├── ...  └── sample_077/

  Each sample represents one typing session from a participant, anonymized and numbered sequentially.

  File Descriptions per Session

  Video File

  - output_video.mp4 - Recording of the typing session showing hand position    - Typical file size: 400-700 MB per session    - Displays finger tracking, key presses, and hand skeleton visualization    - Synchronized with all sensor data streams

  Sensor Data Files (CSV Format)

  1. imu_data.csv

  - IMU sensor readings from ring-mounted device  - Columns: Timestamp, GyroX, GyroY, GyroZ, AccX, AccY, AccZ  - Captures motion data for touch and gesture detection  - Sampling rate: ~30 Hz

  2. voltage_data.csv

  - Raw electrical signal readings for touch detection  - Columns: Timestamp, Voltage  - Used to detect palm contacts via body impedance changes  - Signal: 1 kHz, 5V sinusoidal

  3. voltage_touch.csv

  - Processed touch events detected from electrical signals  - Columns: Timestamp, Touch (binary: 1 = touch detected)  - Auto-labeled ground truth for touch events  - Generated via peak detection with user-specific calibration

  4. pinching_distance.csv

  - Distance measurements between thumb and index finger  - Columns: Timestamp, Pinch_distance  - Used for gesture recognition and hand state analysis  - Normalized distance values

  5. pinching.csv

  - Binary pinching events  - Columns: Timestamp, Pinching  - Indicates when pinch gestures are detected  - Used for hand state classification

  6. red_circle.csv

  - Hand tracking data for calibration and key targeting  - Columns: Timestamp, letter_to_type, finger_coordinate, finger_coordinate_window  - Contains finger position trajectories and target letters  - Includes sliding window of recent finger coordinates

  7. key_collection_bayes.csv

  - Detailed key press events with spatial-temporal data  - Large file containing Bayesian decoder inputs for each touch event  - Used for training key classification models  - Includes finger position windows and ground truth labels

  8. frame_timestamps.csv

  - Video frame timestamps for synchronization  - Columns: FrameIndex, Timestamp  - Enables precise alignment between video and sensor data  - Critical for multimodal data fusion

  9. config.py

  - Session configuration file with all experimental parameters    - Filter parameters: Kalman, EMA, Hampel, Median, Adaptive EMA settings    - Keyboard layout: QWERTY coordinates, key positions (26 letters + space)    - Touch detection: Thresholds, window sizes, calibration settings    - Camera settings: Resolution (640x480), FPS (60), MediaPipe parameters    - IMU settings: Serial port, baud rate (500000), sampling parameters    - Language model: Paths to GPT-2, Markov models, and decoder files    - User calibration: Individualized touch sensitivity parameters

  Data Collection Protocol

  Phase 1 - Random Keys (Calibration):

  - Random keys highlighted on virtual keyboard  - Users tap highlighted keys with real-time finger position feedback  - Purpose: System calibration and baseline data collection  - Enables personalized touch detection thresholds

  Phase 2 - Sentence Typing (Evaluation):

  - Users type sentences from MacKenzie dataset  - Real-time progress tracking (typed letters highlighted)  - Visual feedback disabled to capture natural eyes-off typing  - Hand skeleton blinks blue briefly to confirm each touch detection  - Captures spatial-temporal tap sequences for training

  Technical Specifications

  IMU Sensor

  - Ring-mounted on index finger  - 3-axis gyroscope + 3-axis accelerometer  - Sampling rate: ~30 Hz  - Baud rate: 500000  - Used for motion-based touch detection

  Electrical Touch Detection

  - 1 kHz, 5V sinusoidal signal passed through body  - Electrodes placed on back of palm and forearm  - Current: ~5mA (safe, below 10mA safety limit)  - Peak detection algorithm with personalized SNR calibration  - Window length: 20 ms, minimum peak distance: 80 ms

  Camera

  - Wide-FOV camera (necklace-mounted, simulating AR head-camera)  - Resolution: 640x480 pixels  - Max FPS: 60  - Hand tracking via MediaPipe  - Confidence thresholds: Right hand 0.6, Left hand 0.8

  Virtual Keyboard Layout

  - 26-letter QWERTY layout (3 rows)    - Top row: Q W E R T Y U I O P    - Middle row: _ A S D F G H J K L (_ = space)    - Bottom row: Z X C V B N M  - Normalized coordinates [0,1] for key positions  - Projected onto palm using geometric transformation

  Data Processing Pipeline

  1. Signal Acquisition

  Three modalities collected simultaneously:  - Video frames for hand tracking  - IMU data for finger motion  - Electrical signals for touch detection

  2. Touch Detection

  Electrical signal processed via:  - Normalization and peak detection  - User-specific calibration (double-tap protocol)  - Signal-to-Noise Ratio (SNR) analysis for threshold setting

  3. Hand Tracking

  MediaPipe provides:  - 21 3D hand keypoints per hand  - Real-time finger position tracking  - Hand skeleton visualization

  4. Data Synchronization

  All streams aligned via:  - Unified timestamps (Unix time)  - Frame timestamps for video alignment  - Touch event timestamps for sensor fusion

  Potential Applications

  - Time-series neural network training for touch detection  - Multimodal sensor fusion for key classification  - Bayesian language modeling for text prediction  - Hand tracking and gesture recognition in AR/VR  - Eyes-free interaction research  - Adaptive calibration algorithms  - Touch detection robustness evaluation  - Spatial-temporal pattern recognition