Towards Perception-Aware Shared Autonomy: A Multimodal Teleoperation Dataset for Fragile Object Manipulation

<pre>Teleoperated fragile object manipulation is a critical capability in high-stakes domains such as remote hazardous-environment operations and surgical robotics, where perceptual uncertainty can lead to irreversible damage. To understand the perception-control correlation and capture wide range of variation from different users, we present a dataset for analyzing human behavior in this setting, comprising 221 trials from 11 novice participants manipulating objects with 20 visually encoded fragility levels (50--1000~gf), indicating the maximum force can be exerted onto the object. The dataset includes synchronized multimodal observations: robot joint trajectories, gripper force signals, multi-view RGB video, users' perceived object fragility, confidence ratings, and trial outcomes. Our analysis reveals key behavior patterns: a 48\% overall success rate under challenging fragile object constraints, generally under-estimating bias when perceiving object fragility, weak correlation between perception confidence and accuracy, substantial early-contact failures, and failure to provide effective control. We further show that pre-contact kinematics encode users' perception of object fragility while exhibiting strong user-specific baselines. We release this dataset as an open platform to support the development of perception-aware shared autonomy, where systems explicitly model operators' internal estimates of object properties rather than treating observable actions alone as ground truth. This enables research in user behavioral profiling, personalized user modeling, proactive intervention design, and adaptive robot control for damage-sensitive manipulation.</pre>