HaptMR (presented at HCII 2021) advances the conventional Mixed Reality (MR) experience by introducing a semantic-aware haptic feedback system. Unlike traditional solutions that merely simulate touch when users interact with virtual objects, HaptMR introduces “second-order” haptic sensations—users sense different tactile feedback when a virtual object collides with various real-world materials. This unique feature is powered by a distributed architecture: a Hololens 2 for spatial mapping and gesture tracking, an NVIDIA Jetson Nano for deep learning-based material recognition, and a network of fine-tuned vibrotactile actuators embedded in the user’s gloves.

By leveraging real-time computer vision algorithms and robust data exchange over 5GHz Wi-Fi, HaptMR dynamically adjusts vibration intensity and pattern based on both the user’s gestures and the rigidity or hardness of contacted physical objects. The result is an immersive, semantically rich interaction that runs consistently at 60 fps with minimal latency. In user studies involving 18 participants, 77% reported high satisfaction and 61% observed improved intuitiveness in MR interactions. This confirms HaptMR’s potential to redefine multimodal experiences by bridging virtual and physical realms through nuanced tactile feedback.

HaptMR（发表于 HCII 2021）通过引入语义感知触觉反馈系统，为传统混合现实（MR）体验带来了突破性升级。与仅在用户触碰虚拟物体时提供单一触感的传统方法不同，HaptMR 将“二阶”触觉融入系统：当虚拟物体与真实世界中的不同材质碰撞时，用户可感知到相应的差异化触觉。该系统采用分布式架构：Hololens 2 负责空间映射和手势追踪，NVIDIA Jetson Nano 则利用深度学习算法进行材料识别，通过 5GHz Wi-Fi 连接并与嵌入式振动模块配合，实现精准的多层级触觉输出。

借助实时计算机视觉算法和高效数据传输，HaptMR 能根据用户的手势动作以及所接触物体的硬度或刚度，动态调整振动强度和模式。该系统在 60 帧的稳定渲染和极低延迟下运行。用户测试覆盖 18 位参与者，77% 对系统评价甚高，61% 认为整体交互的直观性显著提升。HaptMR 成功地通过精细化的触觉反馈在虚拟与物理世界间架设桥梁，为多模态交互体验树立了新的标杆。